Sep 16 18:38:27.666991 ip-10-0-135-119 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Sep 16 18:38:27.667007 ip-10-0-135-119 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Sep 16 18:38:27.667016 ip-10-0-135-119 systemd[1]: kubelet.service: Failed with result 'resources'.
Sep 16 18:38:27.667246 ip-10-0-135-119 systemd[1]: Failed to start Kubernetes Kubelet.
Sep 16 18:38:37.743544 ip-10-0-135-119 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found.
Sep 16 18:38:37.743562 ip-10-0-135-119 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot c7a78ab00f4e4e619579a93613ec38cd --
Sep 16 18:40:18.822305 ip-10-0-135-119 systemd[1]: Starting Kubernetes Kubelet...
Sep 16 18:40:19.207663 ip-10-0-135-119 kubenswrapper[2578]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 16 18:40:19.207663 ip-10-0-135-119 kubenswrapper[2578]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Sep 16 18:40:19.207663 ip-10-0-135-119 kubenswrapper[2578]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 16 18:40:19.207663 ip-10-0-135-119 kubenswrapper[2578]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Sep 16 18:40:19.207663 ip-10-0-135-119 kubenswrapper[2578]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 16 18:40:19.209439 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:19.209351    2578 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Sep 16 18:40:19.214886 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.214866    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 16 18:40:19.214886 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.214886    2578 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 16 18:40:19.214955 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.214890    2578 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 16 18:40:19.214955 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.214893    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 16 18:40:19.214955 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.214897    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 16 18:40:19.214955 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.214900    2578 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 16 18:40:19.214955 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.214903    2578 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 16 18:40:19.214955 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.214905    2578 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 16 18:40:19.214955 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.214908    2578 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 16 18:40:19.214955 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.214911    2578 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 16 18:40:19.214955 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.214915    2578 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 16 18:40:19.214955 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.214918    2578 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 16 18:40:19.214955 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.214921    2578 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 16 18:40:19.214955 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.214923    2578 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 16 18:40:19.214955 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.214926    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 16 18:40:19.214955 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.214929    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 16 18:40:19.214955 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.214932    2578 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 16 18:40:19.214955 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.214934    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 16 18:40:19.214955 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.214937    2578 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 16 18:40:19.214955 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.214940    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 16 18:40:19.214955 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.214942    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 16 18:40:19.214955 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.214950    2578 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 16 18:40:19.215463 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.214953    2578 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 16 18:40:19.215463 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.214955    2578 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 16 18:40:19.215463 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.214958    2578 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 16 18:40:19.215463 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.214961    2578 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 16 18:40:19.215463 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.214963    2578 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 16 18:40:19.215463 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.214966    2578 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 16 18:40:19.215463 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.214969    2578 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 16 18:40:19.215463 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.214971    2578 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 16 18:40:19.215463 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.214986    2578 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 16 18:40:19.215463 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.214988    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 16 18:40:19.215463 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.214991    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 16 18:40:19.215463 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.214993    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 16 18:40:19.215463 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.214996    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 16 18:40:19.215463 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.214999    2578 feature_gate.go:328] unrecognized feature gate: Example
Sep 16 18:40:19.215463 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.215001    2578 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 16 18:40:19.215463 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.215005    2578 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 16 18:40:19.215463 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.215009    2578 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 16 18:40:19.215463 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.215012    2578 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 16 18:40:19.215463 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.215015    2578 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 16 18:40:19.215926 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.215018    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 16 18:40:19.215926 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.215021    2578 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 16 18:40:19.215926 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.215024    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 16 18:40:19.215926 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.215026    2578 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 16 18:40:19.215926 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.215029    2578 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 16 18:40:19.215926 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.215032    2578 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 16 18:40:19.215926 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.215034    2578 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 16 18:40:19.215926 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.215037    2578 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 16 18:40:19.215926 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.215040    2578 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 16 18:40:19.215926 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.215042    2578 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 16 18:40:19.215926 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.215045    2578 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 16 18:40:19.215926 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.215047    2578 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 16 18:40:19.215926 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.215050    2578 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 16 18:40:19.215926 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.215053    2578 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 16 18:40:19.215926 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.215057    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 16 18:40:19.215926 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.215059    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 16 18:40:19.215926 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.215062    2578 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 16 18:40:19.215926 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.215064    2578 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 16 18:40:19.215926 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.215067    2578 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 16 18:40:19.215926 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.215071    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 16 18:40:19.216459 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.215073    2578 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 16 18:40:19.216459 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.215075    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 16 18:40:19.216459 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.215078    2578 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 16 18:40:19.216459 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.215081    2578 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 16 18:40:19.216459 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.215083    2578 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 16 18:40:19.216459 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.215086    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 16 18:40:19.216459 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.215088    2578 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 16 18:40:19.216459 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.215091    2578 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 16 18:40:19.216459 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.215094    2578 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 16 18:40:19.216459 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.215096    2578 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 16 18:40:19.216459 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.215099    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 16 18:40:19.216459 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.215101    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 16 18:40:19.216459 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.215104    2578 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 16 18:40:19.216459 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.215107    2578 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 16 18:40:19.216459 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.215111    2578 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 16 18:40:19.216459 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.215115    2578 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 16 18:40:19.216459 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.215118    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 16 18:40:19.216459 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.215121    2578 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 16 18:40:19.216459 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.215124    2578 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 16 18:40:19.216911 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.215126    2578 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 16 18:40:19.216911 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.215129    2578 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 16 18:40:19.216911 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.215132    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 16 18:40:19.216911 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.215134    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 16 18:40:19.216911 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.215136    2578 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 16 18:40:19.216911 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.215139    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 16 18:40:19.216911 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.215141    2578 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 16 18:40:19.216911 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.215145    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 16 18:40:19.216911 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.215148    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 16 18:40:19.216911 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.215150    2578 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 16 18:40:19.216911 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.215153    2578 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 16 18:40:19.216911 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.215156    2578 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 16 18:40:19.216911 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.215158    2578 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 16 18:40:19.216911 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.215161    2578 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 16 18:40:19.216911 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.215164    2578 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 16 18:40:19.216911 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.215166    2578 feature_gate.go:328] unrecognized feature gate: Example2
Sep 16 18:40:19.216911 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.215170    2578 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 16 18:40:19.216911 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.215173    2578 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 16 18:40:19.216911 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.215176    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 16 18:40:19.216911 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.215936    2578 feature_gate.go:328] unrecognized feature gate: Example
Sep 16 18:40:19.217417 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.215944    2578 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 16 18:40:19.217417 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.215947    2578 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 16 18:40:19.217417 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.215950    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 16 18:40:19.217417 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.215953    2578 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 16 18:40:19.217417 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.215960    2578 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 16 18:40:19.217417 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.215963    2578 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 16 18:40:19.217417 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.215966    2578 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 16 18:40:19.217417 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.215968    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 16 18:40:19.217417 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.215971    2578 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 16 18:40:19.217417 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.215989    2578 feature_gate.go:328] unrecognized feature gate: Example2
Sep 16 18:40:19.217417 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.215992    2578 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 16 18:40:19.217417 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.215995    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 16 18:40:19.217417 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.215998    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 16 18:40:19.217417 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.216000    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 16 18:40:19.217417 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.216004    2578 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 16 18:40:19.217417 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.216007    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 16 18:40:19.217417 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.216010    2578 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 16 18:40:19.217417 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.216012    2578 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 16 18:40:19.217417 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.216015    2578 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 16 18:40:19.217995 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.216018    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 16 18:40:19.217995 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.216020    2578 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 16 18:40:19.217995 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.216023    2578 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 16 18:40:19.217995 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.216026    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 16 18:40:19.217995 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.216028    2578 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 16 18:40:19.217995 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.216031    2578 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 16 18:40:19.217995 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.216034    2578 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 16 18:40:19.217995 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.216036    2578 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 16 18:40:19.217995 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.216039    2578 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 16 18:40:19.217995 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.216041    2578 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 16 18:40:19.217995 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.216044    2578 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 16 18:40:19.217995 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.216047    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 16 18:40:19.217995 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.216050    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 16 18:40:19.217995 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.216052    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 16 18:40:19.217995 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.216054    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 16 18:40:19.217995 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.216057    2578 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 16 18:40:19.217995 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.216060    2578 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 16 18:40:19.217995 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.216062    2578 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 16 18:40:19.217995 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.216065    2578 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 16 18:40:19.218466 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.216067    2578 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 16 18:40:19.218466 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.216070    2578 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 16 18:40:19.218466 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.216073    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 16 18:40:19.218466 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.216075    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 16 18:40:19.218466 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.216078    2578 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 16 18:40:19.218466 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.216082    2578 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 16 18:40:19.218466 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.216086    2578 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 16 18:40:19.218466 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.216089    2578 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 16 18:40:19.218466 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.216092    2578 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 16 18:40:19.218466 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.216094    2578 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 16 18:40:19.218466 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.216097    2578 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 16 18:40:19.218466 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.216099    2578 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 16 18:40:19.218466 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.216102    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 16 18:40:19.218466 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.216105    2578 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 16 18:40:19.218466 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.216107    2578 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 16 18:40:19.218466 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.216110    2578 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 16 18:40:19.218466 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.216112    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 16 18:40:19.218466 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.216115    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 16 18:40:19.218466 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.216117    2578 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 16 18:40:19.218466 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.216120    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 16 18:40:19.218954 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.216123    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 16 18:40:19.218954 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.216126    2578 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 16 18:40:19.218954 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.216128    2578 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 16 18:40:19.218954 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.216130    2578 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 16 18:40:19.218954 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.216133    2578 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 16 18:40:19.218954 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.216136    2578 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 16 18:40:19.218954 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.216138    2578 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 16 18:40:19.218954 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.216141    2578 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 16 18:40:19.218954 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.216143    2578 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 16 18:40:19.218954 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.216146    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 16 18:40:19.218954 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.216148    2578 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 16 18:40:19.218954 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.216151    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 16 18:40:19.218954 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.216154    2578 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 16 18:40:19.218954 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.216156    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 16 18:40:19.218954 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.216159    2578 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 16 18:40:19.218954 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.216161    2578 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 16 18:40:19.218954 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.216164    2578 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 16 18:40:19.218954 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.216167    2578 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 16 18:40:19.218954 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.216170    2578 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 16 18:40:19.218954 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.216173    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 16 18:40:19.219481 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.216175    2578 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 16 18:40:19.219481 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.216179    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 16 18:40:19.219481 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.216182    2578 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 16 18:40:19.219481 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.216186    2578 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 16 18:40:19.219481 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.216189    2578 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 16 18:40:19.219481 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.216193    2578 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 16 18:40:19.219481 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.216196    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 16 18:40:19.219481 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.216198    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 16 18:40:19.219481 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.216201    2578 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 16 18:40:19.219481 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.216204    2578 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 16 18:40:19.219481 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.216207    2578 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 16 18:40:19.219481 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.216210    2578 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 16 18:40:19.219481 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.216212    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 16 18:40:19.219481 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.216216    2578 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 16 18:40:19.219481 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.216219    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 16 18:40:19.219481 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.216222    2578 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 16 18:40:19.219481 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.216225    2578 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 16 18:40:19.219481 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.216228    2578 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 16 18:40:19.219481 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.216230    2578 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 16 18:40:19.219481 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.216233    2578 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 16 18:40:19.219960 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.216638    2578 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 16 18:40:19.219960 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.216643    2578 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 16 18:40:19.219960 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.216646    2578 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 16 18:40:19.219960 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.216648    2578 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 16 18:40:19.219960 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.216651    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 16 18:40:19.219960 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.216654    2578 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 16 18:40:19.219960 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.216657    2578 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 16 18:40:19.219960 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.216659    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 16 18:40:19.219960 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.216662    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 16 18:40:19.219960 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.216665    2578 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 16 18:40:19.219960 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.216669    2578 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 16 18:40:19.219960 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.216672    2578 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 16 18:40:19.219960 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.216674    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 16 18:40:19.219960 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.216677    2578 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 16 18:40:19.219960 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.216679    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 16 18:40:19.219960 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.216683    2578 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 16 18:40:19.219960 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.216687    2578 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 16 18:40:19.219960 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.216694    2578 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 16 18:40:19.219960 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.216697    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 16 18:40:19.220466 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.216700    2578 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 16 18:40:19.220466 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.216703    2578 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 16 18:40:19.220466 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.216705    2578 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 16 18:40:19.220466 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.216708    2578 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 16 18:40:19.220466 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.216711    2578 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 16 18:40:19.220466 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.216714    2578 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 16 18:40:19.220466 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.216717    2578 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 16 18:40:19.220466 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.216719    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 16 18:40:19.220466 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.216722    2578 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 16 18:40:19.220466 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.216724    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 16 18:40:19.220466 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.216727    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 16 18:40:19.220466 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.216729    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 16 18:40:19.220466 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.216732    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 16 18:40:19.220466 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.216735    2578 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 16 18:40:19.220466 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.216737    2578 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 16 18:40:19.220466 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.216741    2578 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 16 18:40:19.220466 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.216743    2578 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 16 18:40:19.220466 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.216746    2578 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 16 18:40:19.220466 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.216749    2578 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 16 18:40:19.220466 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.216752    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 16 18:40:19.220956 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.216755    2578 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 16 18:40:19.220956 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.216758    2578 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 16 18:40:19.220956 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.216763    2578 feature_gate.go:328] unrecognized feature gate: Example
Sep 16 18:40:19.220956 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.216766    2578 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 16 18:40:19.220956 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.216769    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 16 18:40:19.220956 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.216771    2578 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 16 18:40:19.220956 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.216773    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 16 18:40:19.220956 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.216776    2578 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 16 18:40:19.220956 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.216778    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 16 18:40:19.220956 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.216781    2578 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 16 18:40:19.220956 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.216785    2578 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 16 18:40:19.220956 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.216787    2578 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 16 18:40:19.220956 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.216790    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 16 18:40:19.220956 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.216792    2578 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 16 18:40:19.220956 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.216795    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 16 18:40:19.220956 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.216797    2578 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 16 18:40:19.220956 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.216800    2578 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 16 18:40:19.220956 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.216802    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 16 18:40:19.220956 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.216804    2578 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 16 18:40:19.221487 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.216807    2578 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 16 18:40:19.221487 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.216809    2578 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 16 18:40:19.221487 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.216812    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 16 18:40:19.221487 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.216814    2578 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 16 18:40:19.221487 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.216818    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 16 18:40:19.221487 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.216820    2578 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 16 18:40:19.221487 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.216823    2578 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 16 18:40:19.221487 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.216826    2578 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 16 18:40:19.221487 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.216828    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 16 18:40:19.221487 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.216830    2578 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 16 18:40:19.221487 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.216833    2578 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 16 18:40:19.221487 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.216836    2578 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 16 18:40:19.221487 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.216838    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 16 18:40:19.221487 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.216841    2578 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 16 18:40:19.221487 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.216843    2578 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 16 18:40:19.221487 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.216846    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 16 18:40:19.221487 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.216849    2578 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 16 18:40:19.221487 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.216851    2578 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 16 18:40:19.221487 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.216853    2578 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 16 18:40:19.221487 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.216856    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 16 18:40:19.221964 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.216859    2578 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 16 18:40:19.221964 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.216861    2578 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 16 18:40:19.221964 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.216864    2578 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 16 18:40:19.221964 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.216867    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 16 18:40:19.221964 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.216870    2578 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 16 18:40:19.221964 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.216872    2578 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 16 18:40:19.221964 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.216875    2578 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 16 18:40:19.221964 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.216877    2578 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 16 18:40:19.221964 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.216879    2578 feature_gate.go:328] unrecognized feature gate: Example2
Sep 16 18:40:19.221964 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.216882    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 16 18:40:19.221964 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.216885    2578 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 16 18:40:19.221964 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.216887    2578 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 16 18:40:19.221964 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.216890    2578 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 16 18:40:19.221964 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.216893    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 16 18:40:19.221964 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.216895    2578 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 16 18:40:19.221964 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.216897    2578 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 16 18:40:19.221964 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.216900    2578 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 16 18:40:19.221964 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.216902    2578 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 16 18:40:19.221964 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.216904    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 16 18:40:19.222459 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.216907    2578 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 16 18:40:19.222459 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.216909    2578 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 16 18:40:19.222459 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:19.217005    2578 flags.go:64] FLAG: --address="0.0.0.0"
Sep 16 18:40:19.222459 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:19.217047    2578 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Sep 16 18:40:19.222459 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:19.217055    2578 flags.go:64] FLAG: --anonymous-auth="true"
Sep 16 18:40:19.222459 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:19.217059    2578 flags.go:64] FLAG: --application-metrics-count-limit="100"
Sep 16 18:40:19.222459 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:19.217064    2578 flags.go:64] FLAG: --authentication-token-webhook="false"
Sep 16 18:40:19.222459 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:19.217067    2578 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Sep 16 18:40:19.222459 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:19.217071    2578 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Sep 16 18:40:19.222459 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:19.217076    2578 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Sep 16 18:40:19.222459 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:19.217079    2578 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Sep 16 18:40:19.222459 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:19.217083    2578 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Sep 16 18:40:19.222459 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:19.217086    2578 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Sep 16 18:40:19.222459 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:19.217090    2578 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Sep 16 18:40:19.222459 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:19.217093    2578 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Sep 16 18:40:19.222459 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:19.217096    2578 flags.go:64] FLAG: --cgroup-root=""
Sep 16 18:40:19.222459 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:19.217100    2578 flags.go:64] FLAG: --cgroups-per-qos="true"
Sep 16 18:40:19.222459 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:19.217103    2578 flags.go:64] FLAG: --client-ca-file=""
Sep 16 18:40:19.222459 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:19.217106    2578 flags.go:64] FLAG: --cloud-provider="external"
Sep 16 18:40:19.222459 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:19.217109    2578 flags.go:64] FLAG: --cluster-dns="[]"
Sep 16 18:40:19.222459 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:19.217114    2578 flags.go:64] FLAG: --cluster-domain=""
Sep 16 18:40:19.222459 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:19.217116    2578 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Sep 16 18:40:19.222459 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:19.217120    2578 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Sep 16 18:40:19.222459 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:19.217123    2578 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Sep 16 18:40:19.223058 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:19.217126    2578 flags.go:64] FLAG: --container-log-max-files="5"
Sep 16 18:40:19.223058 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:19.217130    2578 flags.go:64] FLAG: --container-log-max-size="10Mi"
Sep 16 18:40:19.223058 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:19.217133    2578 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Sep 16 18:40:19.223058 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:19.217136    2578 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Sep 16 18:40:19.223058 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:19.217139    2578 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Sep 16 18:40:19.223058 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:19.217142    2578 flags.go:64] FLAG: --contention-profiling="false"
Sep 16 18:40:19.223058 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:19.217145    2578 flags.go:64] FLAG: --cpu-cfs-quota="true"
Sep 16 18:40:19.223058 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:19.217148    2578 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Sep 16 18:40:19.223058 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:19.217152    2578 flags.go:64] FLAG: --cpu-manager-policy="none"
Sep 16 18:40:19.223058 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:19.217155    2578 flags.go:64] FLAG: --cpu-manager-policy-options=""
Sep 16 18:40:19.223058 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:19.217158    2578 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Sep 16 18:40:19.223058 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:19.217161    2578 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Sep 16 18:40:19.223058 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:19.217164    2578 flags.go:64] FLAG: --enable-debugging-handlers="true"
Sep 16 18:40:19.223058 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:19.217167    2578 flags.go:64] FLAG: --enable-load-reader="false"
Sep 16 18:40:19.223058 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:19.217170    2578 flags.go:64] FLAG: --enable-server="true"
Sep 16 18:40:19.223058 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:19.217174    2578 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Sep 16 18:40:19.223058 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:19.217178    2578 flags.go:64] FLAG: --event-burst="100"
Sep 16 18:40:19.223058 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:19.217181    2578 flags.go:64] FLAG: --event-qps="50"
Sep 16 18:40:19.223058 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:19.217184    2578 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Sep 16 18:40:19.223058 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:19.217187    2578 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Sep 16 18:40:19.223058 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:19.217190    2578 flags.go:64] FLAG: --eviction-hard=""
Sep 16 18:40:19.223058 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:19.217193    2578 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Sep 16 18:40:19.223058 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:19.217196    2578 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Sep 16 18:40:19.223058 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:19.217198    2578 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Sep 16 18:40:19.223058 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:19.217202    2578 flags.go:64] FLAG: --eviction-soft=""
Sep 16 18:40:19.223972 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:19.217205    2578 flags.go:64] FLAG: --eviction-soft-grace-period=""
Sep 16 18:40:19.223972 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:19.217208    2578 flags.go:64] FLAG: --exit-on-lock-contention="false"
Sep 16 18:40:19.223972 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:19.217211    2578 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Sep 16 18:40:19.223972 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:19.217214    2578 flags.go:64] FLAG: --experimental-mounter-path=""
Sep 16 18:40:19.223972 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:19.217217    2578 flags.go:64] FLAG: --fail-cgroupv1="false"
Sep 16 18:40:19.223972 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:19.217220    2578 flags.go:64] FLAG: --fail-swap-on="true"
Sep 16 18:40:19.223972 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:19.217223    2578 flags.go:64] FLAG: --feature-gates=""
Sep 16 18:40:19.223972 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:19.217226    2578 flags.go:64] FLAG: --file-check-frequency="20s"
Sep 16 18:40:19.223972 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:19.217229    2578 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Sep 16 18:40:19.223972 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:19.217232    2578 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Sep 16 18:40:19.223972 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:19.217235    2578 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Sep 16 18:40:19.223972 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:19.217238    2578 flags.go:64] FLAG: --healthz-port="10248"
Sep 16 18:40:19.223972 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:19.217241    2578 flags.go:64] FLAG: --help="false"
Sep 16 18:40:19.223972 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:19.217244    2578 flags.go:64] FLAG: --hostname-override="ip-10-0-135-119.ec2.internal"
Sep 16 18:40:19.223972 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:19.217248    2578 flags.go:64] FLAG: --housekeeping-interval="10s"
Sep 16 18:40:19.223972 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:19.217251    2578 flags.go:64] FLAG: --http-check-frequency="20s"
Sep 16 18:40:19.223972 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:19.217254    2578 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Sep 16 18:40:19.223972 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:19.217257    2578 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Sep 16 18:40:19.223972 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:19.217260    2578 flags.go:64] FLAG: --image-gc-high-threshold="85"
Sep 16 18:40:19.223972 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:19.217263    2578 flags.go:64] FLAG: --image-gc-low-threshold="80"
Sep 16 18:40:19.223972 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:19.217266    2578 flags.go:64] FLAG: --image-service-endpoint=""
Sep 16 18:40:19.223972 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:19.217269    2578 flags.go:64] FLAG: --kernel-memcg-notification="false"
Sep 16 18:40:19.223972 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:19.217271    2578 flags.go:64] FLAG: --kube-api-burst="100"
Sep 16 18:40:19.225018 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:19.217274    2578 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Sep 16 18:40:19.225018 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:19.217278    2578 flags.go:64] FLAG: --kube-api-qps="50"
Sep 16 18:40:19.225018 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:19.217280    2578 flags.go:64] FLAG: --kube-reserved=""
Sep 16 18:40:19.225018 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:19.217283    2578 flags.go:64] FLAG: --kube-reserved-cgroup=""
Sep 16 18:40:19.225018 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:19.217286    2578 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Sep 16 18:40:19.225018 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:19.217289    2578 flags.go:64] FLAG: --kubelet-cgroups=""
Sep 16 18:40:19.225018 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:19.217292    2578 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Sep 16 18:40:19.225018 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:19.217295    2578 flags.go:64] FLAG: --lock-file=""
Sep 16 18:40:19.225018 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:19.217299    2578 flags.go:64] FLAG: --log-cadvisor-usage="false"
Sep 16 18:40:19.225018 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:19.217302    2578 flags.go:64] FLAG: --log-flush-frequency="5s"
Sep 16 18:40:19.225018 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:19.217305    2578 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Sep 16 18:40:19.225018 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:19.217310    2578 flags.go:64] FLAG: --log-json-split-stream="false"
Sep 16 18:40:19.225018 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:19.217313    2578 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Sep 16 18:40:19.225018 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:19.217316    2578 flags.go:64] FLAG: --log-text-split-stream="false"
Sep 16 18:40:19.225018 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:19.217319    2578 flags.go:64] FLAG: --logging-format="text"
Sep 16 18:40:19.225018 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:19.217322    2578 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Sep 16 18:40:19.225018 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:19.217325    2578 flags.go:64] FLAG: --make-iptables-util-chains="true"
Sep 16 18:40:19.225018 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:19.217328    2578 flags.go:64] FLAG: --manifest-url=""
Sep 16 18:40:19.225018 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:19.217331    2578 flags.go:64] FLAG: --manifest-url-header=""
Sep 16 18:40:19.225018 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:19.217335    2578 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Sep 16 18:40:19.225018 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:19.217338    2578 flags.go:64] FLAG: --max-open-files="1000000"
Sep 16 18:40:19.225018 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:19.217342    2578 flags.go:64] FLAG: --max-pods="110"
Sep 16 18:40:19.225018 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:19.217345    2578 flags.go:64] FLAG: --maximum-dead-containers="-1"
Sep 16 18:40:19.225018 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:19.217348    2578 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Sep 16 18:40:19.225018 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:19.217351    2578 flags.go:64] FLAG: --memory-manager-policy="None"
Sep 16 18:40:19.225964 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:19.217354    2578 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Sep 16 18:40:19.225964 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:19.217357    2578 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Sep 16 18:40:19.225964 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:19.217360    2578 flags.go:64] FLAG: --node-ip="0.0.0.0"
Sep 16 18:40:19.225964 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:19.217363    2578 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Sep 16 18:40:19.225964 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:19.217370    2578 flags.go:64] FLAG: --node-status-max-images="50"
Sep 16 18:40:19.225964 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:19.217374    2578 flags.go:64] FLAG: --node-status-update-frequency="10s"
Sep 16 18:40:19.225964 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:19.217377    2578 flags.go:64] FLAG: --oom-score-adj="-999"
Sep 16 18:40:19.225964 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:19.217380    2578 flags.go:64] FLAG: --pod-cidr=""
Sep 16 18:40:19.225964 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:19.217383    2578 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:eee392dbe3ac2ed4a38bca29ae3bd486f5b7780406c62c600154cd1906871b10"
Sep 16 18:40:19.225964 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:19.217389    2578 flags.go:64] FLAG: --pod-manifest-path=""
Sep 16 18:40:19.225964 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:19.217392    2578 flags.go:64] FLAG: --pod-max-pids="-1"
Sep 16 18:40:19.225964 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:19.217395    2578 flags.go:64] FLAG: --pods-per-core="0"
Sep 16 18:40:19.225964 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:19.217400    2578 flags.go:64] FLAG: --port="10250"
Sep 16 18:40:19.225964 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:19.217403    2578 flags.go:64] FLAG: --protect-kernel-defaults="false"
Sep 16 18:40:19.225964 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:19.217406    2578 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-04ae9a305d4f3e496"
Sep 16 18:40:19.225964 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:19.217409    2578 flags.go:64] FLAG: --qos-reserved=""
Sep 16 18:40:19.225964 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:19.217413    2578 flags.go:64] FLAG: --read-only-port="10255"
Sep 16 18:40:19.225964 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:19.217416    2578 flags.go:64] FLAG: --register-node="true"
Sep 16 18:40:19.225964 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:19.217419    2578 flags.go:64] FLAG: --register-with-taints=""
Sep 16 18:40:19.225964 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:19.217423    2578 flags.go:64] FLAG: --registry-burst="10"
Sep 16 18:40:19.225964 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:19.217426    2578 flags.go:64] FLAG: --registry-qps="5"
Sep 16 18:40:19.225964 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:19.217429    2578 flags.go:64] FLAG: --reserved-cpus=""
Sep 16 18:40:19.225964 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:19.217432    2578 flags.go:64] FLAG: --reserved-memory=""
Sep 16 18:40:19.225964 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:19.217436    2578 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Sep 16 18:40:19.225964 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:19.217438    2578 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Sep 16 18:40:19.226828 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:19.217442    2578 flags.go:64] FLAG: --rotate-certificates="false"
Sep 16 18:40:19.226828 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:19.217445    2578 flags.go:64] FLAG: --rotate-server-certificates="false"
Sep 16 18:40:19.226828 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:19.217448    2578 flags.go:64] FLAG: --runonce="false"
Sep 16 18:40:19.226828 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:19.217451    2578 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Sep 16 18:40:19.226828 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:19.217454    2578 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Sep 16 18:40:19.226828 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:19.217457    2578 flags.go:64] FLAG: --seccomp-default="false"
Sep 16 18:40:19.226828 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:19.217459    2578 flags.go:64] FLAG: --serialize-image-pulls="true"
Sep 16 18:40:19.226828 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:19.217462    2578 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Sep 16 18:40:19.226828 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:19.217465    2578 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Sep 16 18:40:19.226828 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:19.217468    2578 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Sep 16 18:40:19.226828 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:19.217471    2578 flags.go:64] FLAG: --storage-driver-password="root"
Sep 16 18:40:19.226828 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:19.217474    2578 flags.go:64] FLAG: --storage-driver-secure="false"
Sep 16 18:40:19.226828 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:19.217477    2578 flags.go:64] FLAG: --storage-driver-table="stats"
Sep 16 18:40:19.226828 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:19.217479    2578 flags.go:64] FLAG: --storage-driver-user="root"
Sep 16 18:40:19.226828 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:19.217482    2578 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Sep 16 18:40:19.226828 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:19.217485    2578 flags.go:64] FLAG: --sync-frequency="1m0s"
Sep 16 18:40:19.226828 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:19.217488    2578 flags.go:64] FLAG: --system-cgroups=""
Sep 16 18:40:19.226828 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:19.217491    2578 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Sep 16 18:40:19.226828 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:19.217496    2578 flags.go:64] FLAG: --system-reserved-cgroup=""
Sep 16 18:40:19.226828 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:19.217500    2578 flags.go:64] FLAG: --tls-cert-file=""
Sep 16 18:40:19.226828 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:19.217504    2578 flags.go:64] FLAG: --tls-cipher-suites="[]"
Sep 16 18:40:19.226828 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:19.217507    2578 flags.go:64] FLAG: --tls-min-version=""
Sep 16 18:40:19.226828 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:19.217510    2578 flags.go:64] FLAG: --tls-private-key-file=""
Sep 16 18:40:19.226828 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:19.217514    2578 flags.go:64] FLAG: --topology-manager-policy="none"
Sep 16 18:40:19.226828 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:19.217517    2578 flags.go:64] FLAG: --topology-manager-policy-options=""
Sep 16 18:40:19.227953 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:19.217520    2578 flags.go:64] FLAG: --topology-manager-scope="container"
Sep 16 18:40:19.227953 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:19.217523    2578 flags.go:64] FLAG: --v="2"
Sep 16 18:40:19.227953 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:19.217526    2578 flags.go:64] FLAG: --version="false"
Sep 16 18:40:19.227953 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:19.217530    2578 flags.go:64] FLAG: --vmodule=""
Sep 16 18:40:19.227953 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:19.217534    2578 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Sep 16 18:40:19.227953 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:19.217537    2578 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Sep 16 18:40:19.227953 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.217650    2578 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 16 18:40:19.227953 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.217654    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 16 18:40:19.227953 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.217657    2578 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 16 18:40:19.227953 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.217660    2578 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 16 18:40:19.227953 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.217663    2578 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 16 18:40:19.227953 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.217665    2578 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 16 18:40:19.227953 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.217669    2578 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 16 18:40:19.227953 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.217671    2578 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 16 18:40:19.227953 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.217673    2578 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 16 18:40:19.227953 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.217676    2578 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 16 18:40:19.227953 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.217678    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 16 18:40:19.227953 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.217681    2578 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 16 18:40:19.227953 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.217683    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 16 18:40:19.227953 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.217686    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 16 18:40:19.227953 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.217688    2578 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 16 18:40:19.228694 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.217691    2578 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 16 18:40:19.228694 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.217693    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 16 18:40:19.228694 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.217696    2578 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 16 18:40:19.228694 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.217699    2578 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 16 18:40:19.228694 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.217702    2578 feature_gate.go:328] unrecognized feature gate: Example2
Sep 16 18:40:19.228694 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.217706    2578 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 16 18:40:19.228694 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.217709    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 16 18:40:19.228694 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.217711    2578 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 16 18:40:19.228694 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.217714    2578 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 16 18:40:19.228694 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.217718    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 16 18:40:19.228694 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.217721    2578 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 16 18:40:19.228694 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.217724    2578 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 16 18:40:19.228694 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.217726    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 16 18:40:19.228694 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.217730    2578 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 16 18:40:19.228694 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.217734    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 16 18:40:19.228694 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.217737    2578 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 16 18:40:19.228694 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.217740    2578 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 16 18:40:19.228694 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.217743    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 16 18:40:19.228694 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.217745    2578 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 16 18:40:19.228694 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.217749    2578 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 16 18:40:19.229528 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.217753    2578 feature_gate.go:328] unrecognized feature gate: Example
Sep 16 18:40:19.229528 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.217757    2578 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 16 18:40:19.229528 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.217761    2578 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 16 18:40:19.229528 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.217765    2578 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 16 18:40:19.229528 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.217767    2578 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 16 18:40:19.229528 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.217770    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 16 18:40:19.229528 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.217773    2578 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 16 18:40:19.229528 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.217775    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 16 18:40:19.229528 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.217778    2578 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 16 18:40:19.229528 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.217780    2578 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 16 18:40:19.229528 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.217783    2578 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 16 18:40:19.229528 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.217785    2578 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 16 18:40:19.229528 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.217788    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 16 18:40:19.229528 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.217791    2578 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 16 18:40:19.229528 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.217793    2578 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 16 18:40:19.229528 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.217796    2578 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 16 18:40:19.229528 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.217799    2578 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 16 18:40:19.229528 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.217803    2578 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 16 18:40:19.229528 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.217805    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 16 18:40:19.229528 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.217808    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 16 18:40:19.230169 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.217810    2578 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 16 18:40:19.230169 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.217814    2578 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 16 18:40:19.230169 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.217816    2578 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 16 18:40:19.230169 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.217819    2578 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 16 18:40:19.230169 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.217821    2578 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 16 18:40:19.230169 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.217824    2578 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 16 18:40:19.230169 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.217826    2578 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 16 18:40:19.230169 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.217829    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 16 18:40:19.230169 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.217831    2578 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 16 18:40:19.230169 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.217834    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 16 18:40:19.230169 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.217836    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 16 18:40:19.230169 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.217839    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 16 18:40:19.230169 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.217841    2578 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 16 18:40:19.230169 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.217844    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 16 18:40:19.230169 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.217846    2578 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 16 18:40:19.230169 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.217849    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 16 18:40:19.230169 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.217851    2578 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 16 18:40:19.230169 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.217855    2578 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 16 18:40:19.230169 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.217858    2578 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 16 18:40:19.230808 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.217861    2578 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 16 18:40:19.230808 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.217864    2578 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 16 18:40:19.230808 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.217866    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 16 18:40:19.230808 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.217869    2578 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 16 18:40:19.230808 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.217873    2578 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 16 18:40:19.230808 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.217877    2578 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 16 18:40:19.230808 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.217880    2578 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 16 18:40:19.230808 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.217883    2578 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 16 18:40:19.230808 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.217886    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 16 18:40:19.230808 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.217888    2578 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 16 18:40:19.230808 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.217892    2578 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 16 18:40:19.230808 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.217895    2578 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 16 18:40:19.230808 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.217897    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 16 18:40:19.230808 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.217900    2578 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 16 18:40:19.230808 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.217904    2578 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 16 18:40:19.230808 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.217906    2578 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 16 18:40:19.230808 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.217909    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 16 18:40:19.230808 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.217911    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 16 18:40:19.230808 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.217914    2578 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 16 18:40:19.230808 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.217916    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 16 18:40:19.231339 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.217919    2578 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 16 18:40:19.231339 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.217921    2578 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 16 18:40:19.231339 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.217924    2578 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 16 18:40:19.231339 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.217926    2578 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 16 18:40:19.231339 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.217928    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 16 18:40:19.231339 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:19.217934    2578 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 16 18:40:19.231339 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:19.225658    2578 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Sep 16 18:40:19.231339 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:19.225679    2578 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Sep 16 18:40:19.231339 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:19.226389    2578 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Sep 16 18:40:19.231339 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:19.226403    2578 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Sep 16 18:40:19.231339 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.226493    2578 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 16 18:40:19.231339 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.226501    2578 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 16 18:40:19.231339 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.226506    2578 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 16 18:40:19.231339 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.226511    2578 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 16 18:40:19.231339 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.226516    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 16 18:40:19.231805 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.226521    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 16 18:40:19.231805 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.226526    2578 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 16 18:40:19.231805 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.226530    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 16 18:40:19.231805 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.226534    2578 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 16 18:40:19.231805 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.226538    2578 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 16 18:40:19.231805 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.226543    2578 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 16 18:40:19.231805 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.226547    2578 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 16 18:40:19.231805 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.226552    2578 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 16 18:40:19.231805 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.226556    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 16 18:40:19.231805 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.226561    2578 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 16 18:40:19.231805 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.226565    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 16 18:40:19.231805 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.226570    2578 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 16 18:40:19.231805 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.226576    2578 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 16 18:40:19.231805 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.226581    2578 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 16 18:40:19.231805 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.226585    2578 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 16 18:40:19.231805 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.226592    2578 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 16 18:40:19.231805 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.226598    2578 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 16 18:40:19.231805 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.226603    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 16 18:40:19.231805 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.226608    2578 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 16 18:40:19.232400 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.226612    2578 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 16 18:40:19.232400 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.226617    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 16 18:40:19.232400 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.226622    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 16 18:40:19.232400 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.226626    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 16 18:40:19.232400 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.226630    2578 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 16 18:40:19.232400 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.226635    2578 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 16 18:40:19.232400 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.226639    2578 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 16 18:40:19.232400 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.226644    2578 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 16 18:40:19.232400 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.226648    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 16 18:40:19.232400 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.226652    2578 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 16 18:40:19.232400 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.226656    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 16 18:40:19.232400 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.226660    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 16 18:40:19.232400 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.226667    2578 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 16 18:40:19.232400 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.226673    2578 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 16 18:40:19.232400 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.226678    2578 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 16 18:40:19.232400 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.226682    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 16 18:40:19.232400 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.226687    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 16 18:40:19.232400 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.226691    2578 feature_gate.go:328] unrecognized feature gate: Example2
Sep 16 18:40:19.232400 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.226695    2578 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 16 18:40:19.232849 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.226699    2578 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 16 18:40:19.232849 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.226703    2578 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 16 18:40:19.232849 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.226709    2578 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 16 18:40:19.232849 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.226714    2578 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 16 18:40:19.232849 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.226718    2578 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 16 18:40:19.232849 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.226723    2578 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 16 18:40:19.232849 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.226727    2578 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 16 18:40:19.232849 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.226731    2578 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 16 18:40:19.232849 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.226735    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 16 18:40:19.232849 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.226739    2578 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 16 18:40:19.232849 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.226743    2578 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 16 18:40:19.232849 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.226748    2578 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 16 18:40:19.232849 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.226752    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 16 18:40:19.232849 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.226757    2578 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 16 18:40:19.232849 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.226761    2578 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 16 18:40:19.232849 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.226765    2578 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 16 18:40:19.232849 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.226769    2578 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 16 18:40:19.232849 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.226774    2578 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 16 18:40:19.232849 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.226779    2578 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 16 18:40:19.232849 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.226783    2578 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 16 18:40:19.233412 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.226788    2578 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 16 18:40:19.233412 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.226792    2578 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 16 18:40:19.233412 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.226796    2578 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 16 18:40:19.233412 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.226800    2578 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 16 18:40:19.233412 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.226805    2578 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 16 18:40:19.233412 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.226809    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 16 18:40:19.233412 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.226813    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 16 18:40:19.233412 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.226817    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 16 18:40:19.233412 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.226821    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 16 18:40:19.233412 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.226825    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 16 18:40:19.233412 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.226829    2578 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 16 18:40:19.233412 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.226833    2578 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 16 18:40:19.233412 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.226837    2578 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 16 18:40:19.233412 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.226841    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 16 18:40:19.233412 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.226845    2578 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 16 18:40:19.233412 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.226850    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 16 18:40:19.233412 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.226854    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 16 18:40:19.233412 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.226858    2578 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 16 18:40:19.233412 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.226862    2578 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 16 18:40:19.233412 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.226866    2578 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 16 18:40:19.233900 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.226870    2578 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 16 18:40:19.233900 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.226874    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 16 18:40:19.233900 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.226878    2578 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 16 18:40:19.233900 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.226883    2578 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 16 18:40:19.233900 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.226887    2578 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 16 18:40:19.233900 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.226891    2578 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 16 18:40:19.233900 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.226895    2578 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 16 18:40:19.233900 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.226900    2578 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 16 18:40:19.233900 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.226904    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 16 18:40:19.233900 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.226908    2578 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 16 18:40:19.233900 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.226913    2578 feature_gate.go:328] unrecognized feature gate: Example
Sep 16 18:40:19.233900 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.226917    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 16 18:40:19.233900 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.226922    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 16 18:40:19.233900 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.226926    2578 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 16 18:40:19.233900 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.226930    2578 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 16 18:40:19.233900 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.226934    2578 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 16 18:40:19.234608 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:19.226942    2578 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 16 18:40:19.234608 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.227138    2578 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 16 18:40:19.234608 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.227148    2578 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 16 18:40:19.234608 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.227154    2578 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 16 18:40:19.234608 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.227158    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 16 18:40:19.234608 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.227163    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 16 18:40:19.234608 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.227167    2578 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 16 18:40:19.234608 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.227171    2578 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 16 18:40:19.234608 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.227176    2578 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 16 18:40:19.234608 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.227180    2578 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 16 18:40:19.234608 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.227184    2578 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 16 18:40:19.234608 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.227189    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 16 18:40:19.234608 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.227194    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 16 18:40:19.234608 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.227198    2578 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 16 18:40:19.234608 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.227202    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 16 18:40:19.235376 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.227206    2578 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 16 18:40:19.235376 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.227211    2578 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 16 18:40:19.235376 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.227215    2578 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 16 18:40:19.235376 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.227219    2578 feature_gate.go:328] unrecognized feature gate: Example
Sep 16 18:40:19.235376 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.227223    2578 feature_gate.go:328] unrecognized feature gate: Example2
Sep 16 18:40:19.235376 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.227227    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 16 18:40:19.235376 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.227231    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 16 18:40:19.235376 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.227244    2578 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 16 18:40:19.235376 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.227248    2578 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 16 18:40:19.235376 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.227252    2578 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 16 18:40:19.235376 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.227256    2578 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 16 18:40:19.235376 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.227260    2578 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 16 18:40:19.235376 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.227265    2578 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 16 18:40:19.235376 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.227269    2578 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 16 18:40:19.235376 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.227273    2578 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 16 18:40:19.235376 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.227277    2578 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 16 18:40:19.235376 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.227282    2578 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 16 18:40:19.235376 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.227286    2578 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 16 18:40:19.235376 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.227290    2578 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 16 18:40:19.235376 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.227294    2578 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 16 18:40:19.235951 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.227299    2578 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 16 18:40:19.235951 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.227303    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 16 18:40:19.235951 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.227306    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 16 18:40:19.235951 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.227311    2578 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 16 18:40:19.235951 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.227314    2578 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 16 18:40:19.235951 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.227319    2578 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 16 18:40:19.235951 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.227323    2578 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 16 18:40:19.235951 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.227327    2578 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 16 18:40:19.235951 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.227331    2578 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 16 18:40:19.235951 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.227335    2578 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 16 18:40:19.235951 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.227340    2578 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 16 18:40:19.235951 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.227344    2578 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 16 18:40:19.235951 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.227348    2578 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 16 18:40:19.235951 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.227352    2578 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 16 18:40:19.235951 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.227356    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 16 18:40:19.235951 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.227360    2578 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 16 18:40:19.235951 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.227364    2578 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 16 18:40:19.235951 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.227368    2578 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 16 18:40:19.235951 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.227372    2578 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 16 18:40:19.235951 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.227376    2578 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 16 18:40:19.236450 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.227380    2578 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 16 18:40:19.236450 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.227384    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 16 18:40:19.236450 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.227388    2578 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 16 18:40:19.236450 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.227393    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 16 18:40:19.236450 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.227397    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 16 18:40:19.236450 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.227402    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 16 18:40:19.236450 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.227407    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 16 18:40:19.236450 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.227412    2578 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 16 18:40:19.236450 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.227416    2578 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 16 18:40:19.236450 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.227420    2578 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 16 18:40:19.236450 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.227424    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 16 18:40:19.236450 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.227428    2578 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 16 18:40:19.236450 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.227432    2578 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 16 18:40:19.236450 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.227436    2578 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 16 18:40:19.236450 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.227440    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 16 18:40:19.236450 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.227444    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 16 18:40:19.236450 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.227450    2578 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 16 18:40:19.236450 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.227454    2578 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 16 18:40:19.236450 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.227458    2578 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 16 18:40:19.236908 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.227462    2578 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 16 18:40:19.236908 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.227466    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 16 18:40:19.236908 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.227470    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 16 18:40:19.236908 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.227477    2578 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 16 18:40:19.236908 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.227483    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 16 18:40:19.236908 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.227488    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 16 18:40:19.236908 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.227492    2578 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 16 18:40:19.236908 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.227497    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 16 18:40:19.236908 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.227502    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 16 18:40:19.236908 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.227507    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 16 18:40:19.236908 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.227512    2578 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 16 18:40:19.236908 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.227516    2578 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 16 18:40:19.236908 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.227520    2578 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 16 18:40:19.236908 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.227526    2578 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 16 18:40:19.236908 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.227532    2578 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 16 18:40:19.236908 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.227537    2578 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 16 18:40:19.236908 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.227541    2578 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 16 18:40:19.236908 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.227546    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 16 18:40:19.236908 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.227550    2578 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 16 18:40:19.237430 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.227555    2578 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 16 18:40:19.237430 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.227560    2578 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 16 18:40:19.237430 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.227566    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 16 18:40:19.237430 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.227571    2578 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 16 18:40:19.237430 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.227575    2578 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 16 18:40:19.237430 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.227580    2578 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 16 18:40:19.237430 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:19.227584    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 16 18:40:19.237430 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:19.227592    2578 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 16 18:40:19.237430 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:19.227747    2578 server.go:962] "Client rotation is on, will bootstrap in background"
Sep 16 18:40:19.237430 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:19.230411    2578 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Sep 16 18:40:19.237430 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:19.231402    2578 server.go:1019] "Starting client certificate rotation"
Sep 16 18:40:19.237430 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:19.231506    2578 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 16 18:40:19.237430 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:19.231543    2578 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 16 18:40:19.237745 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:19.234573    2578 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 16 18:40:19.240030 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:19.240003    2578 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 16 18:40:19.251144 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:19.251120    2578 log.go:25] "Validated CRI v1 runtime API"
Sep 16 18:40:19.254091 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:19.254074    2578 log.go:25] "Validated CRI v1 image API"
Sep 16 18:40:19.254900 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:19.254886    2578 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Sep 16 18:40:19.257373 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:19.257354    2578 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 c64079c0-72e1-4e48-82c3-3e4255021230:/dev/nvme0n1p3 f0363b4a-1ac1-49cb-b1cf-b2a0a529ff64:/dev/nvme0n1p4]
Sep 16 18:40:19.257445 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:19.257372    2578 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Sep 16 18:40:19.262436 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:19.262322    2578 manager.go:216] Machine: {Timestamp:2026-09-16 18:40:19.260950656 +0000 UTC m=+0.327442154 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3117685 MemoryCapacity:32812150784 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2ced7d7286d0305cc58b611d62dab4 SystemUUID:ec2ced7d-7286-d030-5cc5-8b611d62dab4 BootID:c7a78ab0-0f4e-4e61-9579-a93613ec38cd Filesystems:[{Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16406073344 Type:vfs Inodes:4005389 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6562430976 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16406077440 Type:vfs Inodes:1048576 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:cb:fa:68:97 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:cb:fa:68:97 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:8e:32:61:95:6d:80 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812150784 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Sep 16 18:40:19.262436 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:19.262428    2578 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Sep 16 18:40:19.262581 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:19.262503    2578 manager.go:232] Version: {KernelVersion:5.14.0-570.140.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260908-1 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Sep 16 18:40:19.265482 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:19.265454    2578 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Sep 16 18:40:19.265636 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:19.265485    2578 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-135-119.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Sep 16 18:40:19.266239 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:19.266228    2578 topology_manager.go:138] "Creating topology manager with none policy"
Sep 16 18:40:19.266304 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:19.266246    2578 container_manager_linux.go:309] "Creating device plugin manager"
Sep 16 18:40:19.266304 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:19.266257    2578 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 16 18:40:19.267164 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:19.267152    2578 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 16 18:40:19.268065 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:19.268054    2578 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Sep 16 18:40:19.268252 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:19.268235    2578 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 16 18:40:19.272338 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:19.272323    2578 state_mem.go:36] "Initialized new in-memory state store"
Sep 16 18:40:19.272464 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:19.272453    2578 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Sep 16 18:40:19.272520 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:19.272491    2578 kubelet.go:486] "Attempting to sync node with API server"
Sep 16 18:40:19.272520 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:19.272506    2578 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Sep 16 18:40:19.272613 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:19.272528    2578 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Sep 16 18:40:19.272613 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:19.272541    2578 kubelet.go:398] "Adding apiserver pod source"
Sep 16 18:40:19.272613 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:19.272553    2578 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Sep 16 18:40:19.272911 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:19.272898    2578 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 16 18:40:19.272994 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:19.272921    2578 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 16 18:40:19.274859 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:19.274843    2578 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.12-5.rhaos4.21.gitd1ec63c.el9" apiVersion="v1"
Sep 16 18:40:19.276600 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:19.276587    2578 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Sep 16 18:40:19.276642 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:19.276617    2578 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Sep 16 18:40:19.278997 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:19.278964    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Sep 16 18:40:19.279077 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:19.278998    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Sep 16 18:40:19.279077 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:19.279005    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Sep 16 18:40:19.279077 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:19.279010    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Sep 16 18:40:19.279077 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:19.279019    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Sep 16 18:40:19.279077 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:19.279024    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Sep 16 18:40:19.279077 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:19.279038    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Sep 16 18:40:19.279077 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:19.279043    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Sep 16 18:40:19.279077 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:19.279051    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Sep 16 18:40:19.279077 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:19.279057    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Sep 16 18:40:19.279077 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:19.279070    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Sep 16 18:40:19.279077 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:19.279076    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Sep 16 18:40:19.279751 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:19.279741    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Sep 16 18:40:19.279779 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:19.279751    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Sep 16 18:40:19.282222 ip-10-0-135-119 kubenswrapper[2578]: E0916 18:40:19.282189    2578 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Sep 16 18:40:19.282222 ip-10-0-135-119 kubenswrapper[2578]: E0916 18:40:19.282209    2578 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-135-119.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Sep 16 18:40:19.282370 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:19.282356    2578 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-135-119.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 16 18:40:19.285399 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:19.285387    2578 server.go:1268] "Started kubelet"
Sep 16 18:40:19.285522 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:19.285501    2578 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Sep 16 18:40:19.285522 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:19.285483    2578 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Sep 16 18:40:19.285615 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:19.285551    2578 server_v1.go:49] "podresources" method="list" useActivePods=true
Sep 16 18:40:19.288710 ip-10-0-135-119 systemd[1]: Started Kubernetes Kubelet.
Sep 16 18:40:19.288926 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:19.288907    2578 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Sep 16 18:40:19.291316 ip-10-0-135-119 kubenswrapper[2578]: E0916 18:40:19.291289    2578 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Sep 16 18:40:19.291922 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:19.291897    2578 server.go:310] "Adding debug handlers to kubelet server"
Sep 16 18:40:19.293623 ip-10-0-135-119 kubenswrapper[2578]: E0916 18:40:19.292606    2578 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-135-119.ec2.internal.18d5e12f153830c8  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-135-119.ec2.internal,UID:ip-10-0-135-119.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-135-119.ec2.internal,},FirstTimestamp:2026-09-16 18:40:19.285356744 +0000 UTC m=+0.351848222,LastTimestamp:2026-09-16 18:40:19.285356744 +0000 UTC m=+0.351848222,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-135-119.ec2.internal,}"
Sep 16 18:40:19.295835 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:19.295816    2578 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Sep 16 18:40:19.296382 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:19.296364    2578 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Sep 16 18:40:19.296473 ip-10-0-135-119 kubenswrapper[2578]: E0916 18:40:19.296454    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-119.ec2.internal\" not found"
Sep 16 18:40:19.296929 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:19.296911    2578 volume_manager.go:311] "The desired_state_of_world populator starts"
Sep 16 18:40:19.296929 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:19.296930    2578 volume_manager.go:313] "Starting Kubelet Volume Manager"
Sep 16 18:40:19.297103 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:19.296910    2578 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Sep 16 18:40:19.297103 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:19.297061    2578 reconstruct.go:97] "Volume reconstruction finished"
Sep 16 18:40:19.297103 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:19.297068    2578 reconciler.go:29] "Reconciler: start to sync state"
Sep 16 18:40:19.297234 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:19.297147    2578 factory.go:55] Registering systemd factory
Sep 16 18:40:19.297234 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:19.297162    2578 factory.go:223] Registration of the systemd container factory successfully
Sep 16 18:40:19.297360 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:19.297347    2578 factory.go:153] Registering CRI-O factory
Sep 16 18:40:19.297360 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:19.297361    2578 factory.go:223] Registration of the crio container factory successfully
Sep 16 18:40:19.297438 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:19.297400    2578 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Sep 16 18:40:19.297438 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:19.297419    2578 factory.go:103] Registering Raw factory
Sep 16 18:40:19.297438 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:19.297429    2578 manager.go:1196] Started watching for new ooms in manager
Sep 16 18:40:19.298227 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:19.298215    2578 manager.go:349] Starting recovery of all containers
Sep 16 18:40:19.307136 ip-10-0-135-119 kubenswrapper[2578]: E0916 18:40:19.307110    2578 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-135-119.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Sep 16 18:40:19.307136 ip-10-0-135-119 kubenswrapper[2578]: E0916 18:40:19.307125    2578 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Sep 16 18:40:19.308238 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:19.308224    2578 manager.go:354] Recovery completed
Sep 16 18:40:19.313081 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:19.313066    2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 18:40:19.314746 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:19.314727    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-119.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 18:40:19.314805 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:19.314758    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-119.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 18:40:19.314805 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:19.314768    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-119.ec2.internal" event="NodeHasSufficientPID"
Sep 16 18:40:19.315351 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:19.315337    2578 cpu_manager.go:222] "Starting CPU manager" policy="none"
Sep 16 18:40:19.315351 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:19.315350    2578 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Sep 16 18:40:19.315501 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:19.315358    2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 18:40:19.315501 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:19.315366    2578 state_mem.go:36] "Initialized new in-memory state store"
Sep 16 18:40:19.316233 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:19.316215    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-119.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 18:40:19.316284 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:19.316252    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-119.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 18:40:19.316284 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:19.316266    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-119.ec2.internal" event="NodeHasSufficientPID"
Sep 16 18:40:19.317526 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:19.317514    2578 policy_none.go:49] "None policy: Start"
Sep 16 18:40:19.317569 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:19.317530    2578 memory_manager.go:187] "Starting memorymanager" policy="None"
Sep 16 18:40:19.317569 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:19.317540    2578 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Sep 16 18:40:19.319472 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:19.319459    2578 policy_none.go:47] "Start"
Sep 16 18:40:19.322148 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:19.322133    2578 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-7r46n"
Sep 16 18:40:19.323377 ip-10-0-135-119 kubenswrapper[2578]: E0916 18:40:19.323297    2578 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-135-119.ec2.internal.18d5e12f16f8a1fa  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-135-119.ec2.internal,UID:ip-10-0-135-119.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-135-119.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-135-119.ec2.internal,},FirstTimestamp:2026-09-16 18:40:19.31474585 +0000 UTC m=+0.381237332,LastTimestamp:2026-09-16 18:40:19.31474585 +0000 UTC m=+0.381237332,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-135-119.ec2.internal,}"
Sep 16 18:40:19.329101 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:19.329082    2578 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-7r46n"
Sep 16 18:40:19.357545 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:19.357531    2578 manager.go:337] "Starting Device Plugin manager"
Sep 16 18:40:19.376883 ip-10-0-135-119 kubenswrapper[2578]: E0916 18:40:19.357569    2578 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Sep 16 18:40:19.376883 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:19.357579    2578 server.go:87] "Starting device plugin registration server"
Sep 16 18:40:19.376883 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:19.357811    2578 eviction_manager.go:189] "Eviction manager: starting control loop"
Sep 16 18:40:19.376883 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:19.357822    2578 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Sep 16 18:40:19.376883 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:19.357940    2578 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Sep 16 18:40:19.376883 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:19.358030    2578 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Sep 16 18:40:19.376883 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:19.358039    2578 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Sep 16 18:40:19.376883 ip-10-0-135-119 kubenswrapper[2578]: E0916 18:40:19.358518    2578 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Sep 16 18:40:19.376883 ip-10-0-135-119 kubenswrapper[2578]: E0916 18:40:19.358560    2578 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-135-119.ec2.internal\" not found"
Sep 16 18:40:19.426288 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:19.426257    2578 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Sep 16 18:40:19.427674 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:19.427655    2578 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Sep 16 18:40:19.427773 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:19.427700    2578 status_manager.go:244] "Starting to sync pod status with apiserver"
Sep 16 18:40:19.427773 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:19.427723    2578 kubelet.go:2448] "Starting kubelet main sync loop"
Sep 16 18:40:19.427773 ip-10-0-135-119 kubenswrapper[2578]: E0916 18:40:19.427770    2578 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Sep 16 18:40:19.430434 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:19.430412    2578 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 16 18:40:19.458589 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:19.458542    2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 18:40:19.459321 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:19.459307    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-119.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 18:40:19.459374 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:19.459333    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-119.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 18:40:19.459374 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:19.459344    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-119.ec2.internal" event="NodeHasSufficientPID"
Sep 16 18:40:19.459374 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:19.459364    2578 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-135-119.ec2.internal"
Sep 16 18:40:19.468739 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:19.468726    2578 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-135-119.ec2.internal"
Sep 16 18:40:19.468790 ip-10-0-135-119 kubenswrapper[2578]: E0916 18:40:19.468744    2578 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-135-119.ec2.internal\": node \"ip-10-0-135-119.ec2.internal\" not found"
Sep 16 18:40:19.503913 ip-10-0-135-119 kubenswrapper[2578]: E0916 18:40:19.503897    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-119.ec2.internal\" not found"
Sep 16 18:40:19.528658 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:19.528639    2578 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["kube-system/kube-apiserver-proxy-ip-10-0-135-119.ec2.internal","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-119.ec2.internal"]
Sep 16 18:40:19.528729 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:19.528695    2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 18:40:19.530086 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:19.530072    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-119.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 18:40:19.530152 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:19.530097    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-119.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 18:40:19.530152 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:19.530106    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-119.ec2.internal" event="NodeHasSufficientPID"
Sep 16 18:40:19.532365 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:19.532353    2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 18:40:19.532477 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:19.532459    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-119.ec2.internal"
Sep 16 18:40:19.532564 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:19.532497    2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 18:40:19.533058 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:19.533045    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-119.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 18:40:19.533136 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:19.533050    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-119.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 18:40:19.533136 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:19.533094    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-119.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 18:40:19.533136 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:19.533112    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-119.ec2.internal" event="NodeHasSufficientPID"
Sep 16 18:40:19.533136 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:19.533069    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-119.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 18:40:19.533254 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:19.533140    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-119.ec2.internal" event="NodeHasSufficientPID"
Sep 16 18:40:19.535851 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:19.535836    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-119.ec2.internal"
Sep 16 18:40:19.535925 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:19.535861    2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 18:40:19.536595 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:19.536581    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-119.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 18:40:19.536655 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:19.536606    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-119.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 18:40:19.536655 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:19.536616    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-119.ec2.internal" event="NodeHasSufficientPID"
Sep 16 18:40:19.559529 ip-10-0-135-119 kubenswrapper[2578]: E0916 18:40:19.559509    2578 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-135-119.ec2.internal\" not found" node="ip-10-0-135-119.ec2.internal"
Sep 16 18:40:19.563896 ip-10-0-135-119 kubenswrapper[2578]: E0916 18:40:19.563882    2578 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-135-119.ec2.internal\" not found" node="ip-10-0-135-119.ec2.internal"
Sep 16 18:40:19.599306 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:19.599281    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/ce93b5afb1de6b9a80d64df7da81462d-config\") pod \"kube-apiserver-proxy-ip-10-0-135-119.ec2.internal\" (UID: \"ce93b5afb1de6b9a80d64df7da81462d\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-119.ec2.internal"
Sep 16 18:40:19.599391 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:19.599314    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/db42fcd462e8e18de823d8d0adf7cc0a-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-119.ec2.internal\" (UID: \"db42fcd462e8e18de823d8d0adf7cc0a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-119.ec2.internal"
Sep 16 18:40:19.599391 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:19.599340    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/db42fcd462e8e18de823d8d0adf7cc0a-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-119.ec2.internal\" (UID: \"db42fcd462e8e18de823d8d0adf7cc0a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-119.ec2.internal"
Sep 16 18:40:19.604392 ip-10-0-135-119 kubenswrapper[2578]: E0916 18:40:19.604376    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-119.ec2.internal\" not found"
Sep 16 18:40:19.699487 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:19.699460    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/ce93b5afb1de6b9a80d64df7da81462d-config\") pod \"kube-apiserver-proxy-ip-10-0-135-119.ec2.internal\" (UID: \"ce93b5afb1de6b9a80d64df7da81462d\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-119.ec2.internal"
Sep 16 18:40:19.699625 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:19.699499    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/db42fcd462e8e18de823d8d0adf7cc0a-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-119.ec2.internal\" (UID: \"db42fcd462e8e18de823d8d0adf7cc0a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-119.ec2.internal"
Sep 16 18:40:19.699625 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:19.699515    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/db42fcd462e8e18de823d8d0adf7cc0a-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-119.ec2.internal\" (UID: \"db42fcd462e8e18de823d8d0adf7cc0a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-119.ec2.internal"
Sep 16 18:40:19.699625 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:19.699555    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/db42fcd462e8e18de823d8d0adf7cc0a-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-119.ec2.internal\" (UID: \"db42fcd462e8e18de823d8d0adf7cc0a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-119.ec2.internal"
Sep 16 18:40:19.699625 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:19.699575    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/ce93b5afb1de6b9a80d64df7da81462d-config\") pod \"kube-apiserver-proxy-ip-10-0-135-119.ec2.internal\" (UID: \"ce93b5afb1de6b9a80d64df7da81462d\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-119.ec2.internal"
Sep 16 18:40:19.699625 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:19.699584    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/db42fcd462e8e18de823d8d0adf7cc0a-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-119.ec2.internal\" (UID: \"db42fcd462e8e18de823d8d0adf7cc0a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-119.ec2.internal"
Sep 16 18:40:19.704559 ip-10-0-135-119 kubenswrapper[2578]: E0916 18:40:19.704545    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-119.ec2.internal\" not found"
Sep 16 18:40:19.805405 ip-10-0-135-119 kubenswrapper[2578]: E0916 18:40:19.805334    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-119.ec2.internal\" not found"
Sep 16 18:40:19.862536 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:19.862512    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-119.ec2.internal"
Sep 16 18:40:19.866115 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:19.866099    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-119.ec2.internal"
Sep 16 18:40:19.906044 ip-10-0-135-119 kubenswrapper[2578]: E0916 18:40:19.905880    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-119.ec2.internal\" not found"
Sep 16 18:40:20.006390 ip-10-0-135-119 kubenswrapper[2578]: E0916 18:40:20.006354    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-119.ec2.internal\" not found"
Sep 16 18:40:20.106887 ip-10-0-135-119 kubenswrapper[2578]: E0916 18:40:20.106811    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-119.ec2.internal\" not found"
Sep 16 18:40:20.110145 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:20.110128    2578 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 16 18:40:20.136911 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:20.136889    2578 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 16 18:40:20.197174 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:20.197144    2578 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-119.ec2.internal"
Sep 16 18:40:20.208786 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:20.208763    2578 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 16 18:40:20.210375 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:20.210362    2578 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-119.ec2.internal"
Sep 16 18:40:20.227906 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:20.227887    2578 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 16 18:40:20.231045 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:20.231030    2578 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Sep 16 18:40:20.231140 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:20.231120    2578 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 16 18:40:20.231193 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:20.231119    2578 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 16 18:40:20.231193 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:20.231143    2578 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 16 18:40:20.273338 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:20.273318    2578 apiserver.go:52] "Watching apiserver"
Sep 16 18:40:20.290329 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:20.290308    2578 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Sep 16 18:40:20.290640 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:20.290621    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-multus/multus-llvsl","openshift-multus/network-metrics-daemon-zfv2x","openshift-ovn-kubernetes/ovnkube-node-k6gvs","kube-system/global-pull-secret-syncer-8ndsh","kube-system/konnectivity-agent-f69wc","openshift-dns/node-resolver-5rrj4","openshift-image-registry/node-ca-bl65n","openshift-network-diagnostics/network-check-target-f287c","openshift-network-operator/iptables-alerter-ncnts","kube-system/kube-apiserver-proxy-ip-10-0-135-119.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wrx94","openshift-cluster-node-tuning-operator/tuned-hqxdq","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-119.ec2.internal","openshift-multus/multus-additional-cni-plugins-dxtkl"]
Sep 16 18:40:20.293018 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:20.293000    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-f287c"
Sep 16 18:40:20.293090 ip-10-0-135-119 kubenswrapper[2578]: E0916 18:40:20.293064    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-f287c" podUID="63abc484-192b-4095-81ea-b5db79acd0ba"
Sep 16 18:40:20.295033 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:20.295020    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zfv2x"
Sep 16 18:40:20.295091 ip-10-0-135-119 kubenswrapper[2578]: E0916 18:40:20.295077    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zfv2x" podUID="1f969e62-dd58-41b0-9419-b2ca445f3ec4"
Sep 16 18:40:20.295903 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:20.295890    2578 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Sep 16 18:40:20.297262 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:20.297251    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-k6gvs"
Sep 16 18:40:20.300050 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:20.299800    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-s6pc4\""
Sep 16 18:40:20.300050 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:20.299888    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Sep 16 18:40:20.300725 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:20.300221    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Sep 16 18:40:20.300725 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:20.300287    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Sep 16 18:40:20.300725 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:20.300288    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Sep 16 18:40:20.300725 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:20.300562    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Sep 16 18:40:20.300725 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:20.300600    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Sep 16 18:40:20.300725 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:20.300678    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-dxtkl"
Sep 16 18:40:20.302416 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:20.302396    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6012728d-aee6-4f93-ab6e-fad616ef36e8-host-run-netns\") pod \"ovnkube-node-k6gvs\" (UID: \"6012728d-aee6-4f93-ab6e-fad616ef36e8\") " pod="openshift-ovn-kubernetes/ovnkube-node-k6gvs"
Sep 16 18:40:20.302511 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:20.302421    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6012728d-aee6-4f93-ab6e-fad616ef36e8-etc-openvswitch\") pod \"ovnkube-node-k6gvs\" (UID: \"6012728d-aee6-4f93-ab6e-fad616ef36e8\") " pod="openshift-ovn-kubernetes/ovnkube-node-k6gvs"
Sep 16 18:40:20.302511 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:20.302435    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/6012728d-aee6-4f93-ab6e-fad616ef36e8-node-log\") pod \"ovnkube-node-k6gvs\" (UID: \"6012728d-aee6-4f93-ab6e-fad616ef36e8\") " pod="openshift-ovn-kubernetes/ovnkube-node-k6gvs"
Sep 16 18:40:20.302511 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:20.302450    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-25hb5\" (UniqueName: \"kubernetes.io/projected/6012728d-aee6-4f93-ab6e-fad616ef36e8-kube-api-access-25hb5\") pod \"ovnkube-node-k6gvs\" (UID: \"6012728d-aee6-4f93-ab6e-fad616ef36e8\") " pod="openshift-ovn-kubernetes/ovnkube-node-k6gvs"
Sep 16 18:40:20.302511 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:20.302469    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-ncnts"
Sep 16 18:40:20.302511 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:20.302487    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/6012728d-aee6-4f93-ab6e-fad616ef36e8-log-socket\") pod \"ovnkube-node-k6gvs\" (UID: \"6012728d-aee6-4f93-ab6e-fad616ef36e8\") " pod="openshift-ovn-kubernetes/ovnkube-node-k6gvs"
Sep 16 18:40:20.302775 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:20.302540    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6012728d-aee6-4f93-ab6e-fad616ef36e8-host-run-ovn-kubernetes\") pod \"ovnkube-node-k6gvs\" (UID: \"6012728d-aee6-4f93-ab6e-fad616ef36e8\") " pod="openshift-ovn-kubernetes/ovnkube-node-k6gvs"
Sep 16 18:40:20.302775 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:20.302572    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6012728d-aee6-4f93-ab6e-fad616ef36e8-host-cni-bin\") pod \"ovnkube-node-k6gvs\" (UID: \"6012728d-aee6-4f93-ab6e-fad616ef36e8\") " pod="openshift-ovn-kubernetes/ovnkube-node-k6gvs"
Sep 16 18:40:20.302775 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:20.302607    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/6012728d-aee6-4f93-ab6e-fad616ef36e8-ovnkube-config\") pod \"ovnkube-node-k6gvs\" (UID: \"6012728d-aee6-4f93-ab6e-fad616ef36e8\") " pod="openshift-ovn-kubernetes/ovnkube-node-k6gvs"
Sep 16 18:40:20.302775 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:20.302652    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/6012728d-aee6-4f93-ab6e-fad616ef36e8-ovn-node-metrics-cert\") pod \"ovnkube-node-k6gvs\" (UID: \"6012728d-aee6-4f93-ab6e-fad616ef36e8\") " pod="openshift-ovn-kubernetes/ovnkube-node-k6gvs"
Sep 16 18:40:20.302775 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:20.302678    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-c7mv8\""
Sep 16 18:40:20.302775 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:20.302677    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/6012728d-aee6-4f93-ab6e-fad616ef36e8-host-cni-netd\") pod \"ovnkube-node-k6gvs\" (UID: \"6012728d-aee6-4f93-ab6e-fad616ef36e8\") " pod="openshift-ovn-kubernetes/ovnkube-node-k6gvs"
Sep 16 18:40:20.303139 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:20.302788    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/91876c9b-d59f-4d80-bda9-f7addebaaee3-system-cni-dir\") pod \"multus-additional-cni-plugins-dxtkl\" (UID: \"91876c9b-d59f-4d80-bda9-f7addebaaee3\") " pod="openshift-multus/multus-additional-cni-plugins-dxtkl"
Sep 16 18:40:20.303139 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:20.302813    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/6012728d-aee6-4f93-ab6e-fad616ef36e8-run-systemd\") pod \"ovnkube-node-k6gvs\" (UID: \"6012728d-aee6-4f93-ab6e-fad616ef36e8\") " pod="openshift-ovn-kubernetes/ovnkube-node-k6gvs"
Sep 16 18:40:20.303139 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:20.302834    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6012728d-aee6-4f93-ab6e-fad616ef36e8-var-lib-openvswitch\") pod \"ovnkube-node-k6gvs\" (UID: \"6012728d-aee6-4f93-ab6e-fad616ef36e8\") " pod="openshift-ovn-kubernetes/ovnkube-node-k6gvs"
Sep 16 18:40:20.303139 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:20.302857    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/91876c9b-d59f-4d80-bda9-f7addebaaee3-tuning-conf-dir\") pod \"multus-additional-cni-plugins-dxtkl\" (UID: \"91876c9b-d59f-4d80-bda9-f7addebaaee3\") " pod="openshift-multus/multus-additional-cni-plugins-dxtkl"
Sep 16 18:40:20.303139 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:20.302893    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/91876c9b-d59f-4d80-bda9-f7addebaaee3-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-dxtkl\" (UID: \"91876c9b-d59f-4d80-bda9-f7addebaaee3\") " pod="openshift-multus/multus-additional-cni-plugins-dxtkl"
Sep 16 18:40:20.303139 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:20.302904    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Sep 16 18:40:20.303139 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:20.302930    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Sep 16 18:40:20.303139 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:20.302930    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Sep 16 18:40:20.303139 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:20.302954    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Sep 16 18:40:20.303139 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:20.302925    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jjh8v\" (UniqueName: \"kubernetes.io/projected/91876c9b-d59f-4d80-bda9-f7addebaaee3-kube-api-access-jjh8v\") pod \"multus-additional-cni-plugins-dxtkl\" (UID: \"91876c9b-d59f-4d80-bda9-f7addebaaee3\") " pod="openshift-multus/multus-additional-cni-plugins-dxtkl"
Sep 16 18:40:20.303139 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:20.303000    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Sep 16 18:40:20.303139 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:20.303020    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6012728d-aee6-4f93-ab6e-fad616ef36e8-run-openvswitch\") pod \"ovnkube-node-k6gvs\" (UID: \"6012728d-aee6-4f93-ab6e-fad616ef36e8\") " pod="openshift-ovn-kubernetes/ovnkube-node-k6gvs"
Sep 16 18:40:20.303139 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:20.303048    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/91876c9b-d59f-4d80-bda9-f7addebaaee3-cnibin\") pod \"multus-additional-cni-plugins-dxtkl\" (UID: \"91876c9b-d59f-4d80-bda9-f7addebaaee3\") " pod="openshift-multus/multus-additional-cni-plugins-dxtkl"
Sep 16 18:40:20.303139 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:20.303070    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/91876c9b-d59f-4d80-bda9-f7addebaaee3-cni-binary-copy\") pod \"multus-additional-cni-plugins-dxtkl\" (UID: \"91876c9b-d59f-4d80-bda9-f7addebaaee3\") " pod="openshift-multus/multus-additional-cni-plugins-dxtkl"
Sep 16 18:40:20.303139 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:20.303086    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/91876c9b-d59f-4d80-bda9-f7addebaaee3-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-dxtkl\" (UID: \"91876c9b-d59f-4d80-bda9-f7addebaaee3\") " pod="openshift-multus/multus-additional-cni-plugins-dxtkl"
Sep 16 18:40:20.303139 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:20.303101    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/6012728d-aee6-4f93-ab6e-fad616ef36e8-host-kubelet\") pod \"ovnkube-node-k6gvs\" (UID: \"6012728d-aee6-4f93-ab6e-fad616ef36e8\") " pod="openshift-ovn-kubernetes/ovnkube-node-k6gvs"
Sep 16 18:40:20.303139 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:20.303145    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/6012728d-aee6-4f93-ab6e-fad616ef36e8-systemd-units\") pod \"ovnkube-node-k6gvs\" (UID: \"6012728d-aee6-4f93-ab6e-fad616ef36e8\") " pod="openshift-ovn-kubernetes/ovnkube-node-k6gvs"
Sep 16 18:40:20.303635 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:20.303160    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/6012728d-aee6-4f93-ab6e-fad616ef36e8-host-slash\") pod \"ovnkube-node-k6gvs\" (UID: \"6012728d-aee6-4f93-ab6e-fad616ef36e8\") " pod="openshift-ovn-kubernetes/ovnkube-node-k6gvs"
Sep 16 18:40:20.303635 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:20.303175    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6012728d-aee6-4f93-ab6e-fad616ef36e8-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-k6gvs\" (UID: \"6012728d-aee6-4f93-ab6e-fad616ef36e8\") " pod="openshift-ovn-kubernetes/ovnkube-node-k6gvs"
Sep 16 18:40:20.303635 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:20.303199    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-77jgz\" (UniqueName: \"kubernetes.io/projected/63abc484-192b-4095-81ea-b5db79acd0ba-kube-api-access-77jgz\") pod \"network-check-target-f287c\" (UID: \"63abc484-192b-4095-81ea-b5db79acd0ba\") " pod="openshift-network-diagnostics/network-check-target-f287c"
Sep 16 18:40:20.303635 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:20.303223    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1f969e62-dd58-41b0-9419-b2ca445f3ec4-metrics-certs\") pod \"network-metrics-daemon-zfv2x\" (UID: \"1f969e62-dd58-41b0-9419-b2ca445f3ec4\") " pod="openshift-multus/network-metrics-daemon-zfv2x"
Sep 16 18:40:20.303635 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:20.303249    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4gc49\" (UniqueName: \"kubernetes.io/projected/1f969e62-dd58-41b0-9419-b2ca445f3ec4-kube-api-access-4gc49\") pod \"network-metrics-daemon-zfv2x\" (UID: \"1f969e62-dd58-41b0-9419-b2ca445f3ec4\") " pod="openshift-multus/network-metrics-daemon-zfv2x"
Sep 16 18:40:20.303635 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:20.303269    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/6012728d-aee6-4f93-ab6e-fad616ef36e8-run-ovn\") pod \"ovnkube-node-k6gvs\" (UID: \"6012728d-aee6-4f93-ab6e-fad616ef36e8\") " pod="openshift-ovn-kubernetes/ovnkube-node-k6gvs"
Sep 16 18:40:20.303635 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:20.303307    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/6012728d-aee6-4f93-ab6e-fad616ef36e8-ovnkube-script-lib\") pod \"ovnkube-node-k6gvs\" (UID: \"6012728d-aee6-4f93-ab6e-fad616ef36e8\") " pod="openshift-ovn-kubernetes/ovnkube-node-k6gvs"
Sep 16 18:40:20.303635 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:20.303330    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/6012728d-aee6-4f93-ab6e-fad616ef36e8-env-overrides\") pod \"ovnkube-node-k6gvs\" (UID: \"6012728d-aee6-4f93-ab6e-fad616ef36e8\") " pod="openshift-ovn-kubernetes/ovnkube-node-k6gvs"
Sep 16 18:40:20.304387 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:20.304371    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Sep 16 18:40:20.304508 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:20.304492    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-92g55\""
Sep 16 18:40:20.304599 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:20.304584    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Sep 16 18:40:20.304679 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:20.304665    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-bl65n"
Sep 16 18:40:20.305035 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:20.305021    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Sep 16 18:40:20.306310 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:20.306292    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Sep 16 18:40:20.306394 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:20.306344    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-87cgd\""
Sep 16 18:40:20.306394 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:20.306345    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Sep 16 18:40:20.306598 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:20.306586    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Sep 16 18:40:20.306938 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:20.306924    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-llvsl"
Sep 16 18:40:20.308623 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:20.308606    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-crk2x\""
Sep 16 18:40:20.308794 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:20.308779    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Sep 16 18:40:20.309251 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:20.309222    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-5rrj4"
Sep 16 18:40:20.311174 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:20.311156    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Sep 16 18:40:20.311373 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:20.311360    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-f69wc"
Sep 16 18:40:20.311548 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:20.311531    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-wh7wb\""
Sep 16 18:40:20.311651 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:20.311640    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Sep 16 18:40:20.313475 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:20.313431    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Sep 16 18:40:20.313475 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:20.313463    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-mgr49\""
Sep 16 18:40:20.313627 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:20.313508    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Sep 16 18:40:20.313767 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:20.313753    2578 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 16 18:40:20.314170 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:20.314140    2578 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 16 18:40:20.314252 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:20.314237    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wrx94"
Sep 16 18:40:20.316079 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:20.316057    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Sep 16 18:40:20.316229 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:20.316210    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Sep 16 18:40:20.316278 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:20.316270    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Sep 16 18:40:20.316336 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:20.316214    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-p2qjd\""
Sep 16 18:40:20.316566 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:20.316551    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-hqxdq"
Sep 16 18:40:20.318777 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:20.318423    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Sep 16 18:40:20.318777 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:20.318419    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Sep 16 18:40:20.318777 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:20.318560    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-jr6mz\""
Sep 16 18:40:20.319686 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:20.319671    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8ndsh"
Sep 16 18:40:20.319739 ip-10-0-135-119 kubenswrapper[2578]: E0916 18:40:20.319724    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8ndsh" podUID="3377e453-9418-49a8-870b-64424f13d694"
Sep 16 18:40:20.333067 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:20.333012    2578 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-15 18:35:19 +0000 UTC" deadline="2028-04-04 16:06:48.988863331 +0000 UTC"
Sep 16 18:40:20.333067 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:20.333065    2578 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13581h26m28.655801436s"
Sep 16 18:40:20.391433 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:20.391402    2578 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-h8scr"
Sep 16 18:40:20.398397 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:20.398378    2578 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Sep 16 18:40:20.403155 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:20.403139    2578 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-h8scr"
Sep 16 18:40:20.404038 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:20.404020    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/62104ee5-c623-4ffd-b3af-9516ffd5f3bf-cnibin\") pod \"multus-llvsl\" (UID: \"62104ee5-c623-4ffd-b3af-9516ffd5f3bf\") " pod="openshift-multus/multus-llvsl"
Sep 16 18:40:20.404091 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:20.404048    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/454c7c51-a475-44f6-be51-1a8f1d29beaf-serviceca\") pod \"node-ca-bl65n\" (UID: \"454c7c51-a475-44f6-be51-1a8f1d29beaf\") " pod="openshift-image-registry/node-ca-bl65n"
Sep 16 18:40:20.404091 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:20.404062    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/92e18a5c-2e80-4f2d-8602-d64ca37db647-etc-systemd\") pod \"tuned-hqxdq\" (UID: \"92e18a5c-2e80-4f2d-8602-d64ca37db647\") " pod="openshift-cluster-node-tuning-operator/tuned-hqxdq"
Sep 16 18:40:20.404091 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:20.404083    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/62104ee5-c623-4ffd-b3af-9516ffd5f3bf-hostroot\") pod \"multus-llvsl\" (UID: \"62104ee5-c623-4ffd-b3af-9516ffd5f3bf\") " pod="openshift-multus/multus-llvsl"
Sep 16 18:40:20.404198 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:20.404155    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/3f2c98a3-c6e7-4610-b207-f7ccde3cb571-kubelet-dir\") pod \"aws-ebs-csi-driver-node-wrx94\" (UID: \"3f2c98a3-c6e7-4610-b207-f7ccde3cb571\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wrx94"
Sep 16 18:40:20.404198 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:20.404177    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/454c7c51-a475-44f6-be51-1a8f1d29beaf-host\") pod \"node-ca-bl65n\" (UID: \"454c7c51-a475-44f6-be51-1a8f1d29beaf\") " pod="openshift-image-registry/node-ca-bl65n"
Sep 16 18:40:20.404198 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:20.404197    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6012728d-aee6-4f93-ab6e-fad616ef36e8-run-openvswitch\") pod \"ovnkube-node-k6gvs\" (UID: \"6012728d-aee6-4f93-ab6e-fad616ef36e8\") " pod="openshift-ovn-kubernetes/ovnkube-node-k6gvs"
Sep 16 18:40:20.404313 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:20.404211    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/f4653546-2f57-4858-a038-9fd7c2139d56-agent-certs\") pod \"konnectivity-agent-f69wc\" (UID: \"f4653546-2f57-4858-a038-9fd7c2139d56\") " pod="kube-system/konnectivity-agent-f69wc"
Sep 16 18:40:20.404313 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:20.404241    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/3f2c98a3-c6e7-4610-b207-f7ccde3cb571-socket-dir\") pod \"aws-ebs-csi-driver-node-wrx94\" (UID: \"3f2c98a3-c6e7-4610-b207-f7ccde3cb571\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wrx94"
Sep 16 18:40:20.404313 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:20.404255    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/92e18a5c-2e80-4f2d-8602-d64ca37db647-etc-sysctl-d\") pod \"tuned-hqxdq\" (UID: \"92e18a5c-2e80-4f2d-8602-d64ca37db647\") " pod="openshift-cluster-node-tuning-operator/tuned-hqxdq"
Sep 16 18:40:20.404313 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:20.404270    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/92e18a5c-2e80-4f2d-8602-d64ca37db647-etc-sysctl-conf\") pod \"tuned-hqxdq\" (UID: \"92e18a5c-2e80-4f2d-8602-d64ca37db647\") " pod="openshift-cluster-node-tuning-operator/tuned-hqxdq"
Sep 16 18:40:20.404313 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:20.404289    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/92e18a5c-2e80-4f2d-8602-d64ca37db647-sys\") pod \"tuned-hqxdq\" (UID: \"92e18a5c-2e80-4f2d-8602-d64ca37db647\") " pod="openshift-cluster-node-tuning-operator/tuned-hqxdq"
Sep 16 18:40:20.404469 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:20.404303    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6012728d-aee6-4f93-ab6e-fad616ef36e8-run-openvswitch\") pod \"ovnkube-node-k6gvs\" (UID: \"6012728d-aee6-4f93-ab6e-fad616ef36e8\") " pod="openshift-ovn-kubernetes/ovnkube-node-k6gvs"
Sep 16 18:40:20.404469 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:20.404319    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/91876c9b-d59f-4d80-bda9-f7addebaaee3-cni-binary-copy\") pod \"multus-additional-cni-plugins-dxtkl\" (UID: \"91876c9b-d59f-4d80-bda9-f7addebaaee3\") " pod="openshift-multus/multus-additional-cni-plugins-dxtkl"
Sep 16 18:40:20.404469 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:20.404378    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/6012728d-aee6-4f93-ab6e-fad616ef36e8-host-kubelet\") pod \"ovnkube-node-k6gvs\" (UID: \"6012728d-aee6-4f93-ab6e-fad616ef36e8\") " pod="openshift-ovn-kubernetes/ovnkube-node-k6gvs"
Sep 16 18:40:20.404469 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:20.404408    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/6012728d-aee6-4f93-ab6e-fad616ef36e8-systemd-units\") pod \"ovnkube-node-k6gvs\" (UID: \"6012728d-aee6-4f93-ab6e-fad616ef36e8\") " pod="openshift-ovn-kubernetes/ovnkube-node-k6gvs"
Sep 16 18:40:20.404469 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:20.404435    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/6012728d-aee6-4f93-ab6e-fad616ef36e8-host-slash\") pod \"ovnkube-node-k6gvs\" (UID: \"6012728d-aee6-4f93-ab6e-fad616ef36e8\") " pod="openshift-ovn-kubernetes/ovnkube-node-k6gvs"
Sep 16 18:40:20.404469 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:20.404465    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/62104ee5-c623-4ffd-b3af-9516ffd5f3bf-cni-binary-copy\") pod \"multus-llvsl\" (UID: \"62104ee5-c623-4ffd-b3af-9516ffd5f3bf\") " pod="openshift-multus/multus-llvsl"
Sep 16 18:40:20.404653 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:20.404488    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/6012728d-aee6-4f93-ab6e-fad616ef36e8-host-slash\") pod \"ovnkube-node-k6gvs\" (UID: \"6012728d-aee6-4f93-ab6e-fad616ef36e8\") " pod="openshift-ovn-kubernetes/ovnkube-node-k6gvs"
Sep 16 18:40:20.404653 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:20.404491    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/6012728d-aee6-4f93-ab6e-fad616ef36e8-systemd-units\") pod \"ovnkube-node-k6gvs\" (UID: \"6012728d-aee6-4f93-ab6e-fad616ef36e8\") " pod="openshift-ovn-kubernetes/ovnkube-node-k6gvs"
Sep 16 18:40:20.404653 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:20.404467    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/6012728d-aee6-4f93-ab6e-fad616ef36e8-host-kubelet\") pod \"ovnkube-node-k6gvs\" (UID: \"6012728d-aee6-4f93-ab6e-fad616ef36e8\") " pod="openshift-ovn-kubernetes/ovnkube-node-k6gvs"
Sep 16 18:40:20.404653 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:20.404489    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/92e18a5c-2e80-4f2d-8602-d64ca37db647-etc-sysconfig\") pod \"tuned-hqxdq\" (UID: \"92e18a5c-2e80-4f2d-8602-d64ca37db647\") " pod="openshift-cluster-node-tuning-operator/tuned-hqxdq"
Sep 16 18:40:20.404653 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:20.404542    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/62104ee5-c623-4ffd-b3af-9516ffd5f3bf-host-var-lib-cni-multus\") pod \"multus-llvsl\" (UID: \"62104ee5-c623-4ffd-b3af-9516ffd5f3bf\") " pod="openshift-multus/multus-llvsl"
Sep 16 18:40:20.404653 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:20.404570    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tkf6g\" (UniqueName: \"kubernetes.io/projected/fc3365ef-dfd3-45fe-b80f-937949d65751-kube-api-access-tkf6g\") pod \"node-resolver-5rrj4\" (UID: \"fc3365ef-dfd3-45fe-b80f-937949d65751\") " pod="openshift-dns/node-resolver-5rrj4"
Sep 16 18:40:20.404653 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:20.404599    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6012728d-aee6-4f93-ab6e-fad616ef36e8-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-k6gvs\" (UID: \"6012728d-aee6-4f93-ab6e-fad616ef36e8\") " pod="openshift-ovn-kubernetes/ovnkube-node-k6gvs"
Sep 16 18:40:20.404653 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:20.404625    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-77jgz\" (UniqueName: \"kubernetes.io/projected/63abc484-192b-4095-81ea-b5db79acd0ba-kube-api-access-77jgz\") pod \"network-check-target-f287c\" (UID: \"63abc484-192b-4095-81ea-b5db79acd0ba\") " pod="openshift-network-diagnostics/network-check-target-f287c"
Sep 16 18:40:20.404653 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:20.404651    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1f969e62-dd58-41b0-9419-b2ca445f3ec4-metrics-certs\") pod \"network-metrics-daemon-zfv2x\" (UID: \"1f969e62-dd58-41b0-9419-b2ca445f3ec4\") " pod="openshift-multus/network-metrics-daemon-zfv2x"
Sep 16 18:40:20.405089 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:20.404676    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4gc49\" (UniqueName: \"kubernetes.io/projected/1f969e62-dd58-41b0-9419-b2ca445f3ec4-kube-api-access-4gc49\") pod \"network-metrics-daemon-zfv2x\" (UID: \"1f969e62-dd58-41b0-9419-b2ca445f3ec4\") " pod="openshift-multus/network-metrics-daemon-zfv2x"
Sep 16 18:40:20.405089 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:20.404683    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6012728d-aee6-4f93-ab6e-fad616ef36e8-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-k6gvs\" (UID: \"6012728d-aee6-4f93-ab6e-fad616ef36e8\") " pod="openshift-ovn-kubernetes/ovnkube-node-k6gvs"
Sep 16 18:40:20.405089 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:20.404700    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/6012728d-aee6-4f93-ab6e-fad616ef36e8-ovnkube-script-lib\") pod \"ovnkube-node-k6gvs\" (UID: \"6012728d-aee6-4f93-ab6e-fad616ef36e8\") " pod="openshift-ovn-kubernetes/ovnkube-node-k6gvs"
Sep 16 18:40:20.405089 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:20.404723    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/3d323f6a-c161-4706-b9d5-8803578a6159-host-slash\") pod \"iptables-alerter-ncnts\" (UID: \"3d323f6a-c161-4706-b9d5-8803578a6159\") " pod="openshift-network-operator/iptables-alerter-ncnts"
Sep 16 18:40:20.405089 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:20.404745    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/6012728d-aee6-4f93-ab6e-fad616ef36e8-env-overrides\") pod \"ovnkube-node-k6gvs\" (UID: \"6012728d-aee6-4f93-ab6e-fad616ef36e8\") " pod="openshift-ovn-kubernetes/ovnkube-node-k6gvs"
Sep 16 18:40:20.405089 ip-10-0-135-119 kubenswrapper[2578]: E0916 18:40:20.404760    2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 18:40:20.405089 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:20.404771    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4wltg\" (UniqueName: \"kubernetes.io/projected/62104ee5-c623-4ffd-b3af-9516ffd5f3bf-kube-api-access-4wltg\") pod \"multus-llvsl\" (UID: \"62104ee5-c623-4ffd-b3af-9516ffd5f3bf\") " pod="openshift-multus/multus-llvsl"
Sep 16 18:40:20.405089 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:20.404795    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/91876c9b-d59f-4d80-bda9-f7addebaaee3-cni-binary-copy\") pod \"multus-additional-cni-plugins-dxtkl\" (UID: \"91876c9b-d59f-4d80-bda9-f7addebaaee3\") " pod="openshift-multus/multus-additional-cni-plugins-dxtkl"
Sep 16 18:40:20.405089 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:20.404798    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fvjll\" (UniqueName: \"kubernetes.io/projected/3d323f6a-c161-4706-b9d5-8803578a6159-kube-api-access-fvjll\") pod \"iptables-alerter-ncnts\" (UID: \"3d323f6a-c161-4706-b9d5-8803578a6159\") " pod="openshift-network-operator/iptables-alerter-ncnts"
Sep 16 18:40:20.405089 ip-10-0-135-119 kubenswrapper[2578]: E0916 18:40:20.404820    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1f969e62-dd58-41b0-9419-b2ca445f3ec4-metrics-certs podName:1f969e62-dd58-41b0-9419-b2ca445f3ec4 nodeName:}" failed. No retries permitted until 2026-09-16 18:40:20.904802987 +0000 UTC m=+1.971294470 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1f969e62-dd58-41b0-9419-b2ca445f3ec4-metrics-certs") pod "network-metrics-daemon-zfv2x" (UID: "1f969e62-dd58-41b0-9419-b2ca445f3ec4") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 18:40:20.405089 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:20.404843    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/92e18a5c-2e80-4f2d-8602-d64ca37db647-etc-modprobe-d\") pod \"tuned-hqxdq\" (UID: \"92e18a5c-2e80-4f2d-8602-d64ca37db647\") " pod="openshift-cluster-node-tuning-operator/tuned-hqxdq"
Sep 16 18:40:20.405089 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:20.404863    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/92e18a5c-2e80-4f2d-8602-d64ca37db647-etc-tuned\") pod \"tuned-hqxdq\" (UID: \"92e18a5c-2e80-4f2d-8602-d64ca37db647\") " pod="openshift-cluster-node-tuning-operator/tuned-hqxdq"
Sep 16 18:40:20.405089 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:20.404896    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rnfqc\" (UniqueName: \"kubernetes.io/projected/92e18a5c-2e80-4f2d-8602-d64ca37db647-kube-api-access-rnfqc\") pod \"tuned-hqxdq\" (UID: \"92e18a5c-2e80-4f2d-8602-d64ca37db647\") " pod="openshift-cluster-node-tuning-operator/tuned-hqxdq"
Sep 16 18:40:20.405089 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:20.404924    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/6012728d-aee6-4f93-ab6e-fad616ef36e8-node-log\") pod \"ovnkube-node-k6gvs\" (UID: \"6012728d-aee6-4f93-ab6e-fad616ef36e8\") " pod="openshift-ovn-kubernetes/ovnkube-node-k6gvs"
Sep 16 18:40:20.405089 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:20.404948    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/62104ee5-c623-4ffd-b3af-9516ffd5f3bf-system-cni-dir\") pod \"multus-llvsl\" (UID: \"62104ee5-c623-4ffd-b3af-9516ffd5f3bf\") " pod="openshift-multus/multus-llvsl"
Sep 16 18:40:20.405089 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:20.404971    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/3f2c98a3-c6e7-4610-b207-f7ccde3cb571-etc-selinux\") pod \"aws-ebs-csi-driver-node-wrx94\" (UID: \"3f2c98a3-c6e7-4610-b207-f7ccde3cb571\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wrx94"
Sep 16 18:40:20.405901 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:20.405000    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/6012728d-aee6-4f93-ab6e-fad616ef36e8-node-log\") pod \"ovnkube-node-k6gvs\" (UID: \"6012728d-aee6-4f93-ab6e-fad616ef36e8\") " pod="openshift-ovn-kubernetes/ovnkube-node-k6gvs"
Sep 16 18:40:20.405901 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:20.405026    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/92e18a5c-2e80-4f2d-8602-d64ca37db647-etc-kubernetes\") pod \"tuned-hqxdq\" (UID: \"92e18a5c-2e80-4f2d-8602-d64ca37db647\") " pod="openshift-cluster-node-tuning-operator/tuned-hqxdq"
Sep 16 18:40:20.405901 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:20.405057    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6012728d-aee6-4f93-ab6e-fad616ef36e8-host-cni-bin\") pod \"ovnkube-node-k6gvs\" (UID: \"6012728d-aee6-4f93-ab6e-fad616ef36e8\") " pod="openshift-ovn-kubernetes/ovnkube-node-k6gvs"
Sep 16 18:40:20.405901 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:20.405083    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fc3365ef-dfd3-45fe-b80f-937949d65751-tmp-dir\") pod \"node-resolver-5rrj4\" (UID: \"fc3365ef-dfd3-45fe-b80f-937949d65751\") " pod="openshift-dns/node-resolver-5rrj4"
Sep 16 18:40:20.405901 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:20.405108    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/f4653546-2f57-4858-a038-9fd7c2139d56-konnectivity-ca\") pod \"konnectivity-agent-f69wc\" (UID: \"f4653546-2f57-4858-a038-9fd7c2139d56\") " pod="kube-system/konnectivity-agent-f69wc"
Sep 16 18:40:20.405901 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:20.405114    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6012728d-aee6-4f93-ab6e-fad616ef36e8-host-cni-bin\") pod \"ovnkube-node-k6gvs\" (UID: \"6012728d-aee6-4f93-ab6e-fad616ef36e8\") " pod="openshift-ovn-kubernetes/ovnkube-node-k6gvs"
Sep 16 18:40:20.405901 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:20.405146    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/6012728d-aee6-4f93-ab6e-fad616ef36e8-host-cni-netd\") pod \"ovnkube-node-k6gvs\" (UID: \"6012728d-aee6-4f93-ab6e-fad616ef36e8\") " pod="openshift-ovn-kubernetes/ovnkube-node-k6gvs"
Sep 16 18:40:20.405901 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:20.405169    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/6012728d-aee6-4f93-ab6e-fad616ef36e8-run-systemd\") pod \"ovnkube-node-k6gvs\" (UID: \"6012728d-aee6-4f93-ab6e-fad616ef36e8\") " pod="openshift-ovn-kubernetes/ovnkube-node-k6gvs"
Sep 16 18:40:20.405901 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:20.405195    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6012728d-aee6-4f93-ab6e-fad616ef36e8-var-lib-openvswitch\") pod \"ovnkube-node-k6gvs\" (UID: \"6012728d-aee6-4f93-ab6e-fad616ef36e8\") " pod="openshift-ovn-kubernetes/ovnkube-node-k6gvs"
Sep 16 18:40:20.405901 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:20.405215    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/6012728d-aee6-4f93-ab6e-fad616ef36e8-run-systemd\") pod \"ovnkube-node-k6gvs\" (UID: \"6012728d-aee6-4f93-ab6e-fad616ef36e8\") " pod="openshift-ovn-kubernetes/ovnkube-node-k6gvs"
Sep 16 18:40:20.405901 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:20.405216    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/6012728d-aee6-4f93-ab6e-fad616ef36e8-host-cni-netd\") pod \"ovnkube-node-k6gvs\" (UID: \"6012728d-aee6-4f93-ab6e-fad616ef36e8\") " pod="openshift-ovn-kubernetes/ovnkube-node-k6gvs"
Sep 16 18:40:20.405901 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:20.405218    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/62104ee5-c623-4ffd-b3af-9516ffd5f3bf-multus-conf-dir\") pod \"multus-llvsl\" (UID: \"62104ee5-c623-4ffd-b3af-9516ffd5f3bf\") " pod="openshift-multus/multus-llvsl"
Sep 16 18:40:20.405901 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:20.405271    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/91876c9b-d59f-4d80-bda9-f7addebaaee3-tuning-conf-dir\") pod \"multus-additional-cni-plugins-dxtkl\" (UID: \"91876c9b-d59f-4d80-bda9-f7addebaaee3\") " pod="openshift-multus/multus-additional-cni-plugins-dxtkl"
Sep 16 18:40:20.405901 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:20.405242    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6012728d-aee6-4f93-ab6e-fad616ef36e8-var-lib-openvswitch\") pod \"ovnkube-node-k6gvs\" (UID: \"6012728d-aee6-4f93-ab6e-fad616ef36e8\") " pod="openshift-ovn-kubernetes/ovnkube-node-k6gvs"
Sep 16 18:40:20.405901 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:20.405298    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/91876c9b-d59f-4d80-bda9-f7addebaaee3-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-dxtkl\" (UID: \"91876c9b-d59f-4d80-bda9-f7addebaaee3\") " pod="openshift-multus/multus-additional-cni-plugins-dxtkl"
Sep 16 18:40:20.405901 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:20.405304    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/6012728d-aee6-4f93-ab6e-fad616ef36e8-env-overrides\") pod \"ovnkube-node-k6gvs\" (UID: \"6012728d-aee6-4f93-ab6e-fad616ef36e8\") " pod="openshift-ovn-kubernetes/ovnkube-node-k6gvs"
Sep 16 18:40:20.405901 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:20.405334    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jjh8v\" (UniqueName: \"kubernetes.io/projected/91876c9b-d59f-4d80-bda9-f7addebaaee3-kube-api-access-jjh8v\") pod \"multus-additional-cni-plugins-dxtkl\" (UID: \"91876c9b-d59f-4d80-bda9-f7addebaaee3\") " pod="openshift-multus/multus-additional-cni-plugins-dxtkl"
Sep 16 18:40:20.406525 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:20.405349    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/6012728d-aee6-4f93-ab6e-fad616ef36e8-ovnkube-script-lib\") pod \"ovnkube-node-k6gvs\" (UID: \"6012728d-aee6-4f93-ab6e-fad616ef36e8\") " pod="openshift-ovn-kubernetes/ovnkube-node-k6gvs"
Sep 16 18:40:20.406525 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:20.405379    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/62104ee5-c623-4ffd-b3af-9516ffd5f3bf-host-run-netns\") pod \"multus-llvsl\" (UID: \"62104ee5-c623-4ffd-b3af-9516ffd5f3bf\") " pod="openshift-multus/multus-llvsl"
Sep 16 18:40:20.406525 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:20.405396    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/3f2c98a3-c6e7-4610-b207-f7ccde3cb571-sys-fs\") pod \"aws-ebs-csi-driver-node-wrx94\" (UID: \"3f2c98a3-c6e7-4610-b207-f7ccde3cb571\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wrx94"
Sep 16 18:40:20.406525 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:20.405412    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/92e18a5c-2e80-4f2d-8602-d64ca37db647-run\") pod \"tuned-hqxdq\" (UID: \"92e18a5c-2e80-4f2d-8602-d64ca37db647\") " pod="openshift-cluster-node-tuning-operator/tuned-hqxdq"
Sep 16 18:40:20.406525 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:20.405432    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/91876c9b-d59f-4d80-bda9-f7addebaaee3-tuning-conf-dir\") pod \"multus-additional-cni-plugins-dxtkl\" (UID: \"91876c9b-d59f-4d80-bda9-f7addebaaee3\") " pod="openshift-multus/multus-additional-cni-plugins-dxtkl"
Sep 16 18:40:20.406525 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:20.405435    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/62104ee5-c623-4ffd-b3af-9516ffd5f3bf-multus-cni-dir\") pod \"multus-llvsl\" (UID: \"62104ee5-c623-4ffd-b3af-9516ffd5f3bf\") " pod="openshift-multus/multus-llvsl"
Sep 16 18:40:20.406525 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:20.405474    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/62104ee5-c623-4ffd-b3af-9516ffd5f3bf-host-run-k8s-cni-cncf-io\") pod \"multus-llvsl\" (UID: \"62104ee5-c623-4ffd-b3af-9516ffd5f3bf\") " pod="openshift-multus/multus-llvsl"
Sep 16 18:40:20.406525 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:20.405493    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/62104ee5-c623-4ffd-b3af-9516ffd5f3bf-etc-kubernetes\") pod \"multus-llvsl\" (UID: \"62104ee5-c623-4ffd-b3af-9516ffd5f3bf\") " pod="openshift-multus/multus-llvsl"
Sep 16 18:40:20.406525 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:20.405510    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/3f2c98a3-c6e7-4610-b207-f7ccde3cb571-registration-dir\") pod \"aws-ebs-csi-driver-node-wrx94\" (UID: \"3f2c98a3-c6e7-4610-b207-f7ccde3cb571\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wrx94"
Sep 16 18:40:20.406525 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:20.405530    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/3377e453-9418-49a8-870b-64424f13d694-kubelet-config\") pod \"global-pull-secret-syncer-8ndsh\" (UID: \"3377e453-9418-49a8-870b-64424f13d694\") " pod="kube-system/global-pull-secret-syncer-8ndsh"
Sep 16 18:40:20.406525 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:20.405546    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8cxqr\" (UniqueName: \"kubernetes.io/projected/454c7c51-a475-44f6-be51-1a8f1d29beaf-kube-api-access-8cxqr\") pod \"node-ca-bl65n\" (UID: \"454c7c51-a475-44f6-be51-1a8f1d29beaf\") " pod="openshift-image-registry/node-ca-bl65n"
Sep 16 18:40:20.406525 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:20.405562    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/91876c9b-d59f-4d80-bda9-f7addebaaee3-cnibin\") pod \"multus-additional-cni-plugins-dxtkl\" (UID: \"91876c9b-d59f-4d80-bda9-f7addebaaee3\") " pod="openshift-multus/multus-additional-cni-plugins-dxtkl"
Sep 16 18:40:20.406525 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:20.405593    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/91876c9b-d59f-4d80-bda9-f7addebaaee3-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-dxtkl\" (UID: \"91876c9b-d59f-4d80-bda9-f7addebaaee3\") " pod="openshift-multus/multus-additional-cni-plugins-dxtkl"
Sep 16 18:40:20.406525 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:20.405613    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/91876c9b-d59f-4d80-bda9-f7addebaaee3-cnibin\") pod \"multus-additional-cni-plugins-dxtkl\" (UID: \"91876c9b-d59f-4d80-bda9-f7addebaaee3\") " pod="openshift-multus/multus-additional-cni-plugins-dxtkl"
Sep 16 18:40:20.406525 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:20.405614    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/92e18a5c-2e80-4f2d-8602-d64ca37db647-var-lib-kubelet\") pod \"tuned-hqxdq\" (UID: \"92e18a5c-2e80-4f2d-8602-d64ca37db647\") " pod="openshift-cluster-node-tuning-operator/tuned-hqxdq"
Sep 16 18:40:20.406525 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:20.405643    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/62104ee5-c623-4ffd-b3af-9516ffd5f3bf-multus-daemon-config\") pod \"multus-llvsl\" (UID: \"62104ee5-c623-4ffd-b3af-9516ffd5f3bf\") " pod="openshift-multus/multus-llvsl"
Sep 16 18:40:20.406525 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:20.405679    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/6012728d-aee6-4f93-ab6e-fad616ef36e8-run-ovn\") pod \"ovnkube-node-k6gvs\" (UID: \"6012728d-aee6-4f93-ab6e-fad616ef36e8\") " pod="openshift-ovn-kubernetes/ovnkube-node-k6gvs"
Sep 16 18:40:20.407031 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:20.405694    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/62104ee5-c623-4ffd-b3af-9516ffd5f3bf-host-var-lib-cni-bin\") pod \"multus-llvsl\" (UID: \"62104ee5-c623-4ffd-b3af-9516ffd5f3bf\") " pod="openshift-multus/multus-llvsl"
Sep 16 18:40:20.407031 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:20.405717    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x54vk\" (UniqueName: \"kubernetes.io/projected/3f2c98a3-c6e7-4610-b207-f7ccde3cb571-kube-api-access-x54vk\") pod \"aws-ebs-csi-driver-node-wrx94\" (UID: \"3f2c98a3-c6e7-4610-b207-f7ccde3cb571\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wrx94"
Sep 16 18:40:20.407031 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:20.405726    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/91876c9b-d59f-4d80-bda9-f7addebaaee3-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-dxtkl\" (UID: \"91876c9b-d59f-4d80-bda9-f7addebaaee3\") " pod="openshift-multus/multus-additional-cni-plugins-dxtkl"
Sep 16 18:40:20.407031 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:20.405741    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/3d323f6a-c161-4706-b9d5-8803578a6159-iptables-alerter-script\") pod \"iptables-alerter-ncnts\" (UID: \"3d323f6a-c161-4706-b9d5-8803578a6159\") " pod="openshift-network-operator/iptables-alerter-ncnts"
Sep 16 18:40:20.407031 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:20.405746    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/6012728d-aee6-4f93-ab6e-fad616ef36e8-run-ovn\") pod \"ovnkube-node-k6gvs\" (UID: \"6012728d-aee6-4f93-ab6e-fad616ef36e8\") " pod="openshift-ovn-kubernetes/ovnkube-node-k6gvs"
Sep 16 18:40:20.407031 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:20.405771    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/92e18a5c-2e80-4f2d-8602-d64ca37db647-tmp\") pod \"tuned-hqxdq\" (UID: \"92e18a5c-2e80-4f2d-8602-d64ca37db647\") " pod="openshift-cluster-node-tuning-operator/tuned-hqxdq"
Sep 16 18:40:20.407031 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:20.405803    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/62104ee5-c623-4ffd-b3af-9516ffd5f3bf-multus-socket-dir-parent\") pod \"multus-llvsl\" (UID: \"62104ee5-c623-4ffd-b3af-9516ffd5f3bf\") " pod="openshift-multus/multus-llvsl"
Sep 16 18:40:20.407031 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:20.405831    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/62104ee5-c623-4ffd-b3af-9516ffd5f3bf-host-var-lib-kubelet\") pod \"multus-llvsl\" (UID: \"62104ee5-c623-4ffd-b3af-9516ffd5f3bf\") " pod="openshift-multus/multus-llvsl"
Sep 16 18:40:20.407031 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:20.405869    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/62104ee5-c623-4ffd-b3af-9516ffd5f3bf-host-run-multus-certs\") pod \"multus-llvsl\" (UID: \"62104ee5-c623-4ffd-b3af-9516ffd5f3bf\") " pod="openshift-multus/multus-llvsl"
Sep 16 18:40:20.407031 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:20.405899    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/3f2c98a3-c6e7-4610-b207-f7ccde3cb571-device-dir\") pod \"aws-ebs-csi-driver-node-wrx94\" (UID: \"3f2c98a3-c6e7-4610-b207-f7ccde3cb571\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wrx94"
Sep 16 18:40:20.407031 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:20.405925    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3377e453-9418-49a8-870b-64424f13d694-original-pull-secret\") pod \"global-pull-secret-syncer-8ndsh\" (UID: \"3377e453-9418-49a8-870b-64424f13d694\") " pod="kube-system/global-pull-secret-syncer-8ndsh"
Sep 16 18:40:20.407031 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:20.405940    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/92e18a5c-2e80-4f2d-8602-d64ca37db647-lib-modules\") pod \"tuned-hqxdq\" (UID: \"92e18a5c-2e80-4f2d-8602-d64ca37db647\") " pod="openshift-cluster-node-tuning-operator/tuned-hqxdq"
Sep 16 18:40:20.407031 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:20.405956    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6012728d-aee6-4f93-ab6e-fad616ef36e8-host-run-netns\") pod \"ovnkube-node-k6gvs\" (UID: \"6012728d-aee6-4f93-ab6e-fad616ef36e8\") " pod="openshift-ovn-kubernetes/ovnkube-node-k6gvs"
Sep 16 18:40:20.407031 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:20.405989    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6012728d-aee6-4f93-ab6e-fad616ef36e8-etc-openvswitch\") pod \"ovnkube-node-k6gvs\" (UID: \"6012728d-aee6-4f93-ab6e-fad616ef36e8\") " pod="openshift-ovn-kubernetes/ovnkube-node-k6gvs"
Sep 16 18:40:20.407031 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:20.406012    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-25hb5\" (UniqueName: \"kubernetes.io/projected/6012728d-aee6-4f93-ab6e-fad616ef36e8-kube-api-access-25hb5\") pod \"ovnkube-node-k6gvs\" (UID: \"6012728d-aee6-4f93-ab6e-fad616ef36e8\") " pod="openshift-ovn-kubernetes/ovnkube-node-k6gvs"
Sep 16 18:40:20.407031 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:20.406027    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6012728d-aee6-4f93-ab6e-fad616ef36e8-host-run-netns\") pod \"ovnkube-node-k6gvs\" (UID: \"6012728d-aee6-4f93-ab6e-fad616ef36e8\") " pod="openshift-ovn-kubernetes/ovnkube-node-k6gvs"
Sep 16 18:40:20.407714 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:20.406036    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/91876c9b-d59f-4d80-bda9-f7addebaaee3-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-dxtkl\" (UID: \"91876c9b-d59f-4d80-bda9-f7addebaaee3\") " pod="openshift-multus/multus-additional-cni-plugins-dxtkl"
Sep 16 18:40:20.407714 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:20.406047    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6012728d-aee6-4f93-ab6e-fad616ef36e8-etc-openvswitch\") pod \"ovnkube-node-k6gvs\" (UID: \"6012728d-aee6-4f93-ab6e-fad616ef36e8\") " pod="openshift-ovn-kubernetes/ovnkube-node-k6gvs"
Sep 16 18:40:20.407714 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:20.406038    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/6012728d-aee6-4f93-ab6e-fad616ef36e8-log-socket\") pod \"ovnkube-node-k6gvs\" (UID: \"6012728d-aee6-4f93-ab6e-fad616ef36e8\") " pod="openshift-ovn-kubernetes/ovnkube-node-k6gvs"
Sep 16 18:40:20.407714 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:20.406072    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/6012728d-aee6-4f93-ab6e-fad616ef36e8-log-socket\") pod \"ovnkube-node-k6gvs\" (UID: \"6012728d-aee6-4f93-ab6e-fad616ef36e8\") " pod="openshift-ovn-kubernetes/ovnkube-node-k6gvs"
Sep 16 18:40:20.407714 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:20.406086    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6012728d-aee6-4f93-ab6e-fad616ef36e8-host-run-ovn-kubernetes\") pod \"ovnkube-node-k6gvs\" (UID: \"6012728d-aee6-4f93-ab6e-fad616ef36e8\") " pod="openshift-ovn-kubernetes/ovnkube-node-k6gvs"
Sep 16 18:40:20.407714 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:20.406109    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/6012728d-aee6-4f93-ab6e-fad616ef36e8-ovnkube-config\") pod \"ovnkube-node-k6gvs\" (UID: \"6012728d-aee6-4f93-ab6e-fad616ef36e8\") " pod="openshift-ovn-kubernetes/ovnkube-node-k6gvs"
Sep 16 18:40:20.407714 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:20.406131    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/6012728d-aee6-4f93-ab6e-fad616ef36e8-ovn-node-metrics-cert\") pod \"ovnkube-node-k6gvs\" (UID: \"6012728d-aee6-4f93-ab6e-fad616ef36e8\") " pod="openshift-ovn-kubernetes/ovnkube-node-k6gvs"
Sep 16 18:40:20.407714 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:20.406138    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6012728d-aee6-4f93-ab6e-fad616ef36e8-host-run-ovn-kubernetes\") pod \"ovnkube-node-k6gvs\" (UID: \"6012728d-aee6-4f93-ab6e-fad616ef36e8\") " pod="openshift-ovn-kubernetes/ovnkube-node-k6gvs"
Sep 16 18:40:20.407714 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:20.406169    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/fc3365ef-dfd3-45fe-b80f-937949d65751-hosts-file\") pod \"node-resolver-5rrj4\" (UID: \"fc3365ef-dfd3-45fe-b80f-937949d65751\") " pod="openshift-dns/node-resolver-5rrj4"
Sep 16 18:40:20.407714 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:20.406198    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/3377e453-9418-49a8-870b-64424f13d694-dbus\") pod \"global-pull-secret-syncer-8ndsh\" (UID: \"3377e453-9418-49a8-870b-64424f13d694\") " pod="kube-system/global-pull-secret-syncer-8ndsh"
Sep 16 18:40:20.407714 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:20.406227    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/92e18a5c-2e80-4f2d-8602-d64ca37db647-host\") pod \"tuned-hqxdq\" (UID: \"92e18a5c-2e80-4f2d-8602-d64ca37db647\") " pod="openshift-cluster-node-tuning-operator/tuned-hqxdq"
Sep 16 18:40:20.407714 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:20.406254    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/91876c9b-d59f-4d80-bda9-f7addebaaee3-system-cni-dir\") pod \"multus-additional-cni-plugins-dxtkl\" (UID: \"91876c9b-d59f-4d80-bda9-f7addebaaee3\") " pod="openshift-multus/multus-additional-cni-plugins-dxtkl"
Sep 16 18:40:20.407714 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:20.406309    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/91876c9b-d59f-4d80-bda9-f7addebaaee3-system-cni-dir\") pod \"multus-additional-cni-plugins-dxtkl\" (UID: \"91876c9b-d59f-4d80-bda9-f7addebaaee3\") " pod="openshift-multus/multus-additional-cni-plugins-dxtkl"
Sep 16 18:40:20.407714 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:20.406410    2578 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Sep 16 18:40:20.407714 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:20.406601    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/6012728d-aee6-4f93-ab6e-fad616ef36e8-ovnkube-config\") pod \"ovnkube-node-k6gvs\" (UID: \"6012728d-aee6-4f93-ab6e-fad616ef36e8\") " pod="openshift-ovn-kubernetes/ovnkube-node-k6gvs"
Sep 16 18:40:20.409860 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:20.409841    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/6012728d-aee6-4f93-ab6e-fad616ef36e8-ovn-node-metrics-cert\") pod \"ovnkube-node-k6gvs\" (UID: \"6012728d-aee6-4f93-ab6e-fad616ef36e8\") " pod="openshift-ovn-kubernetes/ovnkube-node-k6gvs"
Sep 16 18:40:20.417648 ip-10-0-135-119 kubenswrapper[2578]: E0916 18:40:20.417630    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 16 18:40:20.417729 ip-10-0-135-119 kubenswrapper[2578]: E0916 18:40:20.417652    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 16 18:40:20.417729 ip-10-0-135-119 kubenswrapper[2578]: E0916 18:40:20.417662    2578 projected.go:196] Error preparing data for projected volume kube-api-access-77jgz for pod openshift-network-diagnostics/network-check-target-f287c: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 18:40:20.417729 ip-10-0-135-119 kubenswrapper[2578]: E0916 18:40:20.417714    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/63abc484-192b-4095-81ea-b5db79acd0ba-kube-api-access-77jgz podName:63abc484-192b-4095-81ea-b5db79acd0ba nodeName:}" failed. No retries permitted until 2026-09-16 18:40:20.917702615 +0000 UTC m=+1.984194084 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-77jgz" (UniqueName: "kubernetes.io/projected/63abc484-192b-4095-81ea-b5db79acd0ba-kube-api-access-77jgz") pod "network-check-target-f287c" (UID: "63abc484-192b-4095-81ea-b5db79acd0ba") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 18:40:20.424058 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:20.424035    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4gc49\" (UniqueName: \"kubernetes.io/projected/1f969e62-dd58-41b0-9419-b2ca445f3ec4-kube-api-access-4gc49\") pod \"network-metrics-daemon-zfv2x\" (UID: \"1f969e62-dd58-41b0-9419-b2ca445f3ec4\") " pod="openshift-multus/network-metrics-daemon-zfv2x"
Sep 16 18:40:20.424526 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:20.424510    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jjh8v\" (UniqueName: \"kubernetes.io/projected/91876c9b-d59f-4d80-bda9-f7addebaaee3-kube-api-access-jjh8v\") pod \"multus-additional-cni-plugins-dxtkl\" (UID: \"91876c9b-d59f-4d80-bda9-f7addebaaee3\") " pod="openshift-multus/multus-additional-cni-plugins-dxtkl"
Sep 16 18:40:20.424757 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:20.424738    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-25hb5\" (UniqueName: \"kubernetes.io/projected/6012728d-aee6-4f93-ab6e-fad616ef36e8-kube-api-access-25hb5\") pod \"ovnkube-node-k6gvs\" (UID: \"6012728d-aee6-4f93-ab6e-fad616ef36e8\") " pod="openshift-ovn-kubernetes/ovnkube-node-k6gvs"
Sep 16 18:40:20.506720 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:20.506689    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/62104ee5-c623-4ffd-b3af-9516ffd5f3bf-multus-conf-dir\") pod \"multus-llvsl\" (UID: \"62104ee5-c623-4ffd-b3af-9516ffd5f3bf\") " pod="openshift-multus/multus-llvsl"
Sep 16 18:40:20.506720 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:20.506721    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/62104ee5-c623-4ffd-b3af-9516ffd5f3bf-host-run-netns\") pod \"multus-llvsl\" (UID: \"62104ee5-c623-4ffd-b3af-9516ffd5f3bf\") " pod="openshift-multus/multus-llvsl"
Sep 16 18:40:20.506933 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:20.506737    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/3f2c98a3-c6e7-4610-b207-f7ccde3cb571-sys-fs\") pod \"aws-ebs-csi-driver-node-wrx94\" (UID: \"3f2c98a3-c6e7-4610-b207-f7ccde3cb571\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wrx94"
Sep 16 18:40:20.506933 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:20.506760    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/92e18a5c-2e80-4f2d-8602-d64ca37db647-run\") pod \"tuned-hqxdq\" (UID: \"92e18a5c-2e80-4f2d-8602-d64ca37db647\") " pod="openshift-cluster-node-tuning-operator/tuned-hqxdq"
Sep 16 18:40:20.506933 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:20.506783    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/62104ee5-c623-4ffd-b3af-9516ffd5f3bf-multus-cni-dir\") pod \"multus-llvsl\" (UID: \"62104ee5-c623-4ffd-b3af-9516ffd5f3bf\") " pod="openshift-multus/multus-llvsl"
Sep 16 18:40:20.506933 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:20.506788    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/62104ee5-c623-4ffd-b3af-9516ffd5f3bf-multus-conf-dir\") pod \"multus-llvsl\" (UID: \"62104ee5-c623-4ffd-b3af-9516ffd5f3bf\") " pod="openshift-multus/multus-llvsl"
Sep 16 18:40:20.506933 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:20.506807    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/62104ee5-c623-4ffd-b3af-9516ffd5f3bf-host-run-netns\") pod \"multus-llvsl\" (UID: \"62104ee5-c623-4ffd-b3af-9516ffd5f3bf\") " pod="openshift-multus/multus-llvsl"
Sep 16 18:40:20.506933 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:20.506832    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/62104ee5-c623-4ffd-b3af-9516ffd5f3bf-host-run-k8s-cni-cncf-io\") pod \"multus-llvsl\" (UID: \"62104ee5-c623-4ffd-b3af-9516ffd5f3bf\") " pod="openshift-multus/multus-llvsl"
Sep 16 18:40:20.506933 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:20.506856    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/62104ee5-c623-4ffd-b3af-9516ffd5f3bf-host-run-k8s-cni-cncf-io\") pod \"multus-llvsl\" (UID: \"62104ee5-c623-4ffd-b3af-9516ffd5f3bf\") " pod="openshift-multus/multus-llvsl"
Sep 16 18:40:20.506933 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:20.506869    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/62104ee5-c623-4ffd-b3af-9516ffd5f3bf-etc-kubernetes\") pod \"multus-llvsl\" (UID: \"62104ee5-c623-4ffd-b3af-9516ffd5f3bf\") " pod="openshift-multus/multus-llvsl"
Sep 16 18:40:20.506933 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:20.506832    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/3f2c98a3-c6e7-4610-b207-f7ccde3cb571-sys-fs\") pod \"aws-ebs-csi-driver-node-wrx94\" (UID: \"3f2c98a3-c6e7-4610-b207-f7ccde3cb571\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wrx94"
Sep 16 18:40:20.506933 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:20.506887    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/3f2c98a3-c6e7-4610-b207-f7ccde3cb571-registration-dir\") pod \"aws-ebs-csi-driver-node-wrx94\" (UID: \"3f2c98a3-c6e7-4610-b207-f7ccde3cb571\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wrx94"
Sep 16 18:40:20.506933 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:20.506903    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/3377e453-9418-49a8-870b-64424f13d694-kubelet-config\") pod \"global-pull-secret-syncer-8ndsh\" (UID: \"3377e453-9418-49a8-870b-64424f13d694\") " pod="kube-system/global-pull-secret-syncer-8ndsh"
Sep 16 18:40:20.506933 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:20.506913    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/62104ee5-c623-4ffd-b3af-9516ffd5f3bf-etc-kubernetes\") pod \"multus-llvsl\" (UID: \"62104ee5-c623-4ffd-b3af-9516ffd5f3bf\") " pod="openshift-multus/multus-llvsl"
Sep 16 18:40:20.506933 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:20.506915    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/92e18a5c-2e80-4f2d-8602-d64ca37db647-run\") pod \"tuned-hqxdq\" (UID: \"92e18a5c-2e80-4f2d-8602-d64ca37db647\") " pod="openshift-cluster-node-tuning-operator/tuned-hqxdq"
Sep 16 18:40:20.506933 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:20.506922    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8cxqr\" (UniqueName: \"kubernetes.io/projected/454c7c51-a475-44f6-be51-1a8f1d29beaf-kube-api-access-8cxqr\") pod \"node-ca-bl65n\" (UID: \"454c7c51-a475-44f6-be51-1a8f1d29beaf\") " pod="openshift-image-registry/node-ca-bl65n"
Sep 16 18:40:20.506933 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:20.506917    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/62104ee5-c623-4ffd-b3af-9516ffd5f3bf-multus-cni-dir\") pod \"multus-llvsl\" (UID: \"62104ee5-c623-4ffd-b3af-9516ffd5f3bf\") " pod="openshift-multus/multus-llvsl"
Sep 16 18:40:20.507698 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:20.506948    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/92e18a5c-2e80-4f2d-8602-d64ca37db647-var-lib-kubelet\") pod \"tuned-hqxdq\" (UID: \"92e18a5c-2e80-4f2d-8602-d64ca37db647\") " pod="openshift-cluster-node-tuning-operator/tuned-hqxdq"
Sep 16 18:40:20.507698 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:20.506949    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/3f2c98a3-c6e7-4610-b207-f7ccde3cb571-registration-dir\") pod \"aws-ebs-csi-driver-node-wrx94\" (UID: \"3f2c98a3-c6e7-4610-b207-f7ccde3cb571\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wrx94"
Sep 16 18:40:20.507698 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:20.506972    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/62104ee5-c623-4ffd-b3af-9516ffd5f3bf-multus-daemon-config\") pod \"multus-llvsl\" (UID: \"62104ee5-c623-4ffd-b3af-9516ffd5f3bf\") " pod="openshift-multus/multus-llvsl"
Sep 16 18:40:20.507698 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:20.507000    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/3377e453-9418-49a8-870b-64424f13d694-kubelet-config\") pod \"global-pull-secret-syncer-8ndsh\" (UID: \"3377e453-9418-49a8-870b-64424f13d694\") " pod="kube-system/global-pull-secret-syncer-8ndsh"
Sep 16 18:40:20.507698 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:20.507021    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/62104ee5-c623-4ffd-b3af-9516ffd5f3bf-host-var-lib-cni-bin\") pod \"multus-llvsl\" (UID: \"62104ee5-c623-4ffd-b3af-9516ffd5f3bf\") " pod="openshift-multus/multus-llvsl"
Sep 16 18:40:20.507698 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:20.507033    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/92e18a5c-2e80-4f2d-8602-d64ca37db647-var-lib-kubelet\") pod \"tuned-hqxdq\" (UID: \"92e18a5c-2e80-4f2d-8602-d64ca37db647\") " pod="openshift-cluster-node-tuning-operator/tuned-hqxdq"
Sep 16 18:40:20.507698 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:20.507045    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x54vk\" (UniqueName: \"kubernetes.io/projected/3f2c98a3-c6e7-4610-b207-f7ccde3cb571-kube-api-access-x54vk\") pod \"aws-ebs-csi-driver-node-wrx94\" (UID: \"3f2c98a3-c6e7-4610-b207-f7ccde3cb571\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wrx94"
Sep 16 18:40:20.507698 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:20.507064    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/62104ee5-c623-4ffd-b3af-9516ffd5f3bf-host-var-lib-cni-bin\") pod \"multus-llvsl\" (UID: \"62104ee5-c623-4ffd-b3af-9516ffd5f3bf\") " pod="openshift-multus/multus-llvsl"
Sep 16 18:40:20.507698 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:20.507063    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/3d323f6a-c161-4706-b9d5-8803578a6159-iptables-alerter-script\") pod \"iptables-alerter-ncnts\" (UID: \"3d323f6a-c161-4706-b9d5-8803578a6159\") " pod="openshift-network-operator/iptables-alerter-ncnts"
Sep 16 18:40:20.507698 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:20.507092    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/92e18a5c-2e80-4f2d-8602-d64ca37db647-tmp\") pod \"tuned-hqxdq\" (UID: \"92e18a5c-2e80-4f2d-8602-d64ca37db647\") " pod="openshift-cluster-node-tuning-operator/tuned-hqxdq"
Sep 16 18:40:20.507698 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:20.507124    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/62104ee5-c623-4ffd-b3af-9516ffd5f3bf-multus-socket-dir-parent\") pod \"multus-llvsl\" (UID: \"62104ee5-c623-4ffd-b3af-9516ffd5f3bf\") " pod="openshift-multus/multus-llvsl"
Sep 16 18:40:20.507698 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:20.507148    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/62104ee5-c623-4ffd-b3af-9516ffd5f3bf-host-var-lib-kubelet\") pod \"multus-llvsl\" (UID: \"62104ee5-c623-4ffd-b3af-9516ffd5f3bf\") " pod="openshift-multus/multus-llvsl"
Sep 16 18:40:20.507698 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:20.507171    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/62104ee5-c623-4ffd-b3af-9516ffd5f3bf-host-run-multus-certs\") pod \"multus-llvsl\" (UID: \"62104ee5-c623-4ffd-b3af-9516ffd5f3bf\") " pod="openshift-multus/multus-llvsl"
Sep 16 18:40:20.507698 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:20.507194    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/3f2c98a3-c6e7-4610-b207-f7ccde3cb571-device-dir\") pod \"aws-ebs-csi-driver-node-wrx94\" (UID: \"3f2c98a3-c6e7-4610-b207-f7ccde3cb571\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wrx94"
Sep 16 18:40:20.507698 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:20.507216    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3377e453-9418-49a8-870b-64424f13d694-original-pull-secret\") pod \"global-pull-secret-syncer-8ndsh\" (UID: \"3377e453-9418-49a8-870b-64424f13d694\") " pod="kube-system/global-pull-secret-syncer-8ndsh"
Sep 16 18:40:20.507698 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:20.507216    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/62104ee5-c623-4ffd-b3af-9516ffd5f3bf-host-var-lib-kubelet\") pod \"multus-llvsl\" (UID: \"62104ee5-c623-4ffd-b3af-9516ffd5f3bf\") " pod="openshift-multus/multus-llvsl"
Sep 16 18:40:20.507698 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:20.507247    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/92e18a5c-2e80-4f2d-8602-d64ca37db647-lib-modules\") pod \"tuned-hqxdq\" (UID: \"92e18a5c-2e80-4f2d-8602-d64ca37db647\") " pod="openshift-cluster-node-tuning-operator/tuned-hqxdq"
Sep 16 18:40:20.508482 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:20.507278    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/fc3365ef-dfd3-45fe-b80f-937949d65751-hosts-file\") pod \"node-resolver-5rrj4\" (UID: \"fc3365ef-dfd3-45fe-b80f-937949d65751\") " pod="openshift-dns/node-resolver-5rrj4"
Sep 16 18:40:20.508482 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:20.507297    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/3f2c98a3-c6e7-4610-b207-f7ccde3cb571-device-dir\") pod \"aws-ebs-csi-driver-node-wrx94\" (UID: \"3f2c98a3-c6e7-4610-b207-f7ccde3cb571\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wrx94"
Sep 16 18:40:20.508482 ip-10-0-135-119 kubenswrapper[2578]: E0916 18:40:20.507317    2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 16 18:40:20.508482 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:20.507346    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/fc3365ef-dfd3-45fe-b80f-937949d65751-hosts-file\") pod \"node-resolver-5rrj4\" (UID: \"fc3365ef-dfd3-45fe-b80f-937949d65751\") " pod="openshift-dns/node-resolver-5rrj4"
Sep 16 18:40:20.508482 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:20.507355    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/3377e453-9418-49a8-870b-64424f13d694-dbus\") pod \"global-pull-secret-syncer-8ndsh\" (UID: \"3377e453-9418-49a8-870b-64424f13d694\") " pod="kube-system/global-pull-secret-syncer-8ndsh"
Sep 16 18:40:20.508482 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:20.507300    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/3377e453-9418-49a8-870b-64424f13d694-dbus\") pod \"global-pull-secret-syncer-8ndsh\" (UID: \"3377e453-9418-49a8-870b-64424f13d694\") " pod="kube-system/global-pull-secret-syncer-8ndsh"
Sep 16 18:40:20.508482 ip-10-0-135-119 kubenswrapper[2578]: E0916 18:40:20.507373    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3377e453-9418-49a8-870b-64424f13d694-original-pull-secret podName:3377e453-9418-49a8-870b-64424f13d694 nodeName:}" failed. No retries permitted until 2026-09-16 18:40:21.007354833 +0000 UTC m=+2.073846306 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/3377e453-9418-49a8-870b-64424f13d694-original-pull-secret") pod "global-pull-secret-syncer-8ndsh" (UID: "3377e453-9418-49a8-870b-64424f13d694") : object "kube-system"/"original-pull-secret" not registered
Sep 16 18:40:20.508482 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:20.507371    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/92e18a5c-2e80-4f2d-8602-d64ca37db647-lib-modules\") pod \"tuned-hqxdq\" (UID: \"92e18a5c-2e80-4f2d-8602-d64ca37db647\") " pod="openshift-cluster-node-tuning-operator/tuned-hqxdq"
Sep 16 18:40:20.508482 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:20.507395    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/62104ee5-c623-4ffd-b3af-9516ffd5f3bf-host-run-multus-certs\") pod \"multus-llvsl\" (UID: \"62104ee5-c623-4ffd-b3af-9516ffd5f3bf\") " pod="openshift-multus/multus-llvsl"
Sep 16 18:40:20.508482 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:20.507413    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/92e18a5c-2e80-4f2d-8602-d64ca37db647-host\") pod \"tuned-hqxdq\" (UID: \"92e18a5c-2e80-4f2d-8602-d64ca37db647\") " pod="openshift-cluster-node-tuning-operator/tuned-hqxdq"
Sep 16 18:40:20.508482 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:20.507439    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/62104ee5-c623-4ffd-b3af-9516ffd5f3bf-cnibin\") pod \"multus-llvsl\" (UID: \"62104ee5-c623-4ffd-b3af-9516ffd5f3bf\") " pod="openshift-multus/multus-llvsl"
Sep 16 18:40:20.508482 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:20.507461    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/454c7c51-a475-44f6-be51-1a8f1d29beaf-serviceca\") pod \"node-ca-bl65n\" (UID: \"454c7c51-a475-44f6-be51-1a8f1d29beaf\") " pod="openshift-image-registry/node-ca-bl65n"
Sep 16 18:40:20.508482 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:20.507482    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/92e18a5c-2e80-4f2d-8602-d64ca37db647-etc-systemd\") pod \"tuned-hqxdq\" (UID: \"92e18a5c-2e80-4f2d-8602-d64ca37db647\") " pod="openshift-cluster-node-tuning-operator/tuned-hqxdq"
Sep 16 18:40:20.508482 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:20.507491    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/92e18a5c-2e80-4f2d-8602-d64ca37db647-host\") pod \"tuned-hqxdq\" (UID: \"92e18a5c-2e80-4f2d-8602-d64ca37db647\") " pod="openshift-cluster-node-tuning-operator/tuned-hqxdq"
Sep 16 18:40:20.508482 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:20.507506    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/62104ee5-c623-4ffd-b3af-9516ffd5f3bf-hostroot\") pod \"multus-llvsl\" (UID: \"62104ee5-c623-4ffd-b3af-9516ffd5f3bf\") " pod="openshift-multus/multus-llvsl"
Sep 16 18:40:20.508482 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:20.507522    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/62104ee5-c623-4ffd-b3af-9516ffd5f3bf-cnibin\") pod \"multus-llvsl\" (UID: \"62104ee5-c623-4ffd-b3af-9516ffd5f3bf\") " pod="openshift-multus/multus-llvsl"
Sep 16 18:40:20.508482 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:20.507529    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/3f2c98a3-c6e7-4610-b207-f7ccde3cb571-kubelet-dir\") pod \"aws-ebs-csi-driver-node-wrx94\" (UID: \"3f2c98a3-c6e7-4610-b207-f7ccde3cb571\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wrx94"
Sep 16 18:40:20.508482 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:20.507542    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/62104ee5-c623-4ffd-b3af-9516ffd5f3bf-multus-daemon-config\") pod \"multus-llvsl\" (UID: \"62104ee5-c623-4ffd-b3af-9516ffd5f3bf\") " pod="openshift-multus/multus-llvsl"
Sep 16 18:40:20.509180 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:20.507560    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/92e18a5c-2e80-4f2d-8602-d64ca37db647-etc-systemd\") pod \"tuned-hqxdq\" (UID: \"92e18a5c-2e80-4f2d-8602-d64ca37db647\") " pod="openshift-cluster-node-tuning-operator/tuned-hqxdq"
Sep 16 18:40:20.509180 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:20.507566    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/454c7c51-a475-44f6-be51-1a8f1d29beaf-host\") pod \"node-ca-bl65n\" (UID: \"454c7c51-a475-44f6-be51-1a8f1d29beaf\") " pod="openshift-image-registry/node-ca-bl65n"
Sep 16 18:40:20.509180 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:20.507571    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/3f2c98a3-c6e7-4610-b207-f7ccde3cb571-kubelet-dir\") pod \"aws-ebs-csi-driver-node-wrx94\" (UID: \"3f2c98a3-c6e7-4610-b207-f7ccde3cb571\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wrx94"
Sep 16 18:40:20.509180 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:20.507585    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/3d323f6a-c161-4706-b9d5-8803578a6159-iptables-alerter-script\") pod \"iptables-alerter-ncnts\" (UID: \"3d323f6a-c161-4706-b9d5-8803578a6159\") " pod="openshift-network-operator/iptables-alerter-ncnts"
Sep 16 18:40:20.509180 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:20.507597    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/62104ee5-c623-4ffd-b3af-9516ffd5f3bf-multus-socket-dir-parent\") pod \"multus-llvsl\" (UID: \"62104ee5-c623-4ffd-b3af-9516ffd5f3bf\") " pod="openshift-multus/multus-llvsl"
Sep 16 18:40:20.509180 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:20.507593    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/f4653546-2f57-4858-a038-9fd7c2139d56-agent-certs\") pod \"konnectivity-agent-f69wc\" (UID: \"f4653546-2f57-4858-a038-9fd7c2139d56\") " pod="kube-system/konnectivity-agent-f69wc"
Sep 16 18:40:20.509180 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:20.507610    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/454c7c51-a475-44f6-be51-1a8f1d29beaf-host\") pod \"node-ca-bl65n\" (UID: \"454c7c51-a475-44f6-be51-1a8f1d29beaf\") " pod="openshift-image-registry/node-ca-bl65n"
Sep 16 18:40:20.509180 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:20.507605    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/62104ee5-c623-4ffd-b3af-9516ffd5f3bf-hostroot\") pod \"multus-llvsl\" (UID: \"62104ee5-c623-4ffd-b3af-9516ffd5f3bf\") " pod="openshift-multus/multus-llvsl"
Sep 16 18:40:20.509180 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:20.507630    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/3f2c98a3-c6e7-4610-b207-f7ccde3cb571-socket-dir\") pod \"aws-ebs-csi-driver-node-wrx94\" (UID: \"3f2c98a3-c6e7-4610-b207-f7ccde3cb571\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wrx94"
Sep 16 18:40:20.509180 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:20.507654    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/92e18a5c-2e80-4f2d-8602-d64ca37db647-etc-sysctl-d\") pod \"tuned-hqxdq\" (UID: \"92e18a5c-2e80-4f2d-8602-d64ca37db647\") " pod="openshift-cluster-node-tuning-operator/tuned-hqxdq"
Sep 16 18:40:20.509180 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:20.507698    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/92e18a5c-2e80-4f2d-8602-d64ca37db647-etc-sysctl-conf\") pod \"tuned-hqxdq\" (UID: \"92e18a5c-2e80-4f2d-8602-d64ca37db647\") " pod="openshift-cluster-node-tuning-operator/tuned-hqxdq"
Sep 16 18:40:20.509180 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:20.507719    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/92e18a5c-2e80-4f2d-8602-d64ca37db647-sys\") pod \"tuned-hqxdq\" (UID: \"92e18a5c-2e80-4f2d-8602-d64ca37db647\") " pod="openshift-cluster-node-tuning-operator/tuned-hqxdq"
Sep 16 18:40:20.509180 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:20.507735    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/3f2c98a3-c6e7-4610-b207-f7ccde3cb571-socket-dir\") pod \"aws-ebs-csi-driver-node-wrx94\" (UID: \"3f2c98a3-c6e7-4610-b207-f7ccde3cb571\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wrx94"
Sep 16 18:40:20.509180 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:20.507745    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/62104ee5-c623-4ffd-b3af-9516ffd5f3bf-cni-binary-copy\") pod \"multus-llvsl\" (UID: \"62104ee5-c623-4ffd-b3af-9516ffd5f3bf\") " pod="openshift-multus/multus-llvsl"
Sep 16 18:40:20.509180 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:20.507768    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/92e18a5c-2e80-4f2d-8602-d64ca37db647-etc-sysconfig\") pod \"tuned-hqxdq\" (UID: \"92e18a5c-2e80-4f2d-8602-d64ca37db647\") " pod="openshift-cluster-node-tuning-operator/tuned-hqxdq"
Sep 16 18:40:20.509180 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:20.507784    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/92e18a5c-2e80-4f2d-8602-d64ca37db647-sys\") pod \"tuned-hqxdq\" (UID: \"92e18a5c-2e80-4f2d-8602-d64ca37db647\") " pod="openshift-cluster-node-tuning-operator/tuned-hqxdq"
Sep 16 18:40:20.509180 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:20.507791    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/62104ee5-c623-4ffd-b3af-9516ffd5f3bf-host-var-lib-cni-multus\") pod \"multus-llvsl\" (UID: \"62104ee5-c623-4ffd-b3af-9516ffd5f3bf\") " pod="openshift-multus/multus-llvsl"
Sep 16 18:40:20.509180 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:20.507832    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tkf6g\" (UniqueName: \"kubernetes.io/projected/fc3365ef-dfd3-45fe-b80f-937949d65751-kube-api-access-tkf6g\") pod \"node-resolver-5rrj4\" (UID: \"fc3365ef-dfd3-45fe-b80f-937949d65751\") " pod="openshift-dns/node-resolver-5rrj4"
Sep 16 18:40:20.509715 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:20.507842    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/62104ee5-c623-4ffd-b3af-9516ffd5f3bf-host-var-lib-cni-multus\") pod \"multus-llvsl\" (UID: \"62104ee5-c623-4ffd-b3af-9516ffd5f3bf\") " pod="openshift-multus/multus-llvsl"
Sep 16 18:40:20.509715 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:20.507861    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/92e18a5c-2e80-4f2d-8602-d64ca37db647-etc-sysctl-d\") pod \"tuned-hqxdq\" (UID: \"92e18a5c-2e80-4f2d-8602-d64ca37db647\") " pod="openshift-cluster-node-tuning-operator/tuned-hqxdq"
Sep 16 18:40:20.509715 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:20.507871    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/92e18a5c-2e80-4f2d-8602-d64ca37db647-etc-sysctl-conf\") pod \"tuned-hqxdq\" (UID: \"92e18a5c-2e80-4f2d-8602-d64ca37db647\") " pod="openshift-cluster-node-tuning-operator/tuned-hqxdq"
Sep 16 18:40:20.509715 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:20.507930    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/3d323f6a-c161-4706-b9d5-8803578a6159-host-slash\") pod \"iptables-alerter-ncnts\" (UID: \"3d323f6a-c161-4706-b9d5-8803578a6159\") " pod="openshift-network-operator/iptables-alerter-ncnts"
Sep 16 18:40:20.509715 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:20.507922    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/92e18a5c-2e80-4f2d-8602-d64ca37db647-etc-sysconfig\") pod \"tuned-hqxdq\" (UID: \"92e18a5c-2e80-4f2d-8602-d64ca37db647\") " pod="openshift-cluster-node-tuning-operator/tuned-hqxdq"
Sep 16 18:40:20.509715 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:20.507958    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4wltg\" (UniqueName: \"kubernetes.io/projected/62104ee5-c623-4ffd-b3af-9516ffd5f3bf-kube-api-access-4wltg\") pod \"multus-llvsl\" (UID: \"62104ee5-c623-4ffd-b3af-9516ffd5f3bf\") " pod="openshift-multus/multus-llvsl"
Sep 16 18:40:20.509715 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:20.507998    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/3d323f6a-c161-4706-b9d5-8803578a6159-host-slash\") pod \"iptables-alerter-ncnts\" (UID: \"3d323f6a-c161-4706-b9d5-8803578a6159\") " pod="openshift-network-operator/iptables-alerter-ncnts"
Sep 16 18:40:20.509715 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:20.508000    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fvjll\" (UniqueName: \"kubernetes.io/projected/3d323f6a-c161-4706-b9d5-8803578a6159-kube-api-access-fvjll\") pod \"iptables-alerter-ncnts\" (UID: \"3d323f6a-c161-4706-b9d5-8803578a6159\") " pod="openshift-network-operator/iptables-alerter-ncnts"
Sep 16 18:40:20.509715 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:20.508029    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/92e18a5c-2e80-4f2d-8602-d64ca37db647-etc-modprobe-d\") pod \"tuned-hqxdq\" (UID: \"92e18a5c-2e80-4f2d-8602-d64ca37db647\") " pod="openshift-cluster-node-tuning-operator/tuned-hqxdq"
Sep 16 18:40:20.509715 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:20.508050    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/92e18a5c-2e80-4f2d-8602-d64ca37db647-etc-tuned\") pod \"tuned-hqxdq\" (UID: \"92e18a5c-2e80-4f2d-8602-d64ca37db647\") " pod="openshift-cluster-node-tuning-operator/tuned-hqxdq"
Sep 16 18:40:20.509715 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:20.508090    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/454c7c51-a475-44f6-be51-1a8f1d29beaf-serviceca\") pod \"node-ca-bl65n\" (UID: \"454c7c51-a475-44f6-be51-1a8f1d29beaf\") " pod="openshift-image-registry/node-ca-bl65n"
Sep 16 18:40:20.509715 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:20.508198    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rnfqc\" (UniqueName: \"kubernetes.io/projected/92e18a5c-2e80-4f2d-8602-d64ca37db647-kube-api-access-rnfqc\") pod \"tuned-hqxdq\" (UID: \"92e18a5c-2e80-4f2d-8602-d64ca37db647\") " pod="openshift-cluster-node-tuning-operator/tuned-hqxdq"
Sep 16 18:40:20.509715 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:20.508206    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/92e18a5c-2e80-4f2d-8602-d64ca37db647-etc-modprobe-d\") pod \"tuned-hqxdq\" (UID: \"92e18a5c-2e80-4f2d-8602-d64ca37db647\") " pod="openshift-cluster-node-tuning-operator/tuned-hqxdq"
Sep 16 18:40:20.509715 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:20.508229    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/62104ee5-c623-4ffd-b3af-9516ffd5f3bf-system-cni-dir\") pod \"multus-llvsl\" (UID: \"62104ee5-c623-4ffd-b3af-9516ffd5f3bf\") " pod="openshift-multus/multus-llvsl"
Sep 16 18:40:20.509715 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:20.508255    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/3f2c98a3-c6e7-4610-b207-f7ccde3cb571-etc-selinux\") pod \"aws-ebs-csi-driver-node-wrx94\" (UID: \"3f2c98a3-c6e7-4610-b207-f7ccde3cb571\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wrx94"
Sep 16 18:40:20.509715 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:20.508279    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/92e18a5c-2e80-4f2d-8602-d64ca37db647-etc-kubernetes\") pod \"tuned-hqxdq\" (UID: \"92e18a5c-2e80-4f2d-8602-d64ca37db647\") " pod="openshift-cluster-node-tuning-operator/tuned-hqxdq"
Sep 16 18:40:20.509715 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:20.508304    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fc3365ef-dfd3-45fe-b80f-937949d65751-tmp-dir\") pod \"node-resolver-5rrj4\" (UID: \"fc3365ef-dfd3-45fe-b80f-937949d65751\") " pod="openshift-dns/node-resolver-5rrj4"
Sep 16 18:40:20.509715 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:20.508329    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/f4653546-2f57-4858-a038-9fd7c2139d56-konnectivity-ca\") pod \"konnectivity-agent-f69wc\" (UID: \"f4653546-2f57-4858-a038-9fd7c2139d56\") " pod="kube-system/konnectivity-agent-f69wc"
Sep 16 18:40:20.510269 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:20.508332    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/62104ee5-c623-4ffd-b3af-9516ffd5f3bf-cni-binary-copy\") pod \"multus-llvsl\" (UID: \"62104ee5-c623-4ffd-b3af-9516ffd5f3bf\") " pod="openshift-multus/multus-llvsl"
Sep 16 18:40:20.510269 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:20.508413    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/62104ee5-c623-4ffd-b3af-9516ffd5f3bf-system-cni-dir\") pod \"multus-llvsl\" (UID: \"62104ee5-c623-4ffd-b3af-9516ffd5f3bf\") " pod="openshift-multus/multus-llvsl"
Sep 16 18:40:20.510269 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:20.508465    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/92e18a5c-2e80-4f2d-8602-d64ca37db647-etc-kubernetes\") pod \"tuned-hqxdq\" (UID: \"92e18a5c-2e80-4f2d-8602-d64ca37db647\") " pod="openshift-cluster-node-tuning-operator/tuned-hqxdq"
Sep 16 18:40:20.510269 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:20.508530    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/3f2c98a3-c6e7-4610-b207-f7ccde3cb571-etc-selinux\") pod \"aws-ebs-csi-driver-node-wrx94\" (UID: \"3f2c98a3-c6e7-4610-b207-f7ccde3cb571\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wrx94"
Sep 16 18:40:20.510269 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:20.508838    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fc3365ef-dfd3-45fe-b80f-937949d65751-tmp-dir\") pod \"node-resolver-5rrj4\" (UID: \"fc3365ef-dfd3-45fe-b80f-937949d65751\") " pod="openshift-dns/node-resolver-5rrj4"
Sep 16 18:40:20.510269 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:20.508849    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/f4653546-2f57-4858-a038-9fd7c2139d56-konnectivity-ca\") pod \"konnectivity-agent-f69wc\" (UID: \"f4653546-2f57-4858-a038-9fd7c2139d56\") " pod="kube-system/konnectivity-agent-f69wc"
Sep 16 18:40:20.510269 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:20.509603    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/92e18a5c-2e80-4f2d-8602-d64ca37db647-tmp\") pod \"tuned-hqxdq\" (UID: \"92e18a5c-2e80-4f2d-8602-d64ca37db647\") " pod="openshift-cluster-node-tuning-operator/tuned-hqxdq"
Sep 16 18:40:20.510269 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:20.509918    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/f4653546-2f57-4858-a038-9fd7c2139d56-agent-certs\") pod \"konnectivity-agent-f69wc\" (UID: \"f4653546-2f57-4858-a038-9fd7c2139d56\") " pod="kube-system/konnectivity-agent-f69wc"
Sep 16 18:40:20.510269 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:20.510088    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/92e18a5c-2e80-4f2d-8602-d64ca37db647-etc-tuned\") pod \"tuned-hqxdq\" (UID: \"92e18a5c-2e80-4f2d-8602-d64ca37db647\") " pod="openshift-cluster-node-tuning-operator/tuned-hqxdq"
Sep 16 18:40:20.516534 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:20.516514    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x54vk\" (UniqueName: \"kubernetes.io/projected/3f2c98a3-c6e7-4610-b207-f7ccde3cb571-kube-api-access-x54vk\") pod \"aws-ebs-csi-driver-node-wrx94\" (UID: \"3f2c98a3-c6e7-4610-b207-f7ccde3cb571\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wrx94"
Sep 16 18:40:20.516665 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:20.516650    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tkf6g\" (UniqueName: \"kubernetes.io/projected/fc3365ef-dfd3-45fe-b80f-937949d65751-kube-api-access-tkf6g\") pod \"node-resolver-5rrj4\" (UID: \"fc3365ef-dfd3-45fe-b80f-937949d65751\") " pod="openshift-dns/node-resolver-5rrj4"
Sep 16 18:40:20.516869 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:20.516854    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8cxqr\" (UniqueName: \"kubernetes.io/projected/454c7c51-a475-44f6-be51-1a8f1d29beaf-kube-api-access-8cxqr\") pod \"node-ca-bl65n\" (UID: \"454c7c51-a475-44f6-be51-1a8f1d29beaf\") " pod="openshift-image-registry/node-ca-bl65n"
Sep 16 18:40:20.521211 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:20.521188    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fvjll\" (UniqueName: \"kubernetes.io/projected/3d323f6a-c161-4706-b9d5-8803578a6159-kube-api-access-fvjll\") pod \"iptables-alerter-ncnts\" (UID: \"3d323f6a-c161-4706-b9d5-8803578a6159\") " pod="openshift-network-operator/iptables-alerter-ncnts"
Sep 16 18:40:20.521298 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:20.521236    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rnfqc\" (UniqueName: \"kubernetes.io/projected/92e18a5c-2e80-4f2d-8602-d64ca37db647-kube-api-access-rnfqc\") pod \"tuned-hqxdq\" (UID: \"92e18a5c-2e80-4f2d-8602-d64ca37db647\") " pod="openshift-cluster-node-tuning-operator/tuned-hqxdq"
Sep 16 18:40:20.521929 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:20.521913    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4wltg\" (UniqueName: \"kubernetes.io/projected/62104ee5-c623-4ffd-b3af-9516ffd5f3bf-kube-api-access-4wltg\") pod \"multus-llvsl\" (UID: \"62104ee5-c623-4ffd-b3af-9516ffd5f3bf\") " pod="openshift-multus/multus-llvsl"
Sep 16 18:40:20.545201 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:20.545171    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb42fcd462e8e18de823d8d0adf7cc0a.slice/crio-301f3fae50c6ba818e8bded7df2d73b1eddb79139ffe1781f48017ca2a9d5504 WatchSource:0}: Error finding container 301f3fae50c6ba818e8bded7df2d73b1eddb79139ffe1781f48017ca2a9d5504: Status 404 returned error can't find the container with id 301f3fae50c6ba818e8bded7df2d73b1eddb79139ffe1781f48017ca2a9d5504
Sep 16 18:40:20.545633 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:20.545616    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce93b5afb1de6b9a80d64df7da81462d.slice/crio-15a4d5c7f5a9bf5e5a0ee423eaaea3fe475939b79056512b11e11ace2f3b7792 WatchSource:0}: Error finding container 15a4d5c7f5a9bf5e5a0ee423eaaea3fe475939b79056512b11e11ace2f3b7792: Status 404 returned error can't find the container with id 15a4d5c7f5a9bf5e5a0ee423eaaea3fe475939b79056512b11e11ace2f3b7792
Sep 16 18:40:20.549991 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:20.549964    2578 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 16 18:40:20.629668 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:20.629650    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-k6gvs"
Sep 16 18:40:20.639261 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:20.639228    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6012728d_aee6_4f93_ab6e_fad616ef36e8.slice/crio-e75f04cc0edf2060118904620bb51b8c9a52e1e15cfa58ee450ca9f3662b99b4 WatchSource:0}: Error finding container e75f04cc0edf2060118904620bb51b8c9a52e1e15cfa58ee450ca9f3662b99b4: Status 404 returned error can't find the container with id e75f04cc0edf2060118904620bb51b8c9a52e1e15cfa58ee450ca9f3662b99b4
Sep 16 18:40:20.651414 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:20.651398    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-dxtkl"
Sep 16 18:40:20.658202 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:20.658186    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-ncnts"
Sep 16 18:40:20.659565 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:20.659546    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod91876c9b_d59f_4d80_bda9_f7addebaaee3.slice/crio-82ae39762edae23aa803b56c13eacf1cdaef3d60c0a627440f10e6c5d79cc698 WatchSource:0}: Error finding container 82ae39762edae23aa803b56c13eacf1cdaef3d60c0a627440f10e6c5d79cc698: Status 404 returned error can't find the container with id 82ae39762edae23aa803b56c13eacf1cdaef3d60c0a627440f10e6c5d79cc698
Sep 16 18:40:20.666797 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:20.666777    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d323f6a_c161_4706_b9d5_8803578a6159.slice/crio-caee374f08cd2e7acb2426881050edad424f6ce0e8549412af181e7b116dc155 WatchSource:0}: Error finding container caee374f08cd2e7acb2426881050edad424f6ce0e8549412af181e7b116dc155: Status 404 returned error can't find the container with id caee374f08cd2e7acb2426881050edad424f6ce0e8549412af181e7b116dc155
Sep 16 18:40:20.668498 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:20.668480    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-bl65n"
Sep 16 18:40:20.676224 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:20.676201    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod454c7c51_a475_44f6_be51_1a8f1d29beaf.slice/crio-a7aca1498bc3795d5f6df3fb21cb4ee2ec6cacd27fbe8aa3f3f2bd6d74ec1cc6 WatchSource:0}: Error finding container a7aca1498bc3795d5f6df3fb21cb4ee2ec6cacd27fbe8aa3f3f2bd6d74ec1cc6: Status 404 returned error can't find the container with id a7aca1498bc3795d5f6df3fb21cb4ee2ec6cacd27fbe8aa3f3f2bd6d74ec1cc6
Sep 16 18:40:20.684081 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:20.684060    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-llvsl"
Sep 16 18:40:20.688940 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:20.688922    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-5rrj4"
Sep 16 18:40:20.692534 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:20.692518    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod62104ee5_c623_4ffd_b3af_9516ffd5f3bf.slice/crio-ef4d695e3c4f8dff40f9dca436d7a52271612aa152b356f9a6b0efb1565e3149 WatchSource:0}: Error finding container ef4d695e3c4f8dff40f9dca436d7a52271612aa152b356f9a6b0efb1565e3149: Status 404 returned error can't find the container with id ef4d695e3c4f8dff40f9dca436d7a52271612aa152b356f9a6b0efb1565e3149
Sep 16 18:40:20.697696 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:20.697671    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc3365ef_dfd3_45fe_b80f_937949d65751.slice/crio-e1a29ac3a90bed890e8ddbca7c855c54a4037e25012a2413b9cc67c9b597855b WatchSource:0}: Error finding container e1a29ac3a90bed890e8ddbca7c855c54a4037e25012a2413b9cc67c9b597855b: Status 404 returned error can't find the container with id e1a29ac3a90bed890e8ddbca7c855c54a4037e25012a2413b9cc67c9b597855b
Sep 16 18:40:20.710145 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:20.710125    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-f69wc"
Sep 16 18:40:20.731211 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:20.731191    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wrx94"
Sep 16 18:40:20.736041 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:20.736023    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-hqxdq"
Sep 16 18:40:20.739209 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:20.739186    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f2c98a3_c6e7_4610_b207_f7ccde3cb571.slice/crio-38c202e28561540c79226909ce63c356016a314d70f3b13b8a3c2acba73c6c81 WatchSource:0}: Error finding container 38c202e28561540c79226909ce63c356016a314d70f3b13b8a3c2acba73c6c81: Status 404 returned error can't find the container with id 38c202e28561540c79226909ce63c356016a314d70f3b13b8a3c2acba73c6c81
Sep 16 18:40:20.744886 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:20.744862    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod92e18a5c_2e80_4f2d_8602_d64ca37db647.slice/crio-c34cecc1162b0e0de780c7a4172ff366e0278699f23d56a81b1c0628821d3706 WatchSource:0}: Error finding container c34cecc1162b0e0de780c7a4172ff366e0278699f23d56a81b1c0628821d3706: Status 404 returned error can't find the container with id c34cecc1162b0e0de780c7a4172ff366e0278699f23d56a81b1c0628821d3706
Sep 16 18:40:20.746481 ip-10-0-135-119 kubenswrapper[2578]: E0916 18:40:20.746462    2578 kuberuntime_manager.go:1459] "Unhandled Error" err="container tuned start failed in pod tuned-hqxdq_openshift-cluster-node-tuning-operator(92e18a5c-2e80-4f2d-8602-d64ca37db647): ErrImagePull: pull QPS exceeded" logger="UnhandledError"
Sep 16 18:40:20.746526 ip-10-0-135-119 kubenswrapper[2578]: E0916 18:40:20.746496    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"tuned\" with ErrImagePull: \"pull QPS exceeded\"" pod="openshift-cluster-node-tuning-operator/tuned-hqxdq" podUID="92e18a5c-2e80-4f2d-8602-d64ca37db647"
Sep 16 18:40:20.911596 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:20.911509    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1f969e62-dd58-41b0-9419-b2ca445f3ec4-metrics-certs\") pod \"network-metrics-daemon-zfv2x\" (UID: \"1f969e62-dd58-41b0-9419-b2ca445f3ec4\") " pod="openshift-multus/network-metrics-daemon-zfv2x"
Sep 16 18:40:20.911752 ip-10-0-135-119 kubenswrapper[2578]: E0916 18:40:20.911640    2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 18:40:20.911752 ip-10-0-135-119 kubenswrapper[2578]: E0916 18:40:20.911714    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1f969e62-dd58-41b0-9419-b2ca445f3ec4-metrics-certs podName:1f969e62-dd58-41b0-9419-b2ca445f3ec4 nodeName:}" failed. No retries permitted until 2026-09-16 18:40:21.91169376 +0000 UTC m=+2.978185230 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1f969e62-dd58-41b0-9419-b2ca445f3ec4-metrics-certs") pod "network-metrics-daemon-zfv2x" (UID: "1f969e62-dd58-41b0-9419-b2ca445f3ec4") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 18:40:21.012834 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:21.012791    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3377e453-9418-49a8-870b-64424f13d694-original-pull-secret\") pod \"global-pull-secret-syncer-8ndsh\" (UID: \"3377e453-9418-49a8-870b-64424f13d694\") " pod="kube-system/global-pull-secret-syncer-8ndsh"
Sep 16 18:40:21.013058 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:21.012866    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-77jgz\" (UniqueName: \"kubernetes.io/projected/63abc484-192b-4095-81ea-b5db79acd0ba-kube-api-access-77jgz\") pod \"network-check-target-f287c\" (UID: \"63abc484-192b-4095-81ea-b5db79acd0ba\") " pod="openshift-network-diagnostics/network-check-target-f287c"
Sep 16 18:40:21.013058 ip-10-0-135-119 kubenswrapper[2578]: E0916 18:40:21.012981    2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 16 18:40:21.013058 ip-10-0-135-119 kubenswrapper[2578]: E0916 18:40:21.013039    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 16 18:40:21.013058 ip-10-0-135-119 kubenswrapper[2578]: E0916 18:40:21.013058    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 16 18:40:21.013277 ip-10-0-135-119 kubenswrapper[2578]: E0916 18:40:21.013073    2578 projected.go:196] Error preparing data for projected volume kube-api-access-77jgz for pod openshift-network-diagnostics/network-check-target-f287c: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 18:40:21.013277 ip-10-0-135-119 kubenswrapper[2578]: E0916 18:40:21.013080    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3377e453-9418-49a8-870b-64424f13d694-original-pull-secret podName:3377e453-9418-49a8-870b-64424f13d694 nodeName:}" failed. No retries permitted until 2026-09-16 18:40:22.01305763 +0000 UTC m=+3.079549116 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/3377e453-9418-49a8-870b-64424f13d694-original-pull-secret") pod "global-pull-secret-syncer-8ndsh" (UID: "3377e453-9418-49a8-870b-64424f13d694") : object "kube-system"/"original-pull-secret" not registered
Sep 16 18:40:21.013277 ip-10-0-135-119 kubenswrapper[2578]: E0916 18:40:21.013128    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/63abc484-192b-4095-81ea-b5db79acd0ba-kube-api-access-77jgz podName:63abc484-192b-4095-81ea-b5db79acd0ba nodeName:}" failed. No retries permitted until 2026-09-16 18:40:22.013107818 +0000 UTC m=+3.079599294 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-77jgz" (UniqueName: "kubernetes.io/projected/63abc484-192b-4095-81ea-b5db79acd0ba-kube-api-access-77jgz") pod "network-check-target-f287c" (UID: "63abc484-192b-4095-81ea-b5db79acd0ba") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 18:40:21.406973 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:21.406902    2578 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-15 18:35:20 +0000 UTC" deadline="2028-03-31 06:48:02.359443747 +0000 UTC"
Sep 16 18:40:21.406973 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:21.406971    2578 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13476h7m40.952477418s"
Sep 16 18:40:21.429363 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:21.429324    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-f287c"
Sep 16 18:40:21.429573 ip-10-0-135-119 kubenswrapper[2578]: E0916 18:40:21.429456    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-f287c" podUID="63abc484-192b-4095-81ea-b5db79acd0ba"
Sep 16 18:40:21.429897 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:21.429873    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zfv2x"
Sep 16 18:40:21.430014 ip-10-0-135-119 kubenswrapper[2578]: E0916 18:40:21.429994    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zfv2x" podUID="1f969e62-dd58-41b0-9419-b2ca445f3ec4"
Sep 16 18:40:21.430110 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:21.430094    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8ndsh"
Sep 16 18:40:21.430206 ip-10-0-135-119 kubenswrapper[2578]: E0916 18:40:21.430186    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8ndsh" podUID="3377e453-9418-49a8-870b-64424f13d694"
Sep 16 18:40:21.446772 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:21.446715    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-hqxdq" event={"ID":"92e18a5c-2e80-4f2d-8602-d64ca37db647","Type":"ContainerStarted","Data":"c34cecc1162b0e0de780c7a4172ff366e0278699f23d56a81b1c0628821d3706"}
Sep 16 18:40:21.450645 ip-10-0-135-119 kubenswrapper[2578]: E0916 18:40:21.450614    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"tuned\" with ImagePullBackOff: \"Back-off pulling image \\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:667babb345aeb39d2681913fd91a5c163d48d4ab8a3ba0d3156502181ea1177a\\\": ErrImagePull: pull QPS exceeded\"" pod="openshift-cluster-node-tuning-operator/tuned-hqxdq" podUID="92e18a5c-2e80-4f2d-8602-d64ca37db647"
Sep 16 18:40:21.458855 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:21.458826    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wrx94" event={"ID":"3f2c98a3-c6e7-4610-b207-f7ccde3cb571","Type":"ContainerStarted","Data":"38c202e28561540c79226909ce63c356016a314d70f3b13b8a3c2acba73c6c81"}
Sep 16 18:40:21.473604 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:21.473565    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-f69wc" event={"ID":"f4653546-2f57-4858-a038-9fd7c2139d56","Type":"ContainerStarted","Data":"e22f1a504501ff7d88dcb2f9fd3402ce9902ddd22cc21f01cb7f6436f9afdbbd"}
Sep 16 18:40:21.495129 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:21.495099    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-llvsl" event={"ID":"62104ee5-c623-4ffd-b3af-9516ffd5f3bf","Type":"ContainerStarted","Data":"ef4d695e3c4f8dff40f9dca436d7a52271612aa152b356f9a6b0efb1565e3149"}
Sep 16 18:40:21.512554 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:21.512520    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-bl65n" event={"ID":"454c7c51-a475-44f6-be51-1a8f1d29beaf","Type":"ContainerStarted","Data":"a7aca1498bc3795d5f6df3fb21cb4ee2ec6cacd27fbe8aa3f3f2bd6d74ec1cc6"}
Sep 16 18:40:21.524158 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:21.524131    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-ncnts" event={"ID":"3d323f6a-c161-4706-b9d5-8803578a6159","Type":"ContainerStarted","Data":"caee374f08cd2e7acb2426881050edad424f6ce0e8549412af181e7b116dc155"}
Sep 16 18:40:21.532221 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:21.532139    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-dxtkl" event={"ID":"91876c9b-d59f-4d80-bda9-f7addebaaee3","Type":"ContainerStarted","Data":"82ae39762edae23aa803b56c13eacf1cdaef3d60c0a627440f10e6c5d79cc698"}
Sep 16 18:40:21.538506 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:21.537849    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-k6gvs" event={"ID":"6012728d-aee6-4f93-ab6e-fad616ef36e8","Type":"ContainerStarted","Data":"e75f04cc0edf2060118904620bb51b8c9a52e1e15cfa58ee450ca9f3662b99b4"}
Sep 16 18:40:21.544318 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:21.544293    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-5rrj4" event={"ID":"fc3365ef-dfd3-45fe-b80f-937949d65751","Type":"ContainerStarted","Data":"e1a29ac3a90bed890e8ddbca7c855c54a4037e25012a2413b9cc67c9b597855b"}
Sep 16 18:40:21.560317 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:21.560290    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-119.ec2.internal" event={"ID":"ce93b5afb1de6b9a80d64df7da81462d","Type":"ContainerStarted","Data":"15a4d5c7f5a9bf5e5a0ee423eaaea3fe475939b79056512b11e11ace2f3b7792"}
Sep 16 18:40:21.571611 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:21.571582    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-119.ec2.internal" event={"ID":"db42fcd462e8e18de823d8d0adf7cc0a","Type":"ContainerStarted","Data":"301f3fae50c6ba818e8bded7df2d73b1eddb79139ffe1781f48017ca2a9d5504"}
Sep 16 18:40:21.802796 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:21.802720    2578 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 16 18:40:21.919783 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:21.919748    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1f969e62-dd58-41b0-9419-b2ca445f3ec4-metrics-certs\") pod \"network-metrics-daemon-zfv2x\" (UID: \"1f969e62-dd58-41b0-9419-b2ca445f3ec4\") " pod="openshift-multus/network-metrics-daemon-zfv2x"
Sep 16 18:40:21.919972 ip-10-0-135-119 kubenswrapper[2578]: E0916 18:40:21.919925    2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 18:40:21.920075 ip-10-0-135-119 kubenswrapper[2578]: E0916 18:40:21.919992    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1f969e62-dd58-41b0-9419-b2ca445f3ec4-metrics-certs podName:1f969e62-dd58-41b0-9419-b2ca445f3ec4 nodeName:}" failed. No retries permitted until 2026-09-16 18:40:23.919972857 +0000 UTC m=+4.986464344 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1f969e62-dd58-41b0-9419-b2ca445f3ec4-metrics-certs") pod "network-metrics-daemon-zfv2x" (UID: "1f969e62-dd58-41b0-9419-b2ca445f3ec4") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 18:40:22.020325 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:22.020280    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-77jgz\" (UniqueName: \"kubernetes.io/projected/63abc484-192b-4095-81ea-b5db79acd0ba-kube-api-access-77jgz\") pod \"network-check-target-f287c\" (UID: \"63abc484-192b-4095-81ea-b5db79acd0ba\") " pod="openshift-network-diagnostics/network-check-target-f287c"
Sep 16 18:40:22.020612 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:22.020373    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3377e453-9418-49a8-870b-64424f13d694-original-pull-secret\") pod \"global-pull-secret-syncer-8ndsh\" (UID: \"3377e453-9418-49a8-870b-64424f13d694\") " pod="kube-system/global-pull-secret-syncer-8ndsh"
Sep 16 18:40:22.020612 ip-10-0-135-119 kubenswrapper[2578]: E0916 18:40:22.020507    2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 16 18:40:22.020612 ip-10-0-135-119 kubenswrapper[2578]: E0916 18:40:22.020584    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3377e453-9418-49a8-870b-64424f13d694-original-pull-secret podName:3377e453-9418-49a8-870b-64424f13d694 nodeName:}" failed. No retries permitted until 2026-09-16 18:40:24.020564902 +0000 UTC m=+5.087056375 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/3377e453-9418-49a8-870b-64424f13d694-original-pull-secret") pod "global-pull-secret-syncer-8ndsh" (UID: "3377e453-9418-49a8-870b-64424f13d694") : object "kube-system"/"original-pull-secret" not registered
Sep 16 18:40:22.021058 ip-10-0-135-119 kubenswrapper[2578]: E0916 18:40:22.021034    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 16 18:40:22.021167 ip-10-0-135-119 kubenswrapper[2578]: E0916 18:40:22.021075    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 16 18:40:22.021167 ip-10-0-135-119 kubenswrapper[2578]: E0916 18:40:22.021089    2578 projected.go:196] Error preparing data for projected volume kube-api-access-77jgz for pod openshift-network-diagnostics/network-check-target-f287c: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 18:40:22.021167 ip-10-0-135-119 kubenswrapper[2578]: E0916 18:40:22.021137    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/63abc484-192b-4095-81ea-b5db79acd0ba-kube-api-access-77jgz podName:63abc484-192b-4095-81ea-b5db79acd0ba nodeName:}" failed. No retries permitted until 2026-09-16 18:40:24.021120244 +0000 UTC m=+5.087611717 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-77jgz" (UniqueName: "kubernetes.io/projected/63abc484-192b-4095-81ea-b5db79acd0ba-kube-api-access-77jgz") pod "network-check-target-f287c" (UID: "63abc484-192b-4095-81ea-b5db79acd0ba") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 18:40:22.318347 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:22.317690    2578 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 16 18:40:22.415509 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:22.415457    2578 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-15 18:35:20 +0000 UTC" deadline="2028-03-31 12:08:37.618243207 +0000 UTC"
Sep 16 18:40:22.415509 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:22.415510    2578 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13481h28m15.202738521s"
Sep 16 18:40:23.096215 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:23.095967    2578 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 16 18:40:23.431743 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:23.430924    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8ndsh"
Sep 16 18:40:23.431743 ip-10-0-135-119 kubenswrapper[2578]: E0916 18:40:23.431075    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8ndsh" podUID="3377e453-9418-49a8-870b-64424f13d694"
Sep 16 18:40:23.431743 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:23.431471    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-f287c"
Sep 16 18:40:23.431743 ip-10-0-135-119 kubenswrapper[2578]: E0916 18:40:23.431560    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-f287c" podUID="63abc484-192b-4095-81ea-b5db79acd0ba"
Sep 16 18:40:23.431743 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:23.431610    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zfv2x"
Sep 16 18:40:23.431743 ip-10-0-135-119 kubenswrapper[2578]: E0916 18:40:23.431696    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zfv2x" podUID="1f969e62-dd58-41b0-9419-b2ca445f3ec4"
Sep 16 18:40:23.945290 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:23.945252    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1f969e62-dd58-41b0-9419-b2ca445f3ec4-metrics-certs\") pod \"network-metrics-daemon-zfv2x\" (UID: \"1f969e62-dd58-41b0-9419-b2ca445f3ec4\") " pod="openshift-multus/network-metrics-daemon-zfv2x"
Sep 16 18:40:23.945484 ip-10-0-135-119 kubenswrapper[2578]: E0916 18:40:23.945440    2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 18:40:23.945555 ip-10-0-135-119 kubenswrapper[2578]: E0916 18:40:23.945510    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1f969e62-dd58-41b0-9419-b2ca445f3ec4-metrics-certs podName:1f969e62-dd58-41b0-9419-b2ca445f3ec4 nodeName:}" failed. No retries permitted until 2026-09-16 18:40:27.945488969 +0000 UTC m=+9.011980479 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1f969e62-dd58-41b0-9419-b2ca445f3ec4-metrics-certs") pod "network-metrics-daemon-zfv2x" (UID: "1f969e62-dd58-41b0-9419-b2ca445f3ec4") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 18:40:24.046426 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:24.046383    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-77jgz\" (UniqueName: \"kubernetes.io/projected/63abc484-192b-4095-81ea-b5db79acd0ba-kube-api-access-77jgz\") pod \"network-check-target-f287c\" (UID: \"63abc484-192b-4095-81ea-b5db79acd0ba\") " pod="openshift-network-diagnostics/network-check-target-f287c"
Sep 16 18:40:24.046594 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:24.046480    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3377e453-9418-49a8-870b-64424f13d694-original-pull-secret\") pod \"global-pull-secret-syncer-8ndsh\" (UID: \"3377e453-9418-49a8-870b-64424f13d694\") " pod="kube-system/global-pull-secret-syncer-8ndsh"
Sep 16 18:40:24.046641 ip-10-0-135-119 kubenswrapper[2578]: E0916 18:40:24.046613    2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 16 18:40:24.046694 ip-10-0-135-119 kubenswrapper[2578]: E0916 18:40:24.046681    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3377e453-9418-49a8-870b-64424f13d694-original-pull-secret podName:3377e453-9418-49a8-870b-64424f13d694 nodeName:}" failed. No retries permitted until 2026-09-16 18:40:28.046660417 +0000 UTC m=+9.113151927 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/3377e453-9418-49a8-870b-64424f13d694-original-pull-secret") pod "global-pull-secret-syncer-8ndsh" (UID: "3377e453-9418-49a8-870b-64424f13d694") : object "kube-system"/"original-pull-secret" not registered
Sep 16 18:40:24.047201 ip-10-0-135-119 kubenswrapper[2578]: E0916 18:40:24.047181    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 16 18:40:24.047251 ip-10-0-135-119 kubenswrapper[2578]: E0916 18:40:24.047211    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 16 18:40:24.047251 ip-10-0-135-119 kubenswrapper[2578]: E0916 18:40:24.047226    2578 projected.go:196] Error preparing data for projected volume kube-api-access-77jgz for pod openshift-network-diagnostics/network-check-target-f287c: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 18:40:24.047362 ip-10-0-135-119 kubenswrapper[2578]: E0916 18:40:24.047276    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/63abc484-192b-4095-81ea-b5db79acd0ba-kube-api-access-77jgz podName:63abc484-192b-4095-81ea-b5db79acd0ba nodeName:}" failed. No retries permitted until 2026-09-16 18:40:28.047259954 +0000 UTC m=+9.113751425 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-77jgz" (UniqueName: "kubernetes.io/projected/63abc484-192b-4095-81ea-b5db79acd0ba-kube-api-access-77jgz") pod "network-check-target-f287c" (UID: "63abc484-192b-4095-81ea-b5db79acd0ba") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 18:40:25.428339 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:25.428301    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-f287c"
Sep 16 18:40:25.428734 ip-10-0-135-119 kubenswrapper[2578]: E0916 18:40:25.428423    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-f287c" podUID="63abc484-192b-4095-81ea-b5db79acd0ba"
Sep 16 18:40:25.428786 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:25.428774    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zfv2x"
Sep 16 18:40:25.428884 ip-10-0-135-119 kubenswrapper[2578]: E0916 18:40:25.428864    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zfv2x" podUID="1f969e62-dd58-41b0-9419-b2ca445f3ec4"
Sep 16 18:40:25.428953 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:25.428938    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8ndsh"
Sep 16 18:40:25.429048 ip-10-0-135-119 kubenswrapper[2578]: E0916 18:40:25.429031    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8ndsh" podUID="3377e453-9418-49a8-870b-64424f13d694"
Sep 16 18:40:27.428460 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:27.428428    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8ndsh"
Sep 16 18:40:27.428894 ip-10-0-135-119 kubenswrapper[2578]: E0916 18:40:27.428561    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8ndsh" podUID="3377e453-9418-49a8-870b-64424f13d694"
Sep 16 18:40:27.429008 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:27.428929    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-f287c"
Sep 16 18:40:27.429077 ip-10-0-135-119 kubenswrapper[2578]: E0916 18:40:27.429043    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-f287c" podUID="63abc484-192b-4095-81ea-b5db79acd0ba"
Sep 16 18:40:27.429221 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:27.429191    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zfv2x"
Sep 16 18:40:27.429337 ip-10-0-135-119 kubenswrapper[2578]: E0916 18:40:27.429311    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zfv2x" podUID="1f969e62-dd58-41b0-9419-b2ca445f3ec4"
Sep 16 18:40:27.980044 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:27.980004    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1f969e62-dd58-41b0-9419-b2ca445f3ec4-metrics-certs\") pod \"network-metrics-daemon-zfv2x\" (UID: \"1f969e62-dd58-41b0-9419-b2ca445f3ec4\") " pod="openshift-multus/network-metrics-daemon-zfv2x"
Sep 16 18:40:27.980234 ip-10-0-135-119 kubenswrapper[2578]: E0916 18:40:27.980183    2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 18:40:27.980298 ip-10-0-135-119 kubenswrapper[2578]: E0916 18:40:27.980247    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1f969e62-dd58-41b0-9419-b2ca445f3ec4-metrics-certs podName:1f969e62-dd58-41b0-9419-b2ca445f3ec4 nodeName:}" failed. No retries permitted until 2026-09-16 18:40:35.980227981 +0000 UTC m=+17.046719467 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1f969e62-dd58-41b0-9419-b2ca445f3ec4-metrics-certs") pod "network-metrics-daemon-zfv2x" (UID: "1f969e62-dd58-41b0-9419-b2ca445f3ec4") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 18:40:28.081280 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:28.081236    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3377e453-9418-49a8-870b-64424f13d694-original-pull-secret\") pod \"global-pull-secret-syncer-8ndsh\" (UID: \"3377e453-9418-49a8-870b-64424f13d694\") " pod="kube-system/global-pull-secret-syncer-8ndsh"
Sep 16 18:40:28.081468 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:28.081298    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-77jgz\" (UniqueName: \"kubernetes.io/projected/63abc484-192b-4095-81ea-b5db79acd0ba-kube-api-access-77jgz\") pod \"network-check-target-f287c\" (UID: \"63abc484-192b-4095-81ea-b5db79acd0ba\") " pod="openshift-network-diagnostics/network-check-target-f287c"
Sep 16 18:40:28.086090 ip-10-0-135-119 kubenswrapper[2578]: E0916 18:40:28.086062    2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 16 18:40:28.086217 ip-10-0-135-119 kubenswrapper[2578]: E0916 18:40:28.086163    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3377e453-9418-49a8-870b-64424f13d694-original-pull-secret podName:3377e453-9418-49a8-870b-64424f13d694 nodeName:}" failed. No retries permitted until 2026-09-16 18:40:36.086143609 +0000 UTC m=+17.152635094 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/3377e453-9418-49a8-870b-64424f13d694-original-pull-secret") pod "global-pull-secret-syncer-8ndsh" (UID: "3377e453-9418-49a8-870b-64424f13d694") : object "kube-system"/"original-pull-secret" not registered
Sep 16 18:40:28.086644 ip-10-0-135-119 kubenswrapper[2578]: E0916 18:40:28.086623    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 16 18:40:28.086747 ip-10-0-135-119 kubenswrapper[2578]: E0916 18:40:28.086656    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 16 18:40:28.086747 ip-10-0-135-119 kubenswrapper[2578]: E0916 18:40:28.086672    2578 projected.go:196] Error preparing data for projected volume kube-api-access-77jgz for pod openshift-network-diagnostics/network-check-target-f287c: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 18:40:28.086747 ip-10-0-135-119 kubenswrapper[2578]: E0916 18:40:28.086727    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/63abc484-192b-4095-81ea-b5db79acd0ba-kube-api-access-77jgz podName:63abc484-192b-4095-81ea-b5db79acd0ba nodeName:}" failed. No retries permitted until 2026-09-16 18:40:36.086704802 +0000 UTC m=+17.153196288 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-77jgz" (UniqueName: "kubernetes.io/projected/63abc484-192b-4095-81ea-b5db79acd0ba-kube-api-access-77jgz") pod "network-check-target-f287c" (UID: "63abc484-192b-4095-81ea-b5db79acd0ba") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 18:40:29.429422 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:29.429391    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8ndsh"
Sep 16 18:40:29.429809 ip-10-0-135-119 kubenswrapper[2578]: E0916 18:40:29.429477    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8ndsh" podUID="3377e453-9418-49a8-870b-64424f13d694"
Sep 16 18:40:29.429876 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:29.429834    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zfv2x"
Sep 16 18:40:29.429969 ip-10-0-135-119 kubenswrapper[2578]: E0916 18:40:29.429943    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zfv2x" podUID="1f969e62-dd58-41b0-9419-b2ca445f3ec4"
Sep 16 18:40:29.430065 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:29.430026    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-f287c"
Sep 16 18:40:29.430147 ip-10-0-135-119 kubenswrapper[2578]: E0916 18:40:29.430123    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-f287c" podUID="63abc484-192b-4095-81ea-b5db79acd0ba"
Sep 16 18:40:31.428795 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:31.428708    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8ndsh"
Sep 16 18:40:31.429223 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:31.428716    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-f287c"
Sep 16 18:40:31.429223 ip-10-0-135-119 kubenswrapper[2578]: E0916 18:40:31.428839    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8ndsh" podUID="3377e453-9418-49a8-870b-64424f13d694"
Sep 16 18:40:31.429223 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:31.428855    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zfv2x"
Sep 16 18:40:31.429223 ip-10-0-135-119 kubenswrapper[2578]: E0916 18:40:31.428944    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-f287c" podUID="63abc484-192b-4095-81ea-b5db79acd0ba"
Sep 16 18:40:31.429223 ip-10-0-135-119 kubenswrapper[2578]: E0916 18:40:31.429047    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zfv2x" podUID="1f969e62-dd58-41b0-9419-b2ca445f3ec4"
Sep 16 18:40:33.428831 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:33.428799    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-f287c"
Sep 16 18:40:33.429312 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:33.428806    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zfv2x"
Sep 16 18:40:33.429312 ip-10-0-135-119 kubenswrapper[2578]: E0916 18:40:33.428905    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-f287c" podUID="63abc484-192b-4095-81ea-b5db79acd0ba"
Sep 16 18:40:33.429312 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:33.428806    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8ndsh"
Sep 16 18:40:33.429312 ip-10-0-135-119 kubenswrapper[2578]: E0916 18:40:33.429026    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zfv2x" podUID="1f969e62-dd58-41b0-9419-b2ca445f3ec4"
Sep 16 18:40:33.429312 ip-10-0-135-119 kubenswrapper[2578]: E0916 18:40:33.429077    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8ndsh" podUID="3377e453-9418-49a8-870b-64424f13d694"
Sep 16 18:40:35.428634 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:35.428605    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zfv2x"
Sep 16 18:40:35.428634 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:35.428644    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8ndsh"
Sep 16 18:40:35.429166 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:35.428618    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-f287c"
Sep 16 18:40:35.429166 ip-10-0-135-119 kubenswrapper[2578]: E0916 18:40:35.428740    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zfv2x" podUID="1f969e62-dd58-41b0-9419-b2ca445f3ec4"
Sep 16 18:40:35.429166 ip-10-0-135-119 kubenswrapper[2578]: E0916 18:40:35.428826    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-f287c" podUID="63abc484-192b-4095-81ea-b5db79acd0ba"
Sep 16 18:40:35.429166 ip-10-0-135-119 kubenswrapper[2578]: E0916 18:40:35.428891    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8ndsh" podUID="3377e453-9418-49a8-870b-64424f13d694"
Sep 16 18:40:36.039716 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:36.039687    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1f969e62-dd58-41b0-9419-b2ca445f3ec4-metrics-certs\") pod \"network-metrics-daemon-zfv2x\" (UID: \"1f969e62-dd58-41b0-9419-b2ca445f3ec4\") " pod="openshift-multus/network-metrics-daemon-zfv2x"
Sep 16 18:40:36.039936 ip-10-0-135-119 kubenswrapper[2578]: E0916 18:40:36.039809    2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 18:40:36.039936 ip-10-0-135-119 kubenswrapper[2578]: E0916 18:40:36.039875    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1f969e62-dd58-41b0-9419-b2ca445f3ec4-metrics-certs podName:1f969e62-dd58-41b0-9419-b2ca445f3ec4 nodeName:}" failed. No retries permitted until 2026-09-16 18:40:52.039856573 +0000 UTC m=+33.106348040 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1f969e62-dd58-41b0-9419-b2ca445f3ec4-metrics-certs") pod "network-metrics-daemon-zfv2x" (UID: "1f969e62-dd58-41b0-9419-b2ca445f3ec4") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 18:40:36.140063 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:36.140023    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3377e453-9418-49a8-870b-64424f13d694-original-pull-secret\") pod \"global-pull-secret-syncer-8ndsh\" (UID: \"3377e453-9418-49a8-870b-64424f13d694\") " pod="kube-system/global-pull-secret-syncer-8ndsh"
Sep 16 18:40:36.140225 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:36.140096    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-77jgz\" (UniqueName: \"kubernetes.io/projected/63abc484-192b-4095-81ea-b5db79acd0ba-kube-api-access-77jgz\") pod \"network-check-target-f287c\" (UID: \"63abc484-192b-4095-81ea-b5db79acd0ba\") " pod="openshift-network-diagnostics/network-check-target-f287c"
Sep 16 18:40:36.140225 ip-10-0-135-119 kubenswrapper[2578]: E0916 18:40:36.140175    2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 16 18:40:36.140225 ip-10-0-135-119 kubenswrapper[2578]: E0916 18:40:36.140206    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 16 18:40:36.140225 ip-10-0-135-119 kubenswrapper[2578]: E0916 18:40:36.140225    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 16 18:40:36.140391 ip-10-0-135-119 kubenswrapper[2578]: E0916 18:40:36.140241    2578 projected.go:196] Error preparing data for projected volume kube-api-access-77jgz for pod openshift-network-diagnostics/network-check-target-f287c: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 18:40:36.140391 ip-10-0-135-119 kubenswrapper[2578]: E0916 18:40:36.140245    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3377e453-9418-49a8-870b-64424f13d694-original-pull-secret podName:3377e453-9418-49a8-870b-64424f13d694 nodeName:}" failed. No retries permitted until 2026-09-16 18:40:52.140222612 +0000 UTC m=+33.206714081 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/3377e453-9418-49a8-870b-64424f13d694-original-pull-secret") pod "global-pull-secret-syncer-8ndsh" (UID: "3377e453-9418-49a8-870b-64424f13d694") : object "kube-system"/"original-pull-secret" not registered
Sep 16 18:40:36.140391 ip-10-0-135-119 kubenswrapper[2578]: E0916 18:40:36.140286    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/63abc484-192b-4095-81ea-b5db79acd0ba-kube-api-access-77jgz podName:63abc484-192b-4095-81ea-b5db79acd0ba nodeName:}" failed. No retries permitted until 2026-09-16 18:40:52.140270852 +0000 UTC m=+33.206762321 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-77jgz" (UniqueName: "kubernetes.io/projected/63abc484-192b-4095-81ea-b5db79acd0ba-kube-api-access-77jgz") pod "network-check-target-f287c" (UID: "63abc484-192b-4095-81ea-b5db79acd0ba") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 18:40:37.428448 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:37.428419    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-f287c"
Sep 16 18:40:37.428853 ip-10-0-135-119 kubenswrapper[2578]: E0916 18:40:37.428544    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-f287c" podUID="63abc484-192b-4095-81ea-b5db79acd0ba"
Sep 16 18:40:37.428853 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:37.428561    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zfv2x"
Sep 16 18:40:37.428853 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:37.428418    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8ndsh"
Sep 16 18:40:37.428853 ip-10-0-135-119 kubenswrapper[2578]: E0916 18:40:37.428676    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zfv2x" podUID="1f969e62-dd58-41b0-9419-b2ca445f3ec4"
Sep 16 18:40:37.428853 ip-10-0-135-119 kubenswrapper[2578]: E0916 18:40:37.428764    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8ndsh" podUID="3377e453-9418-49a8-870b-64424f13d694"
Sep 16 18:40:38.640871 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:38.640475    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-k6gvs_6012728d-aee6-4f93-ab6e-fad616ef36e8/ovn-acl-logging/0.log"
Sep 16 18:40:38.641601 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:38.641456    2578 generic.go:356] "Generic (PLEG): container finished" podID="6012728d-aee6-4f93-ab6e-fad616ef36e8" containerID="1212b44ae45e2eaf2a1735068182b97a1880607179526989f29a1762f70e1684" exitCode=1
Sep 16 18:40:38.641601 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:38.641537    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-k6gvs" event={"ID":"6012728d-aee6-4f93-ab6e-fad616ef36e8","Type":"ContainerStarted","Data":"f54ddb2a60bb5bc161898aacdb283819727ab92c21173116dce06499fe86ecf3"}
Sep 16 18:40:38.641688 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:38.641648    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-k6gvs" event={"ID":"6012728d-aee6-4f93-ab6e-fad616ef36e8","Type":"ContainerStarted","Data":"bb702f7ab337eb84c4f86b776217d92dbaee77be39ab58fbe831f475774538d0"}
Sep 16 18:40:38.641730 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:38.641698    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-k6gvs" event={"ID":"6012728d-aee6-4f93-ab6e-fad616ef36e8","Type":"ContainerStarted","Data":"0ce9ab2303b2c09c71b6322108320abe64dbb7eed62605fef8b5f43e6df301e9"}
Sep 16 18:40:38.641780 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:38.641745    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-k6gvs" event={"ID":"6012728d-aee6-4f93-ab6e-fad616ef36e8","Type":"ContainerStarted","Data":"7c40ded766d7700c668675935cdf3441e3ff1c7f16aa31367be48ea2dade000b"}
Sep 16 18:40:38.641843 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:38.641789    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-k6gvs" event={"ID":"6012728d-aee6-4f93-ab6e-fad616ef36e8","Type":"ContainerDied","Data":"1212b44ae45e2eaf2a1735068182b97a1880607179526989f29a1762f70e1684"}
Sep 16 18:40:38.641843 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:38.641836    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-k6gvs" event={"ID":"6012728d-aee6-4f93-ab6e-fad616ef36e8","Type":"ContainerStarted","Data":"cea877dca453b26f5cb32d45b8bb3026a00c8e62f135e8f0637132601af11c2b"}
Sep 16 18:40:38.643692 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:38.643656    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-5rrj4" event={"ID":"fc3365ef-dfd3-45fe-b80f-937949d65751","Type":"ContainerStarted","Data":"0c3bf17f9d46ea18cb4d3480dbc699ead8319b037e6adb70699040d6b2eed83d"}
Sep 16 18:40:38.646072 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:38.646045    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-119.ec2.internal" event={"ID":"ce93b5afb1de6b9a80d64df7da81462d","Type":"ContainerStarted","Data":"b883a3c95827feb26947564bdc3949978a0e284a7f1dd0c82927fe4307c336dc"}
Sep 16 18:40:38.648108 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:38.647972    2578 generic.go:356] "Generic (PLEG): container finished" podID="db42fcd462e8e18de823d8d0adf7cc0a" containerID="138c27268ddfe583a7cfc6c77117d43cc11188c1b2cd1a98732e9291cc951fa1" exitCode=0
Sep 16 18:40:38.648108 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:38.648070    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-119.ec2.internal" event={"ID":"db42fcd462e8e18de823d8d0adf7cc0a","Type":"ContainerDied","Data":"138c27268ddfe583a7cfc6c77117d43cc11188c1b2cd1a98732e9291cc951fa1"}
Sep 16 18:40:38.650142 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:38.650119    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wrx94" event={"ID":"3f2c98a3-c6e7-4610-b207-f7ccde3cb571","Type":"ContainerStarted","Data":"87e98a8dd74061e2074c425514d49ea66293959f0def06918e0e8dbcff45e330"}
Sep 16 18:40:38.651861 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:38.651836    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-f69wc" event={"ID":"f4653546-2f57-4858-a038-9fd7c2139d56","Type":"ContainerStarted","Data":"9713d3d881fa0ab5d1a0ce29934ee99a6bcaf788112d7ac583807b8fce9f8d7f"}
Sep 16 18:40:38.653669 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:38.653636    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-llvsl" event={"ID":"62104ee5-c623-4ffd-b3af-9516ffd5f3bf","Type":"ContainerStarted","Data":"dad59288c2447ab396e9812c065bd74c8bfb05bcf4c83e4bfb9e79d6090fef33"}
Sep 16 18:40:38.655685 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:38.655620    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-bl65n" event={"ID":"454c7c51-a475-44f6-be51-1a8f1d29beaf","Type":"ContainerStarted","Data":"fc432a3301fa8db6e9a0ae43800a2406f455a718a3343d94a7bd1cfa48df2912"}
Sep 16 18:40:38.658377 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:38.658325    2578 generic.go:356] "Generic (PLEG): container finished" podID="91876c9b-d59f-4d80-bda9-f7addebaaee3" containerID="c0342b5445b260ddb9e48c419c257de92e9eb559b204b2422ec44fc52e562bef" exitCode=0
Sep 16 18:40:38.658377 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:38.658353    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-dxtkl" event={"ID":"91876c9b-d59f-4d80-bda9-f7addebaaee3","Type":"ContainerDied","Data":"c0342b5445b260ddb9e48c419c257de92e9eb559b204b2422ec44fc52e562bef"}
Sep 16 18:40:38.660831 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:38.660784    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-5rrj4" podStartSLOduration=1.9146148809999999 podStartE2EDuration="18.660768461s" totalImagesPullingTime="16.74615358s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 18:40:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:40:38.660710213 +0000 UTC m=+19.727201725" watchObservedRunningTime="2026-09-16 18:40:38.660768461 +0000 UTC m=+19.727259952"
Sep 16 18:40:38.698793 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:38.698243    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-bl65n" podStartSLOduration=1.9797516750000002 podStartE2EDuration="18.698221662s" totalImagesPullingTime="16.718469987s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 18:40:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:40:38.676014051 +0000 UTC m=+19.742505559" watchObservedRunningTime="2026-09-16 18:40:38.698221662 +0000 UTC m=+19.764713152"
Sep 16 18:40:38.721933 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:38.721238    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-llvsl" podStartSLOduration=1.652323245 podStartE2EDuration="18.721219975s" totalImagesPullingTime="17.06889673s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 18:40:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:40:38.697872311 +0000 UTC m=+19.764363825" watchObservedRunningTime="2026-09-16 18:40:38.721219975 +0000 UTC m=+19.787711464"
Sep 16 18:40:38.721933 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:38.721638    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-f69wc" podStartSLOduration=1.999689772 podStartE2EDuration="18.721629835s" totalImagesPullingTime="16.721940063s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:40:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:40:38.720948188 +0000 UTC m=+19.787439712" watchObservedRunningTime="2026-09-16 18:40:38.721629835 +0000 UTC m=+19.788121326"
Sep 16 18:40:39.151458 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:39.151251    2578 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Sep 16 18:40:39.367240 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:39.367068    2578 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-16T18:40:39.151456518Z","UUID":"d2931ba0-132c-4f21-a409-36102c3b46d9","Handler":null,"Name":"","Endpoint":""}
Sep 16 18:40:39.368865 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:39.368843    2578 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Sep 16 18:40:39.369021 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:39.368877    2578 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Sep 16 18:40:39.429827 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:39.429789    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8ndsh"
Sep 16 18:40:39.429964 ip-10-0-135-119 kubenswrapper[2578]: E0916 18:40:39.429894    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8ndsh" podUID="3377e453-9418-49a8-870b-64424f13d694"
Sep 16 18:40:39.430050 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:39.430002    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-f287c"
Sep 16 18:40:39.430131 ip-10-0-135-119 kubenswrapper[2578]: E0916 18:40:39.430104    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-f287c" podUID="63abc484-192b-4095-81ea-b5db79acd0ba"
Sep 16 18:40:39.430288 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:39.430270    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zfv2x"
Sep 16 18:40:39.430386 ip-10-0-135-119 kubenswrapper[2578]: E0916 18:40:39.430367    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zfv2x" podUID="1f969e62-dd58-41b0-9419-b2ca445f3ec4"
Sep 16 18:40:39.663948 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:39.663909    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wrx94" event={"ID":"3f2c98a3-c6e7-4610-b207-f7ccde3cb571","Type":"ContainerStarted","Data":"9231fb9df05efd342090c715059336ad6c3087060a8455c87b6a9c6d7f74ff09"}
Sep 16 18:40:39.666841 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:39.666813    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-ncnts" event={"ID":"3d323f6a-c161-4706-b9d5-8803578a6159","Type":"ContainerStarted","Data":"d4aa561f7b2fca97112476dc7e4478f04351e959021d40ebf200f75567c18a60"}
Sep 16 18:40:39.670297 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:39.670269    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-119.ec2.internal" event={"ID":"db42fcd462e8e18de823d8d0adf7cc0a","Type":"ContainerStarted","Data":"65862cc744cbf9d3886a35e831392832d0a52ad064ab6649073400a1e19401a9"}
Sep 16 18:40:39.684645 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:39.684564    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-ncnts" podStartSLOduration=3.93293787 podStartE2EDuration="20.684546133s" totalImagesPullingTime="16.751608263s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 18:40:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:40:39.684149118 +0000 UTC m=+20.750640635" watchObservedRunningTime="2026-09-16 18:40:39.684546133 +0000 UTC m=+20.751037622"
Sep 16 18:40:39.684759 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:39.684706    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-119.ec2.internal" podStartSLOduration=19.684693592 podStartE2EDuration="19.684693592s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 18:40:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:40:38.804431726 +0000 UTC m=+19.870923236" watchObservedRunningTime="2026-09-16 18:40:39.684693592 +0000 UTC m=+20.751185081"
Sep 16 18:40:41.428090 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:41.428052    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zfv2x"
Sep 16 18:40:41.428090 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:41.428085    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8ndsh"
Sep 16 18:40:41.428625 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:41.428111    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-f287c"
Sep 16 18:40:41.428625 ip-10-0-135-119 kubenswrapper[2578]: E0916 18:40:41.428263    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zfv2x" podUID="1f969e62-dd58-41b0-9419-b2ca445f3ec4"
Sep 16 18:40:41.428770 ip-10-0-135-119 kubenswrapper[2578]: E0916 18:40:41.428743    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8ndsh" podUID="3377e453-9418-49a8-870b-64424f13d694"
Sep 16 18:40:41.428856 ip-10-0-135-119 kubenswrapper[2578]: E0916 18:40:41.428836    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-f287c" podUID="63abc484-192b-4095-81ea-b5db79acd0ba"
Sep 16 18:40:43.347235 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:43.347193    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-f69wc"
Sep 16 18:40:43.347881 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:43.347866    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-f69wc"
Sep 16 18:40:43.379539 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:43.379497    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-119.ec2.internal" podStartSLOduration=23.379479887 podStartE2EDuration="23.379479887s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 18:40:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:40:39.70524549 +0000 UTC m=+20.771736997" watchObservedRunningTime="2026-09-16 18:40:43.379479887 +0000 UTC m=+24.445971357"
Sep 16 18:40:43.428237 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:43.428202    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8ndsh"
Sep 16 18:40:43.428424 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:43.428394    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-f287c"
Sep 16 18:40:43.428557 ip-10-0-135-119 kubenswrapper[2578]: E0916 18:40:43.428516    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-f287c" podUID="63abc484-192b-4095-81ea-b5db79acd0ba"
Sep 16 18:40:43.428557 ip-10-0-135-119 kubenswrapper[2578]: E0916 18:40:43.428394    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8ndsh" podUID="3377e453-9418-49a8-870b-64424f13d694"
Sep 16 18:40:43.428557 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:43.428401    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zfv2x"
Sep 16 18:40:43.428686 ip-10-0-135-119 kubenswrapper[2578]: E0916 18:40:43.428604    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zfv2x" podUID="1f969e62-dd58-41b0-9419-b2ca445f3ec4"
Sep 16 18:40:43.679526 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:43.679488    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-f69wc"
Sep 16 18:40:43.680130 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:43.680106    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-f69wc"
Sep 16 18:40:44.683290 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:44.683256    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-hqxdq" event={"ID":"92e18a5c-2e80-4f2d-8602-d64ca37db647","Type":"ContainerStarted","Data":"43858f1f56081a084ad70e3ec627494c899c1d4dacb09b8596b9dd177bd09d4f"}
Sep 16 18:40:44.685356 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:44.685335    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wrx94" event={"ID":"3f2c98a3-c6e7-4610-b207-f7ccde3cb571","Type":"ContainerStarted","Data":"0bc17d334efd676474b14bf2d3fb4ef9bcf50ef6752b2b4ca9b0901b50d20a1a"}
Sep 16 18:40:44.687226 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:44.687205    2578 generic.go:356] "Generic (PLEG): container finished" podID="91876c9b-d59f-4d80-bda9-f7addebaaee3" containerID="c667cbf6379e290827243485b8608a40fe7e7911d903a1e80981650520e07025" exitCode=0
Sep 16 18:40:44.687326 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:44.687260    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-dxtkl" event={"ID":"91876c9b-d59f-4d80-bda9-f7addebaaee3","Type":"ContainerDied","Data":"c667cbf6379e290827243485b8608a40fe7e7911d903a1e80981650520e07025"}
Sep 16 18:40:44.690816 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:44.690686    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-k6gvs_6012728d-aee6-4f93-ab6e-fad616ef36e8/ovn-acl-logging/0.log"
Sep 16 18:40:44.691160 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:44.691141    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-k6gvs" event={"ID":"6012728d-aee6-4f93-ab6e-fad616ef36e8","Type":"ContainerStarted","Data":"29aca9dd49ca17e113362f78959419e8744015cc01980c43235989125444daba"}
Sep 16 18:40:44.747817 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:44.747768    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-hqxdq" podStartSLOduration=1.333070035 podStartE2EDuration="24.747750251s" totalImagesPullingTime="23.414680216s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 18:40:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=1 observedRunningTime="2026-09-16 18:40:44.711688329 +0000 UTC m=+25.778179835" watchObservedRunningTime="2026-09-16 18:40:44.747750251 +0000 UTC m=+25.814241738"
Sep 16 18:40:44.783740 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:44.783690    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wrx94" podStartSLOduration=2.175065487 podStartE2EDuration="24.783680167s" totalImagesPullingTime="22.60861468s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 18:40:20 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:40:44.783389007 +0000 UTC m=+25.849880513" watchObservedRunningTime="2026-09-16 18:40:44.783680167 +0000 UTC m=+25.850171654"
Sep 16 18:40:45.428583 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:45.428551    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8ndsh"
Sep 16 18:40:45.428583 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:45.428575    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-f287c"
Sep 16 18:40:45.428800 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:45.428603    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zfv2x"
Sep 16 18:40:45.428800 ip-10-0-135-119 kubenswrapper[2578]: E0916 18:40:45.428686    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-f287c" podUID="63abc484-192b-4095-81ea-b5db79acd0ba"
Sep 16 18:40:45.428800 ip-10-0-135-119 kubenswrapper[2578]: E0916 18:40:45.428724    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8ndsh" podUID="3377e453-9418-49a8-870b-64424f13d694"
Sep 16 18:40:45.428800 ip-10-0-135-119 kubenswrapper[2578]: E0916 18:40:45.428789    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zfv2x" podUID="1f969e62-dd58-41b0-9419-b2ca445f3ec4"
Sep 16 18:40:46.697254 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:46.697223    2578 generic.go:356] "Generic (PLEG): container finished" podID="91876c9b-d59f-4d80-bda9-f7addebaaee3" containerID="5f7ac377f49a24d25f8d36042944efe8928b636e7c53197e4f8f13d0c052bd9a" exitCode=0
Sep 16 18:40:46.698076 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:46.697315    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-dxtkl" event={"ID":"91876c9b-d59f-4d80-bda9-f7addebaaee3","Type":"ContainerDied","Data":"5f7ac377f49a24d25f8d36042944efe8928b636e7c53197e4f8f13d0c052bd9a"}
Sep 16 18:40:46.700818 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:46.700801    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-k6gvs_6012728d-aee6-4f93-ab6e-fad616ef36e8/ovn-acl-logging/0.log"
Sep 16 18:40:46.701218 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:46.701196    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-k6gvs" event={"ID":"6012728d-aee6-4f93-ab6e-fad616ef36e8","Type":"ContainerStarted","Data":"c47fb3474a6e8a85cba7ec99d755241ac3d5e660a4979cbdd1f5f1a63c4c6b9a"}
Sep 16 18:40:46.701510 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:46.701457    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-k6gvs"
Sep 16 18:40:46.701694 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:46.701676    2578 scope.go:117] "RemoveContainer" containerID="1212b44ae45e2eaf2a1735068182b97a1880607179526989f29a1762f70e1684"
Sep 16 18:40:46.717596 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:46.717575    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-k6gvs"
Sep 16 18:40:47.429054 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:47.428856    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-f287c"
Sep 16 18:40:47.429165 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:47.428856    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8ndsh"
Sep 16 18:40:47.429165 ip-10-0-135-119 kubenswrapper[2578]: E0916 18:40:47.429130    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-f287c" podUID="63abc484-192b-4095-81ea-b5db79acd0ba"
Sep 16 18:40:47.429264 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:47.428907    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zfv2x"
Sep 16 18:40:47.429264 ip-10-0-135-119 kubenswrapper[2578]: E0916 18:40:47.429223    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8ndsh" podUID="3377e453-9418-49a8-870b-64424f13d694"
Sep 16 18:40:47.429358 ip-10-0-135-119 kubenswrapper[2578]: E0916 18:40:47.429277    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zfv2x" podUID="1f969e62-dd58-41b0-9419-b2ca445f3ec4"
Sep 16 18:40:47.706817 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:47.706786    2578 generic.go:356] "Generic (PLEG): container finished" podID="91876c9b-d59f-4d80-bda9-f7addebaaee3" containerID="310d755a7eb98c5a3ef13c5f9a51f368a5bad1ab6f7d17c31c594a689364432f" exitCode=0
Sep 16 18:40:47.707307 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:47.706868    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-dxtkl" event={"ID":"91876c9b-d59f-4d80-bda9-f7addebaaee3","Type":"ContainerDied","Data":"310d755a7eb98c5a3ef13c5f9a51f368a5bad1ab6f7d17c31c594a689364432f"}
Sep 16 18:40:47.711406 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:47.711387    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-k6gvs_6012728d-aee6-4f93-ab6e-fad616ef36e8/ovn-acl-logging/0.log"
Sep 16 18:40:47.711802 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:47.711782    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-k6gvs" event={"ID":"6012728d-aee6-4f93-ab6e-fad616ef36e8","Type":"ContainerStarted","Data":"928793fb7e1fcad26695845057ba09a95430fd2a6348a544ab4b2b7bb98874c4"}
Sep 16 18:40:47.711938 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:47.711867    2578 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
Sep 16 18:40:47.712189 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:47.712160    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-k6gvs"
Sep 16 18:40:47.732377 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:47.732352    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-k6gvs"
Sep 16 18:40:47.759546 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:47.759507    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-k6gvs" podStartSLOduration=11.912169722 podStartE2EDuration="28.759492906s" totalImagesPullingTime="16.847323184s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 18:40:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:40:47.759131925 +0000 UTC m=+28.825623438" watchObservedRunningTime="2026-09-16 18:40:47.759492906 +0000 UTC m=+28.825984394"
Sep 16 18:40:48.451027 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:48.450971    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-f287c"]
Sep 16 18:40:48.451230 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:48.451125    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-f287c"
Sep 16 18:40:48.451309 ip-10-0-135-119 kubenswrapper[2578]: E0916 18:40:48.451222    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-f287c" podUID="63abc484-192b-4095-81ea-b5db79acd0ba"
Sep 16 18:40:48.455276 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:48.455251    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-zfv2x"]
Sep 16 18:40:48.455381 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:48.455350    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zfv2x"
Sep 16 18:40:48.455440 ip-10-0-135-119 kubenswrapper[2578]: E0916 18:40:48.455423    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zfv2x" podUID="1f969e62-dd58-41b0-9419-b2ca445f3ec4"
Sep 16 18:40:48.468034 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:48.468011    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-8ndsh"]
Sep 16 18:40:48.468169 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:48.468113    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8ndsh"
Sep 16 18:40:48.468236 ip-10-0-135-119 kubenswrapper[2578]: E0916 18:40:48.468211    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8ndsh" podUID="3377e453-9418-49a8-870b-64424f13d694"
Sep 16 18:40:48.714036 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:48.713948    2578 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
Sep 16 18:40:48.727689 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:48.727659    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-k6gvs"
Sep 16 18:40:50.428577 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:50.428537    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zfv2x"
Sep 16 18:40:50.428577 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:50.428578    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-f287c"
Sep 16 18:40:50.429069 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:50.428578    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8ndsh"
Sep 16 18:40:50.429069 ip-10-0-135-119 kubenswrapper[2578]: E0916 18:40:50.428661    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zfv2x" podUID="1f969e62-dd58-41b0-9419-b2ca445f3ec4"
Sep 16 18:40:50.429069 ip-10-0-135-119 kubenswrapper[2578]: E0916 18:40:50.428785    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-f287c" podUID="63abc484-192b-4095-81ea-b5db79acd0ba"
Sep 16 18:40:50.429069 ip-10-0-135-119 kubenswrapper[2578]: E0916 18:40:50.428871    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8ndsh" podUID="3377e453-9418-49a8-870b-64424f13d694"
Sep 16 18:40:52.056443 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:52.056367    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1f969e62-dd58-41b0-9419-b2ca445f3ec4-metrics-certs\") pod \"network-metrics-daemon-zfv2x\" (UID: \"1f969e62-dd58-41b0-9419-b2ca445f3ec4\") " pod="openshift-multus/network-metrics-daemon-zfv2x"
Sep 16 18:40:52.056766 ip-10-0-135-119 kubenswrapper[2578]: E0916 18:40:52.056463    2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 18:40:52.056766 ip-10-0-135-119 kubenswrapper[2578]: E0916 18:40:52.056512    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1f969e62-dd58-41b0-9419-b2ca445f3ec4-metrics-certs podName:1f969e62-dd58-41b0-9419-b2ca445f3ec4 nodeName:}" failed. No retries permitted until 2026-09-16 18:41:24.056497396 +0000 UTC m=+65.122988862 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1f969e62-dd58-41b0-9419-b2ca445f3ec4-metrics-certs") pod "network-metrics-daemon-zfv2x" (UID: "1f969e62-dd58-41b0-9419-b2ca445f3ec4") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 18:40:52.157453 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:52.157423    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3377e453-9418-49a8-870b-64424f13d694-original-pull-secret\") pod \"global-pull-secret-syncer-8ndsh\" (UID: \"3377e453-9418-49a8-870b-64424f13d694\") " pod="kube-system/global-pull-secret-syncer-8ndsh"
Sep 16 18:40:52.157574 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:52.157463    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-77jgz\" (UniqueName: \"kubernetes.io/projected/63abc484-192b-4095-81ea-b5db79acd0ba-kube-api-access-77jgz\") pod \"network-check-target-f287c\" (UID: \"63abc484-192b-4095-81ea-b5db79acd0ba\") " pod="openshift-network-diagnostics/network-check-target-f287c"
Sep 16 18:40:52.157574 ip-10-0-135-119 kubenswrapper[2578]: E0916 18:40:52.157561    2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 16 18:40:52.157633 ip-10-0-135-119 kubenswrapper[2578]: E0916 18:40:52.157614    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3377e453-9418-49a8-870b-64424f13d694-original-pull-secret podName:3377e453-9418-49a8-870b-64424f13d694 nodeName:}" failed. No retries permitted until 2026-09-16 18:41:24.157601171 +0000 UTC m=+65.224092636 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/3377e453-9418-49a8-870b-64424f13d694-original-pull-secret") pod "global-pull-secret-syncer-8ndsh" (UID: "3377e453-9418-49a8-870b-64424f13d694") : object "kube-system"/"original-pull-secret" not registered
Sep 16 18:40:52.157694 ip-10-0-135-119 kubenswrapper[2578]: E0916 18:40:52.157566    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 16 18:40:52.157694 ip-10-0-135-119 kubenswrapper[2578]: E0916 18:40:52.157660    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 16 18:40:52.157694 ip-10-0-135-119 kubenswrapper[2578]: E0916 18:40:52.157672    2578 projected.go:196] Error preparing data for projected volume kube-api-access-77jgz for pod openshift-network-diagnostics/network-check-target-f287c: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 18:40:52.157803 ip-10-0-135-119 kubenswrapper[2578]: E0916 18:40:52.157714    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/63abc484-192b-4095-81ea-b5db79acd0ba-kube-api-access-77jgz podName:63abc484-192b-4095-81ea-b5db79acd0ba nodeName:}" failed. No retries permitted until 2026-09-16 18:41:24.157703779 +0000 UTC m=+65.224195245 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-77jgz" (UniqueName: "kubernetes.io/projected/63abc484-192b-4095-81ea-b5db79acd0ba-kube-api-access-77jgz") pod "network-check-target-f287c" (UID: "63abc484-192b-4095-81ea-b5db79acd0ba") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 18:40:52.427945 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:52.427917    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8ndsh"
Sep 16 18:40:52.427945 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:52.427944    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-f287c"
Sep 16 18:40:52.428193 ip-10-0-135-119 kubenswrapper[2578]: E0916 18:40:52.428037    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8ndsh" podUID="3377e453-9418-49a8-870b-64424f13d694"
Sep 16 18:40:52.428193 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:52.428122    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zfv2x"
Sep 16 18:40:52.428293 ip-10-0-135-119 kubenswrapper[2578]: E0916 18:40:52.428245    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zfv2x" podUID="1f969e62-dd58-41b0-9419-b2ca445f3ec4"
Sep 16 18:40:52.428350 ip-10-0-135-119 kubenswrapper[2578]: E0916 18:40:52.428332    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-f287c" podUID="63abc484-192b-4095-81ea-b5db79acd0ba"
Sep 16 18:40:52.724856 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:52.724782    2578 generic.go:356] "Generic (PLEG): container finished" podID="91876c9b-d59f-4d80-bda9-f7addebaaee3" containerID="72d3bd9152a2ca4bd88d2129b4054fb85ddc0a952518af23592898a985093240" exitCode=0
Sep 16 18:40:52.724856 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:52.724845    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-dxtkl" event={"ID":"91876c9b-d59f-4d80-bda9-f7addebaaee3","Type":"ContainerDied","Data":"72d3bd9152a2ca4bd88d2129b4054fb85ddc0a952518af23592898a985093240"}
Sep 16 18:40:53.730331 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:53.730289    2578 generic.go:356] "Generic (PLEG): container finished" podID="91876c9b-d59f-4d80-bda9-f7addebaaee3" containerID="3336ea74ced10af6f673392ca466543c336c1d89827a02754ad7b3c0323f64e8" exitCode=0
Sep 16 18:40:53.730881 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:53.730372    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-dxtkl" event={"ID":"91876c9b-d59f-4d80-bda9-f7addebaaee3","Type":"ContainerDied","Data":"3336ea74ced10af6f673392ca466543c336c1d89827a02754ad7b3c0323f64e8"}
Sep 16 18:40:54.428296 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:54.428266    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zfv2x"
Sep 16 18:40:54.428459 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:54.428266    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8ndsh"
Sep 16 18:40:54.428459 ip-10-0-135-119 kubenswrapper[2578]: E0916 18:40:54.428373    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zfv2x" podUID="1f969e62-dd58-41b0-9419-b2ca445f3ec4"
Sep 16 18:40:54.428459 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:54.428272    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-f287c"
Sep 16 18:40:54.428571 ip-10-0-135-119 kubenswrapper[2578]: E0916 18:40:54.428449    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8ndsh" podUID="3377e453-9418-49a8-870b-64424f13d694"
Sep 16 18:40:54.428571 ip-10-0-135-119 kubenswrapper[2578]: E0916 18:40:54.428533    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-f287c" podUID="63abc484-192b-4095-81ea-b5db79acd0ba"
Sep 16 18:40:54.735844 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:54.735769    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-dxtkl" event={"ID":"91876c9b-d59f-4d80-bda9-f7addebaaee3","Type":"ContainerStarted","Data":"57d832e438388ccdf8482f5975a3c5a27c292eaa97a4a1b66633b26ff5b5e63e"}
Sep 16 18:40:56.428623 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:56.428433    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-f287c"
Sep 16 18:40:56.429026 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:56.428442    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8ndsh"
Sep 16 18:40:56.429026 ip-10-0-135-119 kubenswrapper[2578]: E0916 18:40:56.428704    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-f287c" podUID="63abc484-192b-4095-81ea-b5db79acd0ba"
Sep 16 18:40:56.429026 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:56.428449    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zfv2x"
Sep 16 18:40:56.429026 ip-10-0-135-119 kubenswrapper[2578]: E0916 18:40:56.428765    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8ndsh" podUID="3377e453-9418-49a8-870b-64424f13d694"
Sep 16 18:40:56.429026 ip-10-0-135-119 kubenswrapper[2578]: E0916 18:40:56.428874    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zfv2x" podUID="1f969e62-dd58-41b0-9419-b2ca445f3ec4"
Sep 16 18:40:58.428695 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:58.428663    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8ndsh"
Sep 16 18:40:58.429116 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:58.428754    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-f287c"
Sep 16 18:40:58.429116 ip-10-0-135-119 kubenswrapper[2578]: E0916 18:40:58.428760    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8ndsh" podUID="3377e453-9418-49a8-870b-64424f13d694"
Sep 16 18:40:58.429116 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:58.428781    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zfv2x"
Sep 16 18:40:58.429116 ip-10-0-135-119 kubenswrapper[2578]: E0916 18:40:58.428857    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-f287c" podUID="63abc484-192b-4095-81ea-b5db79acd0ba"
Sep 16 18:40:58.429116 ip-10-0-135-119 kubenswrapper[2578]: E0916 18:40:58.428919    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zfv2x" podUID="1f969e62-dd58-41b0-9419-b2ca445f3ec4"
Sep 16 18:40:58.703326 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:58.703306    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-119.ec2.internal" event="NodeReady"
Sep 16 18:40:58.703490 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:58.703420    2578 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Sep 16 18:40:58.744145 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:58.744095    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-dxtkl" podStartSLOduration=11.549138598999999 podStartE2EDuration="39.744076893s" totalImagesPullingTime="28.194938294s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 18:40:19 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:40:54.765914999 +0000 UTC m=+35.832406487" watchObservedRunningTime="2026-09-16 18:40:58.744076893 +0000 UTC m=+39.810568382"
Sep 16 18:40:58.744653 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:58.744626    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-9785996cd-z4ckv"]
Sep 16 18:40:58.765632 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:58.765610    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-9785996cd-z4ckv"
Sep 16 18:40:58.768484 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:58.768458    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-9785996cd-z4ckv"]
Sep 16 18:40:58.769730 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:58.769698    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Sep 16 18:40:58.769730 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:58.769712    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-xjxjk\""
Sep 16 18:40:58.769896 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:58.769776    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Sep 16 18:40:58.770836 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:58.770812    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-vpkvv"]
Sep 16 18:40:58.775539 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:58.775521    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Sep 16 18:40:58.776993 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:58.776956    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Sep 16 18:40:58.783835 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:58.783816    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-c42vj"]
Sep 16 18:40:58.783935 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:58.783919    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-vpkvv"
Sep 16 18:40:58.786378 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:58.786360    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Sep 16 18:40:58.786614 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:58.786601    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-67vjb\""
Sep 16 18:40:58.786671 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:58.786613    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Sep 16 18:40:58.805704 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:58.805682    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-vpkvv"]
Sep 16 18:40:58.805791 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:58.805765    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-c42vj"
Sep 16 18:40:58.808551 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:58.808536    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Sep 16 18:40:58.808633 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:58.808588    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-hq9jm\""
Sep 16 18:40:58.808887 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:58.808870    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Sep 16 18:40:58.808967 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:58.808876    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Sep 16 18:40:58.817908 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:58.817891    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-c42vj"]
Sep 16 18:40:58.907693 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:58.907671    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0ef1438e-2929-4fc5-9437-6e5b97e79006-metrics-tls\") pod \"dns-default-vpkvv\" (UID: \"0ef1438e-2929-4fc5-9437-6e5b97e79006\") " pod="openshift-dns/dns-default-vpkvv"
Sep 16 18:40:58.907806 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:58.907702    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/92866a88-93b2-430b-867b-40e3358eb631-registry-certificates\") pod \"image-registry-9785996cd-z4ckv\" (UID: \"92866a88-93b2-430b-867b-40e3358eb631\") " pod="openshift-image-registry/image-registry-9785996cd-z4ckv"
Sep 16 18:40:58.907806 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:58.907719    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fkm8c\" (UniqueName: \"kubernetes.io/projected/92866a88-93b2-430b-867b-40e3358eb631-kube-api-access-fkm8c\") pod \"image-registry-9785996cd-z4ckv\" (UID: \"92866a88-93b2-430b-867b-40e3358eb631\") " pod="openshift-image-registry/image-registry-9785996cd-z4ckv"
Sep 16 18:40:58.907806 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:58.907770    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0ef1438e-2929-4fc5-9437-6e5b97e79006-tmp-dir\") pod \"dns-default-vpkvv\" (UID: \"0ef1438e-2929-4fc5-9437-6e5b97e79006\") " pod="openshift-dns/dns-default-vpkvv"
Sep 16 18:40:58.907806 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:58.907791    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ccaf4d24-a69d-4925-9f2a-8f77d2d3a16d-cert\") pod \"ingress-canary-c42vj\" (UID: \"ccaf4d24-a69d-4925-9f2a-8f77d2d3a16d\") " pod="openshift-ingress-canary/ingress-canary-c42vj"
Sep 16 18:40:58.907806 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:58.907806    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/0ef1438e-2929-4fc5-9437-6e5b97e79006-config-volume\") pod \"dns-default-vpkvv\" (UID: \"0ef1438e-2929-4fc5-9437-6e5b97e79006\") " pod="openshift-dns/dns-default-vpkvv"
Sep 16 18:40:58.908035 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:58.907820    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xmzqn\" (UniqueName: \"kubernetes.io/projected/0ef1438e-2929-4fc5-9437-6e5b97e79006-kube-api-access-xmzqn\") pod \"dns-default-vpkvv\" (UID: \"0ef1438e-2929-4fc5-9437-6e5b97e79006\") " pod="openshift-dns/dns-default-vpkvv"
Sep 16 18:40:58.908035 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:58.907843    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-94x2z\" (UniqueName: \"kubernetes.io/projected/ccaf4d24-a69d-4925-9f2a-8f77d2d3a16d-kube-api-access-94x2z\") pod \"ingress-canary-c42vj\" (UID: \"ccaf4d24-a69d-4925-9f2a-8f77d2d3a16d\") " pod="openshift-ingress-canary/ingress-canary-c42vj"
Sep 16 18:40:58.908035 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:58.907881    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/92866a88-93b2-430b-867b-40e3358eb631-image-registry-private-configuration\") pod \"image-registry-9785996cd-z4ckv\" (UID: \"92866a88-93b2-430b-867b-40e3358eb631\") " pod="openshift-image-registry/image-registry-9785996cd-z4ckv"
Sep 16 18:40:58.908035 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:58.907936    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/92866a88-93b2-430b-867b-40e3358eb631-registry-tls\") pod \"image-registry-9785996cd-z4ckv\" (UID: \"92866a88-93b2-430b-867b-40e3358eb631\") " pod="openshift-image-registry/image-registry-9785996cd-z4ckv"
Sep 16 18:40:58.908035 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:58.907951    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/92866a88-93b2-430b-867b-40e3358eb631-ca-trust-extracted\") pod \"image-registry-9785996cd-z4ckv\" (UID: \"92866a88-93b2-430b-867b-40e3358eb631\") " pod="openshift-image-registry/image-registry-9785996cd-z4ckv"
Sep 16 18:40:58.908035 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:58.907994    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/92866a88-93b2-430b-867b-40e3358eb631-installation-pull-secrets\") pod \"image-registry-9785996cd-z4ckv\" (UID: \"92866a88-93b2-430b-867b-40e3358eb631\") " pod="openshift-image-registry/image-registry-9785996cd-z4ckv"
Sep 16 18:40:58.908035 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:58.908009    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/92866a88-93b2-430b-867b-40e3358eb631-bound-sa-token\") pod \"image-registry-9785996cd-z4ckv\" (UID: \"92866a88-93b2-430b-867b-40e3358eb631\") " pod="openshift-image-registry/image-registry-9785996cd-z4ckv"
Sep 16 18:40:58.908311 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:58.908073    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/92866a88-93b2-430b-867b-40e3358eb631-trusted-ca\") pod \"image-registry-9785996cd-z4ckv\" (UID: \"92866a88-93b2-430b-867b-40e3358eb631\") " pod="openshift-image-registry/image-registry-9785996cd-z4ckv"
Sep 16 18:40:58.931515 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:58.931488    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 18:40:59.008692 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:59.008631    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/92866a88-93b2-430b-867b-40e3358eb631-registry-tls\") pod \"image-registry-9785996cd-z4ckv\" (UID: \"92866a88-93b2-430b-867b-40e3358eb631\") " pod="openshift-image-registry/image-registry-9785996cd-z4ckv"
Sep 16 18:40:59.008692 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:59.008662    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/92866a88-93b2-430b-867b-40e3358eb631-ca-trust-extracted\") pod \"image-registry-9785996cd-z4ckv\" (UID: \"92866a88-93b2-430b-867b-40e3358eb631\") " pod="openshift-image-registry/image-registry-9785996cd-z4ckv"
Sep 16 18:40:59.008878 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:59.008692    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/92866a88-93b2-430b-867b-40e3358eb631-installation-pull-secrets\") pod \"image-registry-9785996cd-z4ckv\" (UID: \"92866a88-93b2-430b-867b-40e3358eb631\") " pod="openshift-image-registry/image-registry-9785996cd-z4ckv"
Sep 16 18:40:59.008878 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:59.008709    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/92866a88-93b2-430b-867b-40e3358eb631-bound-sa-token\") pod \"image-registry-9785996cd-z4ckv\" (UID: \"92866a88-93b2-430b-867b-40e3358eb631\") " pod="openshift-image-registry/image-registry-9785996cd-z4ckv"
Sep 16 18:40:59.008878 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:59.008728    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/92866a88-93b2-430b-867b-40e3358eb631-trusted-ca\") pod \"image-registry-9785996cd-z4ckv\" (UID: \"92866a88-93b2-430b-867b-40e3358eb631\") " pod="openshift-image-registry/image-registry-9785996cd-z4ckv"
Sep 16 18:40:59.008878 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:59.008745    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0ef1438e-2929-4fc5-9437-6e5b97e79006-metrics-tls\") pod \"dns-default-vpkvv\" (UID: \"0ef1438e-2929-4fc5-9437-6e5b97e79006\") " pod="openshift-dns/dns-default-vpkvv"
Sep 16 18:40:59.008878 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:59.008769    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/92866a88-93b2-430b-867b-40e3358eb631-registry-certificates\") pod \"image-registry-9785996cd-z4ckv\" (UID: \"92866a88-93b2-430b-867b-40e3358eb631\") " pod="openshift-image-registry/image-registry-9785996cd-z4ckv"
Sep 16 18:40:59.008878 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:59.008793    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fkm8c\" (UniqueName: \"kubernetes.io/projected/92866a88-93b2-430b-867b-40e3358eb631-kube-api-access-fkm8c\") pod \"image-registry-9785996cd-z4ckv\" (UID: \"92866a88-93b2-430b-867b-40e3358eb631\") " pod="openshift-image-registry/image-registry-9785996cd-z4ckv"
Sep 16 18:40:59.008878 ip-10-0-135-119 kubenswrapper[2578]: E0916 18:40:59.008803    2578 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 16 18:40:59.008878 ip-10-0-135-119 kubenswrapper[2578]: E0916 18:40:59.008820    2578 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-9785996cd-z4ckv: secret "image-registry-tls" not found
Sep 16 18:40:59.008878 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:59.008849    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0ef1438e-2929-4fc5-9437-6e5b97e79006-tmp-dir\") pod \"dns-default-vpkvv\" (UID: \"0ef1438e-2929-4fc5-9437-6e5b97e79006\") " pod="openshift-dns/dns-default-vpkvv"
Sep 16 18:40:59.008878 ip-10-0-135-119 kubenswrapper[2578]: E0916 18:40:59.008851    2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 16 18:40:59.008878 ip-10-0-135-119 kubenswrapper[2578]: E0916 18:40:59.008875    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/92866a88-93b2-430b-867b-40e3358eb631-registry-tls podName:92866a88-93b2-430b-867b-40e3358eb631 nodeName:}" failed. No retries permitted until 2026-09-16 18:40:59.508858042 +0000 UTC m=+40.575349528 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/92866a88-93b2-430b-867b-40e3358eb631-registry-tls") pod "image-registry-9785996cd-z4ckv" (UID: "92866a88-93b2-430b-867b-40e3358eb631") : secret "image-registry-tls" not found
Sep 16 18:40:59.009435 ip-10-0-135-119 kubenswrapper[2578]: E0916 18:40:59.008900    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0ef1438e-2929-4fc5-9437-6e5b97e79006-metrics-tls podName:0ef1438e-2929-4fc5-9437-6e5b97e79006 nodeName:}" failed. No retries permitted until 2026-09-16 18:40:59.508885582 +0000 UTC m=+40.575377048 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/0ef1438e-2929-4fc5-9437-6e5b97e79006-metrics-tls") pod "dns-default-vpkvv" (UID: "0ef1438e-2929-4fc5-9437-6e5b97e79006") : secret "dns-default-metrics-tls" not found
Sep 16 18:40:59.009435 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:59.008916    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ccaf4d24-a69d-4925-9f2a-8f77d2d3a16d-cert\") pod \"ingress-canary-c42vj\" (UID: \"ccaf4d24-a69d-4925-9f2a-8f77d2d3a16d\") " pod="openshift-ingress-canary/ingress-canary-c42vj"
Sep 16 18:40:59.009435 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:59.008938    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/0ef1438e-2929-4fc5-9437-6e5b97e79006-config-volume\") pod \"dns-default-vpkvv\" (UID: \"0ef1438e-2929-4fc5-9437-6e5b97e79006\") " pod="openshift-dns/dns-default-vpkvv"
Sep 16 18:40:59.009435 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:59.008958    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xmzqn\" (UniqueName: \"kubernetes.io/projected/0ef1438e-2929-4fc5-9437-6e5b97e79006-kube-api-access-xmzqn\") pod \"dns-default-vpkvv\" (UID: \"0ef1438e-2929-4fc5-9437-6e5b97e79006\") " pod="openshift-dns/dns-default-vpkvv"
Sep 16 18:40:59.009435 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:59.009011    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-94x2z\" (UniqueName: \"kubernetes.io/projected/ccaf4d24-a69d-4925-9f2a-8f77d2d3a16d-kube-api-access-94x2z\") pod \"ingress-canary-c42vj\" (UID: \"ccaf4d24-a69d-4925-9f2a-8f77d2d3a16d\") " pod="openshift-ingress-canary/ingress-canary-c42vj"
Sep 16 18:40:59.009435 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:59.009041    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/92866a88-93b2-430b-867b-40e3358eb631-image-registry-private-configuration\") pod \"image-registry-9785996cd-z4ckv\" (UID: \"92866a88-93b2-430b-867b-40e3358eb631\") " pod="openshift-image-registry/image-registry-9785996cd-z4ckv"
Sep 16 18:40:59.009435 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:59.009112    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/92866a88-93b2-430b-867b-40e3358eb631-ca-trust-extracted\") pod \"image-registry-9785996cd-z4ckv\" (UID: \"92866a88-93b2-430b-867b-40e3358eb631\") " pod="openshift-image-registry/image-registry-9785996cd-z4ckv"
Sep 16 18:40:59.009772 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:59.009570    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/92866a88-93b2-430b-867b-40e3358eb631-registry-certificates\") pod \"image-registry-9785996cd-z4ckv\" (UID: \"92866a88-93b2-430b-867b-40e3358eb631\") " pod="openshift-image-registry/image-registry-9785996cd-z4ckv"
Sep 16 18:40:59.009772 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:59.009599    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0ef1438e-2929-4fc5-9437-6e5b97e79006-tmp-dir\") pod \"dns-default-vpkvv\" (UID: \"0ef1438e-2929-4fc5-9437-6e5b97e79006\") " pod="openshift-dns/dns-default-vpkvv"
Sep 16 18:40:59.009772 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:59.009621    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/0ef1438e-2929-4fc5-9437-6e5b97e79006-config-volume\") pod \"dns-default-vpkvv\" (UID: \"0ef1438e-2929-4fc5-9437-6e5b97e79006\") " pod="openshift-dns/dns-default-vpkvv"
Sep 16 18:40:59.009901 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:59.009820    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/92866a88-93b2-430b-867b-40e3358eb631-trusted-ca\") pod \"image-registry-9785996cd-z4ckv\" (UID: \"92866a88-93b2-430b-867b-40e3358eb631\") " pod="openshift-image-registry/image-registry-9785996cd-z4ckv"
Sep 16 18:40:59.012904 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:59.012883    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/92866a88-93b2-430b-867b-40e3358eb631-installation-pull-secrets\") pod \"image-registry-9785996cd-z4ckv\" (UID: \"92866a88-93b2-430b-867b-40e3358eb631\") " pod="openshift-image-registry/image-registry-9785996cd-z4ckv"
Sep 16 18:40:59.013000 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:59.012929    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/92866a88-93b2-430b-867b-40e3358eb631-image-registry-private-configuration\") pod \"image-registry-9785996cd-z4ckv\" (UID: \"92866a88-93b2-430b-867b-40e3358eb631\") " pod="openshift-image-registry/image-registry-9785996cd-z4ckv"
Sep 16 18:40:59.013000 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:59.012942    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ccaf4d24-a69d-4925-9f2a-8f77d2d3a16d-cert\") pod \"ingress-canary-c42vj\" (UID: \"ccaf4d24-a69d-4925-9f2a-8f77d2d3a16d\") " pod="openshift-ingress-canary/ingress-canary-c42vj"
Sep 16 18:40:59.020228 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:59.020208    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-94x2z\" (UniqueName: \"kubernetes.io/projected/ccaf4d24-a69d-4925-9f2a-8f77d2d3a16d-kube-api-access-94x2z\") pod \"ingress-canary-c42vj\" (UID: \"ccaf4d24-a69d-4925-9f2a-8f77d2d3a16d\") " pod="openshift-ingress-canary/ingress-canary-c42vj"
Sep 16 18:40:59.020392 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:59.020375    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/92866a88-93b2-430b-867b-40e3358eb631-bound-sa-token\") pod \"image-registry-9785996cd-z4ckv\" (UID: \"92866a88-93b2-430b-867b-40e3358eb631\") " pod="openshift-image-registry/image-registry-9785996cd-z4ckv"
Sep 16 18:40:59.021558 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:59.021539    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fkm8c\" (UniqueName: \"kubernetes.io/projected/92866a88-93b2-430b-867b-40e3358eb631-kube-api-access-fkm8c\") pod \"image-registry-9785996cd-z4ckv\" (UID: \"92866a88-93b2-430b-867b-40e3358eb631\") " pod="openshift-image-registry/image-registry-9785996cd-z4ckv"
Sep 16 18:40:59.021634 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:59.021541    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xmzqn\" (UniqueName: \"kubernetes.io/projected/0ef1438e-2929-4fc5-9437-6e5b97e79006-kube-api-access-xmzqn\") pod \"dns-default-vpkvv\" (UID: \"0ef1438e-2929-4fc5-9437-6e5b97e79006\") " pod="openshift-dns/dns-default-vpkvv"
Sep 16 18:40:59.033796 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:59.033778    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 18:40:59.113192 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:59.113170    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-c42vj"
Sep 16 18:40:59.439395 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:59.439366    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-c42vj"]
Sep 16 18:40:59.443940 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:40:59.443919    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podccaf4d24_a69d_4925_9f2a_8f77d2d3a16d.slice/crio-2a497a7328a3f7f0e5d041c95a854c9807c4974490ba7eb845ea0025206f6c83 WatchSource:0}: Error finding container 2a497a7328a3f7f0e5d041c95a854c9807c4974490ba7eb845ea0025206f6c83: Status 404 returned error can't find the container with id 2a497a7328a3f7f0e5d041c95a854c9807c4974490ba7eb845ea0025206f6c83
Sep 16 18:40:59.513851 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:59.513822    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/92866a88-93b2-430b-867b-40e3358eb631-registry-tls\") pod \"image-registry-9785996cd-z4ckv\" (UID: \"92866a88-93b2-430b-867b-40e3358eb631\") " pod="openshift-image-registry/image-registry-9785996cd-z4ckv"
Sep 16 18:40:59.513965 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:59.513870    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0ef1438e-2929-4fc5-9437-6e5b97e79006-metrics-tls\") pod \"dns-default-vpkvv\" (UID: \"0ef1438e-2929-4fc5-9437-6e5b97e79006\") " pod="openshift-dns/dns-default-vpkvv"
Sep 16 18:40:59.513965 ip-10-0-135-119 kubenswrapper[2578]: E0916 18:40:59.513961    2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 16 18:40:59.514061 ip-10-0-135-119 kubenswrapper[2578]: E0916 18:40:59.513972    2578 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 16 18:40:59.514061 ip-10-0-135-119 kubenswrapper[2578]: E0916 18:40:59.514010    2578 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-9785996cd-z4ckv: secret "image-registry-tls" not found
Sep 16 18:40:59.514061 ip-10-0-135-119 kubenswrapper[2578]: E0916 18:40:59.514032    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0ef1438e-2929-4fc5-9437-6e5b97e79006-metrics-tls podName:0ef1438e-2929-4fc5-9437-6e5b97e79006 nodeName:}" failed. No retries permitted until 2026-09-16 18:41:00.514017387 +0000 UTC m=+41.580508853 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/0ef1438e-2929-4fc5-9437-6e5b97e79006-metrics-tls") pod "dns-default-vpkvv" (UID: "0ef1438e-2929-4fc5-9437-6e5b97e79006") : secret "dns-default-metrics-tls" not found
Sep 16 18:40:59.514061 ip-10-0-135-119 kubenswrapper[2578]: E0916 18:40:59.514052    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/92866a88-93b2-430b-867b-40e3358eb631-registry-tls podName:92866a88-93b2-430b-867b-40e3358eb631 nodeName:}" failed. No retries permitted until 2026-09-16 18:41:00.514040731 +0000 UTC m=+41.580532197 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/92866a88-93b2-430b-867b-40e3358eb631-registry-tls") pod "image-registry-9785996cd-z4ckv" (UID: "92866a88-93b2-430b-867b-40e3358eb631") : secret "image-registry-tls" not found
Sep 16 18:40:59.749330 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:40:59.749262    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-c42vj" event={"ID":"ccaf4d24-a69d-4925-9f2a-8f77d2d3a16d","Type":"ContainerStarted","Data":"2a497a7328a3f7f0e5d041c95a854c9807c4974490ba7eb845ea0025206f6c83"}
Sep 16 18:41:00.429076 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:41:00.428876    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8ndsh"
Sep 16 18:41:00.429245 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:41:00.428901    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zfv2x"
Sep 16 18:41:00.429578 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:41:00.428943    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-f287c"
Sep 16 18:41:00.431571 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:41:00.431554    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 16 18:41:00.432319 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:41:00.432130    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 16 18:41:00.432319 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:41:00.432188    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 16 18:41:00.432319 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:41:00.432269    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 16 18:41:00.432319 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:41:00.432281    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-s9z5g\""
Sep 16 18:41:00.432319 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:41:00.432302    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-g58c8\""
Sep 16 18:41:00.522445 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:41:00.522418    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/92866a88-93b2-430b-867b-40e3358eb631-registry-tls\") pod \"image-registry-9785996cd-z4ckv\" (UID: \"92866a88-93b2-430b-867b-40e3358eb631\") " pod="openshift-image-registry/image-registry-9785996cd-z4ckv"
Sep 16 18:41:00.522865 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:41:00.522464    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0ef1438e-2929-4fc5-9437-6e5b97e79006-metrics-tls\") pod \"dns-default-vpkvv\" (UID: \"0ef1438e-2929-4fc5-9437-6e5b97e79006\") " pod="openshift-dns/dns-default-vpkvv"
Sep 16 18:41:00.522865 ip-10-0-135-119 kubenswrapper[2578]: E0916 18:41:00.522826    2578 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 16 18:41:00.522865 ip-10-0-135-119 kubenswrapper[2578]: E0916 18:41:00.522855    2578 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-9785996cd-z4ckv: secret "image-registry-tls" not found
Sep 16 18:41:00.523037 ip-10-0-135-119 kubenswrapper[2578]: E0916 18:41:00.523008    2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 16 18:41:00.523091 ip-10-0-135-119 kubenswrapper[2578]: E0916 18:41:00.523085    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0ef1438e-2929-4fc5-9437-6e5b97e79006-metrics-tls podName:0ef1438e-2929-4fc5-9437-6e5b97e79006 nodeName:}" failed. No retries permitted until 2026-09-16 18:41:02.5230656 +0000 UTC m=+43.589557081 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/0ef1438e-2929-4fc5-9437-6e5b97e79006-metrics-tls") pod "dns-default-vpkvv" (UID: "0ef1438e-2929-4fc5-9437-6e5b97e79006") : secret "dns-default-metrics-tls" not found
Sep 16 18:41:00.523147 ip-10-0-135-119 kubenswrapper[2578]: E0916 18:41:00.523104    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/92866a88-93b2-430b-867b-40e3358eb631-registry-tls podName:92866a88-93b2-430b-867b-40e3358eb631 nodeName:}" failed. No retries permitted until 2026-09-16 18:41:02.523093978 +0000 UTC m=+43.589585444 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/92866a88-93b2-430b-867b-40e3358eb631-registry-tls") pod "image-registry-9785996cd-z4ckv" (UID: "92866a88-93b2-430b-867b-40e3358eb631") : secret "image-registry-tls" not found
Sep 16 18:41:02.539929 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:41:02.539894    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/92866a88-93b2-430b-867b-40e3358eb631-registry-tls\") pod \"image-registry-9785996cd-z4ckv\" (UID: \"92866a88-93b2-430b-867b-40e3358eb631\") " pod="openshift-image-registry/image-registry-9785996cd-z4ckv"
Sep 16 18:41:02.540358 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:41:02.539951    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0ef1438e-2929-4fc5-9437-6e5b97e79006-metrics-tls\") pod \"dns-default-vpkvv\" (UID: \"0ef1438e-2929-4fc5-9437-6e5b97e79006\") " pod="openshift-dns/dns-default-vpkvv"
Sep 16 18:41:02.540358 ip-10-0-135-119 kubenswrapper[2578]: E0916 18:41:02.540063    2578 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 16 18:41:02.540358 ip-10-0-135-119 kubenswrapper[2578]: E0916 18:41:02.540078    2578 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-9785996cd-z4ckv: secret "image-registry-tls" not found
Sep 16 18:41:02.540358 ip-10-0-135-119 kubenswrapper[2578]: E0916 18:41:02.540100    2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 16 18:41:02.540358 ip-10-0-135-119 kubenswrapper[2578]: E0916 18:41:02.540129    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/92866a88-93b2-430b-867b-40e3358eb631-registry-tls podName:92866a88-93b2-430b-867b-40e3358eb631 nodeName:}" failed. No retries permitted until 2026-09-16 18:41:06.540115325 +0000 UTC m=+47.606606794 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/92866a88-93b2-430b-867b-40e3358eb631-registry-tls") pod "image-registry-9785996cd-z4ckv" (UID: "92866a88-93b2-430b-867b-40e3358eb631") : secret "image-registry-tls" not found
Sep 16 18:41:02.540358 ip-10-0-135-119 kubenswrapper[2578]: E0916 18:41:02.540152    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0ef1438e-2929-4fc5-9437-6e5b97e79006-metrics-tls podName:0ef1438e-2929-4fc5-9437-6e5b97e79006 nodeName:}" failed. No retries permitted until 2026-09-16 18:41:06.540137691 +0000 UTC m=+47.606629157 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/0ef1438e-2929-4fc5-9437-6e5b97e79006-metrics-tls") pod "dns-default-vpkvv" (UID: "0ef1438e-2929-4fc5-9437-6e5b97e79006") : secret "dns-default-metrics-tls" not found
Sep 16 18:41:02.757992 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:41:02.757890    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-c42vj" event={"ID":"ccaf4d24-a69d-4925-9f2a-8f77d2d3a16d","Type":"ContainerStarted","Data":"bbf099cc8f2a2eaf56bca51165e17b6c74bd44e13979e6741e03e2e7bee41613"}
Sep 16 18:41:02.780295 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:41:02.780245    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-c42vj" podStartSLOduration=2.418108498 podStartE2EDuration="4.780228394s" totalImagesPullingTime="2.362119896s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:40:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:41:02.779190553 +0000 UTC m=+43.845682061" watchObservedRunningTime="2026-09-16 18:41:02.780228394 +0000 UTC m=+43.846719893"
Sep 16 18:41:02.805252 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:41:02.805189    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 18:41:02.905655 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:41:02.905620    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 18:41:04.400173 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:41:04.400142    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 18:41:04.482993 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:41:04.482957    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 18:41:04.590664 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:41:04.590637    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 18:41:04.669395 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:41:04.669336    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 18:41:05.323571 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:41:05.323538    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 18:41:05.452903 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:41:05.452876    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 18:41:05.529368 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:41:05.529330    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 18:41:06.568257 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:41:06.568217    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/92866a88-93b2-430b-867b-40e3358eb631-registry-tls\") pod \"image-registry-9785996cd-z4ckv\" (UID: \"92866a88-93b2-430b-867b-40e3358eb631\") " pod="openshift-image-registry/image-registry-9785996cd-z4ckv"
Sep 16 18:41:06.568645 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:41:06.568268    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0ef1438e-2929-4fc5-9437-6e5b97e79006-metrics-tls\") pod \"dns-default-vpkvv\" (UID: \"0ef1438e-2929-4fc5-9437-6e5b97e79006\") " pod="openshift-dns/dns-default-vpkvv"
Sep 16 18:41:06.568645 ip-10-0-135-119 kubenswrapper[2578]: E0916 18:41:06.568359    2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 16 18:41:06.568645 ip-10-0-135-119 kubenswrapper[2578]: E0916 18:41:06.568365    2578 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 16 18:41:06.568645 ip-10-0-135-119 kubenswrapper[2578]: E0916 18:41:06.568385    2578 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-9785996cd-z4ckv: secret "image-registry-tls" not found
Sep 16 18:41:06.568645 ip-10-0-135-119 kubenswrapper[2578]: E0916 18:41:06.568414    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0ef1438e-2929-4fc5-9437-6e5b97e79006-metrics-tls podName:0ef1438e-2929-4fc5-9437-6e5b97e79006 nodeName:}" failed. No retries permitted until 2026-09-16 18:41:14.568401591 +0000 UTC m=+55.634893073 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/0ef1438e-2929-4fc5-9437-6e5b97e79006-metrics-tls") pod "dns-default-vpkvv" (UID: "0ef1438e-2929-4fc5-9437-6e5b97e79006") : secret "dns-default-metrics-tls" not found
Sep 16 18:41:06.568645 ip-10-0-135-119 kubenswrapper[2578]: E0916 18:41:06.568432    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/92866a88-93b2-430b-867b-40e3358eb631-registry-tls podName:92866a88-93b2-430b-867b-40e3358eb631 nodeName:}" failed. No retries permitted until 2026-09-16 18:41:14.568420135 +0000 UTC m=+55.634911601 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/92866a88-93b2-430b-867b-40e3358eb631-registry-tls") pod "image-registry-9785996cd-z4ckv" (UID: "92866a88-93b2-430b-867b-40e3358eb631") : secret "image-registry-tls" not found
Sep 16 18:41:08.804097 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:41:08.804065    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 18:41:09.631277 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:41:09.631244    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 18:41:09.731138 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:41:09.731098    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 18:41:09.805394 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:41:09.805367    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 18:41:11.358063 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:41:11.358026    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 18:41:11.450738 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:41:11.450708    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 18:41:11.880623 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:41:11.880596    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 18:41:11.947866 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:41:11.947834    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 18:41:14.622791 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:41:14.622754    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/92866a88-93b2-430b-867b-40e3358eb631-registry-tls\") pod \"image-registry-9785996cd-z4ckv\" (UID: \"92866a88-93b2-430b-867b-40e3358eb631\") " pod="openshift-image-registry/image-registry-9785996cd-z4ckv"
Sep 16 18:41:14.623286 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:41:14.622815    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0ef1438e-2929-4fc5-9437-6e5b97e79006-metrics-tls\") pod \"dns-default-vpkvv\" (UID: \"0ef1438e-2929-4fc5-9437-6e5b97e79006\") " pod="openshift-dns/dns-default-vpkvv"
Sep 16 18:41:14.623286 ip-10-0-135-119 kubenswrapper[2578]: E0916 18:41:14.622923    2578 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 16 18:41:14.623286 ip-10-0-135-119 kubenswrapper[2578]: E0916 18:41:14.622943    2578 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-9785996cd-z4ckv: secret "image-registry-tls" not found
Sep 16 18:41:14.623286 ip-10-0-135-119 kubenswrapper[2578]: E0916 18:41:14.622949    2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 16 18:41:14.623286 ip-10-0-135-119 kubenswrapper[2578]: E0916 18:41:14.623032    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/92866a88-93b2-430b-867b-40e3358eb631-registry-tls podName:92866a88-93b2-430b-867b-40e3358eb631 nodeName:}" failed. No retries permitted until 2026-09-16 18:41:30.62300984 +0000 UTC m=+71.689501305 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/92866a88-93b2-430b-867b-40e3358eb631-registry-tls") pod "image-registry-9785996cd-z4ckv" (UID: "92866a88-93b2-430b-867b-40e3358eb631") : secret "image-registry-tls" not found
Sep 16 18:41:14.623286 ip-10-0-135-119 kubenswrapper[2578]: E0916 18:41:14.623054    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0ef1438e-2929-4fc5-9437-6e5b97e79006-metrics-tls podName:0ef1438e-2929-4fc5-9437-6e5b97e79006 nodeName:}" failed. No retries permitted until 2026-09-16 18:41:30.623045362 +0000 UTC m=+71.689536829 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/0ef1438e-2929-4fc5-9437-6e5b97e79006-metrics-tls") pod "dns-default-vpkvv" (UID: "0ef1438e-2929-4fc5-9437-6e5b97e79006") : secret "dns-default-metrics-tls" not found
Sep 16 18:41:17.728141 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:41:17.728108    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 18:41:17.815749 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:41:17.815697    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 18:41:19.507244 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:41:19.507213    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 18:41:19.576773 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:41:19.576741    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 18:41:20.729524 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:41:20.729497    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-k6gvs"
Sep 16 18:41:20.894346 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:41:20.894309    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 18:41:21.000628 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:41:21.000556    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 18:41:21.418628 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:41:21.418578    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 18:41:21.487128 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:41:21.487095    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 18:41:24.087075 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:41:24.087019    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1f969e62-dd58-41b0-9419-b2ca445f3ec4-metrics-certs\") pod \"network-metrics-daemon-zfv2x\" (UID: \"1f969e62-dd58-41b0-9419-b2ca445f3ec4\") " pod="openshift-multus/network-metrics-daemon-zfv2x"
Sep 16 18:41:24.089544 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:41:24.089524    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 16 18:41:24.098156 ip-10-0-135-119 kubenswrapper[2578]: E0916 18:41:24.098135    2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Sep 16 18:41:24.098276 ip-10-0-135-119 kubenswrapper[2578]: E0916 18:41:24.098263    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1f969e62-dd58-41b0-9419-b2ca445f3ec4-metrics-certs podName:1f969e62-dd58-41b0-9419-b2ca445f3ec4 nodeName:}" failed. No retries permitted until 2026-09-16 18:42:28.098243225 +0000 UTC m=+129.164734695 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1f969e62-dd58-41b0-9419-b2ca445f3ec4-metrics-certs") pod "network-metrics-daemon-zfv2x" (UID: "1f969e62-dd58-41b0-9419-b2ca445f3ec4") : secret "metrics-daemon-secret" not found
Sep 16 18:41:24.188142 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:41:24.188111    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-77jgz\" (UniqueName: \"kubernetes.io/projected/63abc484-192b-4095-81ea-b5db79acd0ba-kube-api-access-77jgz\") pod \"network-check-target-f287c\" (UID: \"63abc484-192b-4095-81ea-b5db79acd0ba\") " pod="openshift-network-diagnostics/network-check-target-f287c"
Sep 16 18:41:24.188229 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:41:24.188184    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3377e453-9418-49a8-870b-64424f13d694-original-pull-secret\") pod \"global-pull-secret-syncer-8ndsh\" (UID: \"3377e453-9418-49a8-870b-64424f13d694\") " pod="kube-system/global-pull-secret-syncer-8ndsh"
Sep 16 18:41:24.190202 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:41:24.190181    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 16 18:41:24.190304 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:41:24.190204    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 16 18:41:24.200592 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:41:24.200575    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 16 18:41:24.202189 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:41:24.202167    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3377e453-9418-49a8-870b-64424f13d694-original-pull-secret\") pod \"global-pull-secret-syncer-8ndsh\" (UID: \"3377e453-9418-49a8-870b-64424f13d694\") " pod="kube-system/global-pull-secret-syncer-8ndsh"
Sep 16 18:41:24.211310 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:41:24.211291    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-77jgz\" (UniqueName: \"kubernetes.io/projected/63abc484-192b-4095-81ea-b5db79acd0ba-kube-api-access-77jgz\") pod \"network-check-target-f287c\" (UID: \"63abc484-192b-4095-81ea-b5db79acd0ba\") " pod="openshift-network-diagnostics/network-check-target-f287c"
Sep 16 18:41:24.441555 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:41:24.441522    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8ndsh"
Sep 16 18:41:24.455799 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:41:24.455782    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-g58c8\""
Sep 16 18:41:24.464220 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:41:24.464197    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-f287c"
Sep 16 18:41:24.722705 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:41:24.722620    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-8ndsh"]
Sep 16 18:41:24.726190 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:41:24.726161    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3377e453_9418_49a8_870b_64424f13d694.slice/crio-1c21b7a5763e2fd7c7bdef442538d2091e2dd63e52b349cd99fb33bfda15a386 WatchSource:0}: Error finding container 1c21b7a5763e2fd7c7bdef442538d2091e2dd63e52b349cd99fb33bfda15a386: Status 404 returned error can't find the container with id 1c21b7a5763e2fd7c7bdef442538d2091e2dd63e52b349cd99fb33bfda15a386
Sep 16 18:41:24.740682 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:41:24.740655    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-f287c"]
Sep 16 18:41:24.743615 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:41:24.743595    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod63abc484_192b_4095_81ea_b5db79acd0ba.slice/crio-c337cf2672493f52fd633e6fb83a9bccbe421b3c627ed33fcfb8ffc99bc43c22 WatchSource:0}: Error finding container c337cf2672493f52fd633e6fb83a9bccbe421b3c627ed33fcfb8ffc99bc43c22: Status 404 returned error can't find the container with id c337cf2672493f52fd633e6fb83a9bccbe421b3c627ed33fcfb8ffc99bc43c22
Sep 16 18:41:24.813098 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:41:24.813063    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-f287c" event={"ID":"63abc484-192b-4095-81ea-b5db79acd0ba","Type":"ContainerStarted","Data":"c337cf2672493f52fd633e6fb83a9bccbe421b3c627ed33fcfb8ffc99bc43c22"}
Sep 16 18:41:24.814222 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:41:24.814185    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-8ndsh" event={"ID":"3377e453-9418-49a8-870b-64424f13d694","Type":"ContainerStarted","Data":"1c21b7a5763e2fd7c7bdef442538d2091e2dd63e52b349cd99fb33bfda15a386"}
Sep 16 18:41:29.828911 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:41:29.828870    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-f287c" event={"ID":"63abc484-192b-4095-81ea-b5db79acd0ba","Type":"ContainerStarted","Data":"af9bdfeac0ccabaa0c71315a1a6203fa618efa4aabba95e616e639afd4f70358"}
Sep 16 18:41:29.829380 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:41:29.829007    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-f287c"
Sep 16 18:41:29.830352 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:41:29.830328    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-8ndsh" event={"ID":"3377e453-9418-49a8-870b-64424f13d694","Type":"ContainerStarted","Data":"621f0dae81de89eb4d6bb6c692d0da62ba76ab363444112fcd061e90509552b6"}
Sep 16 18:41:29.864035 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:41:29.863995    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-f287c" podStartSLOduration=66.60550396 podStartE2EDuration="1m10.86396611s" totalImagesPullingTime="4.25846215s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:40:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:41:29.863175201 +0000 UTC m=+70.929666701" watchObservedRunningTime="2026-09-16 18:41:29.86396611 +0000 UTC m=+70.930457598"
Sep 16 18:41:29.950505 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:41:29.950458    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-8ndsh" podStartSLOduration=65.670017235 podStartE2EDuration="1m9.950443244s" totalImagesPullingTime="4.280426009s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 18:40:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:41:29.886280962 +0000 UTC m=+70.952772450" watchObservedRunningTime="2026-09-16 18:41:29.950443244 +0000 UTC m=+71.016934732"
Sep 16 18:41:29.950966 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:41:29.950945    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-6f8b768455-99t6g"]
Sep 16 18:41:29.953810 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:41:29.953796    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f8b768455-99t6g"
Sep 16 18:41:29.955829 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:41:29.955797    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-5844b856db-mp7tt"]
Sep 16 18:41:29.955917 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:41:29.955825    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-b82tq\""
Sep 16 18:41:29.958492 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:41:29.958476    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-5844b856db-mp7tt"
Sep 16 18:41:29.960616 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:41:29.960597    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\""
Sep 16 18:41:29.960715 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:41:29.960608    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Sep 16 18:41:29.960715 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:41:29.960666    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Sep 16 18:41:29.960715 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:41:29.960694    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\""
Sep 16 18:41:29.960868 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:41:29.960739    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-fz78z\""
Sep 16 18:41:29.965695 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:41:29.965639    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\""
Sep 16 18:41:29.972330 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:41:29.972312    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f8b768455-99t6g"]
Sep 16 18:41:29.975021 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:41:29.975001    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-5844b856db-mp7tt"]
Sep 16 18:41:30.030731 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:41:30.030703    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/cd2b7515-5e1e-429d-9c17-32a37b492b26-snapshots\") pod \"insights-operator-5844b856db-mp7tt\" (UID: \"cd2b7515-5e1e-429d-9c17-32a37b492b26\") " pod="openshift-insights/insights-operator-5844b856db-mp7tt"
Sep 16 18:41:30.030731 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:41:30.030730    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wbvxb\" (UniqueName: \"kubernetes.io/projected/9ffc57b2-2355-46bf-ba08-987c192fb14d-kube-api-access-wbvxb\") pod \"network-check-source-6f8b768455-99t6g\" (UID: \"9ffc57b2-2355-46bf-ba08-987c192fb14d\") " pod="openshift-network-diagnostics/network-check-source-6f8b768455-99t6g"
Sep 16 18:41:30.030904 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:41:30.030748    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-27q44\" (UniqueName: \"kubernetes.io/projected/cd2b7515-5e1e-429d-9c17-32a37b492b26-kube-api-access-27q44\") pod \"insights-operator-5844b856db-mp7tt\" (UID: \"cd2b7515-5e1e-429d-9c17-32a37b492b26\") " pod="openshift-insights/insights-operator-5844b856db-mp7tt"
Sep 16 18:41:30.030904 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:41:30.030768    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cd2b7515-5e1e-429d-9c17-32a37b492b26-service-ca-bundle\") pod \"insights-operator-5844b856db-mp7tt\" (UID: \"cd2b7515-5e1e-429d-9c17-32a37b492b26\") " pod="openshift-insights/insights-operator-5844b856db-mp7tt"
Sep 16 18:41:30.030904 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:41:30.030854    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/cd2b7515-5e1e-429d-9c17-32a37b492b26-serving-cert\") pod \"insights-operator-5844b856db-mp7tt\" (UID: \"cd2b7515-5e1e-429d-9c17-32a37b492b26\") " pod="openshift-insights/insights-operator-5844b856db-mp7tt"
Sep 16 18:41:30.030904 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:41:30.030887    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/cd2b7515-5e1e-429d-9c17-32a37b492b26-tmp\") pod \"insights-operator-5844b856db-mp7tt\" (UID: \"cd2b7515-5e1e-429d-9c17-32a37b492b26\") " pod="openshift-insights/insights-operator-5844b856db-mp7tt"
Sep 16 18:41:30.030904 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:41:30.030902    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cd2b7515-5e1e-429d-9c17-32a37b492b26-trusted-ca-bundle\") pod \"insights-operator-5844b856db-mp7tt\" (UID: \"cd2b7515-5e1e-429d-9c17-32a37b492b26\") " pod="openshift-insights/insights-operator-5844b856db-mp7tt"
Sep 16 18:41:30.071560 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:41:30.071523    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-696884fb9c-tmrkx"]
Sep 16 18:41:30.074425 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:41:30.074407    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-696884fb9c-tmrkx"
Sep 16 18:41:30.076304 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:41:30.076287    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-p58pb\""
Sep 16 18:41:30.076759 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:41:30.076740    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\""
Sep 16 18:41:30.076759 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:41:30.076759    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\""
Sep 16 18:41:30.076947 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:41:30.076745    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\""
Sep 16 18:41:30.076947 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:41:30.076763    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\""
Sep 16 18:41:30.081646 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:41:30.081625    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\""
Sep 16 18:41:30.090125 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:41:30.090106    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-696884fb9c-tmrkx"]
Sep 16 18:41:30.132016 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:41:30.131990    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/cd2b7515-5e1e-429d-9c17-32a37b492b26-serving-cert\") pod \"insights-operator-5844b856db-mp7tt\" (UID: \"cd2b7515-5e1e-429d-9c17-32a37b492b26\") " pod="openshift-insights/insights-operator-5844b856db-mp7tt"
Sep 16 18:41:30.132123 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:41:30.132044    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/cd2b7515-5e1e-429d-9c17-32a37b492b26-tmp\") pod \"insights-operator-5844b856db-mp7tt\" (UID: \"cd2b7515-5e1e-429d-9c17-32a37b492b26\") " pod="openshift-insights/insights-operator-5844b856db-mp7tt"
Sep 16 18:41:30.132123 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:41:30.132062    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cd2b7515-5e1e-429d-9c17-32a37b492b26-trusted-ca-bundle\") pod \"insights-operator-5844b856db-mp7tt\" (UID: \"cd2b7515-5e1e-429d-9c17-32a37b492b26\") " pod="openshift-insights/insights-operator-5844b856db-mp7tt"
Sep 16 18:41:30.132123 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:41:30.132081    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/cd2b7515-5e1e-429d-9c17-32a37b492b26-snapshots\") pod \"insights-operator-5844b856db-mp7tt\" (UID: \"cd2b7515-5e1e-429d-9c17-32a37b492b26\") " pod="openshift-insights/insights-operator-5844b856db-mp7tt"
Sep 16 18:41:30.132123 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:41:30.132104    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wbvxb\" (UniqueName: \"kubernetes.io/projected/9ffc57b2-2355-46bf-ba08-987c192fb14d-kube-api-access-wbvxb\") pod \"network-check-source-6f8b768455-99t6g\" (UID: \"9ffc57b2-2355-46bf-ba08-987c192fb14d\") " pod="openshift-network-diagnostics/network-check-source-6f8b768455-99t6g"
Sep 16 18:41:30.132123 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:41:30.132123    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-27q44\" (UniqueName: \"kubernetes.io/projected/cd2b7515-5e1e-429d-9c17-32a37b492b26-kube-api-access-27q44\") pod \"insights-operator-5844b856db-mp7tt\" (UID: \"cd2b7515-5e1e-429d-9c17-32a37b492b26\") " pod="openshift-insights/insights-operator-5844b856db-mp7tt"
Sep 16 18:41:30.132391 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:41:30.132149    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/4c2078bb-281a-4aa1-8e38-238897865434-config\") pod \"console-operator-696884fb9c-tmrkx\" (UID: \"4c2078bb-281a-4aa1-8e38-238897865434\") " pod="openshift-console-operator/console-operator-696884fb9c-tmrkx"
Sep 16 18:41:30.132391 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:41:30.132175    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cd2b7515-5e1e-429d-9c17-32a37b492b26-service-ca-bundle\") pod \"insights-operator-5844b856db-mp7tt\" (UID: \"cd2b7515-5e1e-429d-9c17-32a37b492b26\") " pod="openshift-insights/insights-operator-5844b856db-mp7tt"
Sep 16 18:41:30.132391 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:41:30.132222    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/4c2078bb-281a-4aa1-8e38-238897865434-serving-cert\") pod \"console-operator-696884fb9c-tmrkx\" (UID: \"4c2078bb-281a-4aa1-8e38-238897865434\") " pod="openshift-console-operator/console-operator-696884fb9c-tmrkx"
Sep 16 18:41:30.132391 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:41:30.132245    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/4c2078bb-281a-4aa1-8e38-238897865434-trusted-ca\") pod \"console-operator-696884fb9c-tmrkx\" (UID: \"4c2078bb-281a-4aa1-8e38-238897865434\") " pod="openshift-console-operator/console-operator-696884fb9c-tmrkx"
Sep 16 18:41:30.132391 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:41:30.132289    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7wgpg\" (UniqueName: \"kubernetes.io/projected/4c2078bb-281a-4aa1-8e38-238897865434-kube-api-access-7wgpg\") pod \"console-operator-696884fb9c-tmrkx\" (UID: \"4c2078bb-281a-4aa1-8e38-238897865434\") " pod="openshift-console-operator/console-operator-696884fb9c-tmrkx"
Sep 16 18:41:30.132640 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:41:30.132607    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/cd2b7515-5e1e-429d-9c17-32a37b492b26-tmp\") pod \"insights-operator-5844b856db-mp7tt\" (UID: \"cd2b7515-5e1e-429d-9c17-32a37b492b26\") " pod="openshift-insights/insights-operator-5844b856db-mp7tt"
Sep 16 18:41:30.132776 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:41:30.132757    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/cd2b7515-5e1e-429d-9c17-32a37b492b26-snapshots\") pod \"insights-operator-5844b856db-mp7tt\" (UID: \"cd2b7515-5e1e-429d-9c17-32a37b492b26\") " pod="openshift-insights/insights-operator-5844b856db-mp7tt"
Sep 16 18:41:30.132836 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:41:30.132776    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cd2b7515-5e1e-429d-9c17-32a37b492b26-service-ca-bundle\") pod \"insights-operator-5844b856db-mp7tt\" (UID: \"cd2b7515-5e1e-429d-9c17-32a37b492b26\") " pod="openshift-insights/insights-operator-5844b856db-mp7tt"
Sep 16 18:41:30.133015 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:41:30.132970    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cd2b7515-5e1e-429d-9c17-32a37b492b26-trusted-ca-bundle\") pod \"insights-operator-5844b856db-mp7tt\" (UID: \"cd2b7515-5e1e-429d-9c17-32a37b492b26\") " pod="openshift-insights/insights-operator-5844b856db-mp7tt"
Sep 16 18:41:30.134466 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:41:30.134444    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/cd2b7515-5e1e-429d-9c17-32a37b492b26-serving-cert\") pod \"insights-operator-5844b856db-mp7tt\" (UID: \"cd2b7515-5e1e-429d-9c17-32a37b492b26\") " pod="openshift-insights/insights-operator-5844b856db-mp7tt"
Sep 16 18:41:30.141511 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:41:30.141491    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wbvxb\" (UniqueName: \"kubernetes.io/projected/9ffc57b2-2355-46bf-ba08-987c192fb14d-kube-api-access-wbvxb\") pod \"network-check-source-6f8b768455-99t6g\" (UID: \"9ffc57b2-2355-46bf-ba08-987c192fb14d\") " pod="openshift-network-diagnostics/network-check-source-6f8b768455-99t6g"
Sep 16 18:41:30.141771 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:41:30.141750    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-27q44\" (UniqueName: \"kubernetes.io/projected/cd2b7515-5e1e-429d-9c17-32a37b492b26-kube-api-access-27q44\") pod \"insights-operator-5844b856db-mp7tt\" (UID: \"cd2b7515-5e1e-429d-9c17-32a37b492b26\") " pod="openshift-insights/insights-operator-5844b856db-mp7tt"
Sep 16 18:41:30.232955 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:41:30.232915    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/4c2078bb-281a-4aa1-8e38-238897865434-serving-cert\") pod \"console-operator-696884fb9c-tmrkx\" (UID: \"4c2078bb-281a-4aa1-8e38-238897865434\") " pod="openshift-console-operator/console-operator-696884fb9c-tmrkx"
Sep 16 18:41:30.232955 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:41:30.232951    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/4c2078bb-281a-4aa1-8e38-238897865434-trusted-ca\") pod \"console-operator-696884fb9c-tmrkx\" (UID: \"4c2078bb-281a-4aa1-8e38-238897865434\") " pod="openshift-console-operator/console-operator-696884fb9c-tmrkx"
Sep 16 18:41:30.233182 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:41:30.233013    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7wgpg\" (UniqueName: \"kubernetes.io/projected/4c2078bb-281a-4aa1-8e38-238897865434-kube-api-access-7wgpg\") pod \"console-operator-696884fb9c-tmrkx\" (UID: \"4c2078bb-281a-4aa1-8e38-238897865434\") " pod="openshift-console-operator/console-operator-696884fb9c-tmrkx"
Sep 16 18:41:30.233182 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:41:30.233083    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/4c2078bb-281a-4aa1-8e38-238897865434-config\") pod \"console-operator-696884fb9c-tmrkx\" (UID: \"4c2078bb-281a-4aa1-8e38-238897865434\") " pod="openshift-console-operator/console-operator-696884fb9c-tmrkx"
Sep 16 18:41:30.233771 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:41:30.233703    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/4c2078bb-281a-4aa1-8e38-238897865434-config\") pod \"console-operator-696884fb9c-tmrkx\" (UID: \"4c2078bb-281a-4aa1-8e38-238897865434\") " pod="openshift-console-operator/console-operator-696884fb9c-tmrkx"
Sep 16 18:41:30.233925 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:41:30.233904    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/4c2078bb-281a-4aa1-8e38-238897865434-trusted-ca\") pod \"console-operator-696884fb9c-tmrkx\" (UID: \"4c2078bb-281a-4aa1-8e38-238897865434\") " pod="openshift-console-operator/console-operator-696884fb9c-tmrkx"
Sep 16 18:41:30.235411 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:41:30.235392    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/4c2078bb-281a-4aa1-8e38-238897865434-serving-cert\") pod \"console-operator-696884fb9c-tmrkx\" (UID: \"4c2078bb-281a-4aa1-8e38-238897865434\") " pod="openshift-console-operator/console-operator-696884fb9c-tmrkx"
Sep 16 18:41:30.242402 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:41:30.242379    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7wgpg\" (UniqueName: \"kubernetes.io/projected/4c2078bb-281a-4aa1-8e38-238897865434-kube-api-access-7wgpg\") pod \"console-operator-696884fb9c-tmrkx\" (UID: \"4c2078bb-281a-4aa1-8e38-238897865434\") " pod="openshift-console-operator/console-operator-696884fb9c-tmrkx"
Sep 16 18:41:30.263169 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:41:30.263145    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f8b768455-99t6g"
Sep 16 18:41:30.268135 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:41:30.268122    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-5844b856db-mp7tt"
Sep 16 18:41:30.330668 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:41:30.330572    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 18:41:30.382897 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:41:30.382864    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-696884fb9c-tmrkx"
Sep 16 18:41:30.542951 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:41:30.542921    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f8b768455-99t6g"]
Sep 16 18:41:30.545544 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:41:30.545523    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-5844b856db-mp7tt"]
Sep 16 18:41:30.546333 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:41:30.546311    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9ffc57b2_2355_46bf_ba08_987c192fb14d.slice/crio-c795d27a9888fb82f7b76a9e266fea128597a259321b387ad237fc8dd04c1629 WatchSource:0}: Error finding container c795d27a9888fb82f7b76a9e266fea128597a259321b387ad237fc8dd04c1629: Status 404 returned error can't find the container with id c795d27a9888fb82f7b76a9e266fea128597a259321b387ad237fc8dd04c1629
Sep 16 18:41:30.548393 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:41:30.548375    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcd2b7515_5e1e_429d_9c17_32a37b492b26.slice/crio-9bb53a250d5cc22621e5dd5c817344b33a90b1671573b2bc1bc57066178bfb3e WatchSource:0}: Error finding container 9bb53a250d5cc22621e5dd5c817344b33a90b1671573b2bc1bc57066178bfb3e: Status 404 returned error can't find the container with id 9bb53a250d5cc22621e5dd5c817344b33a90b1671573b2bc1bc57066178bfb3e
Sep 16 18:41:30.638824 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:41:30.638798    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0ef1438e-2929-4fc5-9437-6e5b97e79006-metrics-tls\") pod \"dns-default-vpkvv\" (UID: \"0ef1438e-2929-4fc5-9437-6e5b97e79006\") " pod="openshift-dns/dns-default-vpkvv"
Sep 16 18:41:30.638971 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:41:30.638877    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/92866a88-93b2-430b-867b-40e3358eb631-registry-tls\") pod \"image-registry-9785996cd-z4ckv\" (UID: \"92866a88-93b2-430b-867b-40e3358eb631\") " pod="openshift-image-registry/image-registry-9785996cd-z4ckv"
Sep 16 18:41:30.638971 ip-10-0-135-119 kubenswrapper[2578]: E0916 18:41:30.638953    2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 16 18:41:30.638971 ip-10-0-135-119 kubenswrapper[2578]: E0916 18:41:30.638970    2578 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 16 18:41:30.639130 ip-10-0-135-119 kubenswrapper[2578]: E0916 18:41:30.638997    2578 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-9785996cd-z4ckv: secret "image-registry-tls" not found
Sep 16 18:41:30.639130 ip-10-0-135-119 kubenswrapper[2578]: E0916 18:41:30.639044    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0ef1438e-2929-4fc5-9437-6e5b97e79006-metrics-tls podName:0ef1438e-2929-4fc5-9437-6e5b97e79006 nodeName:}" failed. No retries permitted until 2026-09-16 18:42:02.639021729 +0000 UTC m=+103.705513205 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/0ef1438e-2929-4fc5-9437-6e5b97e79006-metrics-tls") pod "dns-default-vpkvv" (UID: "0ef1438e-2929-4fc5-9437-6e5b97e79006") : secret "dns-default-metrics-tls" not found
Sep 16 18:41:30.639130 ip-10-0-135-119 kubenswrapper[2578]: E0916 18:41:30.639065    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/92866a88-93b2-430b-867b-40e3358eb631-registry-tls podName:92866a88-93b2-430b-867b-40e3358eb631 nodeName:}" failed. No retries permitted until 2026-09-16 18:42:02.639053287 +0000 UTC m=+103.705544753 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/92866a88-93b2-430b-867b-40e3358eb631-registry-tls") pod "image-registry-9785996cd-z4ckv" (UID: "92866a88-93b2-430b-867b-40e3358eb631") : secret "image-registry-tls" not found
Sep 16 18:41:30.664676 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:41:30.664648    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-696884fb9c-tmrkx"]
Sep 16 18:41:30.667872 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:41:30.667849    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4c2078bb_281a_4aa1_8e38_238897865434.slice/crio-54f024b6a5cc6edf9e31750ade5e5aa7c312355e09110b3783e33f796a580b8b WatchSource:0}: Error finding container 54f024b6a5cc6edf9e31750ade5e5aa7c312355e09110b3783e33f796a580b8b: Status 404 returned error can't find the container with id 54f024b6a5cc6edf9e31750ade5e5aa7c312355e09110b3783e33f796a580b8b
Sep 16 18:41:30.834034 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:41:30.833926    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-696884fb9c-tmrkx" event={"ID":"4c2078bb-281a-4aa1-8e38-238897865434","Type":"ContainerStarted","Data":"54f024b6a5cc6edf9e31750ade5e5aa7c312355e09110b3783e33f796a580b8b"}
Sep 16 18:41:30.835184 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:41:30.835151    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5844b856db-mp7tt" event={"ID":"cd2b7515-5e1e-429d-9c17-32a37b492b26","Type":"ContainerStarted","Data":"9bb53a250d5cc22621e5dd5c817344b33a90b1671573b2bc1bc57066178bfb3e"}
Sep 16 18:41:30.836735 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:41:30.836711    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f8b768455-99t6g" event={"ID":"9ffc57b2-2355-46bf-ba08-987c192fb14d","Type":"ContainerStarted","Data":"9c20860387f81832e3a0d8ee87cef3fa800ce556c328e74c07fcaaa712670dc6"}
Sep 16 18:41:30.836825 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:41:30.836749    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f8b768455-99t6g" event={"ID":"9ffc57b2-2355-46bf-ba08-987c192fb14d","Type":"ContainerStarted","Data":"c795d27a9888fb82f7b76a9e266fea128597a259321b387ad237fc8dd04c1629"}
Sep 16 18:41:30.865811 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:41:30.865771    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-6f8b768455-99t6g" podStartSLOduration=1.86575656 podStartE2EDuration="1.86575656s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:41:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:41:30.864779758 +0000 UTC m=+71.931271260" watchObservedRunningTime="2026-09-16 18:41:30.86575656 +0000 UTC m=+71.932248047"
Sep 16 18:41:31.662175 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:41:31.662138    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 18:41:32.532365 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:41:32.532334    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 18:41:32.617590 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:41:32.617557    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 18:41:33.588667 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:41:33.588631    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 18:41:33.654008 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:41:33.653962    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 18:41:33.848182 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:41:33.848119    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-tmrkx_4c2078bb-281a-4aa1-8e38-238897865434/console-operator/0.log"
Sep 16 18:41:33.848182 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:41:33.848161    2578 generic.go:356] "Generic (PLEG): container finished" podID="4c2078bb-281a-4aa1-8e38-238897865434" containerID="3581556e46eaa2f1fbdaafaf77bd706d60acbdc33f238f838cbee89a66135056" exitCode=255
Sep 16 18:41:33.848342 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:41:33.848248    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-696884fb9c-tmrkx" event={"ID":"4c2078bb-281a-4aa1-8e38-238897865434","Type":"ContainerDied","Data":"3581556e46eaa2f1fbdaafaf77bd706d60acbdc33f238f838cbee89a66135056"}
Sep 16 18:41:33.848524 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:41:33.848507    2578 scope.go:117] "RemoveContainer" containerID="3581556e46eaa2f1fbdaafaf77bd706d60acbdc33f238f838cbee89a66135056"
Sep 16 18:41:33.849991 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:41:33.849949    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5844b856db-mp7tt" event={"ID":"cd2b7515-5e1e-429d-9c17-32a37b492b26","Type":"ContainerStarted","Data":"4df8798410a4053002f78eb3425d13130f1d94a448c544b48861b45cc71fa118"}
Sep 16 18:41:33.937864 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:41:33.934918    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-operator-5844b856db-mp7tt" podStartSLOduration=2.564641329 podStartE2EDuration="4.934897122s" totalImagesPullingTime="2.370255793s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:41:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:41:33.934087397 +0000 UTC m=+75.000578887" watchObservedRunningTime="2026-09-16 18:41:33.934897122 +0000 UTC m=+75.001388612"
Sep 16 18:41:33.986157 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:41:33.986127    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 18:41:34.005250 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:41:34.005227    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-9kn7r"]
Sep 16 18:41:34.008181 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:41:34.008165    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-9kn7r"
Sep 16 18:41:34.010349 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:41:34.010329    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Sep 16 18:41:34.010432 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:41:34.010334    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Sep 16 18:41:34.010568 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:41:34.010556    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-kxlhk\""
Sep 16 18:41:34.022939 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:41:34.022916    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-9kn7r"]
Sep 16 18:41:34.068632 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:41:34.068604    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/2e9f2573-3279-4102-9777-5ff8f209ed6a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-9kn7r\" (UID: \"2e9f2573-3279-4102-9777-5ff8f209ed6a\") " pod="openshift-insights/insights-runtime-extractor-9kn7r"
Sep 16 18:41:34.068726 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:41:34.068670    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-292df\" (UniqueName: \"kubernetes.io/projected/2e9f2573-3279-4102-9777-5ff8f209ed6a-kube-api-access-292df\") pod \"insights-runtime-extractor-9kn7r\" (UID: \"2e9f2573-3279-4102-9777-5ff8f209ed6a\") " pod="openshift-insights/insights-runtime-extractor-9kn7r"
Sep 16 18:41:34.068763 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:41:34.068729    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/2e9f2573-3279-4102-9777-5ff8f209ed6a-data-volume\") pod \"insights-runtime-extractor-9kn7r\" (UID: \"2e9f2573-3279-4102-9777-5ff8f209ed6a\") " pod="openshift-insights/insights-runtime-extractor-9kn7r"
Sep 16 18:41:34.068801 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:41:34.068777    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/2e9f2573-3279-4102-9777-5ff8f209ed6a-crio-socket\") pod \"insights-runtime-extractor-9kn7r\" (UID: \"2e9f2573-3279-4102-9777-5ff8f209ed6a\") " pod="openshift-insights/insights-runtime-extractor-9kn7r"
Sep 16 18:41:34.068801 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:41:34.068796    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/2e9f2573-3279-4102-9777-5ff8f209ed6a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-9kn7r\" (UID: \"2e9f2573-3279-4102-9777-5ff8f209ed6a\") " pod="openshift-insights/insights-runtime-extractor-9kn7r"
Sep 16 18:41:34.133156 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:41:34.133117    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 18:41:34.169377 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:41:34.169350    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/2e9f2573-3279-4102-9777-5ff8f209ed6a-data-volume\") pod \"insights-runtime-extractor-9kn7r\" (UID: \"2e9f2573-3279-4102-9777-5ff8f209ed6a\") " pod="openshift-insights/insights-runtime-extractor-9kn7r"
Sep 16 18:41:34.169486 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:41:34.169398    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/2e9f2573-3279-4102-9777-5ff8f209ed6a-crio-socket\") pod \"insights-runtime-extractor-9kn7r\" (UID: \"2e9f2573-3279-4102-9777-5ff8f209ed6a\") " pod="openshift-insights/insights-runtime-extractor-9kn7r"
Sep 16 18:41:34.169486 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:41:34.169426    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/2e9f2573-3279-4102-9777-5ff8f209ed6a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-9kn7r\" (UID: \"2e9f2573-3279-4102-9777-5ff8f209ed6a\") " pod="openshift-insights/insights-runtime-extractor-9kn7r"
Sep 16 18:41:34.169486 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:41:34.169451    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/2e9f2573-3279-4102-9777-5ff8f209ed6a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-9kn7r\" (UID: \"2e9f2573-3279-4102-9777-5ff8f209ed6a\") " pod="openshift-insights/insights-runtime-extractor-9kn7r"
Sep 16 18:41:34.169593 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:41:34.169494    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/2e9f2573-3279-4102-9777-5ff8f209ed6a-crio-socket\") pod \"insights-runtime-extractor-9kn7r\" (UID: \"2e9f2573-3279-4102-9777-5ff8f209ed6a\") " pod="openshift-insights/insights-runtime-extractor-9kn7r"
Sep 16 18:41:34.169593 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:41:34.169513    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-292df\" (UniqueName: \"kubernetes.io/projected/2e9f2573-3279-4102-9777-5ff8f209ed6a-kube-api-access-292df\") pod \"insights-runtime-extractor-9kn7r\" (UID: \"2e9f2573-3279-4102-9777-5ff8f209ed6a\") " pod="openshift-insights/insights-runtime-extractor-9kn7r"
Sep 16 18:41:34.169593 ip-10-0-135-119 kubenswrapper[2578]: E0916 18:41:34.169576    2578 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 16 18:41:34.169704 ip-10-0-135-119 kubenswrapper[2578]: E0916 18:41:34.169592    2578 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 16 18:41:34.169704 ip-10-0-135-119 kubenswrapper[2578]: E0916 18:41:34.169646    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2e9f2573-3279-4102-9777-5ff8f209ed6a-insights-runtime-extractor-tls podName:2e9f2573-3279-4102-9777-5ff8f209ed6a nodeName:}" failed. No retries permitted until 2026-09-16 18:41:34.669627599 +0000 UTC m=+75.736119066 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/2e9f2573-3279-4102-9777-5ff8f209ed6a-insights-runtime-extractor-tls") pod "insights-runtime-extractor-9kn7r" (UID: "2e9f2573-3279-4102-9777-5ff8f209ed6a") : secret "insights-runtime-extractor-tls" not found
Sep 16 18:41:34.169704 ip-10-0-135-119 kubenswrapper[2578]: E0916 18:41:34.169673    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/2e9f2573-3279-4102-9777-5ff8f209ed6a-kube-rbac-proxy-cm podName:2e9f2573-3279-4102-9777-5ff8f209ed6a nodeName:}" failed. No retries permitted until 2026-09-16 18:41:34.669662566 +0000 UTC m=+75.736154031 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/2e9f2573-3279-4102-9777-5ff8f209ed6a-kube-rbac-proxy-cm") pod "insights-runtime-extractor-9kn7r" (UID: "2e9f2573-3279-4102-9777-5ff8f209ed6a") : configmap "kube-rbac-proxy" not found
Sep 16 18:41:34.169807 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:41:34.169715    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/2e9f2573-3279-4102-9777-5ff8f209ed6a-data-volume\") pod \"insights-runtime-extractor-9kn7r\" (UID: \"2e9f2573-3279-4102-9777-5ff8f209ed6a\") " pod="openshift-insights/insights-runtime-extractor-9kn7r"
Sep 16 18:41:34.179607 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:41:34.179590    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-292df\" (UniqueName: \"kubernetes.io/projected/2e9f2573-3279-4102-9777-5ff8f209ed6a-kube-api-access-292df\") pod \"insights-runtime-extractor-9kn7r\" (UID: \"2e9f2573-3279-4102-9777-5ff8f209ed6a\") " pod="openshift-insights/insights-runtime-extractor-9kn7r"
Sep 16 18:41:34.566426 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:41:34.566330    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 18:41:34.674649 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:41:34.674620    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/2e9f2573-3279-4102-9777-5ff8f209ed6a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-9kn7r\" (UID: \"2e9f2573-3279-4102-9777-5ff8f209ed6a\") " pod="openshift-insights/insights-runtime-extractor-9kn7r"
Sep 16 18:41:34.674649 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:41:34.674653    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/2e9f2573-3279-4102-9777-5ff8f209ed6a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-9kn7r\" (UID: \"2e9f2573-3279-4102-9777-5ff8f209ed6a\") " pod="openshift-insights/insights-runtime-extractor-9kn7r"
Sep 16 18:41:34.675053 ip-10-0-135-119 kubenswrapper[2578]: E0916 18:41:34.674732    2578 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 16 18:41:34.675053 ip-10-0-135-119 kubenswrapper[2578]: E0916 18:41:34.674765    2578 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 16 18:41:34.675053 ip-10-0-135-119 kubenswrapper[2578]: E0916 18:41:34.674803    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/2e9f2573-3279-4102-9777-5ff8f209ed6a-kube-rbac-proxy-cm podName:2e9f2573-3279-4102-9777-5ff8f209ed6a nodeName:}" failed. No retries permitted until 2026-09-16 18:41:35.674789532 +0000 UTC m=+76.741280998 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/2e9f2573-3279-4102-9777-5ff8f209ed6a-kube-rbac-proxy-cm") pod "insights-runtime-extractor-9kn7r" (UID: "2e9f2573-3279-4102-9777-5ff8f209ed6a") : configmap "kube-rbac-proxy" not found
Sep 16 18:41:34.675053 ip-10-0-135-119 kubenswrapper[2578]: E0916 18:41:34.674828    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2e9f2573-3279-4102-9777-5ff8f209ed6a-insights-runtime-extractor-tls podName:2e9f2573-3279-4102-9777-5ff8f209ed6a nodeName:}" failed. No retries permitted until 2026-09-16 18:41:35.674813494 +0000 UTC m=+76.741304965 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/2e9f2573-3279-4102-9777-5ff8f209ed6a-insights-runtime-extractor-tls") pod "insights-runtime-extractor-9kn7r" (UID: "2e9f2573-3279-4102-9777-5ff8f209ed6a") : secret "insights-runtime-extractor-tls" not found
Sep 16 18:41:34.683110 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:41:34.683090    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 18:41:34.798541 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:41:34.798499    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 18:41:34.854876 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:41:34.854801    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-tmrkx_4c2078bb-281a-4aa1-8e38-238897865434/console-operator/1.log"
Sep 16 18:41:34.855215 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:41:34.855201    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-tmrkx_4c2078bb-281a-4aa1-8e38-238897865434/console-operator/0.log"
Sep 16 18:41:34.855281 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:41:34.855236    2578 generic.go:356] "Generic (PLEG): container finished" podID="4c2078bb-281a-4aa1-8e38-238897865434" containerID="78b91406e9fde86d000ebcb54e7e7f53e6a94f56e2a53952893e99ef2baeb4e7" exitCode=255
Sep 16 18:41:34.855345 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:41:34.855323    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-696884fb9c-tmrkx" event={"ID":"4c2078bb-281a-4aa1-8e38-238897865434","Type":"ContainerDied","Data":"78b91406e9fde86d000ebcb54e7e7f53e6a94f56e2a53952893e99ef2baeb4e7"}
Sep 16 18:41:34.855417 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:41:34.855404    2578 scope.go:117] "RemoveContainer" containerID="3581556e46eaa2f1fbdaafaf77bd706d60acbdc33f238f838cbee89a66135056"
Sep 16 18:41:34.855587 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:41:34.855569    2578 scope.go:117] "RemoveContainer" containerID="78b91406e9fde86d000ebcb54e7e7f53e6a94f56e2a53952893e99ef2baeb4e7"
Sep 16 18:41:34.855778 ip-10-0-135-119 kubenswrapper[2578]: E0916 18:41:34.855748    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-696884fb9c-tmrkx_openshift-console-operator(4c2078bb-281a-4aa1-8e38-238897865434)\"" pod="openshift-console-operator/console-operator-696884fb9c-tmrkx" podUID="4c2078bb-281a-4aa1-8e38-238897865434"
Sep 16 18:41:35.258609 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:41:35.258567    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 18:41:35.348599 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:41:35.348565    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 18:41:35.673192 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:41:35.673169    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-5rrj4_fc3365ef-dfd3-45fe-b80f-937949d65751/dns-node-resolver/0.log"
Sep 16 18:41:35.683603 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:41:35.683575    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/2e9f2573-3279-4102-9777-5ff8f209ed6a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-9kn7r\" (UID: \"2e9f2573-3279-4102-9777-5ff8f209ed6a\") " pod="openshift-insights/insights-runtime-extractor-9kn7r"
Sep 16 18:41:35.683961 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:41:35.683612    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/2e9f2573-3279-4102-9777-5ff8f209ed6a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-9kn7r\" (UID: \"2e9f2573-3279-4102-9777-5ff8f209ed6a\") " pod="openshift-insights/insights-runtime-extractor-9kn7r"
Sep 16 18:41:35.683961 ip-10-0-135-119 kubenswrapper[2578]: E0916 18:41:35.683707    2578 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 16 18:41:35.683961 ip-10-0-135-119 kubenswrapper[2578]: E0916 18:41:35.683726    2578 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 16 18:41:35.683961 ip-10-0-135-119 kubenswrapper[2578]: E0916 18:41:35.683773    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/2e9f2573-3279-4102-9777-5ff8f209ed6a-kube-rbac-proxy-cm podName:2e9f2573-3279-4102-9777-5ff8f209ed6a nodeName:}" failed. No retries permitted until 2026-09-16 18:41:37.683756768 +0000 UTC m=+78.750248236 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/2e9f2573-3279-4102-9777-5ff8f209ed6a-kube-rbac-proxy-cm") pod "insights-runtime-extractor-9kn7r" (UID: "2e9f2573-3279-4102-9777-5ff8f209ed6a") : configmap "kube-rbac-proxy" not found
Sep 16 18:41:35.683961 ip-10-0-135-119 kubenswrapper[2578]: E0916 18:41:35.683791    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2e9f2573-3279-4102-9777-5ff8f209ed6a-insights-runtime-extractor-tls podName:2e9f2573-3279-4102-9777-5ff8f209ed6a nodeName:}" failed. No retries permitted until 2026-09-16 18:41:37.683781798 +0000 UTC m=+78.750273271 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/2e9f2573-3279-4102-9777-5ff8f209ed6a-insights-runtime-extractor-tls") pod "insights-runtime-extractor-9kn7r" (UID: "2e9f2573-3279-4102-9777-5ff8f209ed6a") : secret "insights-runtime-extractor-tls" not found
Sep 16 18:41:35.859895 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:41:35.859860    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-tmrkx_4c2078bb-281a-4aa1-8e38-238897865434/console-operator/1.log"
Sep 16 18:41:35.860180 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:41:35.860166    2578 scope.go:117] "RemoveContainer" containerID="78b91406e9fde86d000ebcb54e7e7f53e6a94f56e2a53952893e99ef2baeb4e7"
Sep 16 18:41:35.860323 ip-10-0-135-119 kubenswrapper[2578]: E0916 18:41:35.860305    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-696884fb9c-tmrkx_openshift-console-operator(4c2078bb-281a-4aa1-8e38-238897865434)\"" pod="openshift-console-operator/console-operator-696884fb9c-tmrkx" podUID="4c2078bb-281a-4aa1-8e38-238897865434"
Sep 16 18:41:36.878961 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:41:36.878930    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-568869fdc5-hf84m"]
Sep 16 18:41:36.883128 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:41:36.883111    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-568869fdc5-hf84m"
Sep 16 18:41:36.886048 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:41:36.886027    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\""
Sep 16 18:41:36.886184 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:41:36.886065    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\""
Sep 16 18:41:36.886706 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:41:36.886687    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\""
Sep 16 18:41:36.886808 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:41:36.886691    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-p57z9\""
Sep 16 18:41:36.886808 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:41:36.886743    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\""
Sep 16 18:41:36.892257 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:41:36.892231    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-568869fdc5-hf84m"]
Sep 16 18:41:36.905592 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:41:36.905553    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 18:41:36.993358 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:41:36.993325    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/0babfe11-5912-4be9-b649-0fa6f2a6735d-signing-cabundle\") pod \"service-ca-568869fdc5-hf84m\" (UID: \"0babfe11-5912-4be9-b649-0fa6f2a6735d\") " pod="openshift-service-ca/service-ca-568869fdc5-hf84m"
Sep 16 18:41:36.993358 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:41:36.993358    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j59lg\" (UniqueName: \"kubernetes.io/projected/0babfe11-5912-4be9-b649-0fa6f2a6735d-kube-api-access-j59lg\") pod \"service-ca-568869fdc5-hf84m\" (UID: \"0babfe11-5912-4be9-b649-0fa6f2a6735d\") " pod="openshift-service-ca/service-ca-568869fdc5-hf84m"
Sep 16 18:41:36.993529 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:41:36.993377    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/0babfe11-5912-4be9-b649-0fa6f2a6735d-signing-key\") pod \"service-ca-568869fdc5-hf84m\" (UID: \"0babfe11-5912-4be9-b649-0fa6f2a6735d\") " pod="openshift-service-ca/service-ca-568869fdc5-hf84m"
Sep 16 18:41:37.073966 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:41:37.073938    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-bl65n_454c7c51-a475-44f6-be51-1a8f1d29beaf/node-ca/0.log"
Sep 16 18:41:37.094295 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:41:37.094268    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/0babfe11-5912-4be9-b649-0fa6f2a6735d-signing-cabundle\") pod \"service-ca-568869fdc5-hf84m\" (UID: \"0babfe11-5912-4be9-b649-0fa6f2a6735d\") " pod="openshift-service-ca/service-ca-568869fdc5-hf84m"
Sep 16 18:41:37.094405 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:41:37.094300    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j59lg\" (UniqueName: \"kubernetes.io/projected/0babfe11-5912-4be9-b649-0fa6f2a6735d-kube-api-access-j59lg\") pod \"service-ca-568869fdc5-hf84m\" (UID: \"0babfe11-5912-4be9-b649-0fa6f2a6735d\") " pod="openshift-service-ca/service-ca-568869fdc5-hf84m"
Sep 16 18:41:37.094405 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:41:37.094316    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/0babfe11-5912-4be9-b649-0fa6f2a6735d-signing-key\") pod \"service-ca-568869fdc5-hf84m\" (UID: \"0babfe11-5912-4be9-b649-0fa6f2a6735d\") " pod="openshift-service-ca/service-ca-568869fdc5-hf84m"
Sep 16 18:41:37.094898 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:41:37.094878    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/0babfe11-5912-4be9-b649-0fa6f2a6735d-signing-cabundle\") pod \"service-ca-568869fdc5-hf84m\" (UID: \"0babfe11-5912-4be9-b649-0fa6f2a6735d\") " pod="openshift-service-ca/service-ca-568869fdc5-hf84m"
Sep 16 18:41:37.096835 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:41:37.096810    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/0babfe11-5912-4be9-b649-0fa6f2a6735d-signing-key\") pod \"service-ca-568869fdc5-hf84m\" (UID: \"0babfe11-5912-4be9-b649-0fa6f2a6735d\") " pod="openshift-service-ca/service-ca-568869fdc5-hf84m"
Sep 16 18:41:37.103185 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:41:37.103160    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j59lg\" (UniqueName: \"kubernetes.io/projected/0babfe11-5912-4be9-b649-0fa6f2a6735d-kube-api-access-j59lg\") pod \"service-ca-568869fdc5-hf84m\" (UID: \"0babfe11-5912-4be9-b649-0fa6f2a6735d\") " pod="openshift-service-ca/service-ca-568869fdc5-hf84m"
Sep 16 18:41:37.192389 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:41:37.192322    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-568869fdc5-hf84m"
Sep 16 18:41:37.465514 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:41:37.465428    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-568869fdc5-hf84m"]
Sep 16 18:41:37.466761 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:41:37.466739    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-c42vj_ccaf4d24-a69d-4925-9f2a-8f77d2d3a16d/serve-healthcheck-canary/0.log"
Sep 16 18:41:37.470273 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:41:37.470248    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0babfe11_5912_4be9_b649_0fa6f2a6735d.slice/crio-8ab2532aba75bf64ed3ae4fbd20869921042a4a9c784d776e657f5ba12888d94 WatchSource:0}: Error finding container 8ab2532aba75bf64ed3ae4fbd20869921042a4a9c784d776e657f5ba12888d94: Status 404 returned error can't find the container with id 8ab2532aba75bf64ed3ae4fbd20869921042a4a9c784d776e657f5ba12888d94
Sep 16 18:41:37.558285 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:41:37.558256    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 18:41:37.649799 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:41:37.649771    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 18:41:37.699368 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:41:37.699341    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/2e9f2573-3279-4102-9777-5ff8f209ed6a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-9kn7r\" (UID: \"2e9f2573-3279-4102-9777-5ff8f209ed6a\") " pod="openshift-insights/insights-runtime-extractor-9kn7r"
Sep 16 18:41:37.699368 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:41:37.699369    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/2e9f2573-3279-4102-9777-5ff8f209ed6a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-9kn7r\" (UID: \"2e9f2573-3279-4102-9777-5ff8f209ed6a\") " pod="openshift-insights/insights-runtime-extractor-9kn7r"
Sep 16 18:41:37.699504 ip-10-0-135-119 kubenswrapper[2578]: E0916 18:41:37.699437    2578 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 16 18:41:37.699504 ip-10-0-135-119 kubenswrapper[2578]: E0916 18:41:37.699488    2578 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 16 18:41:37.699565 ip-10-0-135-119 kubenswrapper[2578]: E0916 18:41:37.699491    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/2e9f2573-3279-4102-9777-5ff8f209ed6a-kube-rbac-proxy-cm podName:2e9f2573-3279-4102-9777-5ff8f209ed6a nodeName:}" failed. No retries permitted until 2026-09-16 18:41:41.699474738 +0000 UTC m=+82.765966214 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/2e9f2573-3279-4102-9777-5ff8f209ed6a-kube-rbac-proxy-cm") pod "insights-runtime-extractor-9kn7r" (UID: "2e9f2573-3279-4102-9777-5ff8f209ed6a") : configmap "kube-rbac-proxy" not found
Sep 16 18:41:37.699565 ip-10-0-135-119 kubenswrapper[2578]: E0916 18:41:37.699553    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2e9f2573-3279-4102-9777-5ff8f209ed6a-insights-runtime-extractor-tls podName:2e9f2573-3279-4102-9777-5ff8f209ed6a nodeName:}" failed. No retries permitted until 2026-09-16 18:41:41.699540957 +0000 UTC m=+82.766032427 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/2e9f2573-3279-4102-9777-5ff8f209ed6a-insights-runtime-extractor-tls") pod "insights-runtime-extractor-9kn7r" (UID: "2e9f2573-3279-4102-9777-5ff8f209ed6a") : secret "insights-runtime-extractor-tls" not found
Sep 16 18:41:37.867121 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:41:37.867044    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-568869fdc5-hf84m" event={"ID":"0babfe11-5912-4be9-b649-0fa6f2a6735d","Type":"ContainerStarted","Data":"8ab2532aba75bf64ed3ae4fbd20869921042a4a9c784d776e657f5ba12888d94"}
Sep 16 18:41:39.874604 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:41:39.874524    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-568869fdc5-hf84m" event={"ID":"0babfe11-5912-4be9-b649-0fa6f2a6735d","Type":"ContainerStarted","Data":"bc808b634f3d1bd6c7f76d7e443a52e4fe643f194485937f3a341ef43fa41313"}
Sep 16 18:41:39.895580 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:41:39.895530    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca/service-ca-568869fdc5-hf84m" podStartSLOduration=1.9643044010000001 podStartE2EDuration="3.895513421s" totalImagesPullingTime="1.93120902s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:41:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:41:39.894303272 +0000 UTC m=+80.960794760" watchObservedRunningTime="2026-09-16 18:41:39.895513421 +0000 UTC m=+80.962004920"
Sep 16 18:41:39.900369 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:41:39.900345    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 18:41:40.072559 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:41:40.072530    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 18:41:40.371035 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:41:40.371000    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 18:41:40.383897 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:41:40.383871    2578 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-696884fb9c-tmrkx"
Sep 16 18:41:40.384058 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:41:40.383923    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-696884fb9c-tmrkx"
Sep 16 18:41:40.384334 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:41:40.384319    2578 scope.go:117] "RemoveContainer" containerID="78b91406e9fde86d000ebcb54e7e7f53e6a94f56e2a53952893e99ef2baeb4e7"
Sep 16 18:41:40.384514 ip-10-0-135-119 kubenswrapper[2578]: E0916 18:41:40.384494    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-696884fb9c-tmrkx_openshift-console-operator(4c2078bb-281a-4aa1-8e38-238897865434)\"" pod="openshift-console-operator/console-operator-696884fb9c-tmrkx" podUID="4c2078bb-281a-4aa1-8e38-238897865434"
Sep 16 18:41:40.467186 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:41:40.467152    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 18:41:40.579008 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:41:40.578890    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 18:41:40.681267 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:41:40.681213    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 18:41:40.970180 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:41:40.970098    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 18:41:41.369693 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:41:41.369600    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 18:41:41.607691 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:41:41.607658    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 18:41:41.734472 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:41:41.734436    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/2e9f2573-3279-4102-9777-5ff8f209ed6a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-9kn7r\" (UID: \"2e9f2573-3279-4102-9777-5ff8f209ed6a\") " pod="openshift-insights/insights-runtime-extractor-9kn7r"
Sep 16 18:41:41.734472 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:41:41.734473    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/2e9f2573-3279-4102-9777-5ff8f209ed6a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-9kn7r\" (UID: \"2e9f2573-3279-4102-9777-5ff8f209ed6a\") " pod="openshift-insights/insights-runtime-extractor-9kn7r"
Sep 16 18:41:41.734689 ip-10-0-135-119 kubenswrapper[2578]: E0916 18:41:41.734555    2578 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 16 18:41:41.734689 ip-10-0-135-119 kubenswrapper[2578]: E0916 18:41:41.734590    2578 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 16 18:41:41.734689 ip-10-0-135-119 kubenswrapper[2578]: E0916 18:41:41.734644    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/2e9f2573-3279-4102-9777-5ff8f209ed6a-kube-rbac-proxy-cm podName:2e9f2573-3279-4102-9777-5ff8f209ed6a nodeName:}" failed. No retries permitted until 2026-09-16 18:41:49.73463074 +0000 UTC m=+90.801122206 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/2e9f2573-3279-4102-9777-5ff8f209ed6a-kube-rbac-proxy-cm") pod "insights-runtime-extractor-9kn7r" (UID: "2e9f2573-3279-4102-9777-5ff8f209ed6a") : configmap "kube-rbac-proxy" not found
Sep 16 18:41:41.734689 ip-10-0-135-119 kubenswrapper[2578]: E0916 18:41:41.734663    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2e9f2573-3279-4102-9777-5ff8f209ed6a-insights-runtime-extractor-tls podName:2e9f2573-3279-4102-9777-5ff8f209ed6a nodeName:}" failed. No retries permitted until 2026-09-16 18:41:49.734649764 +0000 UTC m=+90.801141230 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/2e9f2573-3279-4102-9777-5ff8f209ed6a-insights-runtime-extractor-tls") pod "insights-runtime-extractor-9kn7r" (UID: "2e9f2573-3279-4102-9777-5ff8f209ed6a") : secret "insights-runtime-extractor-tls" not found
Sep 16 18:41:42.174572 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:41:42.174543    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 18:41:42.364441 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:41:42.364408    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 18:41:42.579757 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:41:42.579653    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 18:41:42.976274 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:41:42.976245    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 18:41:43.573279 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:41:43.573238    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 18:41:43.764676 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:41:43.764641    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 18:41:43.925100 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:41:43.925062    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 18:41:44.162625 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:41:44.162580    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 18:41:44.375901 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:41:44.375827    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 18:41:44.762161 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:41:44.762104    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 18:41:44.825099 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:41:44.825067    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 18:41:44.968141 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:41:44.968109    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 18:41:45.360604 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:41:45.360572    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 18:41:46.158458 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:41:46.158423    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 18:41:46.288168 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:41:46.288135    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 18:41:46.563807 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:41:46.563734    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 18:41:46.777056 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:41:46.777026    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 18:41:46.977046 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:41:46.977013    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 18:41:47.141867 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:41:47.141838    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 18:41:47.364182 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:41:47.364109    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 18:41:47.981074 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:41:47.981043    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 18:41:48.569454 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:41:48.569424    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 18:41:48.769801 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:41:48.769773    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 18:41:49.183211 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:41:49.183182    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 18:41:49.371432 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:41:49.371397    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 18:41:49.798025 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:41:49.797970    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/2e9f2573-3279-4102-9777-5ff8f209ed6a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-9kn7r\" (UID: \"2e9f2573-3279-4102-9777-5ff8f209ed6a\") " pod="openshift-insights/insights-runtime-extractor-9kn7r"
Sep 16 18:41:49.798486 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:41:49.798037    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/2e9f2573-3279-4102-9777-5ff8f209ed6a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-9kn7r\" (UID: \"2e9f2573-3279-4102-9777-5ff8f209ed6a\") " pod="openshift-insights/insights-runtime-extractor-9kn7r"
Sep 16 18:41:49.798486 ip-10-0-135-119 kubenswrapper[2578]: E0916 18:41:49.798083    2578 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 16 18:41:49.798486 ip-10-0-135-119 kubenswrapper[2578]: E0916 18:41:49.798181    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2e9f2573-3279-4102-9777-5ff8f209ed6a-insights-runtime-extractor-tls podName:2e9f2573-3279-4102-9777-5ff8f209ed6a nodeName:}" failed. No retries permitted until 2026-09-16 18:42:05.798165362 +0000 UTC m=+106.864656827 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/2e9f2573-3279-4102-9777-5ff8f209ed6a-insights-runtime-extractor-tls") pod "insights-runtime-extractor-9kn7r" (UID: "2e9f2573-3279-4102-9777-5ff8f209ed6a") : secret "insights-runtime-extractor-tls" not found
Sep 16 18:41:49.798683 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:41:49.798637    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/2e9f2573-3279-4102-9777-5ff8f209ed6a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-9kn7r\" (UID: \"2e9f2573-3279-4102-9777-5ff8f209ed6a\") " pod="openshift-insights/insights-runtime-extractor-9kn7r"
Sep 16 18:41:49.968482 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:41:49.968444    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 18:41:50.365171 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:41:50.365142    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 18:41:50.568766 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:41:50.568740    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 18:41:50.782899 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:41:50.782847    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 18:41:52.808146 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:41:52.808114    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 18:41:55.429223 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:41:55.429193    2578 scope.go:117] "RemoveContainer" containerID="78b91406e9fde86d000ebcb54e7e7f53e6a94f56e2a53952893e99ef2baeb4e7"
Sep 16 18:41:55.930047 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:41:55.930017    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-tmrkx_4c2078bb-281a-4aa1-8e38-238897865434/console-operator/2.log"
Sep 16 18:41:55.930406 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:41:55.930391    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-tmrkx_4c2078bb-281a-4aa1-8e38-238897865434/console-operator/1.log"
Sep 16 18:41:55.930455 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:41:55.930424    2578 generic.go:356] "Generic (PLEG): container finished" podID="4c2078bb-281a-4aa1-8e38-238897865434" containerID="8f38b1af4975884bac5c76bc8f8a028e6d32d0bc4be3883c8dd768ea64e92644" exitCode=255
Sep 16 18:41:55.930489 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:41:55.930458    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-696884fb9c-tmrkx" event={"ID":"4c2078bb-281a-4aa1-8e38-238897865434","Type":"ContainerDied","Data":"8f38b1af4975884bac5c76bc8f8a028e6d32d0bc4be3883c8dd768ea64e92644"}
Sep 16 18:41:55.930539 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:41:55.930505    2578 scope.go:117] "RemoveContainer" containerID="78b91406e9fde86d000ebcb54e7e7f53e6a94f56e2a53952893e99ef2baeb4e7"
Sep 16 18:41:55.930792 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:41:55.930769    2578 scope.go:117] "RemoveContainer" containerID="8f38b1af4975884bac5c76bc8f8a028e6d32d0bc4be3883c8dd768ea64e92644"
Sep 16 18:41:55.930970 ip-10-0-135-119 kubenswrapper[2578]: E0916 18:41:55.930950    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-696884fb9c-tmrkx_openshift-console-operator(4c2078bb-281a-4aa1-8e38-238897865434)\"" pod="openshift-console-operator/console-operator-696884fb9c-tmrkx" podUID="4c2078bb-281a-4aa1-8e38-238897865434"
Sep 16 18:41:56.879425 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:41:56.879393    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 18:41:56.935402 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:41:56.935376    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-tmrkx_4c2078bb-281a-4aa1-8e38-238897865434/console-operator/2.log"
Sep 16 18:41:56.946595 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:41:56.946567    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 18:42:00.382971 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:42:00.382929    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-696884fb9c-tmrkx"
Sep 16 18:42:00.383456 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:42:00.383083    2578 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-696884fb9c-tmrkx"
Sep 16 18:42:00.383456 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:42:00.383394    2578 scope.go:117] "RemoveContainer" containerID="8f38b1af4975884bac5c76bc8f8a028e6d32d0bc4be3883c8dd768ea64e92644"
Sep 16 18:42:00.383612 ip-10-0-135-119 kubenswrapper[2578]: E0916 18:42:00.383587    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-696884fb9c-tmrkx_openshift-console-operator(4c2078bb-281a-4aa1-8e38-238897865434)\"" pod="openshift-console-operator/console-operator-696884fb9c-tmrkx" podUID="4c2078bb-281a-4aa1-8e38-238897865434"
Sep 16 18:42:00.839092 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:42:00.839015    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-f287c"
Sep 16 18:42:00.947314 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:42:00.947292    2578 scope.go:117] "RemoveContainer" containerID="8f38b1af4975884bac5c76bc8f8a028e6d32d0bc4be3883c8dd768ea64e92644"
Sep 16 18:42:00.947471 ip-10-0-135-119 kubenswrapper[2578]: E0916 18:42:00.947435    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-696884fb9c-tmrkx_openshift-console-operator(4c2078bb-281a-4aa1-8e38-238897865434)\"" pod="openshift-console-operator/console-operator-696884fb9c-tmrkx" podUID="4c2078bb-281a-4aa1-8e38-238897865434"
Sep 16 18:42:01.672008 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:42:01.671959    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-5b556cd768-md2zx"]
Sep 16 18:42:01.704852 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:42:01.704824    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5b556cd768-md2zx"
Sep 16 18:42:01.707191 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:42:01.707163    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Sep 16 18:42:01.708195 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:42:01.707747    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\""
Sep 16 18:42:01.708195 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:42:01.707752    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Sep 16 18:42:01.708195 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:42:01.708026    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Sep 16 18:42:01.708195 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:42:01.708108    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-5b556cd768-md2zx"]
Sep 16 18:42:01.792083 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:42:01.792049    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q9fkk\" (UniqueName: \"kubernetes.io/projected/dee01b3b-739c-4738-8e9b-d78d2ba95549-kube-api-access-q9fkk\") pod \"klusterlet-addon-workmgr-5b556cd768-md2zx\" (UID: \"dee01b3b-739c-4738-8e9b-d78d2ba95549\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5b556cd768-md2zx"
Sep 16 18:42:01.792258 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:42:01.792158    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/dee01b3b-739c-4738-8e9b-d78d2ba95549-tmp\") pod \"klusterlet-addon-workmgr-5b556cd768-md2zx\" (UID: \"dee01b3b-739c-4738-8e9b-d78d2ba95549\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5b556cd768-md2zx"
Sep 16 18:42:01.792258 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:42:01.792187    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/dee01b3b-739c-4738-8e9b-d78d2ba95549-klusterlet-config\") pod \"klusterlet-addon-workmgr-5b556cd768-md2zx\" (UID: \"dee01b3b-739c-4738-8e9b-d78d2ba95549\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5b556cd768-md2zx"
Sep 16 18:42:01.871414 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:42:01.871387    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 18:42:01.893089 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:42:01.893059    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q9fkk\" (UniqueName: \"kubernetes.io/projected/dee01b3b-739c-4738-8e9b-d78d2ba95549-kube-api-access-q9fkk\") pod \"klusterlet-addon-workmgr-5b556cd768-md2zx\" (UID: \"dee01b3b-739c-4738-8e9b-d78d2ba95549\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5b556cd768-md2zx"
Sep 16 18:42:01.893232 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:42:01.893138    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/dee01b3b-739c-4738-8e9b-d78d2ba95549-tmp\") pod \"klusterlet-addon-workmgr-5b556cd768-md2zx\" (UID: \"dee01b3b-739c-4738-8e9b-d78d2ba95549\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5b556cd768-md2zx"
Sep 16 18:42:01.893232 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:42:01.893162    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/dee01b3b-739c-4738-8e9b-d78d2ba95549-klusterlet-config\") pod \"klusterlet-addon-workmgr-5b556cd768-md2zx\" (UID: \"dee01b3b-739c-4738-8e9b-d78d2ba95549\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5b556cd768-md2zx"
Sep 16 18:42:01.893541 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:42:01.893521    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/dee01b3b-739c-4738-8e9b-d78d2ba95549-tmp\") pod \"klusterlet-addon-workmgr-5b556cd768-md2zx\" (UID: \"dee01b3b-739c-4738-8e9b-d78d2ba95549\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5b556cd768-md2zx"
Sep 16 18:42:01.895817 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:42:01.895790    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/dee01b3b-739c-4738-8e9b-d78d2ba95549-klusterlet-config\") pod \"klusterlet-addon-workmgr-5b556cd768-md2zx\" (UID: \"dee01b3b-739c-4738-8e9b-d78d2ba95549\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5b556cd768-md2zx"
Sep 16 18:42:01.913719 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:42:01.913693    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q9fkk\" (UniqueName: \"kubernetes.io/projected/dee01b3b-739c-4738-8e9b-d78d2ba95549-kube-api-access-q9fkk\") pod \"klusterlet-addon-workmgr-5b556cd768-md2zx\" (UID: \"dee01b3b-739c-4738-8e9b-d78d2ba95549\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5b556cd768-md2zx"
Sep 16 18:42:01.973879 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:42:01.973813    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 18:42:02.015199 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:42:02.015169    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5b556cd768-md2zx"
Sep 16 18:42:02.080431 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:42:02.079767    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 18:42:02.170577 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:42:02.170546    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 18:42:02.288797 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:42:02.288712    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-5b556cd768-md2zx"]
Sep 16 18:42:02.291446 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:42:02.291422    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddee01b3b_739c_4738_8e9b_d78d2ba95549.slice/crio-01f4454d0474b7f9d78c2a583afa44e19de471378024d616f0d88b37fddd04e2 WatchSource:0}: Error finding container 01f4454d0474b7f9d78c2a583afa44e19de471378024d616f0d88b37fddd04e2: Status 404 returned error can't find the container with id 01f4454d0474b7f9d78c2a583afa44e19de471378024d616f0d88b37fddd04e2
Sep 16 18:42:02.700361 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:42:02.700329    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/92866a88-93b2-430b-867b-40e3358eb631-registry-tls\") pod \"image-registry-9785996cd-z4ckv\" (UID: \"92866a88-93b2-430b-867b-40e3358eb631\") " pod="openshift-image-registry/image-registry-9785996cd-z4ckv"
Sep 16 18:42:02.700730 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:42:02.700368    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0ef1438e-2929-4fc5-9437-6e5b97e79006-metrics-tls\") pod \"dns-default-vpkvv\" (UID: \"0ef1438e-2929-4fc5-9437-6e5b97e79006\") " pod="openshift-dns/dns-default-vpkvv"
Sep 16 18:42:02.702772 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:42:02.702751    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0ef1438e-2929-4fc5-9437-6e5b97e79006-metrics-tls\") pod \"dns-default-vpkvv\" (UID: \"0ef1438e-2929-4fc5-9437-6e5b97e79006\") " pod="openshift-dns/dns-default-vpkvv"
Sep 16 18:42:02.702908 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:42:02.702892    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/92866a88-93b2-430b-867b-40e3358eb631-registry-tls\") pod \"image-registry-9785996cd-z4ckv\" (UID: \"92866a88-93b2-430b-867b-40e3358eb631\") " pod="openshift-image-registry/image-registry-9785996cd-z4ckv"
Sep 16 18:42:02.953584 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:42:02.953492    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5b556cd768-md2zx" event={"ID":"dee01b3b-739c-4738-8e9b-d78d2ba95549","Type":"ContainerStarted","Data":"01f4454d0474b7f9d78c2a583afa44e19de471378024d616f0d88b37fddd04e2"}
Sep 16 18:42:02.983024 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:42:02.982996    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-xjxjk\""
Sep 16 18:42:02.991534 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:42:02.991511    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-9785996cd-z4ckv"
Sep 16 18:42:02.993845 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:42:02.993826    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-67vjb\""
Sep 16 18:42:03.001943 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:42:03.001924    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-vpkvv"
Sep 16 18:42:03.300760 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:42:03.300664    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-9785996cd-z4ckv"]
Sep 16 18:42:03.303606 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:42:03.303582    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod92866a88_93b2_430b_867b_40e3358eb631.slice/crio-b1011fdb569f9fd3456804268d92e36caa6505872266be469b3c01515546db3f WatchSource:0}: Error finding container b1011fdb569f9fd3456804268d92e36caa6505872266be469b3c01515546db3f: Status 404 returned error can't find the container with id b1011fdb569f9fd3456804268d92e36caa6505872266be469b3c01515546db3f
Sep 16 18:42:03.322763 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:42:03.322735    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-vpkvv"]
Sep 16 18:42:03.325536 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:42:03.325512    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0ef1438e_2929_4fc5_9437_6e5b97e79006.slice/crio-df1354b14cd12579591817e16ac2ef8d72b7e908377bafa321c14d521434b099 WatchSource:0}: Error finding container df1354b14cd12579591817e16ac2ef8d72b7e908377bafa321c14d521434b099: Status 404 returned error can't find the container with id df1354b14cd12579591817e16ac2ef8d72b7e908377bafa321c14d521434b099
Sep 16 18:42:03.958807 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:42:03.958758    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-vpkvv" event={"ID":"0ef1438e-2929-4fc5-9437-6e5b97e79006","Type":"ContainerStarted","Data":"df1354b14cd12579591817e16ac2ef8d72b7e908377bafa321c14d521434b099"}
Sep 16 18:42:03.960658 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:42:03.960628    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-9785996cd-z4ckv" event={"ID":"92866a88-93b2-430b-867b-40e3358eb631","Type":"ContainerStarted","Data":"a275d6ff5587eea121dda12a18391617704b348ec72ed226db0c8907f3bc1680"}
Sep 16 18:42:03.960783 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:42:03.960673    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-9785996cd-z4ckv" event={"ID":"92866a88-93b2-430b-867b-40e3358eb631","Type":"ContainerStarted","Data":"b1011fdb569f9fd3456804268d92e36caa6505872266be469b3c01515546db3f"}
Sep 16 18:42:03.960959 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:42:03.960940    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-9785996cd-z4ckv"
Sep 16 18:42:04.779066 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:42:04.778918    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 18:42:04.866997 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:42:04.866954    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 18:42:05.831425 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:42:05.831387    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/2e9f2573-3279-4102-9777-5ff8f209ed6a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-9kn7r\" (UID: \"2e9f2573-3279-4102-9777-5ff8f209ed6a\") " pod="openshift-insights/insights-runtime-extractor-9kn7r"
Sep 16 18:42:05.834162 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:42:05.834136    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/2e9f2573-3279-4102-9777-5ff8f209ed6a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-9kn7r\" (UID: \"2e9f2573-3279-4102-9777-5ff8f209ed6a\") " pod="openshift-insights/insights-runtime-extractor-9kn7r"
Sep 16 18:42:06.120204 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:42:06.120124    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-9kn7r"
Sep 16 18:42:06.920063 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:42:06.920009    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 18:42:06.974911 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:42:06.974875    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5b556cd768-md2zx" event={"ID":"dee01b3b-739c-4738-8e9b-d78d2ba95549","Type":"ContainerStarted","Data":"f71d9454a2d7eab194185bc42a327350b86acc5b153b2944b646a0de9dda9ddd"}
Sep 16 18:42:06.975647 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:42:06.975610    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5b556cd768-md2zx"
Sep 16 18:42:06.977334 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:42:06.977298    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-vpkvv" event={"ID":"0ef1438e-2929-4fc5-9437-6e5b97e79006","Type":"ContainerStarted","Data":"730e021bb7a9dc93e3b87fb63156f681d2c5ea26710718ad8135290180fa312c"}
Sep 16 18:42:06.977492 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:42:06.977476    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5b556cd768-md2zx"
Sep 16 18:42:06.993649 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:42:06.993592    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5b556cd768-md2zx" podStartSLOduration=1.45408835 podStartE2EDuration="5.993574686s" totalImagesPullingTime="4.539486336s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:42:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:42:06.993044484 +0000 UTC m=+108.059535973" watchObservedRunningTime="2026-09-16 18:42:06.993574686 +0000 UTC m=+108.060066174"
Sep 16 18:42:06.993843 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:42:06.993819    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-9785996cd-z4ckv" podStartSLOduration=106.993813807 podStartE2EDuration="1m46.993813807s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:40:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:42:03.989795543 +0000 UTC m=+105.056287050" watchObservedRunningTime="2026-09-16 18:42:06.993813807 +0000 UTC m=+108.060305294"
Sep 16 18:42:07.004189 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:42:07.004166    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 18:42:07.104244 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:42:07.104171    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-9kn7r"]
Sep 16 18:42:07.107213 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:42:07.107188    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e9f2573_3279_4102_9777_5ff8f209ed6a.slice/crio-6619cef81a8b13fe4c17b269f8a48c31d905e3ffe1cb0dd5324b3471cca8e495 WatchSource:0}: Error finding container 6619cef81a8b13fe4c17b269f8a48c31d905e3ffe1cb0dd5324b3471cca8e495: Status 404 returned error can't find the container with id 6619cef81a8b13fe4c17b269f8a48c31d905e3ffe1cb0dd5324b3471cca8e495
Sep 16 18:42:07.218120 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:42:07.218093    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 18:42:07.237689 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:42:07.237662    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-rcq4t"]
Sep 16 18:42:07.239839 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:42:07.239816    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-rcq4t"
Sep 16 18:42:07.241868 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:42:07.241837    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\""
Sep 16 18:42:07.242003 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:42:07.241866    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-pg4rq\""
Sep 16 18:42:07.259816 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:42:07.259790    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-rcq4t"]
Sep 16 18:42:07.345920 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:42:07.345887    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/ff22f15f-83ae-4530-81ff-6d81caaf1ddb-tls-certificates\") pod \"prometheus-operator-admission-webhook-5864cd99d7-rcq4t\" (UID: \"ff22f15f-83ae-4530-81ff-6d81caaf1ddb\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-rcq4t"
Sep 16 18:42:07.446559 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:42:07.446530    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/ff22f15f-83ae-4530-81ff-6d81caaf1ddb-tls-certificates\") pod \"prometheus-operator-admission-webhook-5864cd99d7-rcq4t\" (UID: \"ff22f15f-83ae-4530-81ff-6d81caaf1ddb\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-rcq4t"
Sep 16 18:42:07.446702 ip-10-0-135-119 kubenswrapper[2578]: E0916 18:42:07.446638    2578 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-admission-webhook-tls: secret "prometheus-operator-admission-webhook-tls" not found
Sep 16 18:42:07.446702 ip-10-0-135-119 kubenswrapper[2578]: E0916 18:42:07.446691    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ff22f15f-83ae-4530-81ff-6d81caaf1ddb-tls-certificates podName:ff22f15f-83ae-4530-81ff-6d81caaf1ddb nodeName:}" failed. No retries permitted until 2026-09-16 18:42:07.946677771 +0000 UTC m=+109.013169237 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certificates" (UniqueName: "kubernetes.io/secret/ff22f15f-83ae-4530-81ff-6d81caaf1ddb-tls-certificates") pod "prometheus-operator-admission-webhook-5864cd99d7-rcq4t" (UID: "ff22f15f-83ae-4530-81ff-6d81caaf1ddb") : secret "prometheus-operator-admission-webhook-tls" not found
Sep 16 18:42:07.606437 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:42:07.606400    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 18:42:07.950393 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:42:07.950352    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/ff22f15f-83ae-4530-81ff-6d81caaf1ddb-tls-certificates\") pod \"prometheus-operator-admission-webhook-5864cd99d7-rcq4t\" (UID: \"ff22f15f-83ae-4530-81ff-6d81caaf1ddb\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-rcq4t"
Sep 16 18:42:07.952836 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:42:07.952809    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/ff22f15f-83ae-4530-81ff-6d81caaf1ddb-tls-certificates\") pod \"prometheus-operator-admission-webhook-5864cd99d7-rcq4t\" (UID: \"ff22f15f-83ae-4530-81ff-6d81caaf1ddb\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-rcq4t"
Sep 16 18:42:07.982288 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:42:07.982261    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-vpkvv" event={"ID":"0ef1438e-2929-4fc5-9437-6e5b97e79006","Type":"ContainerStarted","Data":"ec6236fadafbf13724af7297682cc4475cdb241b40d7450da2483f506b8a7dbe"}
Sep 16 18:42:07.982408 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:42:07.982380    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-vpkvv"
Sep 16 18:42:07.984227 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:42:07.984204    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-9kn7r" event={"ID":"2e9f2573-3279-4102-9777-5ff8f209ed6a","Type":"ContainerStarted","Data":"38fc2d2d3c5ff7648862517b78aa4a4c7185b8131c4588fe56a6a917abbfc86a"}
Sep 16 18:42:07.984311 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:42:07.984246    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-9kn7r" event={"ID":"2e9f2573-3279-4102-9777-5ff8f209ed6a","Type":"ContainerStarted","Data":"38a8cb3e27da5b495c74ae19adcf365da4cacb5018f299e3b490e791eaacf809"}
Sep 16 18:42:07.984311 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:42:07.984268    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-9kn7r" event={"ID":"2e9f2573-3279-4102-9777-5ff8f209ed6a","Type":"ContainerStarted","Data":"6619cef81a8b13fe4c17b269f8a48c31d905e3ffe1cb0dd5324b3471cca8e495"}
Sep 16 18:42:08.004851 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:42:08.004806    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-vpkvv" podStartSLOduration=66.516282741 podStartE2EDuration="1m10.004793375s" totalImagesPullingTime="3.488510634s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:40:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:42:08.004786787 +0000 UTC m=+109.071278287" watchObservedRunningTime="2026-09-16 18:42:08.004793375 +0000 UTC m=+109.071284863"
Sep 16 18:42:08.153319 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:42:08.153296    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-rcq4t"
Sep 16 18:42:08.422885 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:42:08.422801    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-rcq4t"]
Sep 16 18:42:08.425836 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:42:08.425806    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podff22f15f_83ae_4530_81ff_6d81caaf1ddb.slice/crio-4375e11a8f1c24638f31401737b46768791faf40de854f44ba8d92255dc6b336 WatchSource:0}: Error finding container 4375e11a8f1c24638f31401737b46768791faf40de854f44ba8d92255dc6b336: Status 404 returned error can't find the container with id 4375e11a8f1c24638f31401737b46768791faf40de854f44ba8d92255dc6b336
Sep 16 18:42:08.989322 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:42:08.989281    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-rcq4t" event={"ID":"ff22f15f-83ae-4530-81ff-6d81caaf1ddb","Type":"ContainerStarted","Data":"4375e11a8f1c24638f31401737b46768791faf40de854f44ba8d92255dc6b336"}
Sep 16 18:42:09.996293 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:42:09.996252    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-rcq4t" event={"ID":"ff22f15f-83ae-4530-81ff-6d81caaf1ddb","Type":"ContainerStarted","Data":"cf57f50781f0bc90067179afdf29729f06f93042c11684b9b58693acfecb253a"}
Sep 16 18:42:09.996859 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:42:09.996317    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-rcq4t"
Sep 16 18:42:09.998478 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:42:09.998447    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-9kn7r" event={"ID":"2e9f2573-3279-4102-9777-5ff8f209ed6a","Type":"ContainerStarted","Data":"fe4af1e42c6c83379d31d77f1179af3afc4d51f0095b2a05564cf049c64f0df1"}
Sep 16 18:42:10.001379 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:42:10.001359    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-rcq4t"
Sep 16 18:42:10.013392 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:42:10.013347    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-rcq4t" podStartSLOduration=1.8566059419999998 podStartE2EDuration="3.013334337s" totalImagesPullingTime="1.156728395s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:42:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:42:10.012876205 +0000 UTC m=+111.079367703" watchObservedRunningTime="2026-09-16 18:42:10.013334337 +0000 UTC m=+111.079825824"
Sep 16 18:42:10.053533 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:42:10.051474    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-9kn7r" podStartSLOduration=35.254338329 podStartE2EDuration="37.051455518s" totalImagesPullingTime="1.797117189s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 18:41:33 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:42:10.049646365 +0000 UTC m=+111.116137904" watchObservedRunningTime="2026-09-16 18:42:10.051455518 +0000 UTC m=+111.117947008"
Sep 16 18:42:10.246242 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:42:10.246213    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 18:42:10.320335 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:42:10.320262    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-68bcd7f987-4ncz7"]
Sep 16 18:42:10.322815 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:42:10.322799    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-68bcd7f987-4ncz7"
Sep 16 18:42:10.325937 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:42:10.325915    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Sep 16 18:42:10.326066 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:42:10.325918    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\""
Sep 16 18:42:10.326291 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:42:10.326276    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Sep 16 18:42:10.326340 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:42:10.326302    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Sep 16 18:42:10.326550 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:42:10.326532    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-vxsxh\""
Sep 16 18:42:10.330573 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:42:10.330554    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\""
Sep 16 18:42:10.338468 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:42:10.338449    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-68bcd7f987-4ncz7"]
Sep 16 18:42:10.358245 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:42:10.358212    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 18:42:10.459243 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:42:10.459205    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 18:42:10.470284 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:42:10.470252    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/7254fad8-d179-4cfb-a7eb-2cd5a87578fa-prometheus-operator-tls\") pod \"prometheus-operator-68bcd7f987-4ncz7\" (UID: \"7254fad8-d179-4cfb-a7eb-2cd5a87578fa\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-4ncz7"
Sep 16 18:42:10.470406 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:42:10.470296    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w2kw9\" (UniqueName: \"kubernetes.io/projected/7254fad8-d179-4cfb-a7eb-2cd5a87578fa-kube-api-access-w2kw9\") pod \"prometheus-operator-68bcd7f987-4ncz7\" (UID: \"7254fad8-d179-4cfb-a7eb-2cd5a87578fa\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-4ncz7"
Sep 16 18:42:10.470406 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:42:10.470397    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/7254fad8-d179-4cfb-a7eb-2cd5a87578fa-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-68bcd7f987-4ncz7\" (UID: \"7254fad8-d179-4cfb-a7eb-2cd5a87578fa\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-4ncz7"
Sep 16 18:42:10.470485 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:42:10.470433    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7254fad8-d179-4cfb-a7eb-2cd5a87578fa-metrics-client-ca\") pod \"prometheus-operator-68bcd7f987-4ncz7\" (UID: \"7254fad8-d179-4cfb-a7eb-2cd5a87578fa\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-4ncz7"
Sep 16 18:42:10.571787 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:42:10.571695    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/7254fad8-d179-4cfb-a7eb-2cd5a87578fa-prometheus-operator-tls\") pod \"prometheus-operator-68bcd7f987-4ncz7\" (UID: \"7254fad8-d179-4cfb-a7eb-2cd5a87578fa\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-4ncz7"
Sep 16 18:42:10.571787 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:42:10.571754    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w2kw9\" (UniqueName: \"kubernetes.io/projected/7254fad8-d179-4cfb-a7eb-2cd5a87578fa-kube-api-access-w2kw9\") pod \"prometheus-operator-68bcd7f987-4ncz7\" (UID: \"7254fad8-d179-4cfb-a7eb-2cd5a87578fa\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-4ncz7"
Sep 16 18:42:10.572022 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:42:10.571817    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/7254fad8-d179-4cfb-a7eb-2cd5a87578fa-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-68bcd7f987-4ncz7\" (UID: \"7254fad8-d179-4cfb-a7eb-2cd5a87578fa\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-4ncz7"
Sep 16 18:42:10.572022 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:42:10.571856    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7254fad8-d179-4cfb-a7eb-2cd5a87578fa-metrics-client-ca\") pod \"prometheus-operator-68bcd7f987-4ncz7\" (UID: \"7254fad8-d179-4cfb-a7eb-2cd5a87578fa\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-4ncz7"
Sep 16 18:42:10.572603 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:42:10.572537    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7254fad8-d179-4cfb-a7eb-2cd5a87578fa-metrics-client-ca\") pod \"prometheus-operator-68bcd7f987-4ncz7\" (UID: \"7254fad8-d179-4cfb-a7eb-2cd5a87578fa\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-4ncz7"
Sep 16 18:42:10.574439 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:42:10.574417    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/7254fad8-d179-4cfb-a7eb-2cd5a87578fa-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-68bcd7f987-4ncz7\" (UID: \"7254fad8-d179-4cfb-a7eb-2cd5a87578fa\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-4ncz7"
Sep 16 18:42:10.574517 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:42:10.574470    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/7254fad8-d179-4cfb-a7eb-2cd5a87578fa-prometheus-operator-tls\") pod \"prometheus-operator-68bcd7f987-4ncz7\" (UID: \"7254fad8-d179-4cfb-a7eb-2cd5a87578fa\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-4ncz7"
Sep 16 18:42:10.581848 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:42:10.581829    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w2kw9\" (UniqueName: \"kubernetes.io/projected/7254fad8-d179-4cfb-a7eb-2cd5a87578fa-kube-api-access-w2kw9\") pod \"prometheus-operator-68bcd7f987-4ncz7\" (UID: \"7254fad8-d179-4cfb-a7eb-2cd5a87578fa\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-4ncz7"
Sep 16 18:42:10.632764 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:42:10.632732    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-68bcd7f987-4ncz7"
Sep 16 18:42:10.907145 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:42:10.907117    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-68bcd7f987-4ncz7"]
Sep 16 18:42:10.909780 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:42:10.909754    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7254fad8_d179_4cfb_a7eb_2cd5a87578fa.slice/crio-9adc00f1f003bd28f0f47f245f6101b709b7e567c32e4e58e8b5fb07d223069a WatchSource:0}: Error finding container 9adc00f1f003bd28f0f47f245f6101b709b7e567c32e4e58e8b5fb07d223069a: Status 404 returned error can't find the container with id 9adc00f1f003bd28f0f47f245f6101b709b7e567c32e4e58e8b5fb07d223069a
Sep 16 18:42:11.003066 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:42:11.003034    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-68bcd7f987-4ncz7" event={"ID":"7254fad8-d179-4cfb-a7eb-2cd5a87578fa","Type":"ContainerStarted","Data":"9adc00f1f003bd28f0f47f245f6101b709b7e567c32e4e58e8b5fb07d223069a"}
Sep 16 18:42:11.375907 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:42:11.375824    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 18:42:11.429173 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:42:11.429146    2578 scope.go:117] "RemoveContainer" containerID="8f38b1af4975884bac5c76bc8f8a028e6d32d0bc4be3883c8dd768ea64e92644"
Sep 16 18:42:11.429341 ip-10-0-135-119 kubenswrapper[2578]: E0916 18:42:11.429320    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-696884fb9c-tmrkx_openshift-console-operator(4c2078bb-281a-4aa1-8e38-238897865434)\"" pod="openshift-console-operator/console-operator-696884fb9c-tmrkx" podUID="4c2078bb-281a-4aa1-8e38-238897865434"
Sep 16 18:42:11.479435 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:42:11.479391    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 18:42:12.402220 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:42:12.402190    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 18:42:13.012724 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:42:13.012687    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-68bcd7f987-4ncz7" event={"ID":"7254fad8-d179-4cfb-a7eb-2cd5a87578fa","Type":"ContainerStarted","Data":"c4af732cb93560a854583164a3e5775443a87e5413a3bcab08fc7cfd2353f160"}
Sep 16 18:42:13.012888 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:42:13.012735    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-68bcd7f987-4ncz7" event={"ID":"7254fad8-d179-4cfb-a7eb-2cd5a87578fa","Type":"ContainerStarted","Data":"fa42882175e1f6f39a07b678744ab7fa91497ef94e2ebebb963d94778d48f9d5"}
Sep 16 18:42:13.039170 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:42:13.039127    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-68bcd7f987-4ncz7" podStartSLOduration=1.843907511 podStartE2EDuration="3.039112843s" totalImagesPullingTime="1.195205332s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:42:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:42:13.038296918 +0000 UTC m=+114.104788435" watchObservedRunningTime="2026-09-16 18:42:13.039112843 +0000 UTC m=+114.105604351"
Sep 16 18:42:13.681208 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:42:13.681178    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 18:42:13.757378 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:42:13.757348    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 18:42:14.430062 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:42:14.430023    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 18:42:14.722205 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:42:14.722113    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 18:42:14.880611 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:42:14.880570    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 18:42:14.923247 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:42:14.923209    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-67dcf8c979-8vxlk"]
Sep 16 18:42:14.926591 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:42:14.926569    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-8vxlk"
Sep 16 18:42:14.928763 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:42:14.928738    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\""
Sep 16 18:42:14.929031 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:42:14.929015    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-cb5xd\""
Sep 16 18:42:14.929260 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:42:14.929243    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\""
Sep 16 18:42:14.937476 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:42:14.937434    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-wsn6q"]
Sep 16 18:42:14.940361 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:42:14.940340    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-wsn6q"
Sep 16 18:42:14.941463 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:42:14.941445    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-67dcf8c979-8vxlk"]
Sep 16 18:42:14.942791 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:42:14.942774    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-t5v59\""
Sep 16 18:42:14.942906 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:42:14.942887    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Sep 16 18:42:14.942970 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:42:14.942779    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Sep 16 18:42:14.943167 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:42:14.943151    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Sep 16 18:42:14.996817 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:42:14.996709    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 18:42:15.011175 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:42:15.011150    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/88c01bb5-1e07-406b-bc3a-b7e54f3b8c23-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-67dcf8c979-8vxlk\" (UID: \"88c01bb5-1e07-406b-bc3a-b7e54f3b8c23\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-8vxlk"
Sep 16 18:42:15.011338 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:42:15.011195    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/b38e139a-817f-4138-8f91-70823ef862aa-node-exporter-wtmp\") pod \"node-exporter-wsn6q\" (UID: \"b38e139a-817f-4138-8f91-70823ef862aa\") " pod="openshift-monitoring/node-exporter-wsn6q"
Sep 16 18:42:15.011338 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:42:15.011224    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ssqj7\" (UniqueName: \"kubernetes.io/projected/88c01bb5-1e07-406b-bc3a-b7e54f3b8c23-kube-api-access-ssqj7\") pod \"openshift-state-metrics-67dcf8c979-8vxlk\" (UID: \"88c01bb5-1e07-406b-bc3a-b7e54f3b8c23\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-8vxlk"
Sep 16 18:42:15.011338 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:42:15.011240    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-275qk\" (UniqueName: \"kubernetes.io/projected/b38e139a-817f-4138-8f91-70823ef862aa-kube-api-access-275qk\") pod \"node-exporter-wsn6q\" (UID: \"b38e139a-817f-4138-8f91-70823ef862aa\") " pod="openshift-monitoring/node-exporter-wsn6q"
Sep 16 18:42:15.011338 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:42:15.011268    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/88c01bb5-1e07-406b-bc3a-b7e54f3b8c23-openshift-state-metrics-tls\") pod \"openshift-state-metrics-67dcf8c979-8vxlk\" (UID: \"88c01bb5-1e07-406b-bc3a-b7e54f3b8c23\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-8vxlk"
Sep 16 18:42:15.011338 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:42:15.011291    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b38e139a-817f-4138-8f91-70823ef862aa-sys\") pod \"node-exporter-wsn6q\" (UID: \"b38e139a-817f-4138-8f91-70823ef862aa\") " pod="openshift-monitoring/node-exporter-wsn6q"
Sep 16 18:42:15.011338 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:42:15.011311    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b38e139a-817f-4138-8f91-70823ef862aa-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-wsn6q\" (UID: \"b38e139a-817f-4138-8f91-70823ef862aa\") " pod="openshift-monitoring/node-exporter-wsn6q"
Sep 16 18:42:15.011338 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:42:15.011335    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/b38e139a-817f-4138-8f91-70823ef862aa-node-exporter-accelerators-collector-config\") pod \"node-exporter-wsn6q\" (UID: \"b38e139a-817f-4138-8f91-70823ef862aa\") " pod="openshift-monitoring/node-exporter-wsn6q"
Sep 16 18:42:15.011567 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:42:15.011406    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/b38e139a-817f-4138-8f91-70823ef862aa-node-exporter-tls\") pod \"node-exporter-wsn6q\" (UID: \"b38e139a-817f-4138-8f91-70823ef862aa\") " pod="openshift-monitoring/node-exporter-wsn6q"
Sep 16 18:42:15.011567 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:42:15.011444    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/88c01bb5-1e07-406b-bc3a-b7e54f3b8c23-metrics-client-ca\") pod \"openshift-state-metrics-67dcf8c979-8vxlk\" (UID: \"88c01bb5-1e07-406b-bc3a-b7e54f3b8c23\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-8vxlk"
Sep 16 18:42:15.011567 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:42:15.011465    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/b38e139a-817f-4138-8f91-70823ef862aa-node-exporter-textfile\") pod \"node-exporter-wsn6q\" (UID: \"b38e139a-817f-4138-8f91-70823ef862aa\") " pod="openshift-monitoring/node-exporter-wsn6q"
Sep 16 18:42:15.011567 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:42:15.011483    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b38e139a-817f-4138-8f91-70823ef862aa-metrics-client-ca\") pod \"node-exporter-wsn6q\" (UID: \"b38e139a-817f-4138-8f91-70823ef862aa\") " pod="openshift-monitoring/node-exporter-wsn6q"
Sep 16 18:42:15.011567 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:42:15.011506    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/b38e139a-817f-4138-8f91-70823ef862aa-root\") pod \"node-exporter-wsn6q\" (UID: \"b38e139a-817f-4138-8f91-70823ef862aa\") " pod="openshift-monitoring/node-exporter-wsn6q"
Sep 16 18:42:15.111321 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:42:15.111292    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 18:42:15.112363 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:42:15.112337    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ssqj7\" (UniqueName: \"kubernetes.io/projected/88c01bb5-1e07-406b-bc3a-b7e54f3b8c23-kube-api-access-ssqj7\") pod \"openshift-state-metrics-67dcf8c979-8vxlk\" (UID: \"88c01bb5-1e07-406b-bc3a-b7e54f3b8c23\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-8vxlk"
Sep 16 18:42:15.112491 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:42:15.112370    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-275qk\" (UniqueName: \"kubernetes.io/projected/b38e139a-817f-4138-8f91-70823ef862aa-kube-api-access-275qk\") pod \"node-exporter-wsn6q\" (UID: \"b38e139a-817f-4138-8f91-70823ef862aa\") " pod="openshift-monitoring/node-exporter-wsn6q"
Sep 16 18:42:15.112491 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:42:15.112394    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/88c01bb5-1e07-406b-bc3a-b7e54f3b8c23-openshift-state-metrics-tls\") pod \"openshift-state-metrics-67dcf8c979-8vxlk\" (UID: \"88c01bb5-1e07-406b-bc3a-b7e54f3b8c23\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-8vxlk"
Sep 16 18:42:15.112491 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:42:15.112417    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b38e139a-817f-4138-8f91-70823ef862aa-sys\") pod \"node-exporter-wsn6q\" (UID: \"b38e139a-817f-4138-8f91-70823ef862aa\") " pod="openshift-monitoring/node-exporter-wsn6q"
Sep 16 18:42:15.112491 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:42:15.112441    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b38e139a-817f-4138-8f91-70823ef862aa-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-wsn6q\" (UID: \"b38e139a-817f-4138-8f91-70823ef862aa\") " pod="openshift-monitoring/node-exporter-wsn6q"
Sep 16 18:42:15.112491 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:42:15.112467    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/b38e139a-817f-4138-8f91-70823ef862aa-node-exporter-accelerators-collector-config\") pod \"node-exporter-wsn6q\" (UID: \"b38e139a-817f-4138-8f91-70823ef862aa\") " pod="openshift-monitoring/node-exporter-wsn6q"
Sep 16 18:42:15.112753 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:42:15.112496    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/b38e139a-817f-4138-8f91-70823ef862aa-node-exporter-tls\") pod \"node-exporter-wsn6q\" (UID: \"b38e139a-817f-4138-8f91-70823ef862aa\") " pod="openshift-monitoring/node-exporter-wsn6q"
Sep 16 18:42:15.112753 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:42:15.112509    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b38e139a-817f-4138-8f91-70823ef862aa-sys\") pod \"node-exporter-wsn6q\" (UID: \"b38e139a-817f-4138-8f91-70823ef862aa\") " pod="openshift-monitoring/node-exporter-wsn6q"
Sep 16 18:42:15.112753 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:42:15.112527    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/88c01bb5-1e07-406b-bc3a-b7e54f3b8c23-metrics-client-ca\") pod \"openshift-state-metrics-67dcf8c979-8vxlk\" (UID: \"88c01bb5-1e07-406b-bc3a-b7e54f3b8c23\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-8vxlk"
Sep 16 18:42:15.112753 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:42:15.112575    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/b38e139a-817f-4138-8f91-70823ef862aa-node-exporter-textfile\") pod \"node-exporter-wsn6q\" (UID: \"b38e139a-817f-4138-8f91-70823ef862aa\") " pod="openshift-monitoring/node-exporter-wsn6q"
Sep 16 18:42:15.112753 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:42:15.112610    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b38e139a-817f-4138-8f91-70823ef862aa-metrics-client-ca\") pod \"node-exporter-wsn6q\" (UID: \"b38e139a-817f-4138-8f91-70823ef862aa\") " pod="openshift-monitoring/node-exporter-wsn6q"
Sep 16 18:42:15.112753 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:42:15.112638    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/b38e139a-817f-4138-8f91-70823ef862aa-root\") pod \"node-exporter-wsn6q\" (UID: \"b38e139a-817f-4138-8f91-70823ef862aa\") " pod="openshift-monitoring/node-exporter-wsn6q"
Sep 16 18:42:15.112753 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:42:15.112677    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/88c01bb5-1e07-406b-bc3a-b7e54f3b8c23-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-67dcf8c979-8vxlk\" (UID: \"88c01bb5-1e07-406b-bc3a-b7e54f3b8c23\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-8vxlk"
Sep 16 18:42:15.112753 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:42:15.112734    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/b38e139a-817f-4138-8f91-70823ef862aa-node-exporter-wtmp\") pod \"node-exporter-wsn6q\" (UID: \"b38e139a-817f-4138-8f91-70823ef862aa\") " pod="openshift-monitoring/node-exporter-wsn6q"
Sep 16 18:42:15.113128 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:42:15.112915    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/b38e139a-817f-4138-8f91-70823ef862aa-node-exporter-wtmp\") pod \"node-exporter-wsn6q\" (UID: \"b38e139a-817f-4138-8f91-70823ef862aa\") " pod="openshift-monitoring/node-exporter-wsn6q"
Sep 16 18:42:15.113128 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:42:15.112956    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/b38e139a-817f-4138-8f91-70823ef862aa-root\") pod \"node-exporter-wsn6q\" (UID: \"b38e139a-817f-4138-8f91-70823ef862aa\") " pod="openshift-monitoring/node-exporter-wsn6q"
Sep 16 18:42:15.113128 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:42:15.113095    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/b38e139a-817f-4138-8f91-70823ef862aa-node-exporter-textfile\") pod \"node-exporter-wsn6q\" (UID: \"b38e139a-817f-4138-8f91-70823ef862aa\") " pod="openshift-monitoring/node-exporter-wsn6q"
Sep 16 18:42:15.113449 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:42:15.113419    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/88c01bb5-1e07-406b-bc3a-b7e54f3b8c23-metrics-client-ca\") pod \"openshift-state-metrics-67dcf8c979-8vxlk\" (UID: \"88c01bb5-1e07-406b-bc3a-b7e54f3b8c23\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-8vxlk"
Sep 16 18:42:15.113923 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:42:15.113898    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/b38e139a-817f-4138-8f91-70823ef862aa-node-exporter-accelerators-collector-config\") pod \"node-exporter-wsn6q\" (UID: \"b38e139a-817f-4138-8f91-70823ef862aa\") " pod="openshift-monitoring/node-exporter-wsn6q"
Sep 16 18:42:15.114145 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:42:15.114123    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b38e139a-817f-4138-8f91-70823ef862aa-metrics-client-ca\") pod \"node-exporter-wsn6q\" (UID: \"b38e139a-817f-4138-8f91-70823ef862aa\") " pod="openshift-monitoring/node-exporter-wsn6q"
Sep 16 18:42:15.115081 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:42:15.115050    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b38e139a-817f-4138-8f91-70823ef862aa-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-wsn6q\" (UID: \"b38e139a-817f-4138-8f91-70823ef862aa\") " pod="openshift-monitoring/node-exporter-wsn6q"
Sep 16 18:42:15.115442 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:42:15.115424    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/88c01bb5-1e07-406b-bc3a-b7e54f3b8c23-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-67dcf8c979-8vxlk\" (UID: \"88c01bb5-1e07-406b-bc3a-b7e54f3b8c23\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-8vxlk"
Sep 16 18:42:15.115523 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:42:15.115457    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/88c01bb5-1e07-406b-bc3a-b7e54f3b8c23-openshift-state-metrics-tls\") pod \"openshift-state-metrics-67dcf8c979-8vxlk\" (UID: \"88c01bb5-1e07-406b-bc3a-b7e54f3b8c23\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-8vxlk"
Sep 16 18:42:15.115603 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:42:15.115585    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/b38e139a-817f-4138-8f91-70823ef862aa-node-exporter-tls\") pod \"node-exporter-wsn6q\" (UID: \"b38e139a-817f-4138-8f91-70823ef862aa\") " pod="openshift-monitoring/node-exporter-wsn6q"
Sep 16 18:42:15.124612 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:42:15.124576    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ssqj7\" (UniqueName: \"kubernetes.io/projected/88c01bb5-1e07-406b-bc3a-b7e54f3b8c23-kube-api-access-ssqj7\") pod \"openshift-state-metrics-67dcf8c979-8vxlk\" (UID: \"88c01bb5-1e07-406b-bc3a-b7e54f3b8c23\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-8vxlk"
Sep 16 18:42:15.125451 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:42:15.125430    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-275qk\" (UniqueName: \"kubernetes.io/projected/b38e139a-817f-4138-8f91-70823ef862aa-kube-api-access-275qk\") pod \"node-exporter-wsn6q\" (UID: \"b38e139a-817f-4138-8f91-70823ef862aa\") " pod="openshift-monitoring/node-exporter-wsn6q"
Sep 16 18:42:15.226251 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:42:15.226212    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 18:42:15.235959 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:42:15.235935    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-8vxlk"
Sep 16 18:42:15.251840 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:42:15.251779    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-wsn6q"
Sep 16 18:42:15.263335 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:42:15.263263    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb38e139a_817f_4138_8f91_70823ef862aa.slice/crio-9a2690c9f45b76f36135ca0a9d462e88da125901d03bc1cd4a875c59c917c220 WatchSource:0}: Error finding container 9a2690c9f45b76f36135ca0a9d462e88da125901d03bc1cd4a875c59c917c220: Status 404 returned error can't find the container with id 9a2690c9f45b76f36135ca0a9d462e88da125901d03bc1cd4a875c59c917c220
Sep 16 18:42:15.414567 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:42:15.414533    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 18:42:15.529401 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:42:15.529332    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-67dcf8c979-8vxlk"]
Sep 16 18:42:15.532230 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:42:15.532207    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 18:42:15.538236 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:42:15.538197    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod88c01bb5_1e07_406b_bc3a_b7e54f3b8c23.slice/crio-aff0c3fa793371911e9dd369b3c4b17020a6ab69fa816e615e8734f0b896d251 WatchSource:0}: Error finding container aff0c3fa793371911e9dd369b3c4b17020a6ab69fa816e615e8734f0b896d251: Status 404 returned error can't find the container with id aff0c3fa793371911e9dd369b3c4b17020a6ab69fa816e615e8734f0b896d251
Sep 16 18:42:15.754401 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:42:15.754360    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 18:42:15.947185 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:42:15.947151    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 18:42:16.024820 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:42:16.024782    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-wsn6q" event={"ID":"b38e139a-817f-4138-8f91-70823ef862aa","Type":"ContainerStarted","Data":"9a2690c9f45b76f36135ca0a9d462e88da125901d03bc1cd4a875c59c917c220"}
Sep 16 18:42:16.026906 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:42:16.026873    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-8vxlk" event={"ID":"88c01bb5-1e07-406b-bc3a-b7e54f3b8c23","Type":"ContainerStarted","Data":"7b49b7d0a0b4f75258a937ff09a8e0ba9b34ec31caf83830f4555cd3c2e276d4"}
Sep 16 18:42:16.027040 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:42:16.026933    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-8vxlk" event={"ID":"88c01bb5-1e07-406b-bc3a-b7e54f3b8c23","Type":"ContainerStarted","Data":"156e548357bd3276613cd661aed73ff8438100780fdcaf1b939689c6669d406a"}
Sep 16 18:42:16.027040 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:42:16.026955    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-8vxlk" event={"ID":"88c01bb5-1e07-406b-bc3a-b7e54f3b8c23","Type":"ContainerStarted","Data":"aff0c3fa793371911e9dd369b3c4b17020a6ab69fa816e615e8734f0b896d251"}
Sep 16 18:42:16.161443 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:42:16.161402    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 18:42:16.412335 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:42:16.412305    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 18:42:17.031510 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:42:17.031475    2578 generic.go:356] "Generic (PLEG): container finished" podID="b38e139a-817f-4138-8f91-70823ef862aa" containerID="4eca5f117a89e668d3f0f2f40cc4bf36c21359386f2bd3dcd760a768555dcbaa" exitCode=0
Sep 16 18:42:17.031911 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:42:17.031559    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-wsn6q" event={"ID":"b38e139a-817f-4138-8f91-70823ef862aa","Type":"ContainerDied","Data":"4eca5f117a89e668d3f0f2f40cc4bf36c21359386f2bd3dcd760a768555dcbaa"}
Sep 16 18:42:17.033810 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:42:17.033777    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-8vxlk" event={"ID":"88c01bb5-1e07-406b-bc3a-b7e54f3b8c23","Type":"ContainerStarted","Data":"2bf0ab744c706ed08b4104227c061f2da1fe2002b25ea385ac0971fa7e4e40ca"}
Sep 16 18:42:17.108335 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:42:17.108285    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-8vxlk" podStartSLOduration=1.9285016179999999 podStartE2EDuration="3.108265061s" totalImagesPullingTime="1.179763443s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:42:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:42:17.107391651 +0000 UTC m=+118.173883171" watchObservedRunningTime="2026-09-16 18:42:17.108265061 +0000 UTC m=+118.174756546"
Sep 16 18:42:17.992360 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:42:17.992330    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-vpkvv"
Sep 16 18:42:18.039811 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:42:18.039775    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-wsn6q" event={"ID":"b38e139a-817f-4138-8f91-70823ef862aa","Type":"ContainerStarted","Data":"e633ee71864364bb9b026dac42acb12e898a2d5eb323aec7c769ec028950567c"}
Sep 16 18:42:18.040285 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:42:18.039831    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-wsn6q" event={"ID":"b38e139a-817f-4138-8f91-70823ef862aa","Type":"ContainerStarted","Data":"efbec38feb38ab1d4fe22705c9f51104e376e0fa2b81cd5226a3428fe1b6840d"}
Sep 16 18:42:18.444112 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:42:18.444083    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 18:42:18.597299 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:42:18.597263    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 18:42:18.719503 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:42:18.719419    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 18:42:19.510570 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:42:19.510541    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 18:42:19.634832 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:42:19.634776    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-wsn6q" podStartSLOduration=4.439983729 podStartE2EDuration="5.63476059s" totalImagesPullingTime="1.114591388s" totalInitContainerRuntime="80.185473ms" isStatefulPod=true podCreationTimestamp="2026-09-16 18:42:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:42:18.088430781 +0000 UTC m=+119.154922314" watchObservedRunningTime="2026-09-16 18:42:19.63476059 +0000 UTC m=+120.701252078"
Sep 16 18:42:19.637368 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:42:19.635757    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-67669cdfc8-hcgrd"]
Sep 16 18:42:19.640235 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:42:19.640217    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-67669cdfc8-hcgrd"
Sep 16 18:42:19.643379 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:42:19.643354    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\""
Sep 16 18:42:19.643517 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:42:19.643357    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\""
Sep 16 18:42:19.643517 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:42:19.643377    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Sep 16 18:42:19.643517 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:42:19.643390    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-8jvqn\""
Sep 16 18:42:19.643686 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:42:19.643393    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\""
Sep 16 18:42:19.643686 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:42:19.643391    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-96n622fv8ae0a\""
Sep 16 18:42:19.658086 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:42:19.658066    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-67669cdfc8-hcgrd"]
Sep 16 18:42:19.746469 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:42:19.746434    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cvnqw\" (UniqueName: \"kubernetes.io/projected/ab57f3d7-1613-44da-b990-5e4d91689278-kube-api-access-cvnqw\") pod \"metrics-server-67669cdfc8-hcgrd\" (UID: \"ab57f3d7-1613-44da-b990-5e4d91689278\") " pod="openshift-monitoring/metrics-server-67669cdfc8-hcgrd"
Sep 16 18:42:19.746469 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:42:19.746468    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ab57f3d7-1613-44da-b990-5e4d91689278-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-67669cdfc8-hcgrd\" (UID: \"ab57f3d7-1613-44da-b990-5e4d91689278\") " pod="openshift-monitoring/metrics-server-67669cdfc8-hcgrd"
Sep 16 18:42:19.746670 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:42:19.746488    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/ab57f3d7-1613-44da-b990-5e4d91689278-metrics-server-audit-profiles\") pod \"metrics-server-67669cdfc8-hcgrd\" (UID: \"ab57f3d7-1613-44da-b990-5e4d91689278\") " pod="openshift-monitoring/metrics-server-67669cdfc8-hcgrd"
Sep 16 18:42:19.746670 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:42:19.746516    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/ab57f3d7-1613-44da-b990-5e4d91689278-client-ca-bundle\") pod \"metrics-server-67669cdfc8-hcgrd\" (UID: \"ab57f3d7-1613-44da-b990-5e4d91689278\") " pod="openshift-monitoring/metrics-server-67669cdfc8-hcgrd"
Sep 16 18:42:19.746670 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:42:19.746636    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/ab57f3d7-1613-44da-b990-5e4d91689278-secret-metrics-server-tls\") pod \"metrics-server-67669cdfc8-hcgrd\" (UID: \"ab57f3d7-1613-44da-b990-5e4d91689278\") " pod="openshift-monitoring/metrics-server-67669cdfc8-hcgrd"
Sep 16 18:42:19.746765 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:42:19.746672    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/ab57f3d7-1613-44da-b990-5e4d91689278-secret-metrics-server-client-certs\") pod \"metrics-server-67669cdfc8-hcgrd\" (UID: \"ab57f3d7-1613-44da-b990-5e4d91689278\") " pod="openshift-monitoring/metrics-server-67669cdfc8-hcgrd"
Sep 16 18:42:19.746765 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:42:19.746735    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/ab57f3d7-1613-44da-b990-5e4d91689278-audit-log\") pod \"metrics-server-67669cdfc8-hcgrd\" (UID: \"ab57f3d7-1613-44da-b990-5e4d91689278\") " pod="openshift-monitoring/metrics-server-67669cdfc8-hcgrd"
Sep 16 18:42:19.807267 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:42:19.807183    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 18:42:19.847360 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:42:19.847328    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/ab57f3d7-1613-44da-b990-5e4d91689278-audit-log\") pod \"metrics-server-67669cdfc8-hcgrd\" (UID: \"ab57f3d7-1613-44da-b990-5e4d91689278\") " pod="openshift-monitoring/metrics-server-67669cdfc8-hcgrd"
Sep 16 18:42:19.847476 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:42:19.847368    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cvnqw\" (UniqueName: \"kubernetes.io/projected/ab57f3d7-1613-44da-b990-5e4d91689278-kube-api-access-cvnqw\") pod \"metrics-server-67669cdfc8-hcgrd\" (UID: \"ab57f3d7-1613-44da-b990-5e4d91689278\") " pod="openshift-monitoring/metrics-server-67669cdfc8-hcgrd"
Sep 16 18:42:19.847476 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:42:19.847388    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ab57f3d7-1613-44da-b990-5e4d91689278-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-67669cdfc8-hcgrd\" (UID: \"ab57f3d7-1613-44da-b990-5e4d91689278\") " pod="openshift-monitoring/metrics-server-67669cdfc8-hcgrd"
Sep 16 18:42:19.847476 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:42:19.847406    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/ab57f3d7-1613-44da-b990-5e4d91689278-metrics-server-audit-profiles\") pod \"metrics-server-67669cdfc8-hcgrd\" (UID: \"ab57f3d7-1613-44da-b990-5e4d91689278\") " pod="openshift-monitoring/metrics-server-67669cdfc8-hcgrd"
Sep 16 18:42:19.847476 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:42:19.847424    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/ab57f3d7-1613-44da-b990-5e4d91689278-client-ca-bundle\") pod \"metrics-server-67669cdfc8-hcgrd\" (UID: \"ab57f3d7-1613-44da-b990-5e4d91689278\") " pod="openshift-monitoring/metrics-server-67669cdfc8-hcgrd"
Sep 16 18:42:19.847476 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:42:19.847471    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/ab57f3d7-1613-44da-b990-5e4d91689278-secret-metrics-server-tls\") pod \"metrics-server-67669cdfc8-hcgrd\" (UID: \"ab57f3d7-1613-44da-b990-5e4d91689278\") " pod="openshift-monitoring/metrics-server-67669cdfc8-hcgrd"
Sep 16 18:42:19.847729 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:42:19.847495    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/ab57f3d7-1613-44da-b990-5e4d91689278-secret-metrics-server-client-certs\") pod \"metrics-server-67669cdfc8-hcgrd\" (UID: \"ab57f3d7-1613-44da-b990-5e4d91689278\") " pod="openshift-monitoring/metrics-server-67669cdfc8-hcgrd"
Sep 16 18:42:19.847813 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:42:19.847791    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/ab57f3d7-1613-44da-b990-5e4d91689278-audit-log\") pod \"metrics-server-67669cdfc8-hcgrd\" (UID: \"ab57f3d7-1613-44da-b990-5e4d91689278\") " pod="openshift-monitoring/metrics-server-67669cdfc8-hcgrd"
Sep 16 18:42:19.848173 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:42:19.848150    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ab57f3d7-1613-44da-b990-5e4d91689278-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-67669cdfc8-hcgrd\" (UID: \"ab57f3d7-1613-44da-b990-5e4d91689278\") " pod="openshift-monitoring/metrics-server-67669cdfc8-hcgrd"
Sep 16 18:42:19.848851 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:42:19.848822    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/ab57f3d7-1613-44da-b990-5e4d91689278-metrics-server-audit-profiles\") pod \"metrics-server-67669cdfc8-hcgrd\" (UID: \"ab57f3d7-1613-44da-b990-5e4d91689278\") " pod="openshift-monitoring/metrics-server-67669cdfc8-hcgrd"
Sep 16 18:42:19.850158 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:42:19.850135    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/ab57f3d7-1613-44da-b990-5e4d91689278-client-ca-bundle\") pod \"metrics-server-67669cdfc8-hcgrd\" (UID: \"ab57f3d7-1613-44da-b990-5e4d91689278\") " pod="openshift-monitoring/metrics-server-67669cdfc8-hcgrd"
Sep 16 18:42:19.850259 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:42:19.850169    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/ab57f3d7-1613-44da-b990-5e4d91689278-secret-metrics-server-tls\") pod \"metrics-server-67669cdfc8-hcgrd\" (UID: \"ab57f3d7-1613-44da-b990-5e4d91689278\") " pod="openshift-monitoring/metrics-server-67669cdfc8-hcgrd"
Sep 16 18:42:19.850301 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:42:19.850260    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/ab57f3d7-1613-44da-b990-5e4d91689278-secret-metrics-server-client-certs\") pod \"metrics-server-67669cdfc8-hcgrd\" (UID: \"ab57f3d7-1613-44da-b990-5e4d91689278\") " pod="openshift-monitoring/metrics-server-67669cdfc8-hcgrd"
Sep 16 18:42:19.859034 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:42:19.859011    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cvnqw\" (UniqueName: \"kubernetes.io/projected/ab57f3d7-1613-44da-b990-5e4d91689278-kube-api-access-cvnqw\") pod \"metrics-server-67669cdfc8-hcgrd\" (UID: \"ab57f3d7-1613-44da-b990-5e4d91689278\") " pod="openshift-monitoring/metrics-server-67669cdfc8-hcgrd"
Sep 16 18:42:19.949120 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:42:19.949091    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-67669cdfc8-hcgrd"
Sep 16 18:42:20.228216 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:42:20.228189    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-67669cdfc8-hcgrd"]
Sep 16 18:42:20.230526 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:42:20.230496    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podab57f3d7_1613_44da_b990_5e4d91689278.slice/crio-f03b7696d1057740ad0f44e8800bac98f97c751713e475e0d02637fa300cd415 WatchSource:0}: Error finding container f03b7696d1057740ad0f44e8800bac98f97c751713e475e0d02637fa300cd415: Status 404 returned error can't find the container with id f03b7696d1057740ad0f44e8800bac98f97c751713e475e0d02637fa300cd415
Sep 16 18:42:20.246139 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:42:20.246112    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-655cdcc48d-764cf"]
Sep 16 18:42:20.250405 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:42:20.250386    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-655cdcc48d-764cf"
Sep 16 18:42:20.252931 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:42:20.252903    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\""
Sep 16 18:42:20.253045 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:42:20.252955    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\""
Sep 16 18:42:20.253045 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:42:20.253027    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\""
Sep 16 18:42:20.253147 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:42:20.253077    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\""
Sep 16 18:42:20.254852 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:42:20.254836    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-7l6j8\""
Sep 16 18:42:20.254852 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:42:20.254846    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\""
Sep 16 18:42:20.261632 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:42:20.261615    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\""
Sep 16 18:42:20.268562 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:42:20.268545    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-655cdcc48d-764cf"]
Sep 16 18:42:20.350200 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:42:20.350166    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f1465039-5e01-4402-84a7-56d706f786a6-telemeter-trusted-ca-bundle\") pod \"telemeter-client-655cdcc48d-764cf\" (UID: \"f1465039-5e01-4402-84a7-56d706f786a6\") " pod="openshift-monitoring/telemeter-client-655cdcc48d-764cf"
Sep 16 18:42:20.350200 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:42:20.350203    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hcrhg\" (UniqueName: \"kubernetes.io/projected/f1465039-5e01-4402-84a7-56d706f786a6-kube-api-access-hcrhg\") pod \"telemeter-client-655cdcc48d-764cf\" (UID: \"f1465039-5e01-4402-84a7-56d706f786a6\") " pod="openshift-monitoring/telemeter-client-655cdcc48d-764cf"
Sep 16 18:42:20.350418 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:42:20.350265    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f1465039-5e01-4402-84a7-56d706f786a6-serving-certs-ca-bundle\") pod \"telemeter-client-655cdcc48d-764cf\" (UID: \"f1465039-5e01-4402-84a7-56d706f786a6\") " pod="openshift-monitoring/telemeter-client-655cdcc48d-764cf"
Sep 16 18:42:20.350418 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:42:20.350306    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f1465039-5e01-4402-84a7-56d706f786a6-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-655cdcc48d-764cf\" (UID: \"f1465039-5e01-4402-84a7-56d706f786a6\") " pod="openshift-monitoring/telemeter-client-655cdcc48d-764cf"
Sep 16 18:42:20.350418 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:42:20.350368    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/f1465039-5e01-4402-84a7-56d706f786a6-secret-telemeter-client\") pod \"telemeter-client-655cdcc48d-764cf\" (UID: \"f1465039-5e01-4402-84a7-56d706f786a6\") " pod="openshift-monitoring/telemeter-client-655cdcc48d-764cf"
Sep 16 18:42:20.350418 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:42:20.350384    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/f1465039-5e01-4402-84a7-56d706f786a6-telemeter-client-tls\") pod \"telemeter-client-655cdcc48d-764cf\" (UID: \"f1465039-5e01-4402-84a7-56d706f786a6\") " pod="openshift-monitoring/telemeter-client-655cdcc48d-764cf"
Sep 16 18:42:20.350546 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:42:20.350417    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f1465039-5e01-4402-84a7-56d706f786a6-metrics-client-ca\") pod \"telemeter-client-655cdcc48d-764cf\" (UID: \"f1465039-5e01-4402-84a7-56d706f786a6\") " pod="openshift-monitoring/telemeter-client-655cdcc48d-764cf"
Sep 16 18:42:20.350546 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:42:20.350440    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/f1465039-5e01-4402-84a7-56d706f786a6-federate-client-tls\") pod \"telemeter-client-655cdcc48d-764cf\" (UID: \"f1465039-5e01-4402-84a7-56d706f786a6\") " pod="openshift-monitoring/telemeter-client-655cdcc48d-764cf"
Sep 16 18:42:20.451355 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:42:20.451319    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/f1465039-5e01-4402-84a7-56d706f786a6-secret-telemeter-client\") pod \"telemeter-client-655cdcc48d-764cf\" (UID: \"f1465039-5e01-4402-84a7-56d706f786a6\") " pod="openshift-monitoring/telemeter-client-655cdcc48d-764cf"
Sep 16 18:42:20.451355 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:42:20.451359    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/f1465039-5e01-4402-84a7-56d706f786a6-telemeter-client-tls\") pod \"telemeter-client-655cdcc48d-764cf\" (UID: \"f1465039-5e01-4402-84a7-56d706f786a6\") " pod="openshift-monitoring/telemeter-client-655cdcc48d-764cf"
Sep 16 18:42:20.451560 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:42:20.451395    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f1465039-5e01-4402-84a7-56d706f786a6-metrics-client-ca\") pod \"telemeter-client-655cdcc48d-764cf\" (UID: \"f1465039-5e01-4402-84a7-56d706f786a6\") " pod="openshift-monitoring/telemeter-client-655cdcc48d-764cf"
Sep 16 18:42:20.451560 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:42:20.451423    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/f1465039-5e01-4402-84a7-56d706f786a6-federate-client-tls\") pod \"telemeter-client-655cdcc48d-764cf\" (UID: \"f1465039-5e01-4402-84a7-56d706f786a6\") " pod="openshift-monitoring/telemeter-client-655cdcc48d-764cf"
Sep 16 18:42:20.451560 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:42:20.451476    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f1465039-5e01-4402-84a7-56d706f786a6-telemeter-trusted-ca-bundle\") pod \"telemeter-client-655cdcc48d-764cf\" (UID: \"f1465039-5e01-4402-84a7-56d706f786a6\") " pod="openshift-monitoring/telemeter-client-655cdcc48d-764cf"
Sep 16 18:42:20.451560 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:42:20.451498    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hcrhg\" (UniqueName: \"kubernetes.io/projected/f1465039-5e01-4402-84a7-56d706f786a6-kube-api-access-hcrhg\") pod \"telemeter-client-655cdcc48d-764cf\" (UID: \"f1465039-5e01-4402-84a7-56d706f786a6\") " pod="openshift-monitoring/telemeter-client-655cdcc48d-764cf"
Sep 16 18:42:20.451560 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:42:20.451551    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f1465039-5e01-4402-84a7-56d706f786a6-serving-certs-ca-bundle\") pod \"telemeter-client-655cdcc48d-764cf\" (UID: \"f1465039-5e01-4402-84a7-56d706f786a6\") " pod="openshift-monitoring/telemeter-client-655cdcc48d-764cf"
Sep 16 18:42:20.451778 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:42:20.451574    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f1465039-5e01-4402-84a7-56d706f786a6-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-655cdcc48d-764cf\" (UID: \"f1465039-5e01-4402-84a7-56d706f786a6\") " pod="openshift-monitoring/telemeter-client-655cdcc48d-764cf"
Sep 16 18:42:20.452648 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:42:20.452615    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f1465039-5e01-4402-84a7-56d706f786a6-serving-certs-ca-bundle\") pod \"telemeter-client-655cdcc48d-764cf\" (UID: \"f1465039-5e01-4402-84a7-56d706f786a6\") " pod="openshift-monitoring/telemeter-client-655cdcc48d-764cf"
Sep 16 18:42:20.452768 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:42:20.452682    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f1465039-5e01-4402-84a7-56d706f786a6-telemeter-trusted-ca-bundle\") pod \"telemeter-client-655cdcc48d-764cf\" (UID: \"f1465039-5e01-4402-84a7-56d706f786a6\") " pod="openshift-monitoring/telemeter-client-655cdcc48d-764cf"
Sep 16 18:42:20.452842 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:42:20.452772    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f1465039-5e01-4402-84a7-56d706f786a6-metrics-client-ca\") pod \"telemeter-client-655cdcc48d-764cf\" (UID: \"f1465039-5e01-4402-84a7-56d706f786a6\") " pod="openshift-monitoring/telemeter-client-655cdcc48d-764cf"
Sep 16 18:42:20.454249 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:42:20.454229    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/f1465039-5e01-4402-84a7-56d706f786a6-secret-telemeter-client\") pod \"telemeter-client-655cdcc48d-764cf\" (UID: \"f1465039-5e01-4402-84a7-56d706f786a6\") " pod="openshift-monitoring/telemeter-client-655cdcc48d-764cf"
Sep 16 18:42:20.454356 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:42:20.454293    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f1465039-5e01-4402-84a7-56d706f786a6-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-655cdcc48d-764cf\" (UID: \"f1465039-5e01-4402-84a7-56d706f786a6\") " pod="openshift-monitoring/telemeter-client-655cdcc48d-764cf"
Sep 16 18:42:20.454758 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:42:20.454743    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/f1465039-5e01-4402-84a7-56d706f786a6-telemeter-client-tls\") pod \"telemeter-client-655cdcc48d-764cf\" (UID: \"f1465039-5e01-4402-84a7-56d706f786a6\") " pod="openshift-monitoring/telemeter-client-655cdcc48d-764cf"
Sep 16 18:42:20.454803 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:42:20.454784    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/f1465039-5e01-4402-84a7-56d706f786a6-federate-client-tls\") pod \"telemeter-client-655cdcc48d-764cf\" (UID: \"f1465039-5e01-4402-84a7-56d706f786a6\") " pod="openshift-monitoring/telemeter-client-655cdcc48d-764cf"
Sep 16 18:42:20.461034 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:42:20.460991    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hcrhg\" (UniqueName: \"kubernetes.io/projected/f1465039-5e01-4402-84a7-56d706f786a6-kube-api-access-hcrhg\") pod \"telemeter-client-655cdcc48d-764cf\" (UID: \"f1465039-5e01-4402-84a7-56d706f786a6\") " pod="openshift-monitoring/telemeter-client-655cdcc48d-764cf"
Sep 16 18:42:20.559920 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:42:20.559839    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-655cdcc48d-764cf"
Sep 16 18:42:20.666602 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:42:20.666561    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 18:42:20.781522 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:42:20.781483    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 18:42:20.839758 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:42:20.839664    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-655cdcc48d-764cf"]
Sep 16 18:42:20.842867 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:42:20.842843    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf1465039_5e01_4402_84a7_56d706f786a6.slice/crio-d245ae81753bb4c998c694efbb7a9d7543340e3b4c103743202556a6c069edd1 WatchSource:0}: Error finding container d245ae81753bb4c998c694efbb7a9d7543340e3b4c103743202556a6c069edd1: Status 404 returned error can't find the container with id d245ae81753bb4c998c694efbb7a9d7543340e3b4c103743202556a6c069edd1
Sep 16 18:42:21.053058 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:42:21.053010    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-655cdcc48d-764cf" event={"ID":"f1465039-5e01-4402-84a7-56d706f786a6","Type":"ContainerStarted","Data":"d245ae81753bb4c998c694efbb7a9d7543340e3b4c103743202556a6c069edd1"}
Sep 16 18:42:21.054347 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:42:21.054312    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-67669cdfc8-hcgrd" event={"ID":"ab57f3d7-1613-44da-b990-5e4d91689278","Type":"ContainerStarted","Data":"f03b7696d1057740ad0f44e8800bac98f97c751713e475e0d02637fa300cd415"}
Sep 16 18:42:21.913108 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:42:21.913069    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 18:42:21.938526 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:42:21.938496    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 16 18:42:21.942362 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:42:21.942342    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:42:21.946546 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:42:21.946513    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Sep 16 18:42:21.946838 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:42:21.946815    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Sep 16 18:42:21.947063 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:42:21.946845    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Sep 16 18:42:21.947063 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:42:21.946886    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Sep 16 18:42:21.947063 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:42:21.946899    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Sep 16 18:42:21.947470 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:42:21.947317    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Sep 16 18:42:21.947470 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:42:21.947342    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Sep 16 18:42:21.947626 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:42:21.947470    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Sep 16 18:42:21.947626 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:42:21.947590    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Sep 16 18:42:21.947626 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:42:21.947596    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Sep 16 18:42:21.952839 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:42:21.952744    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Sep 16 18:42:21.954240 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:42:21.954222    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Sep 16 18:42:21.956149 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:42:21.956103    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-6o7c2r390kn4a\""
Sep 16 18:42:21.956149 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:42:21.956136    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Sep 16 18:42:21.956313 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:42:21.956171    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-cc7kj\""
Sep 16 18:42:21.959868 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:42:21.958322    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Sep 16 18:42:21.966507 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:42:21.966453    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/686f396c-6b8e-44a8-83b8-4a38dd7b5644-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"686f396c-6b8e-44a8-83b8-4a38dd7b5644\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:42:21.966507 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:42:21.966498    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/686f396c-6b8e-44a8-83b8-4a38dd7b5644-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"686f396c-6b8e-44a8-83b8-4a38dd7b5644\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:42:21.966694 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:42:21.966532    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/686f396c-6b8e-44a8-83b8-4a38dd7b5644-config-out\") pod \"prometheus-k8s-0\" (UID: \"686f396c-6b8e-44a8-83b8-4a38dd7b5644\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:42:21.966694 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:42:21.966593    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/686f396c-6b8e-44a8-83b8-4a38dd7b5644-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"686f396c-6b8e-44a8-83b8-4a38dd7b5644\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:42:21.966694 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:42:21.966639    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/686f396c-6b8e-44a8-83b8-4a38dd7b5644-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"686f396c-6b8e-44a8-83b8-4a38dd7b5644\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:42:21.966694 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:42:21.966670    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/686f396c-6b8e-44a8-83b8-4a38dd7b5644-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"686f396c-6b8e-44a8-83b8-4a38dd7b5644\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:42:21.966902 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:42:21.966714    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/686f396c-6b8e-44a8-83b8-4a38dd7b5644-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"686f396c-6b8e-44a8-83b8-4a38dd7b5644\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:42:21.966902 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:42:21.966748    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/686f396c-6b8e-44a8-83b8-4a38dd7b5644-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"686f396c-6b8e-44a8-83b8-4a38dd7b5644\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:42:21.966902 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:42:21.966797    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/686f396c-6b8e-44a8-83b8-4a38dd7b5644-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"686f396c-6b8e-44a8-83b8-4a38dd7b5644\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:42:21.966902 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:42:21.966836    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/686f396c-6b8e-44a8-83b8-4a38dd7b5644-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"686f396c-6b8e-44a8-83b8-4a38dd7b5644\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:42:21.966902 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:42:21.966887    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/686f396c-6b8e-44a8-83b8-4a38dd7b5644-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"686f396c-6b8e-44a8-83b8-4a38dd7b5644\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:42:21.967143 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:42:21.966916    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/686f396c-6b8e-44a8-83b8-4a38dd7b5644-config\") pod \"prometheus-k8s-0\" (UID: \"686f396c-6b8e-44a8-83b8-4a38dd7b5644\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:42:21.967143 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:42:21.966997    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/686f396c-6b8e-44a8-83b8-4a38dd7b5644-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"686f396c-6b8e-44a8-83b8-4a38dd7b5644\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:42:21.967143 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:42:21.967041    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/686f396c-6b8e-44a8-83b8-4a38dd7b5644-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"686f396c-6b8e-44a8-83b8-4a38dd7b5644\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:42:21.967143 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:42:21.967076    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/686f396c-6b8e-44a8-83b8-4a38dd7b5644-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"686f396c-6b8e-44a8-83b8-4a38dd7b5644\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:42:21.967143 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:42:21.967098    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/686f396c-6b8e-44a8-83b8-4a38dd7b5644-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"686f396c-6b8e-44a8-83b8-4a38dd7b5644\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:42:21.967333 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:42:21.967151    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7wm7v\" (UniqueName: \"kubernetes.io/projected/686f396c-6b8e-44a8-83b8-4a38dd7b5644-kube-api-access-7wm7v\") pod \"prometheus-k8s-0\" (UID: \"686f396c-6b8e-44a8-83b8-4a38dd7b5644\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:42:21.967333 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:42:21.967187    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/686f396c-6b8e-44a8-83b8-4a38dd7b5644-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"686f396c-6b8e-44a8-83b8-4a38dd7b5644\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:42:21.967333 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:42:21.967217    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/686f396c-6b8e-44a8-83b8-4a38dd7b5644-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"686f396c-6b8e-44a8-83b8-4a38dd7b5644\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:42:21.967333 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:42:21.967236    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/686f396c-6b8e-44a8-83b8-4a38dd7b5644-web-config\") pod \"prometheus-k8s-0\" (UID: \"686f396c-6b8e-44a8-83b8-4a38dd7b5644\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:42:21.980689 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:42:21.980661    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 16 18:42:22.061257 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:42:22.061170    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-67669cdfc8-hcgrd" event={"ID":"ab57f3d7-1613-44da-b990-5e4d91689278","Type":"ContainerStarted","Data":"7517c915f5c6a0c39d5c3ad00a4991f8d60bf08e4c2c9439784b81598910cc35"}
Sep 16 18:42:22.068412 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:42:22.068387    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/686f396c-6b8e-44a8-83b8-4a38dd7b5644-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"686f396c-6b8e-44a8-83b8-4a38dd7b5644\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:42:22.068552 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:42:22.068424    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/686f396c-6b8e-44a8-83b8-4a38dd7b5644-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"686f396c-6b8e-44a8-83b8-4a38dd7b5644\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:42:22.068552 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:42:22.068454    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/686f396c-6b8e-44a8-83b8-4a38dd7b5644-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"686f396c-6b8e-44a8-83b8-4a38dd7b5644\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:42:22.068552 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:42:22.068486    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/686f396c-6b8e-44a8-83b8-4a38dd7b5644-config\") pod \"prometheus-k8s-0\" (UID: \"686f396c-6b8e-44a8-83b8-4a38dd7b5644\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:42:22.068552 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:42:22.068538    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/686f396c-6b8e-44a8-83b8-4a38dd7b5644-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"686f396c-6b8e-44a8-83b8-4a38dd7b5644\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:42:22.068764 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:42:22.068569    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/686f396c-6b8e-44a8-83b8-4a38dd7b5644-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"686f396c-6b8e-44a8-83b8-4a38dd7b5644\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:42:22.068764 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:42:22.068593    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/686f396c-6b8e-44a8-83b8-4a38dd7b5644-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"686f396c-6b8e-44a8-83b8-4a38dd7b5644\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:42:22.068764 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:42:22.068613    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/686f396c-6b8e-44a8-83b8-4a38dd7b5644-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"686f396c-6b8e-44a8-83b8-4a38dd7b5644\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:42:22.068764 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:42:22.068644    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7wm7v\" (UniqueName: \"kubernetes.io/projected/686f396c-6b8e-44a8-83b8-4a38dd7b5644-kube-api-access-7wm7v\") pod \"prometheus-k8s-0\" (UID: \"686f396c-6b8e-44a8-83b8-4a38dd7b5644\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:42:22.068764 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:42:22.068684    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/686f396c-6b8e-44a8-83b8-4a38dd7b5644-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"686f396c-6b8e-44a8-83b8-4a38dd7b5644\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:42:22.069036 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:42:22.068950    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/686f396c-6b8e-44a8-83b8-4a38dd7b5644-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"686f396c-6b8e-44a8-83b8-4a38dd7b5644\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:42:22.069036 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:42:22.069008    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/686f396c-6b8e-44a8-83b8-4a38dd7b5644-web-config\") pod \"prometheus-k8s-0\" (UID: \"686f396c-6b8e-44a8-83b8-4a38dd7b5644\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:42:22.069147 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:42:22.069038    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/686f396c-6b8e-44a8-83b8-4a38dd7b5644-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"686f396c-6b8e-44a8-83b8-4a38dd7b5644\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:42:22.069147 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:42:22.069066    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/686f396c-6b8e-44a8-83b8-4a38dd7b5644-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"686f396c-6b8e-44a8-83b8-4a38dd7b5644\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:42:22.069147 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:42:22.069098    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/686f396c-6b8e-44a8-83b8-4a38dd7b5644-config-out\") pod \"prometheus-k8s-0\" (UID: \"686f396c-6b8e-44a8-83b8-4a38dd7b5644\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:42:22.069317 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:42:22.069177    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/686f396c-6b8e-44a8-83b8-4a38dd7b5644-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"686f396c-6b8e-44a8-83b8-4a38dd7b5644\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:42:22.069317 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:42:22.069268    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/686f396c-6b8e-44a8-83b8-4a38dd7b5644-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"686f396c-6b8e-44a8-83b8-4a38dd7b5644\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:42:22.069420 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:42:22.069325    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/686f396c-6b8e-44a8-83b8-4a38dd7b5644-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"686f396c-6b8e-44a8-83b8-4a38dd7b5644\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:42:22.069420 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:42:22.069348    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/686f396c-6b8e-44a8-83b8-4a38dd7b5644-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"686f396c-6b8e-44a8-83b8-4a38dd7b5644\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:42:22.069420 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:42:22.069378    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/686f396c-6b8e-44a8-83b8-4a38dd7b5644-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"686f396c-6b8e-44a8-83b8-4a38dd7b5644\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:42:22.069420 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:42:22.069400    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/686f396c-6b8e-44a8-83b8-4a38dd7b5644-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"686f396c-6b8e-44a8-83b8-4a38dd7b5644\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:42:22.069730 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:42:22.069699    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/686f396c-6b8e-44a8-83b8-4a38dd7b5644-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"686f396c-6b8e-44a8-83b8-4a38dd7b5644\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:42:22.070706 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:42:22.070679    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/686f396c-6b8e-44a8-83b8-4a38dd7b5644-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"686f396c-6b8e-44a8-83b8-4a38dd7b5644\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:42:22.071284 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:42:22.071258    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/686f396c-6b8e-44a8-83b8-4a38dd7b5644-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"686f396c-6b8e-44a8-83b8-4a38dd7b5644\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:42:22.071948 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:42:22.071922    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/686f396c-6b8e-44a8-83b8-4a38dd7b5644-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"686f396c-6b8e-44a8-83b8-4a38dd7b5644\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:42:22.072357 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:42:22.072307    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/686f396c-6b8e-44a8-83b8-4a38dd7b5644-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"686f396c-6b8e-44a8-83b8-4a38dd7b5644\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:42:22.072587 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:42:22.072562    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/686f396c-6b8e-44a8-83b8-4a38dd7b5644-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"686f396c-6b8e-44a8-83b8-4a38dd7b5644\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:42:22.072967 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:42:22.072945    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/686f396c-6b8e-44a8-83b8-4a38dd7b5644-config\") pod \"prometheus-k8s-0\" (UID: \"686f396c-6b8e-44a8-83b8-4a38dd7b5644\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:42:22.075123 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:42:22.074522    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/686f396c-6b8e-44a8-83b8-4a38dd7b5644-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"686f396c-6b8e-44a8-83b8-4a38dd7b5644\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:42:22.075123 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:42:22.074598    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/686f396c-6b8e-44a8-83b8-4a38dd7b5644-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"686f396c-6b8e-44a8-83b8-4a38dd7b5644\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:42:22.075123 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:42:22.075089    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/686f396c-6b8e-44a8-83b8-4a38dd7b5644-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"686f396c-6b8e-44a8-83b8-4a38dd7b5644\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:42:22.075352 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:42:22.075311    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/686f396c-6b8e-44a8-83b8-4a38dd7b5644-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"686f396c-6b8e-44a8-83b8-4a38dd7b5644\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:42:22.075643 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:42:22.075596    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/686f396c-6b8e-44a8-83b8-4a38dd7b5644-config-out\") pod \"prometheus-k8s-0\" (UID: \"686f396c-6b8e-44a8-83b8-4a38dd7b5644\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:42:22.075760 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:42:22.075738    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/686f396c-6b8e-44a8-83b8-4a38dd7b5644-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"686f396c-6b8e-44a8-83b8-4a38dd7b5644\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:42:22.075858 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:42:22.075838    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/686f396c-6b8e-44a8-83b8-4a38dd7b5644-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"686f396c-6b8e-44a8-83b8-4a38dd7b5644\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:42:22.077012 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:42:22.076967    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/686f396c-6b8e-44a8-83b8-4a38dd7b5644-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"686f396c-6b8e-44a8-83b8-4a38dd7b5644\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:42:22.077111 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:42:22.077040    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/686f396c-6b8e-44a8-83b8-4a38dd7b5644-web-config\") pod \"prometheus-k8s-0\" (UID: \"686f396c-6b8e-44a8-83b8-4a38dd7b5644\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:42:22.077639 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:42:22.077621    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/686f396c-6b8e-44a8-83b8-4a38dd7b5644-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"686f396c-6b8e-44a8-83b8-4a38dd7b5644\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:42:22.077773 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:42:22.077753    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/686f396c-6b8e-44a8-83b8-4a38dd7b5644-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"686f396c-6b8e-44a8-83b8-4a38dd7b5644\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:42:22.084988 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:42:22.084949    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7wm7v\" (UniqueName: \"kubernetes.io/projected/686f396c-6b8e-44a8-83b8-4a38dd7b5644-kube-api-access-7wm7v\") pod \"prometheus-k8s-0\" (UID: \"686f396c-6b8e-44a8-83b8-4a38dd7b5644\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:42:22.255744 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:42:22.255701    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:42:22.927952 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:42:22.927893    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-67669cdfc8-hcgrd" podStartSLOduration=2.41836701 podStartE2EDuration="3.92787024s" totalImagesPullingTime="1.50950323s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:42:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:42:22.082262004 +0000 UTC m=+123.148753493" watchObservedRunningTime="2026-09-16 18:42:22.92787024 +0000 UTC m=+123.994361728"
Sep 16 18:42:22.928357 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:42:22.928129    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod686f396c_6b8e_44a8_83b8_4a38dd7b5644.slice/crio-45d9e21c4dc12741800149891dd465b41ff87026ab6b4f9434134ad677c4d16f WatchSource:0}: Error finding container 45d9e21c4dc12741800149891dd465b41ff87026ab6b4f9434134ad677c4d16f: Status 404 returned error can't find the container with id 45d9e21c4dc12741800149891dd465b41ff87026ab6b4f9434134ad677c4d16f
Sep 16 18:42:22.929653 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:42:22.929634    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 16 18:42:23.066544 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:42:23.066506    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-655cdcc48d-764cf" event={"ID":"f1465039-5e01-4402-84a7-56d706f786a6","Type":"ContainerStarted","Data":"c8054639cf6755012060add810e5673f3583308fdc507586fc759a0acb700643"}
Sep 16 18:42:23.067787 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:42:23.067762    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"686f396c-6b8e-44a8-83b8-4a38dd7b5644","Type":"ContainerStarted","Data":"45d9e21c4dc12741800149891dd465b41ff87026ab6b4f9434134ad677c4d16f"}
Sep 16 18:42:23.436015 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:42:23.435726    2578 scope.go:117] "RemoveContainer" containerID="8f38b1af4975884bac5c76bc8f8a028e6d32d0bc4be3883c8dd768ea64e92644"
Sep 16 18:42:23.590361 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:42:23.590326    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-9785996cd-z4ckv"]
Sep 16 18:42:23.595365 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:42:23.595335    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-9785996cd-z4ckv"
Sep 16 18:42:23.724491 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:42:23.724412    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 18:42:23.812166 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:42:23.812138    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 18:42:23.915083 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:42:23.915058    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 18:42:24.022456 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:42:24.022375    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 18:42:24.076244 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:42:24.076016    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-655cdcc48d-764cf" event={"ID":"f1465039-5e01-4402-84a7-56d706f786a6","Type":"ContainerStarted","Data":"92117d250ad989b6e302d970554f617fc30eae3879d84ce4ac112f74157b0d4a"}
Sep 16 18:42:24.076244 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:42:24.076091    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-655cdcc48d-764cf" event={"ID":"f1465039-5e01-4402-84a7-56d706f786a6","Type":"ContainerStarted","Data":"0fc734d8f82a1d6c9dd3b8e60b9a91416075a4ed2b7e2a321ad94586298920b5"}
Sep 16 18:42:24.078476 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:42:24.078451    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-tmrkx_4c2078bb-281a-4aa1-8e38-238897865434/console-operator/2.log"
Sep 16 18:42:24.078612 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:42:24.078561    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-696884fb9c-tmrkx" event={"ID":"4c2078bb-281a-4aa1-8e38-238897865434","Type":"ContainerStarted","Data":"4e9444cadb1fe761dbb644ca131f3065acc4024b4055a1a76dd7d799be89dffa"}
Sep 16 18:42:24.079289 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:42:24.079045    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-696884fb9c-tmrkx"
Sep 16 18:42:24.080746 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:42:24.080718    2578 generic.go:356] "Generic (PLEG): container finished" podID="686f396c-6b8e-44a8-83b8-4a38dd7b5644" containerID="0b50d25dea85bb8df0c47b2ecb83c2827ecda8d645111c4069a97022dfc08826" exitCode=0
Sep 16 18:42:24.080853 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:42:24.080766    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"686f396c-6b8e-44a8-83b8-4a38dd7b5644","Type":"ContainerDied","Data":"0b50d25dea85bb8df0c47b2ecb83c2827ecda8d645111c4069a97022dfc08826"}
Sep 16 18:42:24.084393 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:42:24.084372    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-696884fb9c-tmrkx"
Sep 16 18:42:24.112663 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:42:24.112620    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-655cdcc48d-764cf" podStartSLOduration=1.255887943 podStartE2EDuration="4.112604519s" totalImagesPullingTime="2.856716576s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:42:20 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:42:24.112029437 +0000 UTC m=+125.178520944" watchObservedRunningTime="2026-09-16 18:42:24.112604519 +0000 UTC m=+125.179096007"
Sep 16 18:42:24.183287 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:42:24.183251    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 18:42:24.200555 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:42:24.200499    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console-operator/console-operator-696884fb9c-tmrkx" podStartSLOduration=51.947222553 podStartE2EDuration="54.200479024s" totalImagesPullingTime="2.253256471s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:41:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:42:24.199702607 +0000 UTC m=+125.266194095" watchObservedRunningTime="2026-09-16 18:42:24.200479024 +0000 UTC m=+125.266970515"
Sep 16 18:42:24.314762 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:42:24.314671    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 18:42:24.420282 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:42:24.420241    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 18:42:24.526655 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:42:24.526615    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 18:42:24.727570 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:42:24.727537    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 18:42:24.810548 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:42:24.810515    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 18:42:24.998154 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:42:24.998078    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 18:42:25.097729 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:42:25.097694    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 18:42:26.129273 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:42:26.129203    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 18:42:26.358502 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:42:26.358471    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 18:42:26.648912 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:42:26.648872    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 18:42:26.939282 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:42:26.939193    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 18:42:27.004433 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:42:27.004400    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 18:42:27.196970 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:42:27.196875    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 18:42:27.257339 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:42:27.257309    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 18:42:28.108116 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:42:28.108085    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"686f396c-6b8e-44a8-83b8-4a38dd7b5644","Type":"ContainerStarted","Data":"dcdd869e13d53e7930a087d3164bb4a7ccd54a044d4b668dd0d62205da20b8ec"}
Sep 16 18:42:28.108281 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:42:28.108159    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"686f396c-6b8e-44a8-83b8-4a38dd7b5644","Type":"ContainerStarted","Data":"9eaf47131b84c369eb6ec0858bd64381dd4b855a7ef5d05c1c2006278a3ef205"}
Sep 16 18:42:28.124459 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:42:28.124423    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1f969e62-dd58-41b0-9419-b2ca445f3ec4-metrics-certs\") pod \"network-metrics-daemon-zfv2x\" (UID: \"1f969e62-dd58-41b0-9419-b2ca445f3ec4\") " pod="openshift-multus/network-metrics-daemon-zfv2x"
Sep 16 18:42:28.127035 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:42:28.127013    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1f969e62-dd58-41b0-9419-b2ca445f3ec4-metrics-certs\") pod \"network-metrics-daemon-zfv2x\" (UID: \"1f969e62-dd58-41b0-9419-b2ca445f3ec4\") " pod="openshift-multus/network-metrics-daemon-zfv2x"
Sep 16 18:42:28.349707 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:42:28.349674    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-s9z5g\""
Sep 16 18:42:28.358344 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:42:28.358276    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zfv2x"
Sep 16 18:42:28.640406 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:42:28.640377    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-zfv2x"]
Sep 16 18:42:28.643411 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:42:28.643385    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1f969e62_dd58_41b0_9419_b2ca445f3ec4.slice/crio-ebaa307788949c832393aaf127cf57f06f8d7dd558be6ca87eb7a4ae780579db WatchSource:0}: Error finding container ebaa307788949c832393aaf127cf57f06f8d7dd558be6ca87eb7a4ae780579db: Status 404 returned error can't find the container with id ebaa307788949c832393aaf127cf57f06f8d7dd558be6ca87eb7a4ae780579db
Sep 16 18:42:29.116069 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:42:29.115991    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-zfv2x" event={"ID":"1f969e62-dd58-41b0-9419-b2ca445f3ec4","Type":"ContainerStarted","Data":"ebaa307788949c832393aaf127cf57f06f8d7dd558be6ca87eb7a4ae780579db"}
Sep 16 18:42:30.123350 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:42:30.123317    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-zfv2x" event={"ID":"1f969e62-dd58-41b0-9419-b2ca445f3ec4","Type":"ContainerStarted","Data":"0fc205ef9bd460daf9489350383527c1e92e7d55b50e3319b0304143f3b84f32"}
Sep 16 18:42:30.126788 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:42:30.126761    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"686f396c-6b8e-44a8-83b8-4a38dd7b5644","Type":"ContainerStarted","Data":"0b721324bb03fd14647fff957d46d0041c0482d8cd9c0c19f30703fdf7f2befa"}
Sep 16 18:42:30.126921 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:42:30.126831    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"686f396c-6b8e-44a8-83b8-4a38dd7b5644","Type":"ContainerStarted","Data":"d6f9414e62105b3b9792696ebefee50510fe9a5f723376c1d51128426dc53705"}
Sep 16 18:42:30.126921 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:42:30.126872    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"686f396c-6b8e-44a8-83b8-4a38dd7b5644","Type":"ContainerStarted","Data":"36c4c34d568cb2ba855bd6d8a5dc27d1050d94ef41d0054a823fba9ae34a341c"}
Sep 16 18:42:30.126921 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:42:30.126909    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"686f396c-6b8e-44a8-83b8-4a38dd7b5644","Type":"ContainerStarted","Data":"a64eb6344e289f94222171b6de1abbd7e1a37ff321acc8668bd5cd22800bbbbe"}
Sep 16 18:42:30.157604 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:42:30.157549    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=3.159537336 podStartE2EDuration="9.157531102s" totalImagesPullingTime="5.972767172s" totalInitContainerRuntime="25.226594ms" isStatefulPod=false podCreationTimestamp="2026-09-16 18:42:21 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:42:30.156710173 +0000 UTC m=+131.223201689" watchObservedRunningTime="2026-09-16 18:42:30.157531102 +0000 UTC m=+131.224022617"
Sep 16 18:42:31.134316 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:42:31.134278    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-zfv2x" event={"ID":"1f969e62-dd58-41b0-9419-b2ca445f3ec4","Type":"ContainerStarted","Data":"04e469196eb8dcf8fc4edd43e88c736506542856cc06795463e4a1263473d026"}
Sep 16 18:42:31.154778 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:42:31.154709    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-zfv2x" podStartSLOduration=130.849916546 podStartE2EDuration="2m12.154685991s" totalImagesPullingTime="1.304769445s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:40:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:42:31.153224262 +0000 UTC m=+132.219715761" watchObservedRunningTime="2026-09-16 18:42:31.154685991 +0000 UTC m=+132.221177481"
Sep 16 18:42:31.291156 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:42:31.291111    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 18:42:31.396804 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:42:31.396708    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 18:42:32.256381 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:42:32.256349    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:42:33.185453 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:42:33.185422    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 18:42:33.251570 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:42:33.251539    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 18:42:36.194394 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:42:36.194367    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 18:42:36.254899 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:42:36.254871    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 18:42:37.947370 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:42:37.947332    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 18:42:38.019880 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:42:38.019853    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 18:42:38.726592 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:42:38.726546    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 18:42:38.807889 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:42:38.807858    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 18:42:39.950233 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:42:39.950205    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-67669cdfc8-hcgrd"
Sep 16 18:42:39.950596 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:42:39.950258    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-67669cdfc8-hcgrd"
Sep 16 18:42:42.526732 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:42:42.526694    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 18:42:42.591114 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:42:42.591077    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 18:42:45.925961 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:42:45.925926    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 18:42:46.027080 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:42:46.027042    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 18:42:46.946507 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:42:46.946472    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 18:42:47.019660 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:42:47.019626    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 18:42:48.611935 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:42:48.611874    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-9785996cd-z4ckv" podUID="92866a88-93b2-430b-867b-40e3358eb631" containerName="registry" containerID="cri-o://a275d6ff5587eea121dda12a18391617704b348ec72ed226db0c8907f3bc1680" gracePeriod=30
Sep 16 18:42:48.855282 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:42:48.855256    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-9785996cd-z4ckv"
Sep 16 18:42:49.011572 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:42:49.011529    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fkm8c\" (UniqueName: \"kubernetes.io/projected/92866a88-93b2-430b-867b-40e3358eb631-kube-api-access-fkm8c\") pod \"92866a88-93b2-430b-867b-40e3358eb631\" (UID: \"92866a88-93b2-430b-867b-40e3358eb631\") "
Sep 16 18:42:49.011572 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:42:49.011579    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/92866a88-93b2-430b-867b-40e3358eb631-installation-pull-secrets\") pod \"92866a88-93b2-430b-867b-40e3358eb631\" (UID: \"92866a88-93b2-430b-867b-40e3358eb631\") "
Sep 16 18:42:49.011828 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:42:49.011607    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/92866a88-93b2-430b-867b-40e3358eb631-registry-tls\") pod \"92866a88-93b2-430b-867b-40e3358eb631\" (UID: \"92866a88-93b2-430b-867b-40e3358eb631\") "
Sep 16 18:42:49.011828 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:42:49.011624    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/92866a88-93b2-430b-867b-40e3358eb631-ca-trust-extracted\") pod \"92866a88-93b2-430b-867b-40e3358eb631\" (UID: \"92866a88-93b2-430b-867b-40e3358eb631\") "
Sep 16 18:42:49.011828 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:42:49.011638    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/92866a88-93b2-430b-867b-40e3358eb631-bound-sa-token\") pod \"92866a88-93b2-430b-867b-40e3358eb631\" (UID: \"92866a88-93b2-430b-867b-40e3358eb631\") "
Sep 16 18:42:49.011828 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:42:49.011655    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/92866a88-93b2-430b-867b-40e3358eb631-registry-certificates\") pod \"92866a88-93b2-430b-867b-40e3358eb631\" (UID: \"92866a88-93b2-430b-867b-40e3358eb631\") "
Sep 16 18:42:49.011828 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:42:49.011671    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/92866a88-93b2-430b-867b-40e3358eb631-image-registry-private-configuration\") pod \"92866a88-93b2-430b-867b-40e3358eb631\" (UID: \"92866a88-93b2-430b-867b-40e3358eb631\") "
Sep 16 18:42:49.011828 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:42:49.011706    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/92866a88-93b2-430b-867b-40e3358eb631-trusted-ca\") pod \"92866a88-93b2-430b-867b-40e3358eb631\" (UID: \"92866a88-93b2-430b-867b-40e3358eb631\") "
Sep 16 18:42:49.012348 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:42:49.012284    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/92866a88-93b2-430b-867b-40e3358eb631-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "92866a88-93b2-430b-867b-40e3358eb631" (UID: "92866a88-93b2-430b-867b-40e3358eb631"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:42:49.012348 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:42:49.012306    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/92866a88-93b2-430b-867b-40e3358eb631-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "92866a88-93b2-430b-867b-40e3358eb631" (UID: "92866a88-93b2-430b-867b-40e3358eb631"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:42:49.014966 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:42:49.014918    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/92866a88-93b2-430b-867b-40e3358eb631-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "92866a88-93b2-430b-867b-40e3358eb631" (UID: "92866a88-93b2-430b-867b-40e3358eb631"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:42:49.015101 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:42:49.014957    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/92866a88-93b2-430b-867b-40e3358eb631-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "92866a88-93b2-430b-867b-40e3358eb631" (UID: "92866a88-93b2-430b-867b-40e3358eb631"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:42:49.015101 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:42:49.015012    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/92866a88-93b2-430b-867b-40e3358eb631-kube-api-access-fkm8c" (OuterVolumeSpecName: "kube-api-access-fkm8c") pod "92866a88-93b2-430b-867b-40e3358eb631" (UID: "92866a88-93b2-430b-867b-40e3358eb631"). InnerVolumeSpecName "kube-api-access-fkm8c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:42:49.015101 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:42:49.015032    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/92866a88-93b2-430b-867b-40e3358eb631-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "92866a88-93b2-430b-867b-40e3358eb631" (UID: "92866a88-93b2-430b-867b-40e3358eb631"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:42:49.015218 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:42:49.015114    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/92866a88-93b2-430b-867b-40e3358eb631-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "92866a88-93b2-430b-867b-40e3358eb631" (UID: "92866a88-93b2-430b-867b-40e3358eb631"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:42:49.020810 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:42:49.020784    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/92866a88-93b2-430b-867b-40e3358eb631-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "92866a88-93b2-430b-867b-40e3358eb631" (UID: "92866a88-93b2-430b-867b-40e3358eb631"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:42:49.112664 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:42:49.112640    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fkm8c\" (UniqueName: \"kubernetes.io/projected/92866a88-93b2-430b-867b-40e3358eb631-kube-api-access-fkm8c\") on node \"ip-10-0-135-119.ec2.internal\" DevicePath \"\""
Sep 16 18:42:49.112664 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:42:49.112662    2578 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/92866a88-93b2-430b-867b-40e3358eb631-installation-pull-secrets\") on node \"ip-10-0-135-119.ec2.internal\" DevicePath \"\""
Sep 16 18:42:49.112807 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:42:49.112671    2578 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/92866a88-93b2-430b-867b-40e3358eb631-registry-tls\") on node \"ip-10-0-135-119.ec2.internal\" DevicePath \"\""
Sep 16 18:42:49.112807 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:42:49.112680    2578 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/92866a88-93b2-430b-867b-40e3358eb631-ca-trust-extracted\") on node \"ip-10-0-135-119.ec2.internal\" DevicePath \"\""
Sep 16 18:42:49.112807 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:42:49.112688    2578 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/92866a88-93b2-430b-867b-40e3358eb631-bound-sa-token\") on node \"ip-10-0-135-119.ec2.internal\" DevicePath \"\""
Sep 16 18:42:49.112807 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:42:49.112696    2578 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/92866a88-93b2-430b-867b-40e3358eb631-registry-certificates\") on node \"ip-10-0-135-119.ec2.internal\" DevicePath \"\""
Sep 16 18:42:49.112807 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:42:49.112706    2578 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/92866a88-93b2-430b-867b-40e3358eb631-image-registry-private-configuration\") on node \"ip-10-0-135-119.ec2.internal\" DevicePath \"\""
Sep 16 18:42:49.112807 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:42:49.112715    2578 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/92866a88-93b2-430b-867b-40e3358eb631-trusted-ca\") on node \"ip-10-0-135-119.ec2.internal\" DevicePath \"\""
Sep 16 18:42:49.205088 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:42:49.205059    2578 generic.go:356] "Generic (PLEG): container finished" podID="92866a88-93b2-430b-867b-40e3358eb631" containerID="a275d6ff5587eea121dda12a18391617704b348ec72ed226db0c8907f3bc1680" exitCode=0
Sep 16 18:42:49.205218 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:42:49.205101    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-9785996cd-z4ckv"
Sep 16 18:42:49.205218 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:42:49.205102    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-9785996cd-z4ckv" event={"ID":"92866a88-93b2-430b-867b-40e3358eb631","Type":"ContainerDied","Data":"a275d6ff5587eea121dda12a18391617704b348ec72ed226db0c8907f3bc1680"}
Sep 16 18:42:49.205305 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:42:49.205237    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-9785996cd-z4ckv" event={"ID":"92866a88-93b2-430b-867b-40e3358eb631","Type":"ContainerDied","Data":"b1011fdb569f9fd3456804268d92e36caa6505872266be469b3c01515546db3f"}
Sep 16 18:42:49.205305 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:42:49.205264    2578 scope.go:117] "RemoveContainer" containerID="a275d6ff5587eea121dda12a18391617704b348ec72ed226db0c8907f3bc1680"
Sep 16 18:42:49.218491 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:42:49.218470    2578 scope.go:117] "RemoveContainer" containerID="a275d6ff5587eea121dda12a18391617704b348ec72ed226db0c8907f3bc1680"
Sep 16 18:42:49.218784 ip-10-0-135-119 kubenswrapper[2578]: E0916 18:42:49.218762    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a275d6ff5587eea121dda12a18391617704b348ec72ed226db0c8907f3bc1680\": container with ID starting with a275d6ff5587eea121dda12a18391617704b348ec72ed226db0c8907f3bc1680 not found: ID does not exist" containerID="a275d6ff5587eea121dda12a18391617704b348ec72ed226db0c8907f3bc1680"
Sep 16 18:42:49.218857 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:42:49.218795    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a275d6ff5587eea121dda12a18391617704b348ec72ed226db0c8907f3bc1680"} err="failed to get container status \"a275d6ff5587eea121dda12a18391617704b348ec72ed226db0c8907f3bc1680\": rpc error: code = NotFound desc = could not find container \"a275d6ff5587eea121dda12a18391617704b348ec72ed226db0c8907f3bc1680\": container with ID starting with a275d6ff5587eea121dda12a18391617704b348ec72ed226db0c8907f3bc1680 not found: ID does not exist"
Sep 16 18:42:49.232352 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:42:49.232327    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-9785996cd-z4ckv"]
Sep 16 18:42:49.239044 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:42:49.239027    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-9785996cd-z4ckv"]
Sep 16 18:42:49.433160 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:42:49.433128    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="92866a88-93b2-430b-867b-40e3358eb631" path="/var/lib/kubelet/pods/92866a88-93b2-430b-867b-40e3358eb631/volumes"
Sep 16 18:42:51.929751 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:42:51.929720    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 18:42:51.993669 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:42:51.993629    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 18:42:54.045825 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:42:54.045792    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 18:42:54.148359 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:42:54.148328    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 18:42:54.231029 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:42:54.230996    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 18:42:54.760094 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:42:54.760062    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 18:42:54.824240 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:42:54.824194    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 18:42:56.887368 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:42:56.887339    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 18:42:56.963863 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:42:56.963831    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 18:42:59.954224 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:42:59.954194    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-67669cdfc8-hcgrd"
Sep 16 18:42:59.957887 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:42:59.957870    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-67669cdfc8-hcgrd"
Sep 16 18:43:04.299676 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:43:04.299645    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 18:43:04.374114 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:43:04.374083    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 18:43:09.283202 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:43:09.283168    2578 generic.go:356] "Generic (PLEG): container finished" podID="cd2b7515-5e1e-429d-9c17-32a37b492b26" containerID="4df8798410a4053002f78eb3425d13130f1d94a448c544b48861b45cc71fa118" exitCode=0
Sep 16 18:43:09.283604 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:43:09.283243    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5844b856db-mp7tt" event={"ID":"cd2b7515-5e1e-429d-9c17-32a37b492b26","Type":"ContainerDied","Data":"4df8798410a4053002f78eb3425d13130f1d94a448c544b48861b45cc71fa118"}
Sep 16 18:43:09.283604 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:43:09.283561    2578 scope.go:117] "RemoveContainer" containerID="4df8798410a4053002f78eb3425d13130f1d94a448c544b48861b45cc71fa118"
Sep 16 18:43:09.907235 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:43:09.907198    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 18:43:09.973029 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:43:09.972995    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 18:43:10.289208 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:43:10.289131    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5844b856db-mp7tt" event={"ID":"cd2b7515-5e1e-429d-9c17-32a37b492b26","Type":"ContainerStarted","Data":"1bccc815886fb4ca10beb477e355ce0d2726d6fa0a4e60bbcc18f569f97bb217"}
Sep 16 18:43:11.886200 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:43:11.886163    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 18:43:11.952051 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:43:11.952002    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 18:43:22.256133 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:43:22.256096    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:43:22.277811 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:43:22.277780    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:43:22.364960 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:43:22.364930    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:43:31.615968 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:43:31.615919    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 18:43:31.727080 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:43:31.727042    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 18:43:34.242224 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:43:34.242176    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 18:43:34.307745 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:43:34.307711    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 18:43:34.383492 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:43:34.383459    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 18:43:34.454040 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:43:34.454001    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 18:43:34.983663 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:43:34.983619    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 18:43:40.136142 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:43:40.136103    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 16 18:43:40.136617 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:43:40.136489    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="686f396c-6b8e-44a8-83b8-4a38dd7b5644" containerName="kube-rbac-proxy" containerID="cri-o://d6f9414e62105b3b9792696ebefee50510fe9a5f723376c1d51128426dc53705" gracePeriod=600
Sep 16 18:43:40.136617 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:43:40.136508    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="686f396c-6b8e-44a8-83b8-4a38dd7b5644" containerName="thanos-sidecar" containerID="cri-o://a64eb6344e289f94222171b6de1abbd7e1a37ff321acc8668bd5cd22800bbbbe" gracePeriod=600
Sep 16 18:43:40.136617 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:43:40.136527    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="686f396c-6b8e-44a8-83b8-4a38dd7b5644" containerName="kube-rbac-proxy-web" containerID="cri-o://36c4c34d568cb2ba855bd6d8a5dc27d1050d94ef41d0054a823fba9ae34a341c" gracePeriod=600
Sep 16 18:43:40.136617 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:43:40.136499    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="686f396c-6b8e-44a8-83b8-4a38dd7b5644" containerName="prometheus" containerID="cri-o://9eaf47131b84c369eb6ec0858bd64381dd4b855a7ef5d05c1c2006278a3ef205" gracePeriod=600
Sep 16 18:43:40.136831 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:43:40.136576    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="686f396c-6b8e-44a8-83b8-4a38dd7b5644" containerName="kube-rbac-proxy-thanos" containerID="cri-o://0b721324bb03fd14647fff957d46d0041c0482d8cd9c0c19f30703fdf7f2befa" gracePeriod=600
Sep 16 18:43:40.136831 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:43:40.136780    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="686f396c-6b8e-44a8-83b8-4a38dd7b5644" containerName="config-reloader" containerID="cri-o://dcdd869e13d53e7930a087d3164bb4a7ccd54a044d4b668dd0d62205da20b8ec" gracePeriod=600
Sep 16 18:43:40.432011 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:43:40.431897    2578 generic.go:356] "Generic (PLEG): container finished" podID="686f396c-6b8e-44a8-83b8-4a38dd7b5644" containerID="0b721324bb03fd14647fff957d46d0041c0482d8cd9c0c19f30703fdf7f2befa" exitCode=0
Sep 16 18:43:40.432011 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:43:40.431927    2578 generic.go:356] "Generic (PLEG): container finished" podID="686f396c-6b8e-44a8-83b8-4a38dd7b5644" containerID="d6f9414e62105b3b9792696ebefee50510fe9a5f723376c1d51128426dc53705" exitCode=0
Sep 16 18:43:40.432011 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:43:40.431939    2578 generic.go:356] "Generic (PLEG): container finished" podID="686f396c-6b8e-44a8-83b8-4a38dd7b5644" containerID="a64eb6344e289f94222171b6de1abbd7e1a37ff321acc8668bd5cd22800bbbbe" exitCode=0
Sep 16 18:43:40.432011 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:43:40.431934    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"686f396c-6b8e-44a8-83b8-4a38dd7b5644","Type":"ContainerDied","Data":"0b721324bb03fd14647fff957d46d0041c0482d8cd9c0c19f30703fdf7f2befa"}
Sep 16 18:43:40.432011 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:43:40.431947    2578 generic.go:356] "Generic (PLEG): container finished" podID="686f396c-6b8e-44a8-83b8-4a38dd7b5644" containerID="dcdd869e13d53e7930a087d3164bb4a7ccd54a044d4b668dd0d62205da20b8ec" exitCode=0
Sep 16 18:43:40.432011 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:43:40.432014    2578 generic.go:356] "Generic (PLEG): container finished" podID="686f396c-6b8e-44a8-83b8-4a38dd7b5644" containerID="9eaf47131b84c369eb6ec0858bd64381dd4b855a7ef5d05c1c2006278a3ef205" exitCode=0
Sep 16 18:43:40.432351 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:43:40.432025    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"686f396c-6b8e-44a8-83b8-4a38dd7b5644","Type":"ContainerDied","Data":"d6f9414e62105b3b9792696ebefee50510fe9a5f723376c1d51128426dc53705"}
Sep 16 18:43:40.432351 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:43:40.432072    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"686f396c-6b8e-44a8-83b8-4a38dd7b5644","Type":"ContainerDied","Data":"a64eb6344e289f94222171b6de1abbd7e1a37ff321acc8668bd5cd22800bbbbe"}
Sep 16 18:43:40.432351 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:43:40.432113    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"686f396c-6b8e-44a8-83b8-4a38dd7b5644","Type":"ContainerDied","Data":"dcdd869e13d53e7930a087d3164bb4a7ccd54a044d4b668dd0d62205da20b8ec"}
Sep 16 18:43:40.432351 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:43:40.432143    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"686f396c-6b8e-44a8-83b8-4a38dd7b5644","Type":"ContainerDied","Data":"9eaf47131b84c369eb6ec0858bd64381dd4b855a7ef5d05c1c2006278a3ef205"}
Sep 16 18:43:41.374773 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:43:41.374750    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:43:41.438420 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:43:41.438398    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/686f396c-6b8e-44a8-83b8-4a38dd7b5644-secret-kube-rbac-proxy\") pod \"686f396c-6b8e-44a8-83b8-4a38dd7b5644\" (UID: \"686f396c-6b8e-44a8-83b8-4a38dd7b5644\") "
Sep 16 18:43:41.438562 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:43:41.438425    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/686f396c-6b8e-44a8-83b8-4a38dd7b5644-web-config\") pod \"686f396c-6b8e-44a8-83b8-4a38dd7b5644\" (UID: \"686f396c-6b8e-44a8-83b8-4a38dd7b5644\") "
Sep 16 18:43:41.438617 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:43:41.438578    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/686f396c-6b8e-44a8-83b8-4a38dd7b5644-secret-prometheus-k8s-tls\") pod \"686f396c-6b8e-44a8-83b8-4a38dd7b5644\" (UID: \"686f396c-6b8e-44a8-83b8-4a38dd7b5644\") "
Sep 16 18:43:41.438663 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:43:41.438617    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/686f396c-6b8e-44a8-83b8-4a38dd7b5644-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"686f396c-6b8e-44a8-83b8-4a38dd7b5644\" (UID: \"686f396c-6b8e-44a8-83b8-4a38dd7b5644\") "
Sep 16 18:43:41.438704 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:43:41.438666    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/686f396c-6b8e-44a8-83b8-4a38dd7b5644-secret-metrics-client-certs\") pod \"686f396c-6b8e-44a8-83b8-4a38dd7b5644\" (UID: \"686f396c-6b8e-44a8-83b8-4a38dd7b5644\") "
Sep 16 18:43:41.438764 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:43:41.438708    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7wm7v\" (UniqueName: \"kubernetes.io/projected/686f396c-6b8e-44a8-83b8-4a38dd7b5644-kube-api-access-7wm7v\") pod \"686f396c-6b8e-44a8-83b8-4a38dd7b5644\" (UID: \"686f396c-6b8e-44a8-83b8-4a38dd7b5644\") "
Sep 16 18:43:41.438764 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:43:41.438750    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/686f396c-6b8e-44a8-83b8-4a38dd7b5644-tls-assets\") pod \"686f396c-6b8e-44a8-83b8-4a38dd7b5644\" (UID: \"686f396c-6b8e-44a8-83b8-4a38dd7b5644\") "
Sep 16 18:43:41.438872 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:43:41.438796    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/686f396c-6b8e-44a8-83b8-4a38dd7b5644-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"686f396c-6b8e-44a8-83b8-4a38dd7b5644\" (UID: \"686f396c-6b8e-44a8-83b8-4a38dd7b5644\") "
Sep 16 18:43:41.438872 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:43:41.438828    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/686f396c-6b8e-44a8-83b8-4a38dd7b5644-secret-grpc-tls\") pod \"686f396c-6b8e-44a8-83b8-4a38dd7b5644\" (UID: \"686f396c-6b8e-44a8-83b8-4a38dd7b5644\") "
Sep 16 18:43:41.438872 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:43:41.438853    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/686f396c-6b8e-44a8-83b8-4a38dd7b5644-config-out\") pod \"686f396c-6b8e-44a8-83b8-4a38dd7b5644\" (UID: \"686f396c-6b8e-44a8-83b8-4a38dd7b5644\") "
Sep 16 18:43:41.439050 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:43:41.438878    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/686f396c-6b8e-44a8-83b8-4a38dd7b5644-configmap-serving-certs-ca-bundle\") pod \"686f396c-6b8e-44a8-83b8-4a38dd7b5644\" (UID: \"686f396c-6b8e-44a8-83b8-4a38dd7b5644\") "
Sep 16 18:43:41.439050 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:43:41.438903    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/686f396c-6b8e-44a8-83b8-4a38dd7b5644-thanos-prometheus-http-client-file\") pod \"686f396c-6b8e-44a8-83b8-4a38dd7b5644\" (UID: \"686f396c-6b8e-44a8-83b8-4a38dd7b5644\") "
Sep 16 18:43:41.439050 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:43:41.438932    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/686f396c-6b8e-44a8-83b8-4a38dd7b5644-configmap-metrics-client-ca\") pod \"686f396c-6b8e-44a8-83b8-4a38dd7b5644\" (UID: \"686f396c-6b8e-44a8-83b8-4a38dd7b5644\") "
Sep 16 18:43:41.439050 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:43:41.438959    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/686f396c-6b8e-44a8-83b8-4a38dd7b5644-prometheus-k8s-rulefiles-0\") pod \"686f396c-6b8e-44a8-83b8-4a38dd7b5644\" (UID: \"686f396c-6b8e-44a8-83b8-4a38dd7b5644\") "
Sep 16 18:43:41.439050 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:43:41.439012    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/686f396c-6b8e-44a8-83b8-4a38dd7b5644-config\") pod \"686f396c-6b8e-44a8-83b8-4a38dd7b5644\" (UID: \"686f396c-6b8e-44a8-83b8-4a38dd7b5644\") "
Sep 16 18:43:41.439356 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:43:41.439060    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/686f396c-6b8e-44a8-83b8-4a38dd7b5644-prometheus-k8s-rulefiles-2\") pod \"686f396c-6b8e-44a8-83b8-4a38dd7b5644\" (UID: \"686f396c-6b8e-44a8-83b8-4a38dd7b5644\") "
Sep 16 18:43:41.439356 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:43:41.439082    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/686f396c-6b8e-44a8-83b8-4a38dd7b5644-prometheus-trusted-ca-bundle\") pod \"686f396c-6b8e-44a8-83b8-4a38dd7b5644\" (UID: \"686f396c-6b8e-44a8-83b8-4a38dd7b5644\") "
Sep 16 18:43:41.439356 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:43:41.439111    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/686f396c-6b8e-44a8-83b8-4a38dd7b5644-prometheus-k8s-rulefiles-1\") pod \"686f396c-6b8e-44a8-83b8-4a38dd7b5644\" (UID: \"686f396c-6b8e-44a8-83b8-4a38dd7b5644\") "
Sep 16 18:43:41.439356 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:43:41.439176    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/686f396c-6b8e-44a8-83b8-4a38dd7b5644-prometheus-k8s-db\") pod \"686f396c-6b8e-44a8-83b8-4a38dd7b5644\" (UID: \"686f396c-6b8e-44a8-83b8-4a38dd7b5644\") "
Sep 16 18:43:41.439356 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:43:41.439201    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/686f396c-6b8e-44a8-83b8-4a38dd7b5644-configmap-kubelet-serving-ca-bundle\") pod \"686f396c-6b8e-44a8-83b8-4a38dd7b5644\" (UID: \"686f396c-6b8e-44a8-83b8-4a38dd7b5644\") "
Sep 16 18:43:41.439999 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:43:41.439952    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/686f396c-6b8e-44a8-83b8-4a38dd7b5644-configmap-kubelet-serving-ca-bundle" (OuterVolumeSpecName: "configmap-kubelet-serving-ca-bundle") pod "686f396c-6b8e-44a8-83b8-4a38dd7b5644" (UID: "686f396c-6b8e-44a8-83b8-4a38dd7b5644"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:43:41.440819 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:43:41.440782    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/686f396c-6b8e-44a8-83b8-4a38dd7b5644-prometheus-k8s-rulefiles-0" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-0") pod "686f396c-6b8e-44a8-83b8-4a38dd7b5644" (UID: "686f396c-6b8e-44a8-83b8-4a38dd7b5644"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:43:41.441662 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:43:41.441630    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/686f396c-6b8e-44a8-83b8-4a38dd7b5644-secret-prometheus-k8s-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-tls") pod "686f396c-6b8e-44a8-83b8-4a38dd7b5644" (UID: "686f396c-6b8e-44a8-83b8-4a38dd7b5644"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:43:41.442619 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:43:41.442203    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/686f396c-6b8e-44a8-83b8-4a38dd7b5644-secret-kube-rbac-proxy" (OuterVolumeSpecName: "secret-kube-rbac-proxy") pod "686f396c-6b8e-44a8-83b8-4a38dd7b5644" (UID: "686f396c-6b8e-44a8-83b8-4a38dd7b5644"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:43:41.442619 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:43:41.442403    2578 generic.go:356] "Generic (PLEG): container finished" podID="686f396c-6b8e-44a8-83b8-4a38dd7b5644" containerID="36c4c34d568cb2ba855bd6d8a5dc27d1050d94ef41d0054a823fba9ae34a341c" exitCode=0
Sep 16 18:43:41.442619 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:43:41.442424    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/686f396c-6b8e-44a8-83b8-4a38dd7b5644-secret-metrics-client-certs" (OuterVolumeSpecName: "secret-metrics-client-certs") pod "686f396c-6b8e-44a8-83b8-4a38dd7b5644" (UID: "686f396c-6b8e-44a8-83b8-4a38dd7b5644"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:43:41.442619 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:43:41.442439    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"686f396c-6b8e-44a8-83b8-4a38dd7b5644","Type":"ContainerDied","Data":"36c4c34d568cb2ba855bd6d8a5dc27d1050d94ef41d0054a823fba9ae34a341c"}
Sep 16 18:43:41.442619 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:43:41.442505    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"686f396c-6b8e-44a8-83b8-4a38dd7b5644","Type":"ContainerDied","Data":"45d9e21c4dc12741800149891dd465b41ff87026ab6b4f9434134ad677c4d16f"}
Sep 16 18:43:41.442619 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:43:41.442553    2578 scope.go:117] "RemoveContainer" containerID="0b721324bb03fd14647fff957d46d0041c0482d8cd9c0c19f30703fdf7f2befa"
Sep 16 18:43:41.442993 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:43:41.442687    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/686f396c-6b8e-44a8-83b8-4a38dd7b5644-config" (OuterVolumeSpecName: "config") pod "686f396c-6b8e-44a8-83b8-4a38dd7b5644" (UID: "686f396c-6b8e-44a8-83b8-4a38dd7b5644"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:43:41.442993 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:43:41.442739    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/686f396c-6b8e-44a8-83b8-4a38dd7b5644-configmap-serving-certs-ca-bundle" (OuterVolumeSpecName: "configmap-serving-certs-ca-bundle") pod "686f396c-6b8e-44a8-83b8-4a38dd7b5644" (UID: "686f396c-6b8e-44a8-83b8-4a38dd7b5644"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:43:41.443103 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:43:41.443024    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/686f396c-6b8e-44a8-83b8-4a38dd7b5644-prometheus-k8s-rulefiles-2" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-2") pod "686f396c-6b8e-44a8-83b8-4a38dd7b5644" (UID: "686f396c-6b8e-44a8-83b8-4a38dd7b5644"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:43:41.443103 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:43:41.443076    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/686f396c-6b8e-44a8-83b8-4a38dd7b5644-prometheus-k8s-rulefiles-1" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-1") pod "686f396c-6b8e-44a8-83b8-4a38dd7b5644" (UID: "686f396c-6b8e-44a8-83b8-4a38dd7b5644"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:43:41.444607 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:43:41.443425    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:43:41.444607 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:43:41.443503    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/686f396c-6b8e-44a8-83b8-4a38dd7b5644-prometheus-trusted-ca-bundle" (OuterVolumeSpecName: "prometheus-trusted-ca-bundle") pod "686f396c-6b8e-44a8-83b8-4a38dd7b5644" (UID: "686f396c-6b8e-44a8-83b8-4a38dd7b5644"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:43:41.444607 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:43:41.443708    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/686f396c-6b8e-44a8-83b8-4a38dd7b5644-config-out" (OuterVolumeSpecName: "config-out") pod "686f396c-6b8e-44a8-83b8-4a38dd7b5644" (UID: "686f396c-6b8e-44a8-83b8-4a38dd7b5644"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:43:41.444607 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:43:41.444047    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/686f396c-6b8e-44a8-83b8-4a38dd7b5644-configmap-metrics-client-ca" (OuterVolumeSpecName: "configmap-metrics-client-ca") pod "686f396c-6b8e-44a8-83b8-4a38dd7b5644" (UID: "686f396c-6b8e-44a8-83b8-4a38dd7b5644"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:43:41.444607 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:43:41.444474    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/686f396c-6b8e-44a8-83b8-4a38dd7b5644-secret-prometheus-k8s-thanos-sidecar-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-thanos-sidecar-tls") pod "686f396c-6b8e-44a8-83b8-4a38dd7b5644" (UID: "686f396c-6b8e-44a8-83b8-4a38dd7b5644"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:43:41.444607 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:43:41.444534    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/686f396c-6b8e-44a8-83b8-4a38dd7b5644-prometheus-k8s-db" (OuterVolumeSpecName: "prometheus-k8s-db") pod "686f396c-6b8e-44a8-83b8-4a38dd7b5644" (UID: "686f396c-6b8e-44a8-83b8-4a38dd7b5644"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:43:41.445290 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:43:41.445021    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/686f396c-6b8e-44a8-83b8-4a38dd7b5644-secret-grpc-tls" (OuterVolumeSpecName: "secret-grpc-tls") pod "686f396c-6b8e-44a8-83b8-4a38dd7b5644" (UID: "686f396c-6b8e-44a8-83b8-4a38dd7b5644"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:43:41.445876 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:43:41.445839    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/686f396c-6b8e-44a8-83b8-4a38dd7b5644-thanos-prometheus-http-client-file" (OuterVolumeSpecName: "thanos-prometheus-http-client-file") pod "686f396c-6b8e-44a8-83b8-4a38dd7b5644" (UID: "686f396c-6b8e-44a8-83b8-4a38dd7b5644"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:43:41.446392 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:43:41.446363    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/686f396c-6b8e-44a8-83b8-4a38dd7b5644-secret-prometheus-k8s-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-prometheus-k8s-kube-rbac-proxy-web") pod "686f396c-6b8e-44a8-83b8-4a38dd7b5644" (UID: "686f396c-6b8e-44a8-83b8-4a38dd7b5644"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:43:41.447345 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:43:41.447301    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/686f396c-6b8e-44a8-83b8-4a38dd7b5644-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "686f396c-6b8e-44a8-83b8-4a38dd7b5644" (UID: "686f396c-6b8e-44a8-83b8-4a38dd7b5644"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:43:41.447614 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:43:41.447536    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/686f396c-6b8e-44a8-83b8-4a38dd7b5644-kube-api-access-7wm7v" (OuterVolumeSpecName: "kube-api-access-7wm7v") pod "686f396c-6b8e-44a8-83b8-4a38dd7b5644" (UID: "686f396c-6b8e-44a8-83b8-4a38dd7b5644"). InnerVolumeSpecName "kube-api-access-7wm7v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:43:41.458227 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:43:41.458195    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/686f396c-6b8e-44a8-83b8-4a38dd7b5644-web-config" (OuterVolumeSpecName: "web-config") pod "686f396c-6b8e-44a8-83b8-4a38dd7b5644" (UID: "686f396c-6b8e-44a8-83b8-4a38dd7b5644"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:43:41.465101 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:43:41.465067    2578 scope.go:117] "RemoveContainer" containerID="d6f9414e62105b3b9792696ebefee50510fe9a5f723376c1d51128426dc53705"
Sep 16 18:43:41.471950 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:43:41.471929    2578 scope.go:117] "RemoveContainer" containerID="36c4c34d568cb2ba855bd6d8a5dc27d1050d94ef41d0054a823fba9ae34a341c"
Sep 16 18:43:41.482093 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:43:41.482061    2578 scope.go:117] "RemoveContainer" containerID="a64eb6344e289f94222171b6de1abbd7e1a37ff321acc8668bd5cd22800bbbbe"
Sep 16 18:43:41.488870 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:43:41.488842    2578 scope.go:117] "RemoveContainer" containerID="dcdd869e13d53e7930a087d3164bb4a7ccd54a044d4b668dd0d62205da20b8ec"
Sep 16 18:43:41.495421 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:43:41.495406    2578 scope.go:117] "RemoveContainer" containerID="9eaf47131b84c369eb6ec0858bd64381dd4b855a7ef5d05c1c2006278a3ef205"
Sep 16 18:43:41.502192 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:43:41.502176    2578 scope.go:117] "RemoveContainer" containerID="0b50d25dea85bb8df0c47b2ecb83c2827ecda8d645111c4069a97022dfc08826"
Sep 16 18:43:41.508552 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:43:41.508538    2578 scope.go:117] "RemoveContainer" containerID="0b721324bb03fd14647fff957d46d0041c0482d8cd9c0c19f30703fdf7f2befa"
Sep 16 18:43:41.508782 ip-10-0-135-119 kubenswrapper[2578]: E0916 18:43:41.508764    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0b721324bb03fd14647fff957d46d0041c0482d8cd9c0c19f30703fdf7f2befa\": container with ID starting with 0b721324bb03fd14647fff957d46d0041c0482d8cd9c0c19f30703fdf7f2befa not found: ID does not exist" containerID="0b721324bb03fd14647fff957d46d0041c0482d8cd9c0c19f30703fdf7f2befa"
Sep 16 18:43:41.508828 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:43:41.508792    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0b721324bb03fd14647fff957d46d0041c0482d8cd9c0c19f30703fdf7f2befa"} err="failed to get container status \"0b721324bb03fd14647fff957d46d0041c0482d8cd9c0c19f30703fdf7f2befa\": rpc error: code = NotFound desc = could not find container \"0b721324bb03fd14647fff957d46d0041c0482d8cd9c0c19f30703fdf7f2befa\": container with ID starting with 0b721324bb03fd14647fff957d46d0041c0482d8cd9c0c19f30703fdf7f2befa not found: ID does not exist"
Sep 16 18:43:41.508828 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:43:41.508811    2578 scope.go:117] "RemoveContainer" containerID="d6f9414e62105b3b9792696ebefee50510fe9a5f723376c1d51128426dc53705"
Sep 16 18:43:41.509063 ip-10-0-135-119 kubenswrapper[2578]: E0916 18:43:41.509047    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d6f9414e62105b3b9792696ebefee50510fe9a5f723376c1d51128426dc53705\": container with ID starting with d6f9414e62105b3b9792696ebefee50510fe9a5f723376c1d51128426dc53705 not found: ID does not exist" containerID="d6f9414e62105b3b9792696ebefee50510fe9a5f723376c1d51128426dc53705"
Sep 16 18:43:41.509101 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:43:41.509067    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d6f9414e62105b3b9792696ebefee50510fe9a5f723376c1d51128426dc53705"} err="failed to get container status \"d6f9414e62105b3b9792696ebefee50510fe9a5f723376c1d51128426dc53705\": rpc error: code = NotFound desc = could not find container \"d6f9414e62105b3b9792696ebefee50510fe9a5f723376c1d51128426dc53705\": container with ID starting with d6f9414e62105b3b9792696ebefee50510fe9a5f723376c1d51128426dc53705 not found: ID does not exist"
Sep 16 18:43:41.509101 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:43:41.509081    2578 scope.go:117] "RemoveContainer" containerID="36c4c34d568cb2ba855bd6d8a5dc27d1050d94ef41d0054a823fba9ae34a341c"
Sep 16 18:43:41.509288 ip-10-0-135-119 kubenswrapper[2578]: E0916 18:43:41.509270    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"36c4c34d568cb2ba855bd6d8a5dc27d1050d94ef41d0054a823fba9ae34a341c\": container with ID starting with 36c4c34d568cb2ba855bd6d8a5dc27d1050d94ef41d0054a823fba9ae34a341c not found: ID does not exist" containerID="36c4c34d568cb2ba855bd6d8a5dc27d1050d94ef41d0054a823fba9ae34a341c"
Sep 16 18:43:41.509327 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:43:41.509291    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"36c4c34d568cb2ba855bd6d8a5dc27d1050d94ef41d0054a823fba9ae34a341c"} err="failed to get container status \"36c4c34d568cb2ba855bd6d8a5dc27d1050d94ef41d0054a823fba9ae34a341c\": rpc error: code = NotFound desc = could not find container \"36c4c34d568cb2ba855bd6d8a5dc27d1050d94ef41d0054a823fba9ae34a341c\": container with ID starting with 36c4c34d568cb2ba855bd6d8a5dc27d1050d94ef41d0054a823fba9ae34a341c not found: ID does not exist"
Sep 16 18:43:41.509327 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:43:41.509304    2578 scope.go:117] "RemoveContainer" containerID="a64eb6344e289f94222171b6de1abbd7e1a37ff321acc8668bd5cd22800bbbbe"
Sep 16 18:43:41.509535 ip-10-0-135-119 kubenswrapper[2578]: E0916 18:43:41.509517    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a64eb6344e289f94222171b6de1abbd7e1a37ff321acc8668bd5cd22800bbbbe\": container with ID starting with a64eb6344e289f94222171b6de1abbd7e1a37ff321acc8668bd5cd22800bbbbe not found: ID does not exist" containerID="a64eb6344e289f94222171b6de1abbd7e1a37ff321acc8668bd5cd22800bbbbe"
Sep 16 18:43:41.509597 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:43:41.509543    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a64eb6344e289f94222171b6de1abbd7e1a37ff321acc8668bd5cd22800bbbbe"} err="failed to get container status \"a64eb6344e289f94222171b6de1abbd7e1a37ff321acc8668bd5cd22800bbbbe\": rpc error: code = NotFound desc = could not find container \"a64eb6344e289f94222171b6de1abbd7e1a37ff321acc8668bd5cd22800bbbbe\": container with ID starting with a64eb6344e289f94222171b6de1abbd7e1a37ff321acc8668bd5cd22800bbbbe not found: ID does not exist"
Sep 16 18:43:41.509597 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:43:41.509565    2578 scope.go:117] "RemoveContainer" containerID="dcdd869e13d53e7930a087d3164bb4a7ccd54a044d4b668dd0d62205da20b8ec"
Sep 16 18:43:41.509802 ip-10-0-135-119 kubenswrapper[2578]: E0916 18:43:41.509784    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dcdd869e13d53e7930a087d3164bb4a7ccd54a044d4b668dd0d62205da20b8ec\": container with ID starting with dcdd869e13d53e7930a087d3164bb4a7ccd54a044d4b668dd0d62205da20b8ec not found: ID does not exist" containerID="dcdd869e13d53e7930a087d3164bb4a7ccd54a044d4b668dd0d62205da20b8ec"
Sep 16 18:43:41.509852 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:43:41.509800    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dcdd869e13d53e7930a087d3164bb4a7ccd54a044d4b668dd0d62205da20b8ec"} err="failed to get container status \"dcdd869e13d53e7930a087d3164bb4a7ccd54a044d4b668dd0d62205da20b8ec\": rpc error: code = NotFound desc = could not find container \"dcdd869e13d53e7930a087d3164bb4a7ccd54a044d4b668dd0d62205da20b8ec\": container with ID starting with dcdd869e13d53e7930a087d3164bb4a7ccd54a044d4b668dd0d62205da20b8ec not found: ID does not exist"
Sep 16 18:43:41.509852 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:43:41.509812    2578 scope.go:117] "RemoveContainer" containerID="9eaf47131b84c369eb6ec0858bd64381dd4b855a7ef5d05c1c2006278a3ef205"
Sep 16 18:43:41.510049 ip-10-0-135-119 kubenswrapper[2578]: E0916 18:43:41.510032    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9eaf47131b84c369eb6ec0858bd64381dd4b855a7ef5d05c1c2006278a3ef205\": container with ID starting with 9eaf47131b84c369eb6ec0858bd64381dd4b855a7ef5d05c1c2006278a3ef205 not found: ID does not exist" containerID="9eaf47131b84c369eb6ec0858bd64381dd4b855a7ef5d05c1c2006278a3ef205"
Sep 16 18:43:41.510099 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:43:41.510054    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9eaf47131b84c369eb6ec0858bd64381dd4b855a7ef5d05c1c2006278a3ef205"} err="failed to get container status \"9eaf47131b84c369eb6ec0858bd64381dd4b855a7ef5d05c1c2006278a3ef205\": rpc error: code = NotFound desc = could not find container \"9eaf47131b84c369eb6ec0858bd64381dd4b855a7ef5d05c1c2006278a3ef205\": container with ID starting with 9eaf47131b84c369eb6ec0858bd64381dd4b855a7ef5d05c1c2006278a3ef205 not found: ID does not exist"
Sep 16 18:43:41.510099 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:43:41.510068    2578 scope.go:117] "RemoveContainer" containerID="0b50d25dea85bb8df0c47b2ecb83c2827ecda8d645111c4069a97022dfc08826"
Sep 16 18:43:41.510343 ip-10-0-135-119 kubenswrapper[2578]: E0916 18:43:41.510326    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0b50d25dea85bb8df0c47b2ecb83c2827ecda8d645111c4069a97022dfc08826\": container with ID starting with 0b50d25dea85bb8df0c47b2ecb83c2827ecda8d645111c4069a97022dfc08826 not found: ID does not exist" containerID="0b50d25dea85bb8df0c47b2ecb83c2827ecda8d645111c4069a97022dfc08826"
Sep 16 18:43:41.510412 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:43:41.510348    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0b50d25dea85bb8df0c47b2ecb83c2827ecda8d645111c4069a97022dfc08826"} err="failed to get container status \"0b50d25dea85bb8df0c47b2ecb83c2827ecda8d645111c4069a97022dfc08826\": rpc error: code = NotFound desc = could not find container \"0b50d25dea85bb8df0c47b2ecb83c2827ecda8d645111c4069a97022dfc08826\": container with ID starting with 0b50d25dea85bb8df0c47b2ecb83c2827ecda8d645111c4069a97022dfc08826 not found: ID does not exist"
Sep 16 18:43:41.540397 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:43:41.540375    2578 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/686f396c-6b8e-44a8-83b8-4a38dd7b5644-secret-prometheus-k8s-tls\") on node \"ip-10-0-135-119.ec2.internal\" DevicePath \"\""
Sep 16 18:43:41.540397 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:43:41.540396    2578 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/686f396c-6b8e-44a8-83b8-4a38dd7b5644-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-135-119.ec2.internal\" DevicePath \"\""
Sep 16 18:43:41.540498 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:43:41.540406    2578 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/686f396c-6b8e-44a8-83b8-4a38dd7b5644-secret-metrics-client-certs\") on node \"ip-10-0-135-119.ec2.internal\" DevicePath \"\""
Sep 16 18:43:41.540498 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:43:41.540415    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7wm7v\" (UniqueName: \"kubernetes.io/projected/686f396c-6b8e-44a8-83b8-4a38dd7b5644-kube-api-access-7wm7v\") on node \"ip-10-0-135-119.ec2.internal\" DevicePath \"\""
Sep 16 18:43:41.540498 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:43:41.540424    2578 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/686f396c-6b8e-44a8-83b8-4a38dd7b5644-tls-assets\") on node \"ip-10-0-135-119.ec2.internal\" DevicePath \"\""
Sep 16 18:43:41.540498 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:43:41.540440    2578 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/686f396c-6b8e-44a8-83b8-4a38dd7b5644-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-135-119.ec2.internal\" DevicePath \"\""
Sep 16 18:43:41.540498 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:43:41.540451    2578 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/686f396c-6b8e-44a8-83b8-4a38dd7b5644-secret-grpc-tls\") on node \"ip-10-0-135-119.ec2.internal\" DevicePath \"\""
Sep 16 18:43:41.540498 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:43:41.540459    2578 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/686f396c-6b8e-44a8-83b8-4a38dd7b5644-config-out\") on node \"ip-10-0-135-119.ec2.internal\" DevicePath \"\""
Sep 16 18:43:41.540498 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:43:41.540467    2578 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/686f396c-6b8e-44a8-83b8-4a38dd7b5644-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-135-119.ec2.internal\" DevicePath \"\""
Sep 16 18:43:41.540498 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:43:41.540475    2578 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/686f396c-6b8e-44a8-83b8-4a38dd7b5644-thanos-prometheus-http-client-file\") on node \"ip-10-0-135-119.ec2.internal\" DevicePath \"\""
Sep 16 18:43:41.540498 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:43:41.540484    2578 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/686f396c-6b8e-44a8-83b8-4a38dd7b5644-configmap-metrics-client-ca\") on node \"ip-10-0-135-119.ec2.internal\" DevicePath \"\""
Sep 16 18:43:41.540498 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:43:41.540493    2578 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/686f396c-6b8e-44a8-83b8-4a38dd7b5644-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-135-119.ec2.internal\" DevicePath \"\""
Sep 16 18:43:41.540809 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:43:41.540503    2578 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/686f396c-6b8e-44a8-83b8-4a38dd7b5644-config\") on node \"ip-10-0-135-119.ec2.internal\" DevicePath \"\""
Sep 16 18:43:41.540809 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:43:41.540514    2578 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/686f396c-6b8e-44a8-83b8-4a38dd7b5644-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-135-119.ec2.internal\" DevicePath \"\""
Sep 16 18:43:41.540809 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:43:41.540528    2578 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/686f396c-6b8e-44a8-83b8-4a38dd7b5644-prometheus-trusted-ca-bundle\") on node \"ip-10-0-135-119.ec2.internal\" DevicePath \"\""
Sep 16 18:43:41.540809 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:43:41.540536    2578 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/686f396c-6b8e-44a8-83b8-4a38dd7b5644-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-135-119.ec2.internal\" DevicePath \"\""
Sep 16 18:43:41.540809 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:43:41.540545    2578 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/686f396c-6b8e-44a8-83b8-4a38dd7b5644-prometheus-k8s-db\") on node \"ip-10-0-135-119.ec2.internal\" DevicePath \"\""
Sep 16 18:43:41.540809 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:43:41.540565    2578 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/686f396c-6b8e-44a8-83b8-4a38dd7b5644-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-135-119.ec2.internal\" DevicePath \"\""
Sep 16 18:43:41.540809 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:43:41.540574    2578 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/686f396c-6b8e-44a8-83b8-4a38dd7b5644-secret-kube-rbac-proxy\") on node \"ip-10-0-135-119.ec2.internal\" DevicePath \"\""
Sep 16 18:43:41.540809 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:43:41.540603    2578 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/686f396c-6b8e-44a8-83b8-4a38dd7b5644-web-config\") on node \"ip-10-0-135-119.ec2.internal\" DevicePath \"\""
Sep 16 18:43:41.769037 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:43:41.768959    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 16 18:43:41.776027 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:43:41.776001    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 16 18:43:41.803651 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:43:41.803626    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 16 18:43:41.803917 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:43:41.803904    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="686f396c-6b8e-44a8-83b8-4a38dd7b5644" containerName="kube-rbac-proxy"
Sep 16 18:43:41.803917 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:43:41.803918    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="686f396c-6b8e-44a8-83b8-4a38dd7b5644" containerName="kube-rbac-proxy"
Sep 16 18:43:41.804034 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:43:41.803929    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="92866a88-93b2-430b-867b-40e3358eb631" containerName="registry"
Sep 16 18:43:41.804034 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:43:41.803935    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="92866a88-93b2-430b-867b-40e3358eb631" containerName="registry"
Sep 16 18:43:41.804034 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:43:41.803942    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="686f396c-6b8e-44a8-83b8-4a38dd7b5644" containerName="init-config-reloader"
Sep 16 18:43:41.804034 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:43:41.803948    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="686f396c-6b8e-44a8-83b8-4a38dd7b5644" containerName="init-config-reloader"
Sep 16 18:43:41.804034 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:43:41.803957    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="686f396c-6b8e-44a8-83b8-4a38dd7b5644" containerName="config-reloader"
Sep 16 18:43:41.804034 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:43:41.803962    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="686f396c-6b8e-44a8-83b8-4a38dd7b5644" containerName="config-reloader"
Sep 16 18:43:41.804034 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:43:41.803969    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="686f396c-6b8e-44a8-83b8-4a38dd7b5644" containerName="kube-rbac-proxy-thanos"
Sep 16 18:43:41.804034 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:43:41.803987    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="686f396c-6b8e-44a8-83b8-4a38dd7b5644" containerName="kube-rbac-proxy-thanos"
Sep 16 18:43:41.804034 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:43:41.803994    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="686f396c-6b8e-44a8-83b8-4a38dd7b5644" containerName="prometheus"
Sep 16 18:43:41.804034 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:43:41.803999    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="686f396c-6b8e-44a8-83b8-4a38dd7b5644" containerName="prometheus"
Sep 16 18:43:41.804034 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:43:41.804005    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="686f396c-6b8e-44a8-83b8-4a38dd7b5644" containerName="thanos-sidecar"
Sep 16 18:43:41.804034 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:43:41.804010    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="686f396c-6b8e-44a8-83b8-4a38dd7b5644" containerName="thanos-sidecar"
Sep 16 18:43:41.804034 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:43:41.804021    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="686f396c-6b8e-44a8-83b8-4a38dd7b5644" containerName="kube-rbac-proxy-web"
Sep 16 18:43:41.804034 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:43:41.804026    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="686f396c-6b8e-44a8-83b8-4a38dd7b5644" containerName="kube-rbac-proxy-web"
Sep 16 18:43:41.804482 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:43:41.804084    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="686f396c-6b8e-44a8-83b8-4a38dd7b5644" containerName="prometheus"
Sep 16 18:43:41.804482 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:43:41.804093    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="686f396c-6b8e-44a8-83b8-4a38dd7b5644" containerName="thanos-sidecar"
Sep 16 18:43:41.804482 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:43:41.804099    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="686f396c-6b8e-44a8-83b8-4a38dd7b5644" containerName="kube-rbac-proxy"
Sep 16 18:43:41.804482 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:43:41.804104    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="686f396c-6b8e-44a8-83b8-4a38dd7b5644" containerName="config-reloader"
Sep 16 18:43:41.804482 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:43:41.804111    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="92866a88-93b2-430b-867b-40e3358eb631" containerName="registry"
Sep 16 18:43:41.804482 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:43:41.804116    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="686f396c-6b8e-44a8-83b8-4a38dd7b5644" containerName="kube-rbac-proxy-thanos"
Sep 16 18:43:41.804482 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:43:41.804122    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="686f396c-6b8e-44a8-83b8-4a38dd7b5644" containerName="kube-rbac-proxy-web"
Sep 16 18:43:41.807572 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:43:41.807554    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:43:41.811242 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:43:41.811216    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Sep 16 18:43:41.811375 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:43:41.811343    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Sep 16 18:43:41.811446 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:43:41.811421    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Sep 16 18:43:41.811496 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:43:41.811445    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Sep 16 18:43:41.811544 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:43:41.811219    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Sep 16 18:43:41.811604 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:43:41.811553    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Sep 16 18:43:41.811604 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:43:41.811581    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-6o7c2r390kn4a\""
Sep 16 18:43:41.811788 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:43:41.811648    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Sep 16 18:43:41.811788 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:43:41.811666    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Sep 16 18:43:41.811788 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:43:41.811731    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Sep 16 18:43:41.811788 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:43:41.811746    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Sep 16 18:43:41.812072 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:43:41.811789    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Sep 16 18:43:41.812072 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:43:41.812018    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-cc7kj\""
Sep 16 18:43:41.812186 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:43:41.812124    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Sep 16 18:43:41.813432 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:43:41.813408    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Sep 16 18:43:41.816184 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:43:41.816145    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Sep 16 18:43:41.830453 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:43:41.830433    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 16 18:43:41.944012 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:43:41.943967    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/f501e149-38f4-4f16-aaed-225fcd0faf34-config\") pod \"prometheus-k8s-0\" (UID: \"f501e149-38f4-4f16-aaed-225fcd0faf34\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:43:41.944188 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:43:41.944021    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l9lj5\" (UniqueName: \"kubernetes.io/projected/f501e149-38f4-4f16-aaed-225fcd0faf34-kube-api-access-l9lj5\") pod \"prometheus-k8s-0\" (UID: \"f501e149-38f4-4f16-aaed-225fcd0faf34\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:43:41.944188 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:43:41.944091    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/f501e149-38f4-4f16-aaed-225fcd0faf34-config-out\") pod \"prometheus-k8s-0\" (UID: \"f501e149-38f4-4f16-aaed-225fcd0faf34\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:43:41.944188 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:43:41.944110    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/f501e149-38f4-4f16-aaed-225fcd0faf34-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"f501e149-38f4-4f16-aaed-225fcd0faf34\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:43:41.944188 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:43:41.944129    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/f501e149-38f4-4f16-aaed-225fcd0faf34-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"f501e149-38f4-4f16-aaed-225fcd0faf34\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:43:41.944188 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:43:41.944154    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/f501e149-38f4-4f16-aaed-225fcd0faf34-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"f501e149-38f4-4f16-aaed-225fcd0faf34\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:43:41.944188 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:43:41.944170    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f501e149-38f4-4f16-aaed-225fcd0faf34-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"f501e149-38f4-4f16-aaed-225fcd0faf34\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:43:41.944490 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:43:41.944215    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/f501e149-38f4-4f16-aaed-225fcd0faf34-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"f501e149-38f4-4f16-aaed-225fcd0faf34\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:43:41.944490 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:43:41.944256    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/f501e149-38f4-4f16-aaed-225fcd0faf34-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"f501e149-38f4-4f16-aaed-225fcd0faf34\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:43:41.944490 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:43:41.944285    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f501e149-38f4-4f16-aaed-225fcd0faf34-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"f501e149-38f4-4f16-aaed-225fcd0faf34\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:43:41.944490 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:43:41.944311    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/f501e149-38f4-4f16-aaed-225fcd0faf34-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"f501e149-38f4-4f16-aaed-225fcd0faf34\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:43:41.944490 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:43:41.944344    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/f501e149-38f4-4f16-aaed-225fcd0faf34-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"f501e149-38f4-4f16-aaed-225fcd0faf34\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:43:41.944490 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:43:41.944367    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/f501e149-38f4-4f16-aaed-225fcd0faf34-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"f501e149-38f4-4f16-aaed-225fcd0faf34\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:43:41.944490 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:43:41.944392    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/f501e149-38f4-4f16-aaed-225fcd0faf34-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"f501e149-38f4-4f16-aaed-225fcd0faf34\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:43:41.944490 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:43:41.944417    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/f501e149-38f4-4f16-aaed-225fcd0faf34-web-config\") pod \"prometheus-k8s-0\" (UID: \"f501e149-38f4-4f16-aaed-225fcd0faf34\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:43:41.944490 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:43:41.944441    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/f501e149-38f4-4f16-aaed-225fcd0faf34-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"f501e149-38f4-4f16-aaed-225fcd0faf34\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:43:41.944783 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:43:41.944509    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f501e149-38f4-4f16-aaed-225fcd0faf34-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"f501e149-38f4-4f16-aaed-225fcd0faf34\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:43:41.944783 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:43:41.944532    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f501e149-38f4-4f16-aaed-225fcd0faf34-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"f501e149-38f4-4f16-aaed-225fcd0faf34\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:43:41.944783 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:43:41.944565    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f501e149-38f4-4f16-aaed-225fcd0faf34-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"f501e149-38f4-4f16-aaed-225fcd0faf34\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:43:41.944783 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:43:41.944590    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f501e149-38f4-4f16-aaed-225fcd0faf34-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"f501e149-38f4-4f16-aaed-225fcd0faf34\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:43:42.045343 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:43:42.045274    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l9lj5\" (UniqueName: \"kubernetes.io/projected/f501e149-38f4-4f16-aaed-225fcd0faf34-kube-api-access-l9lj5\") pod \"prometheus-k8s-0\" (UID: \"f501e149-38f4-4f16-aaed-225fcd0faf34\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:43:42.045343 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:43:42.045308    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/f501e149-38f4-4f16-aaed-225fcd0faf34-config-out\") pod \"prometheus-k8s-0\" (UID: \"f501e149-38f4-4f16-aaed-225fcd0faf34\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:43:42.045343 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:43:42.045326    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/f501e149-38f4-4f16-aaed-225fcd0faf34-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"f501e149-38f4-4f16-aaed-225fcd0faf34\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:43:42.045343 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:43:42.045345    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/f501e149-38f4-4f16-aaed-225fcd0faf34-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"f501e149-38f4-4f16-aaed-225fcd0faf34\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:43:42.045646 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:43:42.045370    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/f501e149-38f4-4f16-aaed-225fcd0faf34-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"f501e149-38f4-4f16-aaed-225fcd0faf34\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:43:42.045646 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:43:42.045394    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f501e149-38f4-4f16-aaed-225fcd0faf34-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"f501e149-38f4-4f16-aaed-225fcd0faf34\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:43:42.045646 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:43:42.045420    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/f501e149-38f4-4f16-aaed-225fcd0faf34-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"f501e149-38f4-4f16-aaed-225fcd0faf34\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:43:42.045646 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:43:42.045448    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/f501e149-38f4-4f16-aaed-225fcd0faf34-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"f501e149-38f4-4f16-aaed-225fcd0faf34\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:43:42.045646 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:43:42.045471    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f501e149-38f4-4f16-aaed-225fcd0faf34-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"f501e149-38f4-4f16-aaed-225fcd0faf34\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:43:42.045646 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:43:42.045496    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/f501e149-38f4-4f16-aaed-225fcd0faf34-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"f501e149-38f4-4f16-aaed-225fcd0faf34\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:43:42.045646 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:43:42.045519    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/f501e149-38f4-4f16-aaed-225fcd0faf34-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"f501e149-38f4-4f16-aaed-225fcd0faf34\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:43:42.045646 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:43:42.045541    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/f501e149-38f4-4f16-aaed-225fcd0faf34-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"f501e149-38f4-4f16-aaed-225fcd0faf34\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:43:42.045646 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:43:42.045571    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/f501e149-38f4-4f16-aaed-225fcd0faf34-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"f501e149-38f4-4f16-aaed-225fcd0faf34\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:43:42.045646 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:43:42.045603    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/f501e149-38f4-4f16-aaed-225fcd0faf34-web-config\") pod \"prometheus-k8s-0\" (UID: \"f501e149-38f4-4f16-aaed-225fcd0faf34\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:43:42.045646 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:43:42.045628    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/f501e149-38f4-4f16-aaed-225fcd0faf34-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"f501e149-38f4-4f16-aaed-225fcd0faf34\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:43:42.046261 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:43:42.045672    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f501e149-38f4-4f16-aaed-225fcd0faf34-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"f501e149-38f4-4f16-aaed-225fcd0faf34\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:43:42.046261 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:43:42.045699    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f501e149-38f4-4f16-aaed-225fcd0faf34-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"f501e149-38f4-4f16-aaed-225fcd0faf34\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:43:42.046261 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:43:42.045738    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/f501e149-38f4-4f16-aaed-225fcd0faf34-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"f501e149-38f4-4f16-aaed-225fcd0faf34\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:43:42.046261 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:43:42.045740    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f501e149-38f4-4f16-aaed-225fcd0faf34-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"f501e149-38f4-4f16-aaed-225fcd0faf34\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:43:42.046261 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:43:42.045793    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f501e149-38f4-4f16-aaed-225fcd0faf34-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"f501e149-38f4-4f16-aaed-225fcd0faf34\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:43:42.046261 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:43:42.045827    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/f501e149-38f4-4f16-aaed-225fcd0faf34-config\") pod \"prometheus-k8s-0\" (UID: \"f501e149-38f4-4f16-aaed-225fcd0faf34\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:43:42.046261 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:43:42.046023    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/f501e149-38f4-4f16-aaed-225fcd0faf34-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"f501e149-38f4-4f16-aaed-225fcd0faf34\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:43:42.049197 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:43:42.048812    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f501e149-38f4-4f16-aaed-225fcd0faf34-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"f501e149-38f4-4f16-aaed-225fcd0faf34\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:43:42.049197 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:43:42.048855    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/f501e149-38f4-4f16-aaed-225fcd0faf34-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"f501e149-38f4-4f16-aaed-225fcd0faf34\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:43:42.049197 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:43:42.049011    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/f501e149-38f4-4f16-aaed-225fcd0faf34-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"f501e149-38f4-4f16-aaed-225fcd0faf34\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:43:42.049197 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:43:42.049037    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/f501e149-38f4-4f16-aaed-225fcd0faf34-config\") pod \"prometheus-k8s-0\" (UID: \"f501e149-38f4-4f16-aaed-225fcd0faf34\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:43:42.049197 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:43:42.049064    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/f501e149-38f4-4f16-aaed-225fcd0faf34-config-out\") pod \"prometheus-k8s-0\" (UID: \"f501e149-38f4-4f16-aaed-225fcd0faf34\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:43:42.049197 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:43:42.049092    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/f501e149-38f4-4f16-aaed-225fcd0faf34-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"f501e149-38f4-4f16-aaed-225fcd0faf34\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:43:42.049197 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:43:42.049130    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/f501e149-38f4-4f16-aaed-225fcd0faf34-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"f501e149-38f4-4f16-aaed-225fcd0faf34\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:43:42.050029 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:43:42.049364    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/f501e149-38f4-4f16-aaed-225fcd0faf34-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"f501e149-38f4-4f16-aaed-225fcd0faf34\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:43:42.050029 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:43:42.049412    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/f501e149-38f4-4f16-aaed-225fcd0faf34-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"f501e149-38f4-4f16-aaed-225fcd0faf34\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:43:42.050029 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:43:42.049644    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f501e149-38f4-4f16-aaed-225fcd0faf34-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"f501e149-38f4-4f16-aaed-225fcd0faf34\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:43:42.050029 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:43:42.049745    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f501e149-38f4-4f16-aaed-225fcd0faf34-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"f501e149-38f4-4f16-aaed-225fcd0faf34\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:43:42.050029 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:43:42.049841    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f501e149-38f4-4f16-aaed-225fcd0faf34-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"f501e149-38f4-4f16-aaed-225fcd0faf34\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:43:42.050300 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:43:42.050144    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f501e149-38f4-4f16-aaed-225fcd0faf34-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"f501e149-38f4-4f16-aaed-225fcd0faf34\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:43:42.051355 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:43:42.051327    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/f501e149-38f4-4f16-aaed-225fcd0faf34-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"f501e149-38f4-4f16-aaed-225fcd0faf34\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:43:42.051534 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:43:42.051513    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/f501e149-38f4-4f16-aaed-225fcd0faf34-web-config\") pod \"prometheus-k8s-0\" (UID: \"f501e149-38f4-4f16-aaed-225fcd0faf34\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:43:42.051688 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:43:42.051670    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f501e149-38f4-4f16-aaed-225fcd0faf34-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"f501e149-38f4-4f16-aaed-225fcd0faf34\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:43:42.052723 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:43:42.052703    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/f501e149-38f4-4f16-aaed-225fcd0faf34-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"f501e149-38f4-4f16-aaed-225fcd0faf34\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:43:42.054105 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:43:42.054085    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l9lj5\" (UniqueName: \"kubernetes.io/projected/f501e149-38f4-4f16-aaed-225fcd0faf34-kube-api-access-l9lj5\") pod \"prometheus-k8s-0\" (UID: \"f501e149-38f4-4f16-aaed-225fcd0faf34\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:43:42.119338 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:43:42.119289    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:43:42.396872 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:43:42.396842    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 16 18:43:42.399204 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:43:42.399174    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf501e149_38f4_4f16_aaed_225fcd0faf34.slice/crio-fbc88aa25fc2a4ffaabc1f7cdc1d6b9de9c3315b8e12d3243872211a73cf6406 WatchSource:0}: Error finding container fbc88aa25fc2a4ffaabc1f7cdc1d6b9de9c3315b8e12d3243872211a73cf6406: Status 404 returned error can't find the container with id fbc88aa25fc2a4ffaabc1f7cdc1d6b9de9c3315b8e12d3243872211a73cf6406
Sep 16 18:43:42.448177 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:43:42.448153    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f501e149-38f4-4f16-aaed-225fcd0faf34","Type":"ContainerStarted","Data":"fbc88aa25fc2a4ffaabc1f7cdc1d6b9de9c3315b8e12d3243872211a73cf6406"}
Sep 16 18:43:43.433059 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:43:43.433025    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="686f396c-6b8e-44a8-83b8-4a38dd7b5644" path="/var/lib/kubelet/pods/686f396c-6b8e-44a8-83b8-4a38dd7b5644/volumes"
Sep 16 18:43:43.439805 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:43:43.439778    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 18:43:43.453845 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:43:43.453816    2578 generic.go:356] "Generic (PLEG): container finished" podID="f501e149-38f4-4f16-aaed-225fcd0faf34" containerID="e34c132a0b136d2e830cefa8d9b86a4fb82b13bc101afa6cf0ac91713cbf00b1" exitCode=0
Sep 16 18:43:43.453930 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:43:43.453895    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f501e149-38f4-4f16-aaed-225fcd0faf34","Type":"ContainerDied","Data":"e34c132a0b136d2e830cefa8d9b86a4fb82b13bc101afa6cf0ac91713cbf00b1"}
Sep 16 18:43:43.540347 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:43:43.540320    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 18:43:44.461269 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:43:44.461227    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f501e149-38f4-4f16-aaed-225fcd0faf34","Type":"ContainerStarted","Data":"da0e9cbfdd489734029a044b3cd84e6728292cc9d655ebb6ff3b11519d34587a"}
Sep 16 18:43:44.461636 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:43:44.461321    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f501e149-38f4-4f16-aaed-225fcd0faf34","Type":"ContainerStarted","Data":"e992b16b6ac3991d17f3629c795d45b3bdf5d16c1006ddf751391a214491a86a"}
Sep 16 18:43:44.461636 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:43:44.461352    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f501e149-38f4-4f16-aaed-225fcd0faf34","Type":"ContainerStarted","Data":"ed82017aa14afcbdc9c44974063526ed51387cf6788a2e15534ffa954df099ff"}
Sep 16 18:43:44.461636 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:43:44.461378    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f501e149-38f4-4f16-aaed-225fcd0faf34","Type":"ContainerStarted","Data":"ea84661e1e2f9040e3b990b9db666c41049d1818ac331d5ca2f6460bd9b12fe9"}
Sep 16 18:43:44.461636 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:43:44.461416    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f501e149-38f4-4f16-aaed-225fcd0faf34","Type":"ContainerStarted","Data":"604c9b410ce2ec929d72f019d49da2a492778249def99e65bc5af1d93bada397"}
Sep 16 18:43:44.461636 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:43:44.461438    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f501e149-38f4-4f16-aaed-225fcd0faf34","Type":"ContainerStarted","Data":"143035a1385af8b27b42fc6ad1c98268eb70b27118dd9c51e4dde63b2a17939b"}
Sep 16 18:43:44.489360 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:43:44.489311    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=3.470676786 podStartE2EDuration="3.489295151s" totalImagesPullingTime="0s" totalInitContainerRuntime="18.618365ms" isStatefulPod=false podCreationTimestamp="2026-09-16 18:43:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:43:44.487965304 +0000 UTC m=+205.554456847" watchObservedRunningTime="2026-09-16 18:43:44.489295151 +0000 UTC m=+205.555786639"
Sep 16 18:43:45.201456 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:43:45.201417    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 18:43:45.302245 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:43:45.302199    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 18:43:45.364316 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:43:45.364285    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 18:43:47.120245 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:43:47.120205    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:43:56.900329 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:43:56.900294    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 18:43:56.995135 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:43:56.995106    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 18:44:40.126595 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:44:40.126552    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 18:44:40.181150 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:44:40.181122    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 18:44:42.120296 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:44:42.120237    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:44:42.137024 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:44:42.136959    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:44:42.707344 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:44:42.707311    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:44:51.868185 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:44:51.868138    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 18:44:51.969771 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:44:51.969734    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 18:44:52.072792 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:44:52.072758    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 18:45:02.409108 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:45:02.409072    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 18:45:14.344686 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:45:14.344642    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 18:45:14.458082 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:45:14.458031    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 18:45:17.208707 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:45:17.208672    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 18:45:17.319864 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:45:17.319828    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 18:45:19.310707 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:45:19.310668    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-tmrkx_4c2078bb-281a-4aa1-8e38-238897865434/console-operator/2.log"
Sep 16 18:45:19.314386 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:45:19.314356    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-tmrkx_4c2078bb-281a-4aa1-8e38-238897865434/console-operator/2.log"
Sep 16 18:45:19.320910 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:45:19.320882    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-k6gvs_6012728d-aee6-4f93-ab6e-fad616ef36e8/ovn-acl-logging/0.log"
Sep 16 18:45:19.325682 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:45:19.325652    2578 kubelet.go:1648] "Image garbage collection succeeded"
Sep 16 18:45:19.330107 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:45:19.330084    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-k6gvs_6012728d-aee6-4f93-ab6e-fad616ef36e8/ovn-acl-logging/0.log"
Sep 16 18:45:20.402424 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:45:20.402372    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 18:45:20.684111 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:45:20.684028    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 18:45:20.902284 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:45:20.902250    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 18:45:21.305546 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:45:21.305493    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 18:45:21.404097 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:45:21.404057    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 18:45:21.510370 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:45:21.510340    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 18:45:21.710367 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:45:21.710337    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 18:45:21.808799 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:45:21.808768    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 18:45:21.904562 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:45:21.904532    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 18:45:21.966638 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:45:21.966532    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 18:46:13.196847 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:46:13.196800    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 18:46:13.306120 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:46:13.306090    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 18:46:21.694527 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:46:21.694494    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 18:46:24.384925 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:46:24.384883    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 18:46:25.156660 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:46:25.156629    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 18:46:28.213198 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:46:28.213154    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 18:46:28.307276 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:46:28.307236    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 18:46:29.552226 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:46:29.552189    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 18:46:29.658864 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:46:29.658833    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 18:46:29.751422 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:46:29.751389    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 18:46:29.981191 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:46:29.981151    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 18:46:30.094476 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:46:30.094445    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 18:46:32.023329 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:46:32.023283    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 18:46:32.400850 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:46:32.400820    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-jdd9q"]
Sep 16 18:46:32.404083 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:46:32.404067    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-jdd9q"
Sep 16 18:46:32.406484 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:46:32.406462    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-mdlm4\""
Sep 16 18:46:32.406603 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:46:32.406469    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\""
Sep 16 18:46:32.407059 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:46:32.407042    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\""
Sep 16 18:46:32.417073 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:46:32.417044    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-jdd9q"]
Sep 16 18:46:32.430246 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:46:32.430225    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-km2tm\" (UniqueName: \"kubernetes.io/projected/630ed374-a506-4df2-8b53-6144003df89e-kube-api-access-km2tm\") pod \"cert-manager-cainjector-559cdd8c87-jdd9q\" (UID: \"630ed374-a506-4df2-8b53-6144003df89e\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-jdd9q"
Sep 16 18:46:32.430345 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:46:32.430261    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/630ed374-a506-4df2-8b53-6144003df89e-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-jdd9q\" (UID: \"630ed374-a506-4df2-8b53-6144003df89e\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-jdd9q"
Sep 16 18:46:32.531131 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:46:32.531094    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-km2tm\" (UniqueName: \"kubernetes.io/projected/630ed374-a506-4df2-8b53-6144003df89e-kube-api-access-km2tm\") pod \"cert-manager-cainjector-559cdd8c87-jdd9q\" (UID: \"630ed374-a506-4df2-8b53-6144003df89e\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-jdd9q"
Sep 16 18:46:32.531131 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:46:32.531141    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/630ed374-a506-4df2-8b53-6144003df89e-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-jdd9q\" (UID: \"630ed374-a506-4df2-8b53-6144003df89e\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-jdd9q"
Sep 16 18:46:32.541049 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:46:32.541024    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/630ed374-a506-4df2-8b53-6144003df89e-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-jdd9q\" (UID: \"630ed374-a506-4df2-8b53-6144003df89e\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-jdd9q"
Sep 16 18:46:32.541180 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:46:32.541090    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-km2tm\" (UniqueName: \"kubernetes.io/projected/630ed374-a506-4df2-8b53-6144003df89e-kube-api-access-km2tm\") pod \"cert-manager-cainjector-559cdd8c87-jdd9q\" (UID: \"630ed374-a506-4df2-8b53-6144003df89e\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-jdd9q"
Sep 16 18:46:32.732142 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:46:32.732048    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-jdd9q"
Sep 16 18:46:33.008212 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:46:33.008140    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-jdd9q"]
Sep 16 18:46:33.011268 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:46:33.011241    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod630ed374_a506_4df2_8b53_6144003df89e.slice/crio-78286a6f24fa428e43845b185c0c9e7aaf935ebd2717c5284ea3711a7f3ede7e WatchSource:0}: Error finding container 78286a6f24fa428e43845b185c0c9e7aaf935ebd2717c5284ea3711a7f3ede7e: Status 404 returned error can't find the container with id 78286a6f24fa428e43845b185c0c9e7aaf935ebd2717c5284ea3711a7f3ede7e
Sep 16 18:46:33.013283 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:46:33.013267    2578 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 16 18:46:33.120327 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:46:33.120291    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-jdd9q" event={"ID":"630ed374-a506-4df2-8b53-6144003df89e","Type":"ContainerStarted","Data":"78286a6f24fa428e43845b185c0c9e7aaf935ebd2717c5284ea3711a7f3ede7e"}
Sep 16 18:46:33.678672 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:46:33.678635    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 18:46:34.997135 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:46:34.997104    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 18:46:35.107694 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:46:35.107661    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 18:46:37.138587 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:46:37.138557    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-jdd9q" event={"ID":"630ed374-a506-4df2-8b53-6144003df89e","Type":"ContainerStarted","Data":"184a2ecda5290140680e7e048f92cd02519817d982ee1c2db2a2f75878dbea58"}
Sep 16 18:46:37.163764 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:46:37.163716    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-559cdd8c87-jdd9q" podStartSLOduration=2.078018298 podStartE2EDuration="5.163700572s" totalImagesPullingTime="3.085682274s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:46:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:46:37.163392327 +0000 UTC m=+378.229883839" watchObservedRunningTime="2026-09-16 18:46:37.163700572 +0000 UTC m=+378.230192060"
Sep 16 18:46:40.584507 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:46:40.584478    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 18:46:40.802266 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:46:40.802231    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 18:46:40.898532 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:46:40.898505    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 18:46:41.909101 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:46:41.909065    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 18:46:42.007572 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:46:42.007541    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 18:46:43.029540 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:46:43.029503    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-r9d6z"]
Sep 16 18:46:43.032766 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:46:43.032743    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-r9d6z"
Sep 16 18:46:43.034863 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:46:43.034841    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\""
Sep 16 18:46:43.034961 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:46:43.034841    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"openshift-lws-operator-dockercfg-pgx5h\""
Sep 16 18:46:43.035511 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:46:43.035490    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\""
Sep 16 18:46:43.044697 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:46:43.044675    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-r9d6z"]
Sep 16 18:46:43.118899 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:46:43.118858    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xjwf6\" (UniqueName: \"kubernetes.io/projected/7f7087fe-a489-4aee-8e5e-c6b21fb64f8b-kube-api-access-xjwf6\") pod \"openshift-lws-operator-fd8ccff4c-r9d6z\" (UID: \"7f7087fe-a489-4aee-8e5e-c6b21fb64f8b\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-r9d6z"
Sep 16 18:46:43.119095 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:46:43.118908    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7f7087fe-a489-4aee-8e5e-c6b21fb64f8b-tmp\") pod \"openshift-lws-operator-fd8ccff4c-r9d6z\" (UID: \"7f7087fe-a489-4aee-8e5e-c6b21fb64f8b\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-r9d6z"
Sep 16 18:46:43.219305 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:46:43.219267    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xjwf6\" (UniqueName: \"kubernetes.io/projected/7f7087fe-a489-4aee-8e5e-c6b21fb64f8b-kube-api-access-xjwf6\") pod \"openshift-lws-operator-fd8ccff4c-r9d6z\" (UID: \"7f7087fe-a489-4aee-8e5e-c6b21fb64f8b\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-r9d6z"
Sep 16 18:46:43.219493 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:46:43.219319    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7f7087fe-a489-4aee-8e5e-c6b21fb64f8b-tmp\") pod \"openshift-lws-operator-fd8ccff4c-r9d6z\" (UID: \"7f7087fe-a489-4aee-8e5e-c6b21fb64f8b\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-r9d6z"
Sep 16 18:46:43.219730 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:46:43.219708    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7f7087fe-a489-4aee-8e5e-c6b21fb64f8b-tmp\") pod \"openshift-lws-operator-fd8ccff4c-r9d6z\" (UID: \"7f7087fe-a489-4aee-8e5e-c6b21fb64f8b\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-r9d6z"
Sep 16 18:46:43.232488 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:46:43.232462    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xjwf6\" (UniqueName: \"kubernetes.io/projected/7f7087fe-a489-4aee-8e5e-c6b21fb64f8b-kube-api-access-xjwf6\") pod \"openshift-lws-operator-fd8ccff4c-r9d6z\" (UID: \"7f7087fe-a489-4aee-8e5e-c6b21fb64f8b\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-r9d6z"
Sep 16 18:46:43.341128 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:46:43.341039    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-r9d6z"
Sep 16 18:46:43.625279 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:46:43.625203    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-r9d6z"]
Sep 16 18:46:43.628933 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:46:43.628905    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7f7087fe_a489_4aee_8e5e_c6b21fb64f8b.slice/crio-492e01e6f108e88fca1552df62aad9ea4c9b36b4f1df2400a3724f4369e2c507 WatchSource:0}: Error finding container 492e01e6f108e88fca1552df62aad9ea4c9b36b4f1df2400a3724f4369e2c507: Status 404 returned error can't find the container with id 492e01e6f108e88fca1552df62aad9ea4c9b36b4f1df2400a3724f4369e2c507
Sep 16 18:46:44.167066 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:46:44.167032    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-r9d6z" event={"ID":"7f7087fe-a489-4aee-8e5e-c6b21fb64f8b","Type":"ContainerStarted","Data":"492e01e6f108e88fca1552df62aad9ea4c9b36b4f1df2400a3724f4369e2c507"}
Sep 16 18:46:46.178490 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:46:46.178451    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-r9d6z" event={"ID":"7f7087fe-a489-4aee-8e5e-c6b21fb64f8b","Type":"ContainerStarted","Data":"3f88cadaab66fda0ac77008ef55fa5e4d776be6683fedf9eea3b01432176750e"}
Sep 16 18:46:46.198437 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:46:46.198378    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-r9d6z" podStartSLOduration=0.791506299 podStartE2EDuration="3.198357989s" totalImagesPullingTime="2.40685169s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:46:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:46:46.19666552 +0000 UTC m=+387.263157000" watchObservedRunningTime="2026-09-16 18:46:46.198357989 +0000 UTC m=+387.264849482"
Sep 16 18:46:52.988586 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:46:52.988548    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 18:46:57.797339 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:46:57.797301    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 18:46:57.874277 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:46:57.874243    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 18:46:58.820607 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:46:58.820565    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 18:46:59.235595 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:46:59.235561    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 18:46:59.332451 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:46:59.332418    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-846d5d4569-4tqdv"]
Sep 16 18:46:59.335720 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:46:59.335701    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-846d5d4569-4tqdv"
Sep 16 18:46:59.337915 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:46:59.337890    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\""
Sep 16 18:46:59.337915 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:46:59.337903    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\""
Sep 16 18:46:59.338733 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:46:59.338708    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\""
Sep 16 18:46:59.340314 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:46:59.340293    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-8qb4s\""
Sep 16 18:46:59.353466 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:46:59.353444    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-846d5d4569-4tqdv"]
Sep 16 18:46:59.362039 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:46:59.362006    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 18:46:59.466483 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:46:59.466452    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/339be3f7-ae11-4ab9-8a58-7aff3f24da03-manager-config\") pod \"lws-controller-manager-846d5d4569-4tqdv\" (UID: \"339be3f7-ae11-4ab9-8a58-7aff3f24da03\") " pod="openshift-lws-operator/lws-controller-manager-846d5d4569-4tqdv"
Sep 16 18:46:59.466675 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:46:59.466494    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/339be3f7-ae11-4ab9-8a58-7aff3f24da03-metrics-cert\") pod \"lws-controller-manager-846d5d4569-4tqdv\" (UID: \"339be3f7-ae11-4ab9-8a58-7aff3f24da03\") " pod="openshift-lws-operator/lws-controller-manager-846d5d4569-4tqdv"
Sep 16 18:46:59.466675 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:46:59.466517    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5dl7c\" (UniqueName: \"kubernetes.io/projected/339be3f7-ae11-4ab9-8a58-7aff3f24da03-kube-api-access-5dl7c\") pod \"lws-controller-manager-846d5d4569-4tqdv\" (UID: \"339be3f7-ae11-4ab9-8a58-7aff3f24da03\") " pod="openshift-lws-operator/lws-controller-manager-846d5d4569-4tqdv"
Sep 16 18:46:59.466675 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:46:59.466651    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/339be3f7-ae11-4ab9-8a58-7aff3f24da03-cert\") pod \"lws-controller-manager-846d5d4569-4tqdv\" (UID: \"339be3f7-ae11-4ab9-8a58-7aff3f24da03\") " pod="openshift-lws-operator/lws-controller-manager-846d5d4569-4tqdv"
Sep 16 18:46:59.568040 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:46:59.567943    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/339be3f7-ae11-4ab9-8a58-7aff3f24da03-cert\") pod \"lws-controller-manager-846d5d4569-4tqdv\" (UID: \"339be3f7-ae11-4ab9-8a58-7aff3f24da03\") " pod="openshift-lws-operator/lws-controller-manager-846d5d4569-4tqdv"
Sep 16 18:46:59.568332 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:46:59.568305    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/339be3f7-ae11-4ab9-8a58-7aff3f24da03-manager-config\") pod \"lws-controller-manager-846d5d4569-4tqdv\" (UID: \"339be3f7-ae11-4ab9-8a58-7aff3f24da03\") " pod="openshift-lws-operator/lws-controller-manager-846d5d4569-4tqdv"
Sep 16 18:46:59.568460 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:46:59.568351    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/339be3f7-ae11-4ab9-8a58-7aff3f24da03-metrics-cert\") pod \"lws-controller-manager-846d5d4569-4tqdv\" (UID: \"339be3f7-ae11-4ab9-8a58-7aff3f24da03\") " pod="openshift-lws-operator/lws-controller-manager-846d5d4569-4tqdv"
Sep 16 18:46:59.568460 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:46:59.568376    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5dl7c\" (UniqueName: \"kubernetes.io/projected/339be3f7-ae11-4ab9-8a58-7aff3f24da03-kube-api-access-5dl7c\") pod \"lws-controller-manager-846d5d4569-4tqdv\" (UID: \"339be3f7-ae11-4ab9-8a58-7aff3f24da03\") " pod="openshift-lws-operator/lws-controller-manager-846d5d4569-4tqdv"
Sep 16 18:46:59.568882 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:46:59.568860    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/339be3f7-ae11-4ab9-8a58-7aff3f24da03-manager-config\") pod \"lws-controller-manager-846d5d4569-4tqdv\" (UID: \"339be3f7-ae11-4ab9-8a58-7aff3f24da03\") " pod="openshift-lws-operator/lws-controller-manager-846d5d4569-4tqdv"
Sep 16 18:46:59.570786 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:46:59.570756    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/339be3f7-ae11-4ab9-8a58-7aff3f24da03-cert\") pod \"lws-controller-manager-846d5d4569-4tqdv\" (UID: \"339be3f7-ae11-4ab9-8a58-7aff3f24da03\") " pod="openshift-lws-operator/lws-controller-manager-846d5d4569-4tqdv"
Sep 16 18:46:59.570786 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:46:59.570777    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/339be3f7-ae11-4ab9-8a58-7aff3f24da03-metrics-cert\") pod \"lws-controller-manager-846d5d4569-4tqdv\" (UID: \"339be3f7-ae11-4ab9-8a58-7aff3f24da03\") " pod="openshift-lws-operator/lws-controller-manager-846d5d4569-4tqdv"
Sep 16 18:46:59.577116 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:46:59.577092    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5dl7c\" (UniqueName: \"kubernetes.io/projected/339be3f7-ae11-4ab9-8a58-7aff3f24da03-kube-api-access-5dl7c\") pod \"lws-controller-manager-846d5d4569-4tqdv\" (UID: \"339be3f7-ae11-4ab9-8a58-7aff3f24da03\") " pod="openshift-lws-operator/lws-controller-manager-846d5d4569-4tqdv"
Sep 16 18:46:59.649290 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:46:59.649254    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-846d5d4569-4tqdv"
Sep 16 18:46:59.939522 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:46:59.939496    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-846d5d4569-4tqdv"]
Sep 16 18:46:59.941488 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:46:59.941462    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod339be3f7_ae11_4ab9_8a58_7aff3f24da03.slice/crio-76ba311e2e951b074db8c8c10da90241e993d15b4812ca08297f5751740d291d WatchSource:0}: Error finding container 76ba311e2e951b074db8c8c10da90241e993d15b4812ca08297f5751740d291d: Status 404 returned error can't find the container with id 76ba311e2e951b074db8c8c10da90241e993d15b4812ca08297f5751740d291d
Sep 16 18:47:00.235401 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:47:00.235309    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-846d5d4569-4tqdv" event={"ID":"339be3f7-ae11-4ab9-8a58-7aff3f24da03","Type":"ContainerStarted","Data":"76ba311e2e951b074db8c8c10da90241e993d15b4812ca08297f5751740d291d"}
Sep 16 18:47:01.681242 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:47:01.681210    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 18:47:01.784191 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:47:01.784157    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 18:47:02.244583 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:47:02.244551    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-846d5d4569-4tqdv" event={"ID":"339be3f7-ae11-4ab9-8a58-7aff3f24da03","Type":"ContainerStarted","Data":"73ee128a746d344aec3cb24293ec960a8a8e42da5e4f676945ba9a2959fcbc9b"}
Sep 16 18:47:02.244758 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:47:02.244648    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-846d5d4569-4tqdv"
Sep 16 18:47:04.688920 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:47:04.688880    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 18:47:04.793088 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:47:04.793052    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 18:47:07.721845 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:47:07.721818    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 18:47:10.177669 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:47:10.177620    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 18:47:10.292770 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:47:10.292733    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 18:47:13.251773 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:47:13.251736    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-846d5d4569-4tqdv"
Sep 16 18:47:13.281089 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:47:13.281037    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-846d5d4569-4tqdv" podStartSLOduration=13.026217045 podStartE2EDuration="14.28102163s" totalImagesPullingTime="1.254804585s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:46:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:47:02.265003328 +0000 UTC m=+403.331494818" watchObservedRunningTime="2026-09-16 18:47:13.28102163 +0000 UTC m=+414.347513117"
Sep 16 18:47:28.022273 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:47:28.022229    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 18:47:30.082948 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:47:30.082911    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-5ffb4b4ccb-4djgv"]
Sep 16 18:47:30.088718 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:47:30.088699    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-5ffb4b4ccb-4djgv"
Sep 16 18:47:30.090909 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:47:30.090887    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Sep 16 18:47:30.091403 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:47:30.091385    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Sep 16 18:47:30.091505 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:47:30.091403    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-v7m2x\""
Sep 16 18:47:30.104916 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:47:30.104890    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-5ffb4b4ccb-4djgv"]
Sep 16 18:47:30.122772 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:47:30.122742    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j64wn\" (UniqueName: \"kubernetes.io/projected/734cbe34-515a-4d3a-bdb5-b6914ab46948-kube-api-access-j64wn\") pod \"limitador-operator-controller-manager-5ffb4b4ccb-4djgv\" (UID: \"734cbe34-515a-4d3a-bdb5-b6914ab46948\") " pod="kuadrant-system/limitador-operator-controller-manager-5ffb4b4ccb-4djgv"
Sep 16 18:47:30.224125 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:47:30.224082    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j64wn\" (UniqueName: \"kubernetes.io/projected/734cbe34-515a-4d3a-bdb5-b6914ab46948-kube-api-access-j64wn\") pod \"limitador-operator-controller-manager-5ffb4b4ccb-4djgv\" (UID: \"734cbe34-515a-4d3a-bdb5-b6914ab46948\") " pod="kuadrant-system/limitador-operator-controller-manager-5ffb4b4ccb-4djgv"
Sep 16 18:47:30.237087 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:47:30.237054    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j64wn\" (UniqueName: \"kubernetes.io/projected/734cbe34-515a-4d3a-bdb5-b6914ab46948-kube-api-access-j64wn\") pod \"limitador-operator-controller-manager-5ffb4b4ccb-4djgv\" (UID: \"734cbe34-515a-4d3a-bdb5-b6914ab46948\") " pod="kuadrant-system/limitador-operator-controller-manager-5ffb4b4ccb-4djgv"
Sep 16 18:47:30.398433 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:47:30.398399    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-5ffb4b4ccb-4djgv"
Sep 16 18:47:30.628469 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:47:30.628434    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 18:47:30.675719 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:47:30.675682    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-5ffb4b4ccb-4djgv"]
Sep 16 18:47:30.678154 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:47:30.678130    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod734cbe34_515a_4d3a_bdb5_b6914ab46948.slice/crio-5327f250c88b51568f0f47d2d34b76e5a8bf440c2907e6c442c2bd29e0cd6e7c WatchSource:0}: Error finding container 5327f250c88b51568f0f47d2d34b76e5a8bf440c2907e6c442c2bd29e0cd6e7c: Status 404 returned error can't find the container with id 5327f250c88b51568f0f47d2d34b76e5a8bf440c2907e6c442c2bd29e0cd6e7c
Sep 16 18:47:31.087164 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:47:31.087076    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 18:47:31.368690 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:47:31.368588    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-5ffb4b4ccb-4djgv" event={"ID":"734cbe34-515a-4d3a-bdb5-b6914ab46948","Type":"ContainerStarted","Data":"5327f250c88b51568f0f47d2d34b76e5a8bf440c2907e6c442c2bd29e0cd6e7c"}
Sep 16 18:47:31.443819 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:47:31.443732    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 18:47:33.379555 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:47:33.379522    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-5ffb4b4ccb-4djgv" event={"ID":"734cbe34-515a-4d3a-bdb5-b6914ab46948","Type":"ContainerStarted","Data":"64a1b535397306155f351f2dea7b52dd8572cc90fa02970fc0d31447bcc8389c"}
Sep 16 18:47:33.379931 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:47:33.379653    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-5ffb4b4ccb-4djgv"
Sep 16 18:47:33.398203 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:47:33.398150    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-5ffb4b4ccb-4djgv" podStartSLOduration=1.044644395 podStartE2EDuration="3.398135129s" totalImagesPullingTime="2.353490734s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:47:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:47:33.396212018 +0000 UTC m=+434.462703526" watchObservedRunningTime="2026-09-16 18:47:33.398135129 +0000 UTC m=+434.464626617"
Sep 16 18:47:33.859760 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:47:33.859675    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 18:47:34.623533 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:47:34.623497    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 18:47:35.015201 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:47:35.015151    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 18:47:35.148907 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:47:35.148856    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 18:47:36.473217 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:47:36.473186    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 18:47:36.567632 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:47:36.567590    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 18:47:37.644190 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:47:37.644140    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 18:47:37.853640 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:47:37.853602    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 18:47:39.437294 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:47:39.437257    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-65dc8f76f5-4f2xb"]
Sep 16 18:47:39.443686 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:47:39.442392    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-65dc8f76f5-4f2xb"
Sep 16 18:47:39.445217 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:47:39.445191    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-b2qm8\""
Sep 16 18:47:39.445217 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:47:39.445193    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\""
Sep 16 18:47:39.453792 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:47:39.453772    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-65dc8f76f5-4f2xb"]
Sep 16 18:47:39.506787 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:47:39.506749    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cmjln\" (UniqueName: \"kubernetes.io/projected/3f409571-9229-4506-898f-145d06ea77b2-kube-api-access-cmjln\") pod \"dns-operator-controller-manager-65dc8f76f5-4f2xb\" (UID: \"3f409571-9229-4506-898f-145d06ea77b2\") " pod="kuadrant-system/dns-operator-controller-manager-65dc8f76f5-4f2xb"
Sep 16 18:47:39.607500 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:47:39.607463    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cmjln\" (UniqueName: \"kubernetes.io/projected/3f409571-9229-4506-898f-145d06ea77b2-kube-api-access-cmjln\") pod \"dns-operator-controller-manager-65dc8f76f5-4f2xb\" (UID: \"3f409571-9229-4506-898f-145d06ea77b2\") " pod="kuadrant-system/dns-operator-controller-manager-65dc8f76f5-4f2xb"
Sep 16 18:47:39.619172 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:47:39.619134    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cmjln\" (UniqueName: \"kubernetes.io/projected/3f409571-9229-4506-898f-145d06ea77b2-kube-api-access-cmjln\") pod \"dns-operator-controller-manager-65dc8f76f5-4f2xb\" (UID: \"3f409571-9229-4506-898f-145d06ea77b2\") " pod="kuadrant-system/dns-operator-controller-manager-65dc8f76f5-4f2xb"
Sep 16 18:47:39.753876 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:47:39.753784    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-65dc8f76f5-4f2xb"
Sep 16 18:47:39.999210 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:47:39.999178    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 18:47:40.027167 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:47:40.027142    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-65dc8f76f5-4f2xb"]
Sep 16 18:47:40.029754 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:47:40.029730    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f409571_9229_4506_898f_145d06ea77b2.slice/crio-af5f5cc472ad6efe1bebf2e62d56c7754dbbc92a8506a1ab15085c1a27b9803e WatchSource:0}: Error finding container af5f5cc472ad6efe1bebf2e62d56c7754dbbc92a8506a1ab15085c1a27b9803e: Status 404 returned error can't find the container with id af5f5cc472ad6efe1bebf2e62d56c7754dbbc92a8506a1ab15085c1a27b9803e
Sep 16 18:47:40.110339 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:47:40.110297    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 18:47:40.224092 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:47:40.224057    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 18:47:40.412595 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:47:40.412559    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-65dc8f76f5-4f2xb" event={"ID":"3f409571-9229-4506-898f-145d06ea77b2","Type":"ContainerStarted","Data":"af5f5cc472ad6efe1bebf2e62d56c7754dbbc92a8506a1ab15085c1a27b9803e"}
Sep 16 18:47:40.643290 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:47:40.643215    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 18:47:40.744101 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:47:40.744021    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 18:47:40.843327 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:47:40.843289    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 18:47:41.350428 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:47:41.350392    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-fbdd4f445-cvtk5"]
Sep 16 18:47:41.354139 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:47:41.354116    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-cvtk5"
Sep 16 18:47:41.356449 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:47:41.356427    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-pnvqr\""
Sep 16 18:47:41.356702 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:47:41.356658    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\""
Sep 16 18:47:41.356815 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:47:41.356791    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\""
Sep 16 18:47:41.363726 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:47:41.363680    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-fbdd4f445-cvtk5"]
Sep 16 18:47:41.381664 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:47:41.381635    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 18:47:41.430966 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:47:41.430932    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/2a534cde-c0df-4cbd-bd26-e3f08bb0468a-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-cvtk5\" (UID: \"2a534cde-c0df-4cbd-bd26-e3f08bb0468a\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-cvtk5"
Sep 16 18:47:41.431173 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:47:41.431074    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/2a534cde-c0df-4cbd-bd26-e3f08bb0468a-nginx-conf\") pod \"kuadrant-console-plugin-fbdd4f445-cvtk5\" (UID: \"2a534cde-c0df-4cbd-bd26-e3f08bb0468a\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-cvtk5"
Sep 16 18:47:41.431173 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:47:41.431115    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s52jb\" (UniqueName: \"kubernetes.io/projected/2a534cde-c0df-4cbd-bd26-e3f08bb0468a-kube-api-access-s52jb\") pod \"kuadrant-console-plugin-fbdd4f445-cvtk5\" (UID: \"2a534cde-c0df-4cbd-bd26-e3f08bb0468a\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-cvtk5"
Sep 16 18:47:41.532552 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:47:41.532509    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s52jb\" (UniqueName: \"kubernetes.io/projected/2a534cde-c0df-4cbd-bd26-e3f08bb0468a-kube-api-access-s52jb\") pod \"kuadrant-console-plugin-fbdd4f445-cvtk5\" (UID: \"2a534cde-c0df-4cbd-bd26-e3f08bb0468a\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-cvtk5"
Sep 16 18:47:41.532744 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:47:41.532662    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/2a534cde-c0df-4cbd-bd26-e3f08bb0468a-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-cvtk5\" (UID: \"2a534cde-c0df-4cbd-bd26-e3f08bb0468a\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-cvtk5"
Sep 16 18:47:41.532821 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:47:41.532760    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/2a534cde-c0df-4cbd-bd26-e3f08bb0468a-nginx-conf\") pod \"kuadrant-console-plugin-fbdd4f445-cvtk5\" (UID: \"2a534cde-c0df-4cbd-bd26-e3f08bb0468a\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-cvtk5"
Sep 16 18:47:41.532821 ip-10-0-135-119 kubenswrapper[2578]: E0916 18:47:41.532778    2578 secret.go:189] Couldn't get secret kuadrant-system/plugin-serving-cert: secret "plugin-serving-cert" not found
Sep 16 18:47:41.532919 ip-10-0-135-119 kubenswrapper[2578]: E0916 18:47:41.532867    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2a534cde-c0df-4cbd-bd26-e3f08bb0468a-plugin-serving-cert podName:2a534cde-c0df-4cbd-bd26-e3f08bb0468a nodeName:}" failed. No retries permitted until 2026-09-16 18:47:42.032840231 +0000 UTC m=+443.099331713 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "plugin-serving-cert" (UniqueName: "kubernetes.io/secret/2a534cde-c0df-4cbd-bd26-e3f08bb0468a-plugin-serving-cert") pod "kuadrant-console-plugin-fbdd4f445-cvtk5" (UID: "2a534cde-c0df-4cbd-bd26-e3f08bb0468a") : secret "plugin-serving-cert" not found
Sep 16 18:47:41.533512 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:47:41.533490    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/2a534cde-c0df-4cbd-bd26-e3f08bb0468a-nginx-conf\") pod \"kuadrant-console-plugin-fbdd4f445-cvtk5\" (UID: \"2a534cde-c0df-4cbd-bd26-e3f08bb0468a\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-cvtk5"
Sep 16 18:47:41.541677 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:47:41.541644    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s52jb\" (UniqueName: \"kubernetes.io/projected/2a534cde-c0df-4cbd-bd26-e3f08bb0468a-kube-api-access-s52jb\") pod \"kuadrant-console-plugin-fbdd4f445-cvtk5\" (UID: \"2a534cde-c0df-4cbd-bd26-e3f08bb0468a\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-cvtk5"
Sep 16 18:47:41.719348 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:47:41.719303    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 18:47:42.038149 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:47:42.038061    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/2a534cde-c0df-4cbd-bd26-e3f08bb0468a-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-cvtk5\" (UID: \"2a534cde-c0df-4cbd-bd26-e3f08bb0468a\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-cvtk5"
Sep 16 18:47:42.040609 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:47:42.040581    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/2a534cde-c0df-4cbd-bd26-e3f08bb0468a-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-cvtk5\" (UID: \"2a534cde-c0df-4cbd-bd26-e3f08bb0468a\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-cvtk5"
Sep 16 18:47:42.266725 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:47:42.266678    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-cvtk5"
Sep 16 18:47:42.423097 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:47:42.423052    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-65dc8f76f5-4f2xb" event={"ID":"3f409571-9229-4506-898f-145d06ea77b2","Type":"ContainerStarted","Data":"01f946c86acf1acd0de7d1f47bbe4b9af18544fceeadd6dd0527e53b1a8281b4"}
Sep 16 18:47:42.423274 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:47:42.423118    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-65dc8f76f5-4f2xb"
Sep 16 18:47:42.445421 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:47:42.445367    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-65dc8f76f5-4f2xb" podStartSLOduration=1.768809804 podStartE2EDuration="3.44534849s" totalImagesPullingTime="1.676538686s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:47:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:47:42.444222788 +0000 UTC m=+443.510714301" watchObservedRunningTime="2026-09-16 18:47:42.44534849 +0000 UTC m=+443.511839978"
Sep 16 18:47:42.542647 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:47:42.542615    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-fbdd4f445-cvtk5"]
Sep 16 18:47:42.545454 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:47:42.545419    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2a534cde_c0df_4cbd_bd26_e3f08bb0468a.slice/crio-5e5d56f8f621116919aa5f8b7320144bb1f098769218b857564b6e61a30c1fa9 WatchSource:0}: Error finding container 5e5d56f8f621116919aa5f8b7320144bb1f098769218b857564b6e61a30c1fa9: Status 404 returned error can't find the container with id 5e5d56f8f621116919aa5f8b7320144bb1f098769218b857564b6e61a30c1fa9
Sep 16 18:47:43.436926 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:47:43.436860    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-cvtk5" event={"ID":"2a534cde-c0df-4cbd-bd26-e3f08bb0468a","Type":"ContainerStarted","Data":"5e5d56f8f621116919aa5f8b7320144bb1f098769218b857564b6e61a30c1fa9"}
Sep 16 18:47:44.386064 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:47:44.386030    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-5ffb4b4ccb-4djgv"
Sep 16 18:47:44.471873 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:47:44.471833    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 18:47:44.567923 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:47:44.567892    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 18:47:44.869349 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:47:44.869267    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 18:47:44.962117 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:47:44.962082    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 18:47:45.068095 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:47:45.068057    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 18:47:45.165755 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:47:45.165710    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 18:47:47.456118 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:47:47.456081    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-cvtk5" event={"ID":"2a534cde-c0df-4cbd-bd26-e3f08bb0468a","Type":"ContainerStarted","Data":"f1747ec9d9b11ea6507d9b0781fc8052373576cea15e31d49c4be404c048d27f"}
Sep 16 18:47:47.478433 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:47:47.478385    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-cvtk5" podStartSLOduration=2.136732404 podStartE2EDuration="6.478371051s" totalImagesPullingTime="4.341638647s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:47:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:47:47.476096899 +0000 UTC m=+448.542588387" watchObservedRunningTime="2026-09-16 18:47:47.478371051 +0000 UTC m=+448.544862538"
Sep 16 18:47:51.310167 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:47:51.310134    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 18:47:51.414459 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:47:51.414431    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 18:47:51.510516 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:47:51.510478    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 18:47:51.614282 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:47:51.614185    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 18:47:53.059320 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:47:53.059288    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 18:47:53.163137 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:47:53.163099    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 18:47:53.435855 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:47:53.435829    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-65dc8f76f5-4f2xb"
Sep 16 18:47:55.498865 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:47:55.498831    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 18:47:55.566832 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:47:55.566802    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 18:47:59.869099 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:47:59.869065    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 18:48:00.014953 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:48:00.014918    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 18:48:03.462767 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:48:03.462730    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 18:48:03.527782 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:48:03.527750    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 18:48:03.856308 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:48:03.856217    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 18:48:03.923496 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:48:03.923464    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 18:48:04.046095 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:48:04.046058    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 18:48:04.146765 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:48:04.146732    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 18:48:12.919580 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:48:12.919527    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 18:48:13.012138 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:48:13.012103    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 18:48:15.346618 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:48:15.346572    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 18:48:15.461305 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:48:15.461271    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 18:48:22.767847 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:48:22.767806    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 18:48:22.876094 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:48:22.876061    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 18:48:22.978674 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:48:22.978619    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 18:48:23.730106 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:48:23.730055    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 18:48:23.826437 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:48:23.826401    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 18:48:23.874885 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:48:23.874849    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-59c974bd48-5rnfv"]
Sep 16 18:48:23.896479 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:48:23.896449    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-59c974bd48-5rnfv"]
Sep 16 18:48:23.896626 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:48:23.896543    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-5rnfv"
Sep 16 18:48:23.898938 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:48:23.898917    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-jf5d4\""
Sep 16 18:48:23.992049 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:48:23.991944    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qfn4p\" (UniqueName: \"kubernetes.io/projected/fb7eb867-31c5-46e4-9234-9e0fe683d740-kube-api-access-qfn4p\") pod \"authorino-59c974bd48-5rnfv\" (UID: \"fb7eb867-31c5-46e4-9234-9e0fe683d740\") " pod="kuadrant-system/authorino-59c974bd48-5rnfv"
Sep 16 18:48:24.093446 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:48:24.093411    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qfn4p\" (UniqueName: \"kubernetes.io/projected/fb7eb867-31c5-46e4-9234-9e0fe683d740-kube-api-access-qfn4p\") pod \"authorino-59c974bd48-5rnfv\" (UID: \"fb7eb867-31c5-46e4-9234-9e0fe683d740\") " pod="kuadrant-system/authorino-59c974bd48-5rnfv"
Sep 16 18:48:24.103705 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:48:24.103679    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qfn4p\" (UniqueName: \"kubernetes.io/projected/fb7eb867-31c5-46e4-9234-9e0fe683d740-kube-api-access-qfn4p\") pod \"authorino-59c974bd48-5rnfv\" (UID: \"fb7eb867-31c5-46e4-9234-9e0fe683d740\") " pod="kuadrant-system/authorino-59c974bd48-5rnfv"
Sep 16 18:48:24.205758 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:48:24.205719    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-5rnfv"
Sep 16 18:48:24.484129 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:48:24.484097    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-59c974bd48-5rnfv"]
Sep 16 18:48:24.487515 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:48:24.487489    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfb7eb867_31c5_46e4_9234_9e0fe683d740.slice/crio-7fdbb5f519c644bf4c0efc8f217c506518c2fe4553d2978776f0b2e276953ee4 WatchSource:0}: Error finding container 7fdbb5f519c644bf4c0efc8f217c506518c2fe4553d2978776f0b2e276953ee4: Status 404 returned error can't find the container with id 7fdbb5f519c644bf4c0efc8f217c506518c2fe4553d2978776f0b2e276953ee4
Sep 16 18:48:24.625783 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:48:24.625697    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-5rnfv" event={"ID":"fb7eb867-31c5-46e4-9234-9e0fe683d740","Type":"ContainerStarted","Data":"7fdbb5f519c644bf4c0efc8f217c506518c2fe4553d2978776f0b2e276953ee4"}
Sep 16 18:48:28.644887 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:48:28.644851    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-5rnfv" event={"ID":"fb7eb867-31c5-46e4-9234-9e0fe683d740","Type":"ContainerStarted","Data":"ebb3aa4b8aae92fe184d5777546402881b0417e6110b001b886c370ac1af5ba5"}
Sep 16 18:48:28.662780 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:48:28.662725    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-59c974bd48-5rnfv" podStartSLOduration=1.952558421 podStartE2EDuration="5.662707303s" totalImagesPullingTime="3.710148882s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:48:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:48:28.662611991 +0000 UTC m=+489.729103490" watchObservedRunningTime="2026-09-16 18:48:28.662707303 +0000 UTC m=+489.729198793"
Sep 16 18:48:43.368630 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:48:43.368592    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 18:48:43.620355 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:48:43.620280    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 18:48:49.409021 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:48:49.408961    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-59c974bd48-5rnfv"]
Sep 16 18:48:49.409517 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:48:49.409186    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-59c974bd48-5rnfv" podUID="fb7eb867-31c5-46e4-9234-9e0fe683d740" containerName="authorino" containerID="cri-o://ebb3aa4b8aae92fe184d5777546402881b0417e6110b001b886c370ac1af5ba5" gracePeriod=30
Sep 16 18:48:49.663516 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:48:49.663453    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-5rnfv"
Sep 16 18:48:49.741529 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:48:49.741497    2578 generic.go:356] "Generic (PLEG): container finished" podID="fb7eb867-31c5-46e4-9234-9e0fe683d740" containerID="ebb3aa4b8aae92fe184d5777546402881b0417e6110b001b886c370ac1af5ba5" exitCode=0
Sep 16 18:48:49.741529 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:48:49.741528    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-5rnfv"
Sep 16 18:48:49.741741 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:48:49.741567    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-5rnfv" event={"ID":"fb7eb867-31c5-46e4-9234-9e0fe683d740","Type":"ContainerDied","Data":"ebb3aa4b8aae92fe184d5777546402881b0417e6110b001b886c370ac1af5ba5"}
Sep 16 18:48:49.741741 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:48:49.741608    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-5rnfv" event={"ID":"fb7eb867-31c5-46e4-9234-9e0fe683d740","Type":"ContainerDied","Data":"7fdbb5f519c644bf4c0efc8f217c506518c2fe4553d2978776f0b2e276953ee4"}
Sep 16 18:48:49.741741 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:48:49.741625    2578 scope.go:117] "RemoveContainer" containerID="ebb3aa4b8aae92fe184d5777546402881b0417e6110b001b886c370ac1af5ba5"
Sep 16 18:48:49.750187 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:48:49.750169    2578 scope.go:117] "RemoveContainer" containerID="ebb3aa4b8aae92fe184d5777546402881b0417e6110b001b886c370ac1af5ba5"
Sep 16 18:48:49.750442 ip-10-0-135-119 kubenswrapper[2578]: E0916 18:48:49.750426    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ebb3aa4b8aae92fe184d5777546402881b0417e6110b001b886c370ac1af5ba5\": container with ID starting with ebb3aa4b8aae92fe184d5777546402881b0417e6110b001b886c370ac1af5ba5 not found: ID does not exist" containerID="ebb3aa4b8aae92fe184d5777546402881b0417e6110b001b886c370ac1af5ba5"
Sep 16 18:48:49.750488 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:48:49.750452    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ebb3aa4b8aae92fe184d5777546402881b0417e6110b001b886c370ac1af5ba5"} err="failed to get container status \"ebb3aa4b8aae92fe184d5777546402881b0417e6110b001b886c370ac1af5ba5\": rpc error: code = NotFound desc = could not find container \"ebb3aa4b8aae92fe184d5777546402881b0417e6110b001b886c370ac1af5ba5\": container with ID starting with ebb3aa4b8aae92fe184d5777546402881b0417e6110b001b886c370ac1af5ba5 not found: ID does not exist"
Sep 16 18:48:49.822925 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:48:49.822890    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qfn4p\" (UniqueName: \"kubernetes.io/projected/fb7eb867-31c5-46e4-9234-9e0fe683d740-kube-api-access-qfn4p\") pod \"fb7eb867-31c5-46e4-9234-9e0fe683d740\" (UID: \"fb7eb867-31c5-46e4-9234-9e0fe683d740\") "
Sep 16 18:48:49.825274 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:48:49.825245    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fb7eb867-31c5-46e4-9234-9e0fe683d740-kube-api-access-qfn4p" (OuterVolumeSpecName: "kube-api-access-qfn4p") pod "fb7eb867-31c5-46e4-9234-9e0fe683d740" (UID: "fb7eb867-31c5-46e4-9234-9e0fe683d740"). InnerVolumeSpecName "kube-api-access-qfn4p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:48:49.923931 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:48:49.923846    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qfn4p\" (UniqueName: \"kubernetes.io/projected/fb7eb867-31c5-46e4-9234-9e0fe683d740-kube-api-access-qfn4p\") on node \"ip-10-0-135-119.ec2.internal\" DevicePath \"\""
Sep 16 18:48:50.073017 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:48:50.072989    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-59c974bd48-5rnfv"]
Sep 16 18:48:50.076066 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:48:50.076041    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-59c974bd48-5rnfv"]
Sep 16 18:48:51.434466 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:48:51.434434    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fb7eb867-31c5-46e4-9234-9e0fe683d740" path="/var/lib/kubelet/pods/fb7eb867-31c5-46e4-9234-9e0fe683d740/volumes"
Sep 16 18:48:52.009802 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:48:52.009766    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 18:48:52.112593 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:48:52.112561    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 18:49:21.849498 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:49:21.849461    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 18:49:21.927441 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:49:21.927407    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 18:49:21.952737 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:49:21.952707    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/llmisvc-controller-manager-6878c65f99-xl6qv"]
Sep 16 18:49:21.953124 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:49:21.953107    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fb7eb867-31c5-46e4-9234-9e0fe683d740" containerName="authorino"
Sep 16 18:49:21.953214 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:49:21.953126    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="fb7eb867-31c5-46e4-9234-9e0fe683d740" containerName="authorino"
Sep 16 18:49:21.953268 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:49:21.953248    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="fb7eb867-31c5-46e4-9234-9e0fe683d740" containerName="authorino"
Sep 16 18:49:21.957338 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:49:21.957304    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/llmisvc-controller-manager-6878c65f99-xl6qv"
Sep 16 18:49:21.959394 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:49:21.959373    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"llmisvc-webhook-server-cert\""
Sep 16 18:49:21.959394 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:49:21.959390    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"llmisvc-controller-manager-dockercfg-rcsz9\""
Sep 16 18:49:21.959637 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:49:21.959618    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
Sep 16 18:49:21.959908 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:49:21.959892    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
Sep 16 18:49:21.965317 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:49:21.965294    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/llmisvc-controller-manager-6878c65f99-xl6qv"]
Sep 16 18:49:21.974942 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:49:21.974916    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/seaweedfs-64568bcd49-sm5gv"]
Sep 16 18:49:21.978024 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:49:21.978006    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-64568bcd49-sm5gv"
Sep 16 18:49:21.980514 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:49:21.980498    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-f9k9l\""
Sep 16 18:49:21.980889 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:49:21.980869    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"mlpipeline-s3-artifact\""
Sep 16 18:49:21.988302 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:49:21.988275    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-64568bcd49-sm5gv"]
Sep 16 18:49:22.043712 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:49:22.043683    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 18:49:22.095749 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:49:22.095720    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/28083719-d2d0-41dd-93e2-08a91886fd0c-cert\") pod \"llmisvc-controller-manager-6878c65f99-xl6qv\" (UID: \"28083719-d2d0-41dd-93e2-08a91886fd0c\") " pod="kserve/llmisvc-controller-manager-6878c65f99-xl6qv"
Sep 16 18:49:22.095891 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:49:22.095755    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fq6t9\" (UniqueName: \"kubernetes.io/projected/28083719-d2d0-41dd-93e2-08a91886fd0c-kube-api-access-fq6t9\") pod \"llmisvc-controller-manager-6878c65f99-xl6qv\" (UID: \"28083719-d2d0-41dd-93e2-08a91886fd0c\") " pod="kserve/llmisvc-controller-manager-6878c65f99-xl6qv"
Sep 16 18:49:22.095891 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:49:22.095790    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/b930538f-7cf6-4a9d-a43f-d96aa93595dc-data\") pod \"seaweedfs-64568bcd49-sm5gv\" (UID: \"b930538f-7cf6-4a9d-a43f-d96aa93595dc\") " pod="kserve/seaweedfs-64568bcd49-sm5gv"
Sep 16 18:49:22.095891 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:49:22.095834    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jzz5b\" (UniqueName: \"kubernetes.io/projected/b930538f-7cf6-4a9d-a43f-d96aa93595dc-kube-api-access-jzz5b\") pod \"seaweedfs-64568bcd49-sm5gv\" (UID: \"b930538f-7cf6-4a9d-a43f-d96aa93595dc\") " pod="kserve/seaweedfs-64568bcd49-sm5gv"
Sep 16 18:49:22.144114 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:49:22.144081    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 18:49:22.196994 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:49:22.196954    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/28083719-d2d0-41dd-93e2-08a91886fd0c-cert\") pod \"llmisvc-controller-manager-6878c65f99-xl6qv\" (UID: \"28083719-d2d0-41dd-93e2-08a91886fd0c\") " pod="kserve/llmisvc-controller-manager-6878c65f99-xl6qv"
Sep 16 18:49:22.197120 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:49:22.197021    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fq6t9\" (UniqueName: \"kubernetes.io/projected/28083719-d2d0-41dd-93e2-08a91886fd0c-kube-api-access-fq6t9\") pod \"llmisvc-controller-manager-6878c65f99-xl6qv\" (UID: \"28083719-d2d0-41dd-93e2-08a91886fd0c\") " pod="kserve/llmisvc-controller-manager-6878c65f99-xl6qv"
Sep 16 18:49:22.197120 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:49:22.197059    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/b930538f-7cf6-4a9d-a43f-d96aa93595dc-data\") pod \"seaweedfs-64568bcd49-sm5gv\" (UID: \"b930538f-7cf6-4a9d-a43f-d96aa93595dc\") " pod="kserve/seaweedfs-64568bcd49-sm5gv"
Sep 16 18:49:22.197120 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:49:22.197081    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jzz5b\" (UniqueName: \"kubernetes.io/projected/b930538f-7cf6-4a9d-a43f-d96aa93595dc-kube-api-access-jzz5b\") pod \"seaweedfs-64568bcd49-sm5gv\" (UID: \"b930538f-7cf6-4a9d-a43f-d96aa93595dc\") " pod="kserve/seaweedfs-64568bcd49-sm5gv"
Sep 16 18:49:22.197454 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:49:22.197435    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/b930538f-7cf6-4a9d-a43f-d96aa93595dc-data\") pod \"seaweedfs-64568bcd49-sm5gv\" (UID: \"b930538f-7cf6-4a9d-a43f-d96aa93595dc\") " pod="kserve/seaweedfs-64568bcd49-sm5gv"
Sep 16 18:49:22.199619 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:49:22.199595    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/28083719-d2d0-41dd-93e2-08a91886fd0c-cert\") pod \"llmisvc-controller-manager-6878c65f99-xl6qv\" (UID: \"28083719-d2d0-41dd-93e2-08a91886fd0c\") " pod="kserve/llmisvc-controller-manager-6878c65f99-xl6qv"
Sep 16 18:49:22.205971 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:49:22.205949    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fq6t9\" (UniqueName: \"kubernetes.io/projected/28083719-d2d0-41dd-93e2-08a91886fd0c-kube-api-access-fq6t9\") pod \"llmisvc-controller-manager-6878c65f99-xl6qv\" (UID: \"28083719-d2d0-41dd-93e2-08a91886fd0c\") " pod="kserve/llmisvc-controller-manager-6878c65f99-xl6qv"
Sep 16 18:49:22.206364 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:49:22.206346    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jzz5b\" (UniqueName: \"kubernetes.io/projected/b930538f-7cf6-4a9d-a43f-d96aa93595dc-kube-api-access-jzz5b\") pod \"seaweedfs-64568bcd49-sm5gv\" (UID: \"b930538f-7cf6-4a9d-a43f-d96aa93595dc\") " pod="kserve/seaweedfs-64568bcd49-sm5gv"
Sep 16 18:49:22.246003 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:49:22.245935    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 18:49:22.268632 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:49:22.268607    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/llmisvc-controller-manager-6878c65f99-xl6qv"
Sep 16 18:49:22.287757 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:49:22.287734    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-64568bcd49-sm5gv"
Sep 16 18:49:22.546935 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:49:22.546911    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/llmisvc-controller-manager-6878c65f99-xl6qv"]
Sep 16 18:49:22.549035 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:49:22.549008    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 18:49:22.549657 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:49:22.549632    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod28083719_d2d0_41dd_93e2_08a91886fd0c.slice/crio-79a38365a11cc92f482b160c440bd388bdd965bef246554c4aec962a96b4706c WatchSource:0}: Error finding container 79a38365a11cc92f482b160c440bd388bdd965bef246554c4aec962a96b4706c: Status 404 returned error can't find the container with id 79a38365a11cc92f482b160c440bd388bdd965bef246554c4aec962a96b4706c
Sep 16 18:49:22.572279 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:49:22.572259    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-64568bcd49-sm5gv"]
Sep 16 18:49:22.574505 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:49:22.574485    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb930538f_7cf6_4a9d_a43f_d96aa93595dc.slice/crio-c2024d3e28bc81351fc56365fcb69acfc6c56303b929c32708729da1af7348b9 WatchSource:0}: Error finding container c2024d3e28bc81351fc56365fcb69acfc6c56303b929c32708729da1af7348b9: Status 404 returned error can't find the container with id c2024d3e28bc81351fc56365fcb69acfc6c56303b929c32708729da1af7348b9
Sep 16 18:49:22.646569 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:49:22.646541    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 18:49:22.718727 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:49:22.718674    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 18:49:22.888554 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:49:22.888523    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-64568bcd49-sm5gv" event={"ID":"b930538f-7cf6-4a9d-a43f-d96aa93595dc","Type":"ContainerStarted","Data":"c2024d3e28bc81351fc56365fcb69acfc6c56303b929c32708729da1af7348b9"}
Sep 16 18:49:22.889629 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:49:22.889608    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/llmisvc-controller-manager-6878c65f99-xl6qv" event={"ID":"28083719-d2d0-41dd-93e2-08a91886fd0c","Type":"ContainerStarted","Data":"79a38365a11cc92f482b160c440bd388bdd965bef246554c4aec962a96b4706c"}
Sep 16 18:49:26.913403 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:49:26.913367    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-64568bcd49-sm5gv" event={"ID":"b930538f-7cf6-4a9d-a43f-d96aa93595dc","Type":"ContainerStarted","Data":"5284b8f63acabf139d8007bae3be880e94a60014169ffe025f00843a8e481aca"}
Sep 16 18:49:26.913847 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:49:26.913494    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/seaweedfs-64568bcd49-sm5gv"
Sep 16 18:49:26.914915 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:49:26.914893    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/llmisvc-controller-manager-6878c65f99-xl6qv" event={"ID":"28083719-d2d0-41dd-93e2-08a91886fd0c","Type":"ContainerStarted","Data":"1ebe060848bb521ee84fa067b8558a5e4c9fbece136f201bad92386315507101"}
Sep 16 18:49:26.915056 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:49:26.914946    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/llmisvc-controller-manager-6878c65f99-xl6qv"
Sep 16 18:49:26.931662 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:49:26.931621    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/seaweedfs-64568bcd49-sm5gv" podStartSLOduration=2.117413719 podStartE2EDuration="5.931607866s" totalImagesPullingTime="3.814194147s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:49:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:49:26.929651328 +0000 UTC m=+547.996142845" watchObservedRunningTime="2026-09-16 18:49:26.931607866 +0000 UTC m=+547.998099353"
Sep 16 18:49:26.948573 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:49:26.948471    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/llmisvc-controller-manager-6878c65f99-xl6qv" podStartSLOduration=2.200330365 podStartE2EDuration="5.948454765s" totalImagesPullingTime="3.7481244s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:49:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:49:26.948216624 +0000 UTC m=+548.014708135" watchObservedRunningTime="2026-09-16 18:49:26.948454765 +0000 UTC m=+548.014946255"
Sep 16 18:49:32.922079 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:49:32.922047    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/seaweedfs-64568bcd49-sm5gv"
Sep 16 18:49:57.921926 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:49:57.921889    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/llmisvc-controller-manager-6878c65f99-xl6qv"
Sep 16 18:50:19.350474 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:50:19.350439    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-tmrkx_4c2078bb-281a-4aa1-8e38-238897865434/console-operator/2.log"
Sep 16 18:50:19.355080 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:50:19.355058    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-tmrkx_4c2078bb-281a-4aa1-8e38-238897865434/console-operator/2.log"
Sep 16 18:50:19.358532 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:50:19.358512    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-k6gvs_6012728d-aee6-4f93-ab6e-fad616ef36e8/ovn-acl-logging/0.log"
Sep 16 18:50:19.363613 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:50:19.363592    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-k6gvs_6012728d-aee6-4f93-ab6e-fad616ef36e8/ovn-acl-logging/0.log"
Sep 16 18:50:39.085801 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:50:39.085766    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 18:50:39.139932 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:50:39.139897    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/odh-model-controller-7b966d7984-svwvp"]
Sep 16 18:50:39.143547 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:50:39.143527    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/odh-model-controller-7b966d7984-svwvp"
Sep 16 18:50:39.146887 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:50:39.146860    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-webhook-cert\""
Sep 16 18:50:39.147033 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:50:39.146897    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-metrics-tls\""
Sep 16 18:50:39.147107 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:50:39.147023    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-dockercfg-7x845\""
Sep 16 18:50:39.155039 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:50:39.155017    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/odh-model-controller-7b966d7984-svwvp"]
Sep 16 18:50:39.169100 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:50:39.169074    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 18:50:39.211270 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:50:39.211240    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0e4010eb-b58d-43e3-a2ce-c2717edaf822-cert\") pod \"odh-model-controller-7b966d7984-svwvp\" (UID: \"0e4010eb-b58d-43e3-a2ce-c2717edaf822\") " pod="kserve/odh-model-controller-7b966d7984-svwvp"
Sep 16 18:50:39.211409 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:50:39.211284    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0e4010eb-b58d-43e3-a2ce-c2717edaf822-metrics-certs\") pod \"odh-model-controller-7b966d7984-svwvp\" (UID: \"0e4010eb-b58d-43e3-a2ce-c2717edaf822\") " pod="kserve/odh-model-controller-7b966d7984-svwvp"
Sep 16 18:50:39.211409 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:50:39.211362    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hpg7v\" (UniqueName: \"kubernetes.io/projected/0e4010eb-b58d-43e3-a2ce-c2717edaf822-kube-api-access-hpg7v\") pod \"odh-model-controller-7b966d7984-svwvp\" (UID: \"0e4010eb-b58d-43e3-a2ce-c2717edaf822\") " pod="kserve/odh-model-controller-7b966d7984-svwvp"
Sep 16 18:50:39.310120 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:50:39.310084    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 18:50:39.312304 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:50:39.312279    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0e4010eb-b58d-43e3-a2ce-c2717edaf822-cert\") pod \"odh-model-controller-7b966d7984-svwvp\" (UID: \"0e4010eb-b58d-43e3-a2ce-c2717edaf822\") " pod="kserve/odh-model-controller-7b966d7984-svwvp"
Sep 16 18:50:39.312413 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:50:39.312311    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0e4010eb-b58d-43e3-a2ce-c2717edaf822-metrics-certs\") pod \"odh-model-controller-7b966d7984-svwvp\" (UID: \"0e4010eb-b58d-43e3-a2ce-c2717edaf822\") " pod="kserve/odh-model-controller-7b966d7984-svwvp"
Sep 16 18:50:39.312413 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:50:39.312359    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hpg7v\" (UniqueName: \"kubernetes.io/projected/0e4010eb-b58d-43e3-a2ce-c2717edaf822-kube-api-access-hpg7v\") pod \"odh-model-controller-7b966d7984-svwvp\" (UID: \"0e4010eb-b58d-43e3-a2ce-c2717edaf822\") " pod="kserve/odh-model-controller-7b966d7984-svwvp"
Sep 16 18:50:39.312524 ip-10-0-135-119 kubenswrapper[2578]: E0916 18:50:39.312430    2578 secret.go:189] Couldn't get secret kserve/odh-model-controller-webhook-cert: secret "odh-model-controller-webhook-cert" not found
Sep 16 18:50:39.312524 ip-10-0-135-119 kubenswrapper[2578]: E0916 18:50:39.312447    2578 secret.go:189] Couldn't get secret kserve/odh-model-controller-metrics-tls: secret "odh-model-controller-metrics-tls" not found
Sep 16 18:50:39.312524 ip-10-0-135-119 kubenswrapper[2578]: E0916 18:50:39.312485    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0e4010eb-b58d-43e3-a2ce-c2717edaf822-cert podName:0e4010eb-b58d-43e3-a2ce-c2717edaf822 nodeName:}" failed. No retries permitted until 2026-09-16 18:50:39.812466851 +0000 UTC m=+620.878958322 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/0e4010eb-b58d-43e3-a2ce-c2717edaf822-cert") pod "odh-model-controller-7b966d7984-svwvp" (UID: "0e4010eb-b58d-43e3-a2ce-c2717edaf822") : secret "odh-model-controller-webhook-cert" not found
Sep 16 18:50:39.312524 ip-10-0-135-119 kubenswrapper[2578]: E0916 18:50:39.312516    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0e4010eb-b58d-43e3-a2ce-c2717edaf822-metrics-certs podName:0e4010eb-b58d-43e3-a2ce-c2717edaf822 nodeName:}" failed. No retries permitted until 2026-09-16 18:50:39.812498723 +0000 UTC m=+620.878990205 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0e4010eb-b58d-43e3-a2ce-c2717edaf822-metrics-certs") pod "odh-model-controller-7b966d7984-svwvp" (UID: "0e4010eb-b58d-43e3-a2ce-c2717edaf822") : secret "odh-model-controller-metrics-tls" not found
Sep 16 18:50:39.323535 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:50:39.323513    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hpg7v\" (UniqueName: \"kubernetes.io/projected/0e4010eb-b58d-43e3-a2ce-c2717edaf822-kube-api-access-hpg7v\") pod \"odh-model-controller-7b966d7984-svwvp\" (UID: \"0e4010eb-b58d-43e3-a2ce-c2717edaf822\") " pod="kserve/odh-model-controller-7b966d7984-svwvp"
Sep 16 18:50:39.509298 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:50:39.509265    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 18:50:39.611955 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:50:39.611925    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 18:50:39.816814 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:50:39.816729    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0e4010eb-b58d-43e3-a2ce-c2717edaf822-cert\") pod \"odh-model-controller-7b966d7984-svwvp\" (UID: \"0e4010eb-b58d-43e3-a2ce-c2717edaf822\") " pod="kserve/odh-model-controller-7b966d7984-svwvp"
Sep 16 18:50:39.816814 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:50:39.816766    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0e4010eb-b58d-43e3-a2ce-c2717edaf822-metrics-certs\") pod \"odh-model-controller-7b966d7984-svwvp\" (UID: \"0e4010eb-b58d-43e3-a2ce-c2717edaf822\") " pod="kserve/odh-model-controller-7b966d7984-svwvp"
Sep 16 18:50:39.819305 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:50:39.819272    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0e4010eb-b58d-43e3-a2ce-c2717edaf822-metrics-certs\") pod \"odh-model-controller-7b966d7984-svwvp\" (UID: \"0e4010eb-b58d-43e3-a2ce-c2717edaf822\") " pod="kserve/odh-model-controller-7b966d7984-svwvp"
Sep 16 18:50:39.819456 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:50:39.819434    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0e4010eb-b58d-43e3-a2ce-c2717edaf822-cert\") pod \"odh-model-controller-7b966d7984-svwvp\" (UID: \"0e4010eb-b58d-43e3-a2ce-c2717edaf822\") " pod="kserve/odh-model-controller-7b966d7984-svwvp"
Sep 16 18:50:40.057071 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:50:40.057038    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/odh-model-controller-7b966d7984-svwvp"
Sep 16 18:50:40.331135 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:50:40.331109    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/odh-model-controller-7b966d7984-svwvp"]
Sep 16 18:50:40.333108 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:50:40.333083    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e4010eb_b58d_43e3_a2ce_c2717edaf822.slice/crio-b197c377b232b1420545de1395271a31f2de9da0b24d8b06c0506c648e398427 WatchSource:0}: Error finding container b197c377b232b1420545de1395271a31f2de9da0b24d8b06c0506c648e398427: Status 404 returned error can't find the container with id b197c377b232b1420545de1395271a31f2de9da0b24d8b06c0506c648e398427
Sep 16 18:50:40.421379 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:50:40.421337    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 18:50:40.523575 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:50:40.523486    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 18:50:41.262942 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:50:41.262905    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/odh-model-controller-7b966d7984-svwvp" event={"ID":"0e4010eb-b58d-43e3-a2ce-c2717edaf822","Type":"ContainerStarted","Data":"b197c377b232b1420545de1395271a31f2de9da0b24d8b06c0506c648e398427"}
Sep 16 18:50:43.275435 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:50:43.275349    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/odh-model-controller-7b966d7984-svwvp" event={"ID":"0e4010eb-b58d-43e3-a2ce-c2717edaf822","Type":"ContainerStarted","Data":"ff4bd72f3807aff4d20322269870320b49ed18ee9cccfdd250ba6a215fb77b59"}
Sep 16 18:50:43.275829 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:50:43.275445    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/odh-model-controller-7b966d7984-svwvp"
Sep 16 18:50:43.296883 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:50:43.296833    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/odh-model-controller-7b966d7984-svwvp" podStartSLOduration=1.703083619 podStartE2EDuration="4.296814775s" totalImagesPullingTime="2.593731156s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:50:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:50:43.295532089 +0000 UTC m=+624.362023597" watchObservedRunningTime="2026-09-16 18:50:43.296814775 +0000 UTC m=+624.363306263"
Sep 16 18:50:49.991117 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:50:49.991076    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 18:50:54.281301 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:50:54.281263    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/odh-model-controller-7b966d7984-svwvp"
Sep 16 18:50:55.270580 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:50:55.270544    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 18:50:55.370855 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:50:55.370820    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 18:51:05.567784 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:51:05.567741    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 18:51:05.670623 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:51:05.670590    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 18:51:13.899528 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:51:13.899493    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 18:51:13.968476 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:51:13.968436    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 18:51:38.003203 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:51:38.003169    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 18:52:12.373917 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:52:12.373880    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 18:52:12.474032 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:52:12.473999    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 18:52:25.115486 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:52:25.115448    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 18:52:25.213574 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:52:25.213534    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 18:52:33.571017 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:52:33.570959    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 18:52:33.967300 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:52:33.967260    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-h4bjw"]
Sep 16 18:52:33.971194 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:52:33.971168    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-h4bjw"
Sep 16 18:52:33.974173 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:52:33.974142    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-flow-control-smoke-4f8df83c\"/\"openshift-service-ca.crt\""
Sep 16 18:52:33.974325 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:52:33.974142    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-flow-control-smoke-4f8df83c\"/\"kube-root-ca.crt\""
Sep 16 18:52:33.974325 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:52:33.974153    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-flow-control-smoke-4f8df83c\"/\"default-dockercfg-pv4rj\""
Sep 16 18:52:33.974441 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:52:33.974409    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-flow-control-smoke-4f8df83c\"/\"fc-smoke-test-kserve-self-signed-certs\""
Sep 16 18:52:33.974525 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:52:33.974508    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-flow-control-smoke-4f8df83c\"/\"fc-smoke-test-epp-sa-dockercfg-dpj2b\""
Sep 16 18:52:33.981505 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:52:33.981477    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-h4bjw"]
Sep 16 18:52:34.021311 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:52:34.021276    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 18:52:34.106648 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:52:34.106608    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5d5tr\" (UniqueName: \"kubernetes.io/projected/8cbd3490-a60b-4e2b-9bf9-7ac45f50d6e8-kube-api-access-5d5tr\") pod \"fc-smoke-test-kserve-router-scheduler-76d5658f88-h4bjw\" (UID: \"8cbd3490-a60b-4e2b-9bf9-7ac45f50d6e8\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-h4bjw"
Sep 16 18:52:34.106843 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:52:34.106703    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8cbd3490-a60b-4e2b-9bf9-7ac45f50d6e8-tls-certs\") pod \"fc-smoke-test-kserve-router-scheduler-76d5658f88-h4bjw\" (UID: \"8cbd3490-a60b-4e2b-9bf9-7ac45f50d6e8\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-h4bjw"
Sep 16 18:52:34.207838 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:52:34.207793    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5d5tr\" (UniqueName: \"kubernetes.io/projected/8cbd3490-a60b-4e2b-9bf9-7ac45f50d6e8-kube-api-access-5d5tr\") pod \"fc-smoke-test-kserve-router-scheduler-76d5658f88-h4bjw\" (UID: \"8cbd3490-a60b-4e2b-9bf9-7ac45f50d6e8\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-h4bjw"
Sep 16 18:52:34.208074 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:52:34.207850    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8cbd3490-a60b-4e2b-9bf9-7ac45f50d6e8-tls-certs\") pod \"fc-smoke-test-kserve-router-scheduler-76d5658f88-h4bjw\" (UID: \"8cbd3490-a60b-4e2b-9bf9-7ac45f50d6e8\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-h4bjw"
Sep 16 18:52:34.210591 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:52:34.210555    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8cbd3490-a60b-4e2b-9bf9-7ac45f50d6e8-tls-certs\") pod \"fc-smoke-test-kserve-router-scheduler-76d5658f88-h4bjw\" (UID: \"8cbd3490-a60b-4e2b-9bf9-7ac45f50d6e8\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-h4bjw"
Sep 16 18:52:34.217111 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:52:34.217084    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5d5tr\" (UniqueName: \"kubernetes.io/projected/8cbd3490-a60b-4e2b-9bf9-7ac45f50d6e8-kube-api-access-5d5tr\") pod \"fc-smoke-test-kserve-router-scheduler-76d5658f88-h4bjw\" (UID: \"8cbd3490-a60b-4e2b-9bf9-7ac45f50d6e8\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-h4bjw"
Sep 16 18:52:34.283289 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:52:34.283189    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-h4bjw"
Sep 16 18:52:34.577646 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:52:34.577564    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-h4bjw"]
Sep 16 18:52:34.581436 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:52:34.581408    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8cbd3490_a60b_4e2b_9bf9_7ac45f50d6e8.slice/crio-6306a6ee9ba5a74df2721603ff0d6a25234f5bb5a85ac516365074afb38d464f WatchSource:0}: Error finding container 6306a6ee9ba5a74df2721603ff0d6a25234f5bb5a85ac516365074afb38d464f: Status 404 returned error can't find the container with id 6306a6ee9ba5a74df2721603ff0d6a25234f5bb5a85ac516365074afb38d464f
Sep 16 18:52:34.583740 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:52:34.583719    2578 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 16 18:52:34.787086 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:52:34.787046    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-h4bjw" event={"ID":"8cbd3490-a60b-4e2b-9bf9-7ac45f50d6e8","Type":"ContainerStarted","Data":"6306a6ee9ba5a74df2721603ff0d6a25234f5bb5a85ac516365074afb38d464f"}
Sep 16 18:52:37.808597 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:52:37.808556    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-h4bjw" event={"ID":"8cbd3490-a60b-4e2b-9bf9-7ac45f50d6e8","Type":"ContainerStarted","Data":"1d9c5994f56c210f2562080659b550a9615d3aeea9cabc31cc48a379886c0c79"}
Sep 16 18:52:37.809042 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:52:37.808706    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-h4bjw"
Sep 16 18:52:37.826966 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:52:37.826912    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-h4bjw" podStartSLOduration=2.448826579 podStartE2EDuration="4.826892946s" totalImagesPullingTime="2.378066367s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:52:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:52:37.825406008 +0000 UTC m=+738.891897496" watchObservedRunningTime="2026-09-16 18:52:37.826892946 +0000 UTC m=+738.893384433"
Sep 16 18:52:38.799425 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:52:38.799392    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 18:52:39.056585 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:52:39.056493    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 18:52:39.177425 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:52:39.177389    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 18:52:39.333097 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:52:39.333017    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schecmtwr"]
Sep 16 18:52:39.336557 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:52:39.336534    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schecmtwr"
Sep 16 18:52:39.340414 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:52:39.340386    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"default-dockercfg-xnzpq\""
Sep 16 18:52:39.340565 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:52:39.340407    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"llmisv32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-self-signed-certs\""
Sep 16 18:52:39.340659 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:52:39.340639    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"kube-root-ca.crt\""
Sep 16 18:52:39.340744 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:52:39.340728    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"llmisvc-model-fb-opt-125m-route-dc21cb14-epp-sa-dockercfg-f9zsh\""
Sep 16 18:52:39.340817 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:52:39.340743    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"openshift-service-ca.crt\""
Sep 16 18:52:39.351883 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:52:39.351854    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schecmtwr"]
Sep 16 18:52:39.372375 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:52:39.372340    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 18:52:39.456533 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:52:39.456496    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/94783b3f-5090-493d-88c2-2d0e523bad12-tls-certs\") pod \"llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schecmtwr\" (UID: \"94783b3f-5090-493d-88c2-2d0e523bad12\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schecmtwr"
Sep 16 18:52:39.456533 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:52:39.456540    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6rp79\" (UniqueName: \"kubernetes.io/projected/94783b3f-5090-493d-88c2-2d0e523bad12-kube-api-access-6rp79\") pod \"llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schecmtwr\" (UID: \"94783b3f-5090-493d-88c2-2d0e523bad12\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schecmtwr"
Sep 16 18:52:39.557640 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:52:39.557606    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/94783b3f-5090-493d-88c2-2d0e523bad12-tls-certs\") pod \"llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schecmtwr\" (UID: \"94783b3f-5090-493d-88c2-2d0e523bad12\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schecmtwr"
Sep 16 18:52:39.557813 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:52:39.557648    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6rp79\" (UniqueName: \"kubernetes.io/projected/94783b3f-5090-493d-88c2-2d0e523bad12-kube-api-access-6rp79\") pod \"llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schecmtwr\" (UID: \"94783b3f-5090-493d-88c2-2d0e523bad12\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schecmtwr"
Sep 16 18:52:39.560416 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:52:39.560389    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/94783b3f-5090-493d-88c2-2d0e523bad12-tls-certs\") pod \"llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schecmtwr\" (UID: \"94783b3f-5090-493d-88c2-2d0e523bad12\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schecmtwr"
Sep 16 18:52:39.566928 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:52:39.566903    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6rp79\" (UniqueName: \"kubernetes.io/projected/94783b3f-5090-493d-88c2-2d0e523bad12-kube-api-access-6rp79\") pod \"llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schecmtwr\" (UID: \"94783b3f-5090-493d-88c2-2d0e523bad12\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schecmtwr"
Sep 16 18:52:39.648446 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:52:39.648411    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schecmtwr"
Sep 16 18:52:39.922836 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:52:39.922810    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schecmtwr"]
Sep 16 18:52:39.926240 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:52:39.926213    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod94783b3f_5090_493d_88c2_2d0e523bad12.slice/crio-d5d1f352bc97ed472a41ccd53bb33f9b5c510a7865b304556d03ec30ffd27710 WatchSource:0}: Error finding container d5d1f352bc97ed472a41ccd53bb33f9b5c510a7865b304556d03ec30ffd27710: Status 404 returned error can't find the container with id d5d1f352bc97ed472a41ccd53bb33f9b5c510a7865b304556d03ec30ffd27710
Sep 16 18:52:40.825896 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:52:40.825857    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schecmtwr" event={"ID":"94783b3f-5090-493d-88c2-2d0e523bad12","Type":"ContainerStarted","Data":"41c10e405dfcd34e6ba5cff11c771e09b9bd6354835a87284f884f6b3f3d6af5"}
Sep 16 18:52:40.826422 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:52:40.825919    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schecmtwr" event={"ID":"94783b3f-5090-493d-88c2-2d0e523bad12","Type":"ContainerStarted","Data":"d5d1f352bc97ed472a41ccd53bb33f9b5c510a7865b304556d03ec30ffd27710"}
Sep 16 18:52:40.826422 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:52:40.825970    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schecmtwr"
Sep 16 18:52:40.845864 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:52:40.845807    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schecmtwr" podStartSLOduration=1.845759713 podStartE2EDuration="1.845759713s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:52:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:52:40.843663953 +0000 UTC m=+741.910155441" watchObservedRunningTime="2026-09-16 18:52:40.845759713 +0000 UTC m=+741.912251531"
Sep 16 18:52:49.795765 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:52:49.795730    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 18:52:56.153135 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:52:56.153100    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 18:53:08.815935 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:53:08.815908    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-h4bjw"
Sep 16 18:53:11.832953 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:53:11.832925    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schecmtwr"
Sep 16 18:53:15.483143 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:53:15.483110    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-h4bjw"]
Sep 16 18:53:15.487583 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:53:15.487553    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 18:53:15.547834 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:53:15.547802    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 18:53:18.816692 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:53:18.816650    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-h4bjw" podUID="8cbd3490-a60b-4e2b-9bf9-7ac45f50d6e8" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 18:53:28.815781 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:53:28.815741    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-h4bjw" podUID="8cbd3490-a60b-4e2b-9bf9-7ac45f50d6e8" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 18:53:30.503998 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:53:30.503930    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-h4bjw" podUID="8cbd3490-a60b-4e2b-9bf9-7ac45f50d6e8" containerName="main" containerID="cri-o://1d9c5994f56c210f2562080659b550a9615d3aeea9cabc31cc48a379886c0c79" gracePeriod=45
Sep 16 18:53:33.552476 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:53:33.552429    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 18:53:33.612164 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:53:33.612133    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 18:53:38.815886 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:53:38.815846    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-h4bjw" podUID="8cbd3490-a60b-4e2b-9bf9-7ac45f50d6e8" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 18:53:38.816287 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:53:38.815950    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-h4bjw"
Sep 16 18:53:48.815890 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:53:48.815810    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-h4bjw" podUID="8cbd3490-a60b-4e2b-9bf9-7ac45f50d6e8" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 18:53:58.816248 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:53:58.816212    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-h4bjw" podUID="8cbd3490-a60b-4e2b-9bf9-7ac45f50d6e8" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 18:54:00.756467 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:54:00.756436    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-h4bjw"
Sep 16 18:54:00.856237 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:54:00.856145    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5d5tr\" (UniqueName: \"kubernetes.io/projected/8cbd3490-a60b-4e2b-9bf9-7ac45f50d6e8-kube-api-access-5d5tr\") pod \"8cbd3490-a60b-4e2b-9bf9-7ac45f50d6e8\" (UID: \"8cbd3490-a60b-4e2b-9bf9-7ac45f50d6e8\") "
Sep 16 18:54:00.856237 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:54:00.856218    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8cbd3490-a60b-4e2b-9bf9-7ac45f50d6e8-tls-certs\") pod \"8cbd3490-a60b-4e2b-9bf9-7ac45f50d6e8\" (UID: \"8cbd3490-a60b-4e2b-9bf9-7ac45f50d6e8\") "
Sep 16 18:54:00.858578 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:54:00.858547    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8cbd3490-a60b-4e2b-9bf9-7ac45f50d6e8-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "8cbd3490-a60b-4e2b-9bf9-7ac45f50d6e8" (UID: "8cbd3490-a60b-4e2b-9bf9-7ac45f50d6e8"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:54:00.858578 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:54:00.858559    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8cbd3490-a60b-4e2b-9bf9-7ac45f50d6e8-kube-api-access-5d5tr" (OuterVolumeSpecName: "kube-api-access-5d5tr") pod "8cbd3490-a60b-4e2b-9bf9-7ac45f50d6e8" (UID: "8cbd3490-a60b-4e2b-9bf9-7ac45f50d6e8"). InnerVolumeSpecName "kube-api-access-5d5tr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:54:00.957152 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:54:00.957116    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8cbd3490-a60b-4e2b-9bf9-7ac45f50d6e8-tls-certs\") on node \"ip-10-0-135-119.ec2.internal\" DevicePath \"\""
Sep 16 18:54:00.957152 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:54:00.957145    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5d5tr\" (UniqueName: \"kubernetes.io/projected/8cbd3490-a60b-4e2b-9bf9-7ac45f50d6e8-kube-api-access-5d5tr\") on node \"ip-10-0-135-119.ec2.internal\" DevicePath \"\""
Sep 16 18:54:01.207904 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:54:01.207865    2578 generic.go:356] "Generic (PLEG): container finished" podID="8cbd3490-a60b-4e2b-9bf9-7ac45f50d6e8" containerID="1d9c5994f56c210f2562080659b550a9615d3aeea9cabc31cc48a379886c0c79" exitCode=0
Sep 16 18:54:01.208084 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:54:01.207909    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-h4bjw"
Sep 16 18:54:01.208084 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:54:01.207922    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-h4bjw" event={"ID":"8cbd3490-a60b-4e2b-9bf9-7ac45f50d6e8","Type":"ContainerDied","Data":"1d9c5994f56c210f2562080659b550a9615d3aeea9cabc31cc48a379886c0c79"}
Sep 16 18:54:01.208084 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:54:01.207971    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-h4bjw" event={"ID":"8cbd3490-a60b-4e2b-9bf9-7ac45f50d6e8","Type":"ContainerDied","Data":"6306a6ee9ba5a74df2721603ff0d6a25234f5bb5a85ac516365074afb38d464f"}
Sep 16 18:54:01.208084 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:54:01.208015    2578 scope.go:117] "RemoveContainer" containerID="1d9c5994f56c210f2562080659b550a9615d3aeea9cabc31cc48a379886c0c79"
Sep 16 18:54:01.217232 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:54:01.217209    2578 scope.go:117] "RemoveContainer" containerID="1d9c5994f56c210f2562080659b550a9615d3aeea9cabc31cc48a379886c0c79"
Sep 16 18:54:01.217475 ip-10-0-135-119 kubenswrapper[2578]: E0916 18:54:01.217449    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1d9c5994f56c210f2562080659b550a9615d3aeea9cabc31cc48a379886c0c79\": container with ID starting with 1d9c5994f56c210f2562080659b550a9615d3aeea9cabc31cc48a379886c0c79 not found: ID does not exist" containerID="1d9c5994f56c210f2562080659b550a9615d3aeea9cabc31cc48a379886c0c79"
Sep 16 18:54:01.217524 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:54:01.217479    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1d9c5994f56c210f2562080659b550a9615d3aeea9cabc31cc48a379886c0c79"} err="failed to get container status \"1d9c5994f56c210f2562080659b550a9615d3aeea9cabc31cc48a379886c0c79\": rpc error: code = NotFound desc = could not find container \"1d9c5994f56c210f2562080659b550a9615d3aeea9cabc31cc48a379886c0c79\": container with ID starting with 1d9c5994f56c210f2562080659b550a9615d3aeea9cabc31cc48a379886c0c79 not found: ID does not exist"
Sep 16 18:54:01.229450 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:54:01.229430    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-h4bjw"]
Sep 16 18:54:01.234330 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:54:01.234309    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-h4bjw"]
Sep 16 18:54:01.433590 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:54:01.433557    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8cbd3490-a60b-4e2b-9bf9-7ac45f50d6e8" path="/var/lib/kubelet/pods/8cbd3490-a60b-4e2b-9bf9-7ac45f50d6e8/volumes"
Sep 16 18:54:05.557708 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:54:05.557667    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 18:54:05.652651 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:54:05.652618    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 18:54:07.754446 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:54:07.754416    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 18:54:12.191654 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:54:12.191618    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 18:54:17.071486 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:54:17.071452    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-rfl44"]
Sep 16 18:54:17.071963 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:54:17.071945    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8cbd3490-a60b-4e2b-9bf9-7ac45f50d6e8" containerName="main"
Sep 16 18:54:17.072030 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:54:17.071968    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="8cbd3490-a60b-4e2b-9bf9-7ac45f50d6e8" containerName="main"
Sep 16 18:54:17.072108 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:54:17.072097    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="8cbd3490-a60b-4e2b-9bf9-7ac45f50d6e8" containerName="main"
Sep 16 18:54:17.075261 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:54:17.075242    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-rfl44"
Sep 16 18:54:17.077752 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:54:17.077730    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-flow-control-smoke-1e98f46d\"/\"default-dockercfg-vxm22\""
Sep 16 18:54:17.078493 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:54:17.078471    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-flow-control-smoke-1e98f46d\"/\"openshift-service-ca.crt\""
Sep 16 18:54:17.078594 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:54:17.078515    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-flow-control-smoke-1e98f46d\"/\"fc-concurrency-test-kserve-self-signed-certs\""
Sep 16 18:54:17.078594 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:54:17.078517    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-flow-control-smoke-1e98f46d\"/\"kube-root-ca.crt\""
Sep 16 18:54:17.085951 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:54:17.085931    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-rfl44"]
Sep 16 18:54:17.116487 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:54:17.116464    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 18:54:17.196751 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:54:17.196719    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3e8d315b-3d65-4bab-861f-2b47cc0b9e10-tls-certs\") pod \"fc-concurrency-test-kserve-b5d44467c-rfl44\" (UID: \"3e8d315b-3d65-4bab-861f-2b47cc0b9e10\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-rfl44"
Sep 16 18:54:17.196906 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:54:17.196765    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3e8d315b-3d65-4bab-861f-2b47cc0b9e10-home\") pod \"fc-concurrency-test-kserve-b5d44467c-rfl44\" (UID: \"3e8d315b-3d65-4bab-861f-2b47cc0b9e10\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-rfl44"
Sep 16 18:54:17.196906 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:54:17.196783    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3e8d315b-3d65-4bab-861f-2b47cc0b9e10-model-cache\") pod \"fc-concurrency-test-kserve-b5d44467c-rfl44\" (UID: \"3e8d315b-3d65-4bab-861f-2b47cc0b9e10\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-rfl44"
Sep 16 18:54:17.196906 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:54:17.196804    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3e8d315b-3d65-4bab-861f-2b47cc0b9e10-tmp-dir\") pod \"fc-concurrency-test-kserve-b5d44467c-rfl44\" (UID: \"3e8d315b-3d65-4bab-861f-2b47cc0b9e10\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-rfl44"
Sep 16 18:54:17.196906 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:54:17.196864    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lk7vq\" (UniqueName: \"kubernetes.io/projected/3e8d315b-3d65-4bab-861f-2b47cc0b9e10-kube-api-access-lk7vq\") pod \"fc-concurrency-test-kserve-b5d44467c-rfl44\" (UID: \"3e8d315b-3d65-4bab-861f-2b47cc0b9e10\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-rfl44"
Sep 16 18:54:17.196906 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:54:17.196900    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3e8d315b-3d65-4bab-861f-2b47cc0b9e10-dshm\") pod \"fc-concurrency-test-kserve-b5d44467c-rfl44\" (UID: \"3e8d315b-3d65-4bab-861f-2b47cc0b9e10\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-rfl44"
Sep 16 18:54:17.297856 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:54:17.297820    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3e8d315b-3d65-4bab-861f-2b47cc0b9e10-dshm\") pod \"fc-concurrency-test-kserve-b5d44467c-rfl44\" (UID: \"3e8d315b-3d65-4bab-861f-2b47cc0b9e10\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-rfl44"
Sep 16 18:54:17.298068 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:54:17.297875    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3e8d315b-3d65-4bab-861f-2b47cc0b9e10-tls-certs\") pod \"fc-concurrency-test-kserve-b5d44467c-rfl44\" (UID: \"3e8d315b-3d65-4bab-861f-2b47cc0b9e10\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-rfl44"
Sep 16 18:54:17.298068 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:54:17.297917    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3e8d315b-3d65-4bab-861f-2b47cc0b9e10-home\") pod \"fc-concurrency-test-kserve-b5d44467c-rfl44\" (UID: \"3e8d315b-3d65-4bab-861f-2b47cc0b9e10\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-rfl44"
Sep 16 18:54:17.298068 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:54:17.297939    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3e8d315b-3d65-4bab-861f-2b47cc0b9e10-model-cache\") pod \"fc-concurrency-test-kserve-b5d44467c-rfl44\" (UID: \"3e8d315b-3d65-4bab-861f-2b47cc0b9e10\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-rfl44"
Sep 16 18:54:17.298068 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:54:17.297962    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3e8d315b-3d65-4bab-861f-2b47cc0b9e10-tmp-dir\") pod \"fc-concurrency-test-kserve-b5d44467c-rfl44\" (UID: \"3e8d315b-3d65-4bab-861f-2b47cc0b9e10\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-rfl44"
Sep 16 18:54:17.298068 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:54:17.298043    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lk7vq\" (UniqueName: \"kubernetes.io/projected/3e8d315b-3d65-4bab-861f-2b47cc0b9e10-kube-api-access-lk7vq\") pod \"fc-concurrency-test-kserve-b5d44467c-rfl44\" (UID: \"3e8d315b-3d65-4bab-861f-2b47cc0b9e10\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-rfl44"
Sep 16 18:54:17.298340 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:54:17.298317    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3e8d315b-3d65-4bab-861f-2b47cc0b9e10-home\") pod \"fc-concurrency-test-kserve-b5d44467c-rfl44\" (UID: \"3e8d315b-3d65-4bab-861f-2b47cc0b9e10\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-rfl44"
Sep 16 18:54:17.298398 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:54:17.298368    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3e8d315b-3d65-4bab-861f-2b47cc0b9e10-model-cache\") pod \"fc-concurrency-test-kserve-b5d44467c-rfl44\" (UID: \"3e8d315b-3d65-4bab-861f-2b47cc0b9e10\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-rfl44"
Sep 16 18:54:17.298454 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:54:17.298395    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3e8d315b-3d65-4bab-861f-2b47cc0b9e10-tmp-dir\") pod \"fc-concurrency-test-kserve-b5d44467c-rfl44\" (UID: \"3e8d315b-3d65-4bab-861f-2b47cc0b9e10\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-rfl44"
Sep 16 18:54:17.300311 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:54:17.300291    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3e8d315b-3d65-4bab-861f-2b47cc0b9e10-dshm\") pod \"fc-concurrency-test-kserve-b5d44467c-rfl44\" (UID: \"3e8d315b-3d65-4bab-861f-2b47cc0b9e10\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-rfl44"
Sep 16 18:54:17.300475 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:54:17.300460    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3e8d315b-3d65-4bab-861f-2b47cc0b9e10-tls-certs\") pod \"fc-concurrency-test-kserve-b5d44467c-rfl44\" (UID: \"3e8d315b-3d65-4bab-861f-2b47cc0b9e10\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-rfl44"
Sep 16 18:54:17.308476 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:54:17.308452    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lk7vq\" (UniqueName: \"kubernetes.io/projected/3e8d315b-3d65-4bab-861f-2b47cc0b9e10-kube-api-access-lk7vq\") pod \"fc-concurrency-test-kserve-b5d44467c-rfl44\" (UID: \"3e8d315b-3d65-4bab-861f-2b47cc0b9e10\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-rfl44"
Sep 16 18:54:17.341675 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:54:17.341593    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-sxdpk"]
Sep 16 18:54:17.345550 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:54:17.345523    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-sxdpk"
Sep 16 18:54:17.348329 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:54:17.348304    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-flow-control-smoke-1e98f46d\"/\"fc-concurrency-test-epp-sa-dockercfg-qgwfm\""
Sep 16 18:54:17.359139 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:54:17.359113    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-sxdpk"]
Sep 16 18:54:17.375838 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:54:17.375809    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 18:54:17.386832 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:54:17.386804    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-rfl44"
Sep 16 18:54:17.500738 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:54:17.500699    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ws2q9\" (UniqueName: \"kubernetes.io/projected/b85f48e8-0608-4f6f-8fe2-adaa05b1c1e8-kube-api-access-ws2q9\") pod \"fc-concurrency-test-kserve-router-scheduler-88dbd48f9-sxdpk\" (UID: \"b85f48e8-0608-4f6f-8fe2-adaa05b1c1e8\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-sxdpk"
Sep 16 18:54:17.500898 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:54:17.500790    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b85f48e8-0608-4f6f-8fe2-adaa05b1c1e8-tls-certs\") pod \"fc-concurrency-test-kserve-router-scheduler-88dbd48f9-sxdpk\" (UID: \"b85f48e8-0608-4f6f-8fe2-adaa05b1c1e8\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-sxdpk"
Sep 16 18:54:17.601952 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:54:17.601864    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ws2q9\" (UniqueName: \"kubernetes.io/projected/b85f48e8-0608-4f6f-8fe2-adaa05b1c1e8-kube-api-access-ws2q9\") pod \"fc-concurrency-test-kserve-router-scheduler-88dbd48f9-sxdpk\" (UID: \"b85f48e8-0608-4f6f-8fe2-adaa05b1c1e8\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-sxdpk"
Sep 16 18:54:17.601952 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:54:17.601928    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b85f48e8-0608-4f6f-8fe2-adaa05b1c1e8-tls-certs\") pod \"fc-concurrency-test-kserve-router-scheduler-88dbd48f9-sxdpk\" (UID: \"b85f48e8-0608-4f6f-8fe2-adaa05b1c1e8\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-sxdpk"
Sep 16 18:54:17.604606 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:54:17.604583    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b85f48e8-0608-4f6f-8fe2-adaa05b1c1e8-tls-certs\") pod \"fc-concurrency-test-kserve-router-scheduler-88dbd48f9-sxdpk\" (UID: \"b85f48e8-0608-4f6f-8fe2-adaa05b1c1e8\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-sxdpk"
Sep 16 18:54:17.610666 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:54:17.610641    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ws2q9\" (UniqueName: \"kubernetes.io/projected/b85f48e8-0608-4f6f-8fe2-adaa05b1c1e8-kube-api-access-ws2q9\") pod \"fc-concurrency-test-kserve-router-scheduler-88dbd48f9-sxdpk\" (UID: \"b85f48e8-0608-4f6f-8fe2-adaa05b1c1e8\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-sxdpk"
Sep 16 18:54:17.656515 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:54:17.656481    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-sxdpk"
Sep 16 18:54:17.661752 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:54:17.661731    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-rfl44"]
Sep 16 18:54:17.666648 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:54:17.666621    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3e8d315b_3d65_4bab_861f_2b47cc0b9e10.slice/crio-be8ea1225df68cf5108ce75da24d58085d6e539ca15acaad164099765e6b6905 WatchSource:0}: Error finding container be8ea1225df68cf5108ce75da24d58085d6e539ca15acaad164099765e6b6905: Status 404 returned error can't find the container with id be8ea1225df68cf5108ce75da24d58085d6e539ca15acaad164099765e6b6905
Sep 16 18:54:17.937079 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:54:17.936960    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-sxdpk"]
Sep 16 18:54:17.939229 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:54:17.939202    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb85f48e8_0608_4f6f_8fe2_adaa05b1c1e8.slice/crio-cca252ff087be3080ea755bf22f5430a927aa344c6a8060337dc9fcd507e5108 WatchSource:0}: Error finding container cca252ff087be3080ea755bf22f5430a927aa344c6a8060337dc9fcd507e5108: Status 404 returned error can't find the container with id cca252ff087be3080ea755bf22f5430a927aa344c6a8060337dc9fcd507e5108
Sep 16 18:54:18.293689 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:54:18.293604    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-sxdpk" event={"ID":"b85f48e8-0608-4f6f-8fe2-adaa05b1c1e8","Type":"ContainerStarted","Data":"f48b2d78701bd601664830655674f4b8ddd71c7ce42e5247781926a6d0bd98fa"}
Sep 16 18:54:18.293689 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:54:18.293670    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-sxdpk" event={"ID":"b85f48e8-0608-4f6f-8fe2-adaa05b1c1e8","Type":"ContainerStarted","Data":"cca252ff087be3080ea755bf22f5430a927aa344c6a8060337dc9fcd507e5108"}
Sep 16 18:54:18.294221 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:54:18.293783    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-sxdpk"
Sep 16 18:54:18.295200 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:54:18.295160    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-rfl44" event={"ID":"3e8d315b-3d65-4bab-861f-2b47cc0b9e10","Type":"ContainerStarted","Data":"be8ea1225df68cf5108ce75da24d58085d6e539ca15acaad164099765e6b6905"}
Sep 16 18:54:18.314067 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:54:18.314010    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-sxdpk" podStartSLOduration=1.313988669 podStartE2EDuration="1.313988669s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:54:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:54:18.310927437 +0000 UTC m=+839.377418954" watchObservedRunningTime="2026-09-16 18:54:18.313988669 +0000 UTC m=+839.380480154"
Sep 16 18:54:18.800338 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:54:18.800306    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 18:54:18.900287 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:54:18.900252    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 18:54:19.302359 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:54:19.302329    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-rfl44" event={"ID":"3e8d315b-3d65-4bab-861f-2b47cc0b9e10","Type":"ContainerStarted","Data":"72ee689b9b0b62d67e9e344a76c9d5751ace50059c71755f6f6284fdee1a3cb0"}
Sep 16 18:54:19.323890 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:54:19.323838    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-rfl44" podStartSLOduration=0.760081523 podStartE2EDuration="2.323817121s" totalImagesPullingTime="1.563735598s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:54:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:54:19.321204751 +0000 UTC m=+840.387696290" watchObservedRunningTime="2026-09-16 18:54:19.323817121 +0000 UTC m=+840.390308611"
Sep 16 18:54:19.387696 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:54:19.387668    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-rfl44"
Sep 16 18:54:27.388047 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:54:27.388012    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-rfl44"
Sep 16 18:54:27.400386 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:54:27.400359    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-rfl44"
Sep 16 18:54:28.356401 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:54:28.356373    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-rfl44"
Sep 16 18:54:43.340046 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:54:43.340006    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 18:54:49.304427 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:54:49.304396    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-sxdpk"
Sep 16 18:54:50.748055 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:54:50.748023    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schecmtwr"]
Sep 16 18:54:50.755339 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:54:50.755315    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 18:54:50.829857 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:54:50.829813    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 18:54:51.832873 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:54:51.832829    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schecmtwr" podUID="94783b3f-5090-493d-88c2-2d0e523bad12" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 18:54:56.626998 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:54:56.626939    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 18:54:56.635309 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:54:56.635283    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-rfl44"]
Sep 16 18:54:56.645426 ip-10-0-135-119 kubenswrapper[2578]: E0916 18:54:56.645181    2578 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 16 18:54:56.645426 ip-10-0-135-119 kubenswrapper[2578]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 18:54:56.645426 ip-10-0-135-119 kubenswrapper[2578]:         , stdout: , stderr: , exit code -1
Sep 16 18:54:56.645426 ip-10-0-135-119 kubenswrapper[2578]:  > containerID="72ee689b9b0b62d67e9e344a76c9d5751ace50059c71755f6f6284fdee1a3cb0" cmd=["/bin/sleep","15"]
Sep 16 18:54:56.645426 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:54:56.645246    2578 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-rfl44" message="" err=<
Sep 16 18:54:56.645426 ip-10-0-135-119 kubenswrapper[2578]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 18:54:56.645426 ip-10-0-135-119 kubenswrapper[2578]:         , stdout: , stderr: , exit code -1
Sep 16 18:54:56.645426 ip-10-0-135-119 kubenswrapper[2578]:  >
Sep 16 18:54:56.645426 ip-10-0-135-119 kubenswrapper[2578]: E0916 18:54:56.645265    2578 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 16 18:54:56.645426 ip-10-0-135-119 kubenswrapper[2578]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 18:54:56.645426 ip-10-0-135-119 kubenswrapper[2578]:         , stdout: , stderr: , exit code -1
Sep 16 18:54:56.645426 ip-10-0-135-119 kubenswrapper[2578]:  > pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-rfl44" podUID="3e8d315b-3d65-4bab-861f-2b47cc0b9e10" containerName="main" containerID="cri-o://72ee689b9b0b62d67e9e344a76c9d5751ace50059c71755f6f6284fdee1a3cb0"
Sep 16 18:54:56.645426 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:54:56.645306    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-rfl44" podUID="3e8d315b-3d65-4bab-861f-2b47cc0b9e10" containerName="main" containerID="cri-o://72ee689b9b0b62d67e9e344a76c9d5751ace50059c71755f6f6284fdee1a3cb0" gracePeriod=60
Sep 16 18:54:56.647841 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:54:56.647818    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-sxdpk"]
Sep 16 18:54:56.727080 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:54:56.727048    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 18:54:56.992820 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:54:56.992799    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-rfl44"
Sep 16 18:54:57.060875 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:54:57.060849    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3e8d315b-3d65-4bab-861f-2b47cc0b9e10-model-cache\") pod \"3e8d315b-3d65-4bab-861f-2b47cc0b9e10\" (UID: \"3e8d315b-3d65-4bab-861f-2b47cc0b9e10\") "
Sep 16 18:54:57.061062 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:54:57.060886    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3e8d315b-3d65-4bab-861f-2b47cc0b9e10-dshm\") pod \"3e8d315b-3d65-4bab-861f-2b47cc0b9e10\" (UID: \"3e8d315b-3d65-4bab-861f-2b47cc0b9e10\") "
Sep 16 18:54:57.061062 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:54:57.061025    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3e8d315b-3d65-4bab-861f-2b47cc0b9e10-tls-certs\") pod \"3e8d315b-3d65-4bab-861f-2b47cc0b9e10\" (UID: \"3e8d315b-3d65-4bab-861f-2b47cc0b9e10\") "
Sep 16 18:54:57.061062 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:54:57.061056    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3e8d315b-3d65-4bab-861f-2b47cc0b9e10-home\") pod \"3e8d315b-3d65-4bab-861f-2b47cc0b9e10\" (UID: \"3e8d315b-3d65-4bab-861f-2b47cc0b9e10\") "
Sep 16 18:54:57.061201 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:54:57.061097    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3e8d315b-3d65-4bab-861f-2b47cc0b9e10-tmp-dir\") pod \"3e8d315b-3d65-4bab-861f-2b47cc0b9e10\" (UID: \"3e8d315b-3d65-4bab-861f-2b47cc0b9e10\") "
Sep 16 18:54:57.061201 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:54:57.061138    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lk7vq\" (UniqueName: \"kubernetes.io/projected/3e8d315b-3d65-4bab-861f-2b47cc0b9e10-kube-api-access-lk7vq\") pod \"3e8d315b-3d65-4bab-861f-2b47cc0b9e10\" (UID: \"3e8d315b-3d65-4bab-861f-2b47cc0b9e10\") "
Sep 16 18:54:57.061303 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:54:57.061205    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e8d315b-3d65-4bab-861f-2b47cc0b9e10-model-cache" (OuterVolumeSpecName: "model-cache") pod "3e8d315b-3d65-4bab-861f-2b47cc0b9e10" (UID: "3e8d315b-3d65-4bab-861f-2b47cc0b9e10"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:54:57.061360 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:54:57.061311    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e8d315b-3d65-4bab-861f-2b47cc0b9e10-home" (OuterVolumeSpecName: "home") pod "3e8d315b-3d65-4bab-861f-2b47cc0b9e10" (UID: "3e8d315b-3d65-4bab-861f-2b47cc0b9e10"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:54:57.061445 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:54:57.061420    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e8d315b-3d65-4bab-861f-2b47cc0b9e10-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "3e8d315b-3d65-4bab-861f-2b47cc0b9e10" (UID: "3e8d315b-3d65-4bab-861f-2b47cc0b9e10"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:54:57.061524 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:54:57.061506    2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3e8d315b-3d65-4bab-861f-2b47cc0b9e10-home\") on node \"ip-10-0-135-119.ec2.internal\" DevicePath \"\""
Sep 16 18:54:57.061579 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:54:57.061530    2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3e8d315b-3d65-4bab-861f-2b47cc0b9e10-tmp-dir\") on node \"ip-10-0-135-119.ec2.internal\" DevicePath \"\""
Sep 16 18:54:57.061579 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:54:57.061546    2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3e8d315b-3d65-4bab-861f-2b47cc0b9e10-model-cache\") on node \"ip-10-0-135-119.ec2.internal\" DevicePath \"\""
Sep 16 18:54:57.063314 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:54:57.063280    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e8d315b-3d65-4bab-861f-2b47cc0b9e10-dshm" (OuterVolumeSpecName: "dshm") pod "3e8d315b-3d65-4bab-861f-2b47cc0b9e10" (UID: "3e8d315b-3d65-4bab-861f-2b47cc0b9e10"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:54:57.063419 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:54:57.063312    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3e8d315b-3d65-4bab-861f-2b47cc0b9e10-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "3e8d315b-3d65-4bab-861f-2b47cc0b9e10" (UID: "3e8d315b-3d65-4bab-861f-2b47cc0b9e10"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:54:57.063419 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:54:57.063397    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3e8d315b-3d65-4bab-861f-2b47cc0b9e10-kube-api-access-lk7vq" (OuterVolumeSpecName: "kube-api-access-lk7vq") pod "3e8d315b-3d65-4bab-861f-2b47cc0b9e10" (UID: "3e8d315b-3d65-4bab-861f-2b47cc0b9e10"). InnerVolumeSpecName "kube-api-access-lk7vq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:54:57.162937 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:54:57.162907    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3e8d315b-3d65-4bab-861f-2b47cc0b9e10-tls-certs\") on node \"ip-10-0-135-119.ec2.internal\" DevicePath \"\""
Sep 16 18:54:57.162937 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:54:57.162934    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lk7vq\" (UniqueName: \"kubernetes.io/projected/3e8d315b-3d65-4bab-861f-2b47cc0b9e10-kube-api-access-lk7vq\") on node \"ip-10-0-135-119.ec2.internal\" DevicePath \"\""
Sep 16 18:54:57.162937 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:54:57.162945    2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3e8d315b-3d65-4bab-861f-2b47cc0b9e10-dshm\") on node \"ip-10-0-135-119.ec2.internal\" DevicePath \"\""
Sep 16 18:54:57.493132 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:54:57.493094    2578 generic.go:356] "Generic (PLEG): container finished" podID="3e8d315b-3d65-4bab-861f-2b47cc0b9e10" containerID="72ee689b9b0b62d67e9e344a76c9d5751ace50059c71755f6f6284fdee1a3cb0" exitCode=0
Sep 16 18:54:57.493275 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:54:57.493158    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-rfl44"
Sep 16 18:54:57.493275 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:54:57.493156    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-rfl44" event={"ID":"3e8d315b-3d65-4bab-861f-2b47cc0b9e10","Type":"ContainerDied","Data":"72ee689b9b0b62d67e9e344a76c9d5751ace50059c71755f6f6284fdee1a3cb0"}
Sep 16 18:54:57.493352 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:54:57.493275    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-rfl44" event={"ID":"3e8d315b-3d65-4bab-861f-2b47cc0b9e10","Type":"ContainerDied","Data":"be8ea1225df68cf5108ce75da24d58085d6e539ca15acaad164099765e6b6905"}
Sep 16 18:54:57.493352 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:54:57.493301    2578 scope.go:117] "RemoveContainer" containerID="72ee689b9b0b62d67e9e344a76c9d5751ace50059c71755f6f6284fdee1a3cb0"
Sep 16 18:54:57.502041 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:54:57.502021    2578 scope.go:117] "RemoveContainer" containerID="72ee689b9b0b62d67e9e344a76c9d5751ace50059c71755f6f6284fdee1a3cb0"
Sep 16 18:54:57.502317 ip-10-0-135-119 kubenswrapper[2578]: E0916 18:54:57.502298    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"72ee689b9b0b62d67e9e344a76c9d5751ace50059c71755f6f6284fdee1a3cb0\": container with ID starting with 72ee689b9b0b62d67e9e344a76c9d5751ace50059c71755f6f6284fdee1a3cb0 not found: ID does not exist" containerID="72ee689b9b0b62d67e9e344a76c9d5751ace50059c71755f6f6284fdee1a3cb0"
Sep 16 18:54:57.502371 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:54:57.502327    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"72ee689b9b0b62d67e9e344a76c9d5751ace50059c71755f6f6284fdee1a3cb0"} err="failed to get container status \"72ee689b9b0b62d67e9e344a76c9d5751ace50059c71755f6f6284fdee1a3cb0\": rpc error: code = NotFound desc = could not find container \"72ee689b9b0b62d67e9e344a76c9d5751ace50059c71755f6f6284fdee1a3cb0\": container with ID starting with 72ee689b9b0b62d67e9e344a76c9d5751ace50059c71755f6f6284fdee1a3cb0 not found: ID does not exist"
Sep 16 18:54:57.511127 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:54:57.511104    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-rfl44"]
Sep 16 18:54:57.515606 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:54:57.515587    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-rfl44"]
Sep 16 18:54:59.304409 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:54:59.304371    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-sxdpk" podUID="b85f48e8-0608-4f6f-8fe2-adaa05b1c1e8" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 18:54:59.434658 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:54:59.434625    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3e8d315b-3d65-4bab-861f-2b47cc0b9e10" path="/var/lib/kubelet/pods/3e8d315b-3d65-4bab-861f-2b47cc0b9e10/volumes"
Sep 16 18:55:01.832310 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:55:01.832270    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schecmtwr" podUID="94783b3f-5090-493d-88c2-2d0e523bad12" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 18:55:05.769146 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:55:05.769102    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schecmtwr" podUID="94783b3f-5090-493d-88c2-2d0e523bad12" containerName="main" containerID="cri-o://41c10e405dfcd34e6ba5cff11c771e09b9bd6354835a87284f884f6b3f3d6af5" gracePeriod=45
Sep 16 18:55:09.304192 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:55:09.304154    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-sxdpk" podUID="b85f48e8-0608-4f6f-8fe2-adaa05b1c1e8" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 18:55:11.667779 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:55:11.667695    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-sxdpk" podUID="b85f48e8-0608-4f6f-8fe2-adaa05b1c1e8" containerName="main" containerID="cri-o://f48b2d78701bd601664830655674f4b8ddd71c7ce42e5247781926a6d0bd98fa" gracePeriod=45
Sep 16 18:55:11.832831 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:55:11.832798    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schecmtwr" podUID="94783b3f-5090-493d-88c2-2d0e523bad12" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 18:55:11.833000 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:55:11.832905    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schecmtwr"
Sep 16 18:55:13.372254 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:55:13.372221    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 18:55:13.471491 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:55:13.471455    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 18:55:19.304062 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:55:19.304026    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-sxdpk" podUID="b85f48e8-0608-4f6f-8fe2-adaa05b1c1e8" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 18:55:19.304581 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:55:19.304144    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-sxdpk"
Sep 16 18:55:19.382325 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:55:19.382300    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-tmrkx_4c2078bb-281a-4aa1-8e38-238897865434/console-operator/2.log"
Sep 16 18:55:19.384281 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:55:19.384249    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-tmrkx_4c2078bb-281a-4aa1-8e38-238897865434/console-operator/2.log"
Sep 16 18:55:19.391422 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:55:19.391400    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-k6gvs_6012728d-aee6-4f93-ab6e-fad616ef36e8/ovn-acl-logging/0.log"
Sep 16 18:55:19.393121 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:55:19.393102    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-k6gvs_6012728d-aee6-4f93-ab6e-fad616ef36e8/ovn-acl-logging/0.log"
Sep 16 18:55:21.832883 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:55:21.832848    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schecmtwr" podUID="94783b3f-5090-493d-88c2-2d0e523bad12" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 18:55:29.304307 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:55:29.304267    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-sxdpk" podUID="b85f48e8-0608-4f6f-8fe2-adaa05b1c1e8" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 18:55:31.832618 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:55:31.832576    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schecmtwr" podUID="94783b3f-5090-493d-88c2-2d0e523bad12" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 18:55:36.019195 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:55:36.019169    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schecmtwr"
Sep 16 18:55:36.107224 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:55:36.107143    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6rp79\" (UniqueName: \"kubernetes.io/projected/94783b3f-5090-493d-88c2-2d0e523bad12-kube-api-access-6rp79\") pod \"94783b3f-5090-493d-88c2-2d0e523bad12\" (UID: \"94783b3f-5090-493d-88c2-2d0e523bad12\") "
Sep 16 18:55:36.107224 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:55:36.107185    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/94783b3f-5090-493d-88c2-2d0e523bad12-tls-certs\") pod \"94783b3f-5090-493d-88c2-2d0e523bad12\" (UID: \"94783b3f-5090-493d-88c2-2d0e523bad12\") "
Sep 16 18:55:36.109553 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:55:36.109524    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/94783b3f-5090-493d-88c2-2d0e523bad12-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "94783b3f-5090-493d-88c2-2d0e523bad12" (UID: "94783b3f-5090-493d-88c2-2d0e523bad12"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:55:36.109642 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:55:36.109524    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/94783b3f-5090-493d-88c2-2d0e523bad12-kube-api-access-6rp79" (OuterVolumeSpecName: "kube-api-access-6rp79") pod "94783b3f-5090-493d-88c2-2d0e523bad12" (UID: "94783b3f-5090-493d-88c2-2d0e523bad12"). InnerVolumeSpecName "kube-api-access-6rp79". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:55:36.208014 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:55:36.207960    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6rp79\" (UniqueName: \"kubernetes.io/projected/94783b3f-5090-493d-88c2-2d0e523bad12-kube-api-access-6rp79\") on node \"ip-10-0-135-119.ec2.internal\" DevicePath \"\""
Sep 16 18:55:36.208014 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:55:36.208012    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/94783b3f-5090-493d-88c2-2d0e523bad12-tls-certs\") on node \"ip-10-0-135-119.ec2.internal\" DevicePath \"\""
Sep 16 18:55:36.680136 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:55:36.680103    2578 generic.go:356] "Generic (PLEG): container finished" podID="94783b3f-5090-493d-88c2-2d0e523bad12" containerID="41c10e405dfcd34e6ba5cff11c771e09b9bd6354835a87284f884f6b3f3d6af5" exitCode=0
Sep 16 18:55:36.680136 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:55:36.680141    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schecmtwr" event={"ID":"94783b3f-5090-493d-88c2-2d0e523bad12","Type":"ContainerDied","Data":"41c10e405dfcd34e6ba5cff11c771e09b9bd6354835a87284f884f6b3f3d6af5"}
Sep 16 18:55:36.680400 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:55:36.680152    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schecmtwr"
Sep 16 18:55:36.680400 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:55:36.680183    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schecmtwr" event={"ID":"94783b3f-5090-493d-88c2-2d0e523bad12","Type":"ContainerDied","Data":"d5d1f352bc97ed472a41ccd53bb33f9b5c510a7865b304556d03ec30ffd27710"}
Sep 16 18:55:36.680400 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:55:36.680205    2578 scope.go:117] "RemoveContainer" containerID="41c10e405dfcd34e6ba5cff11c771e09b9bd6354835a87284f884f6b3f3d6af5"
Sep 16 18:55:36.688872 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:55:36.688854    2578 scope.go:117] "RemoveContainer" containerID="41c10e405dfcd34e6ba5cff11c771e09b9bd6354835a87284f884f6b3f3d6af5"
Sep 16 18:55:36.689147 ip-10-0-135-119 kubenswrapper[2578]: E0916 18:55:36.689126    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"41c10e405dfcd34e6ba5cff11c771e09b9bd6354835a87284f884f6b3f3d6af5\": container with ID starting with 41c10e405dfcd34e6ba5cff11c771e09b9bd6354835a87284f884f6b3f3d6af5 not found: ID does not exist" containerID="41c10e405dfcd34e6ba5cff11c771e09b9bd6354835a87284f884f6b3f3d6af5"
Sep 16 18:55:36.689213 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:55:36.689156    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"41c10e405dfcd34e6ba5cff11c771e09b9bd6354835a87284f884f6b3f3d6af5"} err="failed to get container status \"41c10e405dfcd34e6ba5cff11c771e09b9bd6354835a87284f884f6b3f3d6af5\": rpc error: code = NotFound desc = could not find container \"41c10e405dfcd34e6ba5cff11c771e09b9bd6354835a87284f884f6b3f3d6af5\": container with ID starting with 41c10e405dfcd34e6ba5cff11c771e09b9bd6354835a87284f884f6b3f3d6af5 not found: ID does not exist"
Sep 16 18:55:36.704246 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:55:36.704205    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schecmtwr"]
Sep 16 18:55:36.708953 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:55:36.708931    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schecmtwr"]
Sep 16 18:55:37.434422 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:55:37.434387    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="94783b3f-5090-493d-88c2-2d0e523bad12" path="/var/lib/kubelet/pods/94783b3f-5090-493d-88c2-2d0e523bad12/volumes"
Sep 16 18:55:39.304651 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:55:39.304616    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-sxdpk" podUID="b85f48e8-0608-4f6f-8fe2-adaa05b1c1e8" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 18:55:40.616431 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:55:40.616394    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 18:55:40.715189 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:55:40.715153    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 18:55:41.911009 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:55:41.910971    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-sxdpk"
Sep 16 18:55:41.955021 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:55:41.954924    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ws2q9\" (UniqueName: \"kubernetes.io/projected/b85f48e8-0608-4f6f-8fe2-adaa05b1c1e8-kube-api-access-ws2q9\") pod \"b85f48e8-0608-4f6f-8fe2-adaa05b1c1e8\" (UID: \"b85f48e8-0608-4f6f-8fe2-adaa05b1c1e8\") "
Sep 16 18:55:41.955021 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:55:41.955004    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b85f48e8-0608-4f6f-8fe2-adaa05b1c1e8-tls-certs\") pod \"b85f48e8-0608-4f6f-8fe2-adaa05b1c1e8\" (UID: \"b85f48e8-0608-4f6f-8fe2-adaa05b1c1e8\") "
Sep 16 18:55:41.957187 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:55:41.957161    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b85f48e8-0608-4f6f-8fe2-adaa05b1c1e8-kube-api-access-ws2q9" (OuterVolumeSpecName: "kube-api-access-ws2q9") pod "b85f48e8-0608-4f6f-8fe2-adaa05b1c1e8" (UID: "b85f48e8-0608-4f6f-8fe2-adaa05b1c1e8"). InnerVolumeSpecName "kube-api-access-ws2q9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:55:41.957302 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:55:41.957217    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b85f48e8-0608-4f6f-8fe2-adaa05b1c1e8-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "b85f48e8-0608-4f6f-8fe2-adaa05b1c1e8" (UID: "b85f48e8-0608-4f6f-8fe2-adaa05b1c1e8"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:55:42.060489 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:55:42.056089    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ws2q9\" (UniqueName: \"kubernetes.io/projected/b85f48e8-0608-4f6f-8fe2-adaa05b1c1e8-kube-api-access-ws2q9\") on node \"ip-10-0-135-119.ec2.internal\" DevicePath \"\""
Sep 16 18:55:42.060489 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:55:42.056122    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b85f48e8-0608-4f6f-8fe2-adaa05b1c1e8-tls-certs\") on node \"ip-10-0-135-119.ec2.internal\" DevicePath \"\""
Sep 16 18:55:42.710758 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:55:42.710727    2578 generic.go:356] "Generic (PLEG): container finished" podID="b85f48e8-0608-4f6f-8fe2-adaa05b1c1e8" containerID="f48b2d78701bd601664830655674f4b8ddd71c7ce42e5247781926a6d0bd98fa" exitCode=0
Sep 16 18:55:42.710924 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:55:42.710770    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-sxdpk" event={"ID":"b85f48e8-0608-4f6f-8fe2-adaa05b1c1e8","Type":"ContainerDied","Data":"f48b2d78701bd601664830655674f4b8ddd71c7ce42e5247781926a6d0bd98fa"}
Sep 16 18:55:42.710924 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:55:42.710780    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-sxdpk"
Sep 16 18:55:42.710924 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:55:42.710810    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-sxdpk" event={"ID":"b85f48e8-0608-4f6f-8fe2-adaa05b1c1e8","Type":"ContainerDied","Data":"cca252ff087be3080ea755bf22f5430a927aa344c6a8060337dc9fcd507e5108"}
Sep 16 18:55:42.710924 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:55:42.710832    2578 scope.go:117] "RemoveContainer" containerID="f48b2d78701bd601664830655674f4b8ddd71c7ce42e5247781926a6d0bd98fa"
Sep 16 18:55:42.719211 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:55:42.719194    2578 scope.go:117] "RemoveContainer" containerID="f48b2d78701bd601664830655674f4b8ddd71c7ce42e5247781926a6d0bd98fa"
Sep 16 18:55:42.719466 ip-10-0-135-119 kubenswrapper[2578]: E0916 18:55:42.719447    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f48b2d78701bd601664830655674f4b8ddd71c7ce42e5247781926a6d0bd98fa\": container with ID starting with f48b2d78701bd601664830655674f4b8ddd71c7ce42e5247781926a6d0bd98fa not found: ID does not exist" containerID="f48b2d78701bd601664830655674f4b8ddd71c7ce42e5247781926a6d0bd98fa"
Sep 16 18:55:42.719531 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:55:42.719475    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f48b2d78701bd601664830655674f4b8ddd71c7ce42e5247781926a6d0bd98fa"} err="failed to get container status \"f48b2d78701bd601664830655674f4b8ddd71c7ce42e5247781926a6d0bd98fa\": rpc error: code = NotFound desc = could not find container \"f48b2d78701bd601664830655674f4b8ddd71c7ce42e5247781926a6d0bd98fa\": container with ID starting with f48b2d78701bd601664830655674f4b8ddd71c7ce42e5247781926a6d0bd98fa not found: ID does not exist"
Sep 16 18:55:42.732566 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:55:42.732540    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-sxdpk"]
Sep 16 18:55:42.739129 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:55:42.739109    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-sxdpk"]
Sep 16 18:55:43.434111 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:55:43.434076    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b85f48e8-0608-4f6f-8fe2-adaa05b1c1e8" path="/var/lib/kubelet/pods/b85f48e8-0608-4f6f-8fe2-adaa05b1c1e8/volumes"
Sep 16 18:55:46.646169 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:55:46.646132    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 18:55:46.743336 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:55:46.743309    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 18:55:46.910306 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:55:46.910229    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 18:55:47.017754 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:55:47.017725    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 18:55:48.238892 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:55:48.238858    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 18:55:50.250550 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:55:50.250515    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 18:55:54.058116 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:55:54.058078    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 18:55:55.078030 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:55:55.077997    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 18:55:55.153352 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:55:55.153316    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 18:55:55.647592 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:55:55.647558    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-kzt7p"]
Sep 16 18:55:55.647902 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:55:55.647890    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b85f48e8-0608-4f6f-8fe2-adaa05b1c1e8" containerName="main"
Sep 16 18:55:55.647950 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:55:55.647904    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="b85f48e8-0608-4f6f-8fe2-adaa05b1c1e8" containerName="main"
Sep 16 18:55:55.647950 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:55:55.647923    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3e8d315b-3d65-4bab-861f-2b47cc0b9e10" containerName="main"
Sep 16 18:55:55.647950 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:55:55.647929    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="3e8d315b-3d65-4bab-861f-2b47cc0b9e10" containerName="main"
Sep 16 18:55:55.647950 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:55:55.647935    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="94783b3f-5090-493d-88c2-2d0e523bad12" containerName="main"
Sep 16 18:55:55.647950 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:55:55.647940    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="94783b3f-5090-493d-88c2-2d0e523bad12" containerName="main"
Sep 16 18:55:55.648184 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:55:55.648007    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="3e8d315b-3d65-4bab-861f-2b47cc0b9e10" containerName="main"
Sep 16 18:55:55.648184 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:55:55.648016    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="94783b3f-5090-493d-88c2-2d0e523bad12" containerName="main"
Sep 16 18:55:55.648184 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:55:55.648024    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="b85f48e8-0608-4f6f-8fe2-adaa05b1c1e8" containerName="main"
Sep 16 18:55:55.650966 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:55:55.650949    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-kzt7p"
Sep 16 18:55:55.653107 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:55:55.653081    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-47655ff5\"/\"openshift-service-ca.crt\""
Sep 16 18:55:55.653714 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:55:55.653690    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-47655ff5\"/\"kube-root-ca.crt\""
Sep 16 18:55:55.653822 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:55:55.653717    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-47655ff5\"/\"scheduler-inline-config-test-kserve-self-signed-certs\""
Sep 16 18:55:55.653822 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:55:55.653730    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-47655ff5\"/\"default-dockercfg-8tzw8\""
Sep 16 18:55:55.661099 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:55:55.661079    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-kzt7p"]
Sep 16 18:55:55.773603 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:55:55.773568    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0024e7f2-4253-481c-9709-82b0b268c238-tmp-dir\") pod \"scheduler-inline-config-test-kserve-8655bc757-kzt7p\" (UID: \"0024e7f2-4253-481c-9709-82b0b268c238\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-kzt7p"
Sep 16 18:55:55.773781 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:55:55.773620    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0024e7f2-4253-481c-9709-82b0b268c238-tls-certs\") pod \"scheduler-inline-config-test-kserve-8655bc757-kzt7p\" (UID: \"0024e7f2-4253-481c-9709-82b0b268c238\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-kzt7p"
Sep 16 18:55:55.773781 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:55:55.773652    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4k929\" (UniqueName: \"kubernetes.io/projected/0024e7f2-4253-481c-9709-82b0b268c238-kube-api-access-4k929\") pod \"scheduler-inline-config-test-kserve-8655bc757-kzt7p\" (UID: \"0024e7f2-4253-481c-9709-82b0b268c238\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-kzt7p"
Sep 16 18:55:55.773781 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:55:55.773684    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0024e7f2-4253-481c-9709-82b0b268c238-model-cache\") pod \"scheduler-inline-config-test-kserve-8655bc757-kzt7p\" (UID: \"0024e7f2-4253-481c-9709-82b0b268c238\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-kzt7p"
Sep 16 18:55:55.773781 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:55:55.773706    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0024e7f2-4253-481c-9709-82b0b268c238-dshm\") pod \"scheduler-inline-config-test-kserve-8655bc757-kzt7p\" (UID: \"0024e7f2-4253-481c-9709-82b0b268c238\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-kzt7p"
Sep 16 18:55:55.773937 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:55:55.773795    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0024e7f2-4253-481c-9709-82b0b268c238-home\") pod \"scheduler-inline-config-test-kserve-8655bc757-kzt7p\" (UID: \"0024e7f2-4253-481c-9709-82b0b268c238\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-kzt7p"
Sep 16 18:55:55.874936 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:55:55.874897    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0024e7f2-4253-481c-9709-82b0b268c238-tmp-dir\") pod \"scheduler-inline-config-test-kserve-8655bc757-kzt7p\" (UID: \"0024e7f2-4253-481c-9709-82b0b268c238\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-kzt7p"
Sep 16 18:55:55.875144 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:55:55.874947    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0024e7f2-4253-481c-9709-82b0b268c238-tls-certs\") pod \"scheduler-inline-config-test-kserve-8655bc757-kzt7p\" (UID: \"0024e7f2-4253-481c-9709-82b0b268c238\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-kzt7p"
Sep 16 18:55:55.875144 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:55:55.874964    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4k929\" (UniqueName: \"kubernetes.io/projected/0024e7f2-4253-481c-9709-82b0b268c238-kube-api-access-4k929\") pod \"scheduler-inline-config-test-kserve-8655bc757-kzt7p\" (UID: \"0024e7f2-4253-481c-9709-82b0b268c238\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-kzt7p"
Sep 16 18:55:55.875144 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:55:55.875013    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0024e7f2-4253-481c-9709-82b0b268c238-model-cache\") pod \"scheduler-inline-config-test-kserve-8655bc757-kzt7p\" (UID: \"0024e7f2-4253-481c-9709-82b0b268c238\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-kzt7p"
Sep 16 18:55:55.875144 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:55:55.875032    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0024e7f2-4253-481c-9709-82b0b268c238-dshm\") pod \"scheduler-inline-config-test-kserve-8655bc757-kzt7p\" (UID: \"0024e7f2-4253-481c-9709-82b0b268c238\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-kzt7p"
Sep 16 18:55:55.875144 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:55:55.875083    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0024e7f2-4253-481c-9709-82b0b268c238-home\") pod \"scheduler-inline-config-test-kserve-8655bc757-kzt7p\" (UID: \"0024e7f2-4253-481c-9709-82b0b268c238\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-kzt7p"
Sep 16 18:55:55.875435 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:55:55.875324    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0024e7f2-4253-481c-9709-82b0b268c238-tmp-dir\") pod \"scheduler-inline-config-test-kserve-8655bc757-kzt7p\" (UID: \"0024e7f2-4253-481c-9709-82b0b268c238\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-kzt7p"
Sep 16 18:55:55.875435 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:55:55.875409    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0024e7f2-4253-481c-9709-82b0b268c238-model-cache\") pod \"scheduler-inline-config-test-kserve-8655bc757-kzt7p\" (UID: \"0024e7f2-4253-481c-9709-82b0b268c238\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-kzt7p"
Sep 16 18:55:55.875522 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:55:55.875452    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0024e7f2-4253-481c-9709-82b0b268c238-home\") pod \"scheduler-inline-config-test-kserve-8655bc757-kzt7p\" (UID: \"0024e7f2-4253-481c-9709-82b0b268c238\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-kzt7p"
Sep 16 18:55:55.877475 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:55:55.877446    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0024e7f2-4253-481c-9709-82b0b268c238-dshm\") pod \"scheduler-inline-config-test-kserve-8655bc757-kzt7p\" (UID: \"0024e7f2-4253-481c-9709-82b0b268c238\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-kzt7p"
Sep 16 18:55:55.877781 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:55:55.877765    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0024e7f2-4253-481c-9709-82b0b268c238-tls-certs\") pod \"scheduler-inline-config-test-kserve-8655bc757-kzt7p\" (UID: \"0024e7f2-4253-481c-9709-82b0b268c238\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-kzt7p"
Sep 16 18:55:55.884049 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:55:55.884021    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4k929\" (UniqueName: \"kubernetes.io/projected/0024e7f2-4253-481c-9709-82b0b268c238-kube-api-access-4k929\") pod \"scheduler-inline-config-test-kserve-8655bc757-kzt7p\" (UID: \"0024e7f2-4253-481c-9709-82b0b268c238\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-kzt7p"
Sep 16 18:55:55.962041 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:55:55.961941    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-kzt7p"
Sep 16 18:55:56.242426 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:55:56.242401    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-kzt7p"]
Sep 16 18:55:56.244910 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:55:56.244880    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0024e7f2_4253_481c_9709_82b0b268c238.slice/crio-ede8e2642ef14163f622902a69dc25967ac107c26d0c40bb449a068eaab4cf6a WatchSource:0}: Error finding container ede8e2642ef14163f622902a69dc25967ac107c26d0c40bb449a068eaab4cf6a: Status 404 returned error can't find the container with id ede8e2642ef14163f622902a69dc25967ac107c26d0c40bb449a068eaab4cf6a
Sep 16 18:55:56.778327 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:55:56.778295    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-kzt7p" event={"ID":"0024e7f2-4253-481c-9709-82b0b268c238","Type":"ContainerStarted","Data":"17682a18465f3e36e0b25f3e354be18d4a18f933ccedfd9aac1a268970ebf211"}
Sep 16 18:55:56.778485 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:55:56.778345    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-kzt7p" event={"ID":"0024e7f2-4253-481c-9709-82b0b268c238","Type":"ContainerStarted","Data":"ede8e2642ef14163f622902a69dc25967ac107c26d0c40bb449a068eaab4cf6a"}
Sep 16 18:55:56.796598 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:55:56.796553    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-kzt7p" podStartSLOduration=1.796539892 podStartE2EDuration="1.796539892s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:55:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:55:56.794602722 +0000 UTC m=+937.861094228" watchObservedRunningTime="2026-09-16 18:55:56.796539892 +0000 UTC m=+937.863031380"
Sep 16 18:55:56.962508 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:55:56.962474    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-kzt7p"
Sep 16 18:56:00.302674 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:56:00.302619    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 18:56:00.403636 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:56:00.403598    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 18:56:01.688937 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:56:01.688910    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 18:56:01.849801 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:56:01.849764    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbljd5t"]
Sep 16 18:56:01.854112 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:56:01.854093    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbljd5t"
Sep 16 18:56:01.856137 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:56:01.856116    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-47655ff5\"/\"scheduler-inline-config-test-epp-sa-dockercfg-ncrff\""
Sep 16 18:56:01.860616 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:56:01.860594    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbljd5t"]
Sep 16 18:56:01.876906 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:56:01.876874    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 18:56:02.031653 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:56:02.031552    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d3162ecf-6ee9-4b2b-a8d5-0ab3838f9835-tls-certs\") pod \"scheduler-inline-config-test-kserve-router-scheduler-695cbljd5t\" (UID: \"d3162ecf-6ee9-4b2b-a8d5-0ab3838f9835\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbljd5t"
Sep 16 18:56:02.031653 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:56:02.031607    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4j9wl\" (UniqueName: \"kubernetes.io/projected/d3162ecf-6ee9-4b2b-a8d5-0ab3838f9835-kube-api-access-4j9wl\") pod \"scheduler-inline-config-test-kserve-router-scheduler-695cbljd5t\" (UID: \"d3162ecf-6ee9-4b2b-a8d5-0ab3838f9835\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbljd5t"
Sep 16 18:56:02.133034 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:56:02.132967    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d3162ecf-6ee9-4b2b-a8d5-0ab3838f9835-tls-certs\") pod \"scheduler-inline-config-test-kserve-router-scheduler-695cbljd5t\" (UID: \"d3162ecf-6ee9-4b2b-a8d5-0ab3838f9835\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbljd5t"
Sep 16 18:56:02.133201 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:56:02.133068    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4j9wl\" (UniqueName: \"kubernetes.io/projected/d3162ecf-6ee9-4b2b-a8d5-0ab3838f9835-kube-api-access-4j9wl\") pod \"scheduler-inline-config-test-kserve-router-scheduler-695cbljd5t\" (UID: \"d3162ecf-6ee9-4b2b-a8d5-0ab3838f9835\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbljd5t"
Sep 16 18:56:02.135426 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:56:02.135400    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d3162ecf-6ee9-4b2b-a8d5-0ab3838f9835-tls-certs\") pod \"scheduler-inline-config-test-kserve-router-scheduler-695cbljd5t\" (UID: \"d3162ecf-6ee9-4b2b-a8d5-0ab3838f9835\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbljd5t"
Sep 16 18:56:02.141487 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:56:02.141462    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4j9wl\" (UniqueName: \"kubernetes.io/projected/d3162ecf-6ee9-4b2b-a8d5-0ab3838f9835-kube-api-access-4j9wl\") pod \"scheduler-inline-config-test-kserve-router-scheduler-695cbljd5t\" (UID: \"d3162ecf-6ee9-4b2b-a8d5-0ab3838f9835\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbljd5t"
Sep 16 18:56:02.164450 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:56:02.164422    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbljd5t"
Sep 16 18:56:02.435110 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:56:02.435079    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbljd5t"]
Sep 16 18:56:02.437568 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:56:02.437538    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd3162ecf_6ee9_4b2b_a8d5_0ab3838f9835.slice/crio-28cf21d5359d74caaf1019707a9dee23f620ce95ba29d2bc386c97dc8383de8e WatchSource:0}: Error finding container 28cf21d5359d74caaf1019707a9dee23f620ce95ba29d2bc386c97dc8383de8e: Status 404 returned error can't find the container with id 28cf21d5359d74caaf1019707a9dee23f620ce95ba29d2bc386c97dc8383de8e
Sep 16 18:56:02.808535 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:56:02.808447    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbljd5t" event={"ID":"d3162ecf-6ee9-4b2b-a8d5-0ab3838f9835","Type":"ContainerStarted","Data":"e11ad302c00c76e2be66913aae4886b4c1ac4b346f3236c75bc1df34a1f00428"}
Sep 16 18:56:02.808535 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:56:02.808500    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbljd5t" event={"ID":"d3162ecf-6ee9-4b2b-a8d5-0ab3838f9835","Type":"ContainerStarted","Data":"28cf21d5359d74caaf1019707a9dee23f620ce95ba29d2bc386c97dc8383de8e"}
Sep 16 18:56:02.808535 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:56:02.808533    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbljd5t"
Sep 16 18:56:02.826778 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:56:02.826721    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbljd5t" podStartSLOduration=1.826704189 podStartE2EDuration="1.826704189s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:56:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:56:02.824712081 +0000 UTC m=+943.891203601" watchObservedRunningTime="2026-09-16 18:56:02.826704189 +0000 UTC m=+943.893195676"
Sep 16 18:56:05.962912 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:56:05.962872    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-kzt7p"
Sep 16 18:56:05.974908 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:56:05.974879    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-kzt7p"
Sep 16 18:56:06.841631 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:56:06.841603    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-kzt7p"
Sep 16 18:56:15.281697 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:56:15.281624    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 18:56:17.189814 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:56:17.189777    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 18:56:17.288911 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:56:17.288874    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 18:56:17.400124 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:56:17.400088    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 18:56:17.527406 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:56:17.527373    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 18:56:17.642868 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:56:17.642839    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 18:56:20.209850 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:56:20.209821    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 18:56:22.251842 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:56:22.251805    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 18:56:27.307942 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:56:27.307906    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 18:56:27.942844 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:56:27.942791    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 18:56:29.309306 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:56:29.309225    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 18:56:33.815125 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:56:33.815097    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbljd5t"
Sep 16 18:56:33.820285 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:56:33.820261    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 18:56:35.363571 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:56:35.363536    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 18:56:35.529625 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:56:35.529595    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 18:56:41.510709 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:56:41.510620    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 18:56:43.033757 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:56:43.033700    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 18:56:44.785904 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:56:44.785857    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 18:56:44.878296 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:56:44.878262    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 18:56:45.849570 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:56:45.849536    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 18:56:49.848656 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:56:49.848615    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 18:56:55.446514 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:56:55.446486    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 18:56:56.361155 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:56:56.361068    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 18:57:00.453885 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:57:00.453844    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 18:57:00.679196 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:57:00.679160    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 18:57:02.897443 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:57:02.897407    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbljd5t"]
Sep 16 18:57:02.902839 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:57:02.902805    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 18:57:02.907910 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:57:02.907882    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-kzt7p"]
Sep 16 18:57:02.918895 ip-10-0-135-119 kubenswrapper[2578]: E0916 18:57:02.918860    2578 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 16 18:57:02.918895 ip-10-0-135-119 kubenswrapper[2578]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 18:57:02.918895 ip-10-0-135-119 kubenswrapper[2578]:         , stdout: , stderr: , exit code -1
Sep 16 18:57:02.918895 ip-10-0-135-119 kubenswrapper[2578]:  > containerID="17682a18465f3e36e0b25f3e354be18d4a18f933ccedfd9aac1a268970ebf211" cmd=["/bin/sleep","15"]
Sep 16 18:57:02.919282 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:57:02.918917    2578 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-kzt7p" message="" err=<
Sep 16 18:57:02.919282 ip-10-0-135-119 kubenswrapper[2578]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 18:57:02.919282 ip-10-0-135-119 kubenswrapper[2578]:         , stdout: , stderr: , exit code -1
Sep 16 18:57:02.919282 ip-10-0-135-119 kubenswrapper[2578]:  >
Sep 16 18:57:02.919282 ip-10-0-135-119 kubenswrapper[2578]: E0916 18:57:02.918934    2578 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 16 18:57:02.919282 ip-10-0-135-119 kubenswrapper[2578]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 18:57:02.919282 ip-10-0-135-119 kubenswrapper[2578]:         , stdout: , stderr: , exit code -1
Sep 16 18:57:02.919282 ip-10-0-135-119 kubenswrapper[2578]:  > pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-kzt7p" podUID="0024e7f2-4253-481c-9709-82b0b268c238" containerName="main" containerID="cri-o://17682a18465f3e36e0b25f3e354be18d4a18f933ccedfd9aac1a268970ebf211"
Sep 16 18:57:02.919282 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:57:02.919011    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-kzt7p" podUID="0024e7f2-4253-481c-9709-82b0b268c238" containerName="main" containerID="cri-o://17682a18465f3e36e0b25f3e354be18d4a18f933ccedfd9aac1a268970ebf211" gracePeriod=60
Sep 16 18:57:02.977023 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:57:02.976990    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 18:57:03.266356 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:57:03.266331    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-kzt7p"
Sep 16 18:57:03.376247 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:57:03.376217    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4k929\" (UniqueName: \"kubernetes.io/projected/0024e7f2-4253-481c-9709-82b0b268c238-kube-api-access-4k929\") pod \"0024e7f2-4253-481c-9709-82b0b268c238\" (UID: \"0024e7f2-4253-481c-9709-82b0b268c238\") "
Sep 16 18:57:03.376436 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:57:03.376254    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0024e7f2-4253-481c-9709-82b0b268c238-tmp-dir\") pod \"0024e7f2-4253-481c-9709-82b0b268c238\" (UID: \"0024e7f2-4253-481c-9709-82b0b268c238\") "
Sep 16 18:57:03.376436 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:57:03.376289    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0024e7f2-4253-481c-9709-82b0b268c238-tls-certs\") pod \"0024e7f2-4253-481c-9709-82b0b268c238\" (UID: \"0024e7f2-4253-481c-9709-82b0b268c238\") "
Sep 16 18:57:03.376436 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:57:03.376397    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0024e7f2-4253-481c-9709-82b0b268c238-home\") pod \"0024e7f2-4253-481c-9709-82b0b268c238\" (UID: \"0024e7f2-4253-481c-9709-82b0b268c238\") "
Sep 16 18:57:03.376619 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:57:03.376444    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0024e7f2-4253-481c-9709-82b0b268c238-dshm\") pod \"0024e7f2-4253-481c-9709-82b0b268c238\" (UID: \"0024e7f2-4253-481c-9709-82b0b268c238\") "
Sep 16 18:57:03.376619 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:57:03.376474    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0024e7f2-4253-481c-9709-82b0b268c238-model-cache\") pod \"0024e7f2-4253-481c-9709-82b0b268c238\" (UID: \"0024e7f2-4253-481c-9709-82b0b268c238\") "
Sep 16 18:57:03.376619 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:57:03.376493    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0024e7f2-4253-481c-9709-82b0b268c238-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "0024e7f2-4253-481c-9709-82b0b268c238" (UID: "0024e7f2-4253-481c-9709-82b0b268c238"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:57:03.376780 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:57:03.376633    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0024e7f2-4253-481c-9709-82b0b268c238-home" (OuterVolumeSpecName: "home") pod "0024e7f2-4253-481c-9709-82b0b268c238" (UID: "0024e7f2-4253-481c-9709-82b0b268c238"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:57:03.376780 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:57:03.376772    2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0024e7f2-4253-481c-9709-82b0b268c238-tmp-dir\") on node \"ip-10-0-135-119.ec2.internal\" DevicePath \"\""
Sep 16 18:57:03.376886 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:57:03.376786    2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0024e7f2-4253-481c-9709-82b0b268c238-home\") on node \"ip-10-0-135-119.ec2.internal\" DevicePath \"\""
Sep 16 18:57:03.376886 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:57:03.376786    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0024e7f2-4253-481c-9709-82b0b268c238-model-cache" (OuterVolumeSpecName: "model-cache") pod "0024e7f2-4253-481c-9709-82b0b268c238" (UID: "0024e7f2-4253-481c-9709-82b0b268c238"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:57:03.378701 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:57:03.378676    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0024e7f2-4253-481c-9709-82b0b268c238-dshm" (OuterVolumeSpecName: "dshm") pod "0024e7f2-4253-481c-9709-82b0b268c238" (UID: "0024e7f2-4253-481c-9709-82b0b268c238"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:57:03.378802 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:57:03.378696    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0024e7f2-4253-481c-9709-82b0b268c238-kube-api-access-4k929" (OuterVolumeSpecName: "kube-api-access-4k929") pod "0024e7f2-4253-481c-9709-82b0b268c238" (UID: "0024e7f2-4253-481c-9709-82b0b268c238"). InnerVolumeSpecName "kube-api-access-4k929". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:57:03.378802 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:57:03.378711    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0024e7f2-4253-481c-9709-82b0b268c238-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "0024e7f2-4253-481c-9709-82b0b268c238" (UID: "0024e7f2-4253-481c-9709-82b0b268c238"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:57:03.478052 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:57:03.478016    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0024e7f2-4253-481c-9709-82b0b268c238-tls-certs\") on node \"ip-10-0-135-119.ec2.internal\" DevicePath \"\""
Sep 16 18:57:03.478052 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:57:03.478044    2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0024e7f2-4253-481c-9709-82b0b268c238-dshm\") on node \"ip-10-0-135-119.ec2.internal\" DevicePath \"\""
Sep 16 18:57:03.478052 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:57:03.478055    2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0024e7f2-4253-481c-9709-82b0b268c238-model-cache\") on node \"ip-10-0-135-119.ec2.internal\" DevicePath \"\""
Sep 16 18:57:03.478268 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:57:03.478067    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4k929\" (UniqueName: \"kubernetes.io/projected/0024e7f2-4253-481c-9709-82b0b268c238-kube-api-access-4k929\") on node \"ip-10-0-135-119.ec2.internal\" DevicePath \"\""
Sep 16 18:57:03.814501 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:57:03.814418    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbljd5t" podUID="d3162ecf-6ee9-4b2b-a8d5-0ab3838f9835" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 18:57:04.106467 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:57:04.106379    2578 generic.go:356] "Generic (PLEG): container finished" podID="0024e7f2-4253-481c-9709-82b0b268c238" containerID="17682a18465f3e36e0b25f3e354be18d4a18f933ccedfd9aac1a268970ebf211" exitCode=0
Sep 16 18:57:04.106467 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:57:04.106425    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-kzt7p"
Sep 16 18:57:04.106969 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:57:04.106430    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-kzt7p" event={"ID":"0024e7f2-4253-481c-9709-82b0b268c238","Type":"ContainerDied","Data":"17682a18465f3e36e0b25f3e354be18d4a18f933ccedfd9aac1a268970ebf211"}
Sep 16 18:57:04.106969 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:57:04.106570    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-kzt7p" event={"ID":"0024e7f2-4253-481c-9709-82b0b268c238","Type":"ContainerDied","Data":"ede8e2642ef14163f622902a69dc25967ac107c26d0c40bb449a068eaab4cf6a"}
Sep 16 18:57:04.106969 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:57:04.106611    2578 scope.go:117] "RemoveContainer" containerID="17682a18465f3e36e0b25f3e354be18d4a18f933ccedfd9aac1a268970ebf211"
Sep 16 18:57:04.115328 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:57:04.115291    2578 scope.go:117] "RemoveContainer" containerID="17682a18465f3e36e0b25f3e354be18d4a18f933ccedfd9aac1a268970ebf211"
Sep 16 18:57:04.115540 ip-10-0-135-119 kubenswrapper[2578]: E0916 18:57:04.115524    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"17682a18465f3e36e0b25f3e354be18d4a18f933ccedfd9aac1a268970ebf211\": container with ID starting with 17682a18465f3e36e0b25f3e354be18d4a18f933ccedfd9aac1a268970ebf211 not found: ID does not exist" containerID="17682a18465f3e36e0b25f3e354be18d4a18f933ccedfd9aac1a268970ebf211"
Sep 16 18:57:04.115582 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:57:04.115546    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"17682a18465f3e36e0b25f3e354be18d4a18f933ccedfd9aac1a268970ebf211"} err="failed to get container status \"17682a18465f3e36e0b25f3e354be18d4a18f933ccedfd9aac1a268970ebf211\": rpc error: code = NotFound desc = could not find container \"17682a18465f3e36e0b25f3e354be18d4a18f933ccedfd9aac1a268970ebf211\": container with ID starting with 17682a18465f3e36e0b25f3e354be18d4a18f933ccedfd9aac1a268970ebf211 not found: ID does not exist"
Sep 16 18:57:04.124414 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:57:04.124395    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-kzt7p"]
Sep 16 18:57:04.131404 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:57:04.131383    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-kzt7p"]
Sep 16 18:57:05.434645 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:57:05.434614    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0024e7f2-4253-481c-9709-82b0b268c238" path="/var/lib/kubelet/pods/0024e7f2-4253-481c-9709-82b0b268c238/volumes"
Sep 16 18:57:07.194125 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:57:07.194088    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 18:57:08.056465 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:57:08.056435    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 18:57:08.161564 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:57:08.161530    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 18:57:10.948154 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:57:10.948121    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 18:57:13.814633 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:57:13.814590    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbljd5t" podUID="d3162ecf-6ee9-4b2b-a8d5-0ab3838f9835" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 18:57:17.919578 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:57:17.919530    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbljd5t" podUID="d3162ecf-6ee9-4b2b-a8d5-0ab3838f9835" containerName="main" containerID="cri-o://e11ad302c00c76e2be66913aae4886b4c1ac4b346f3236c75bc1df34a1f00428" gracePeriod=45
Sep 16 18:57:23.814798 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:57:23.814753    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbljd5t" podUID="d3162ecf-6ee9-4b2b-a8d5-0ab3838f9835" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 18:57:23.815177 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:57:23.814859    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbljd5t"
Sep 16 18:57:29.184365 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:57:29.184319    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 18:57:33.814910 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:57:33.814873    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbljd5t" podUID="d3162ecf-6ee9-4b2b-a8d5-0ab3838f9835" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 18:57:43.815085 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:57:43.815046    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbljd5t" podUID="d3162ecf-6ee9-4b2b-a8d5-0ab3838f9835" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 18:57:48.172788 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:57:48.172759    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbljd5t"
Sep 16 18:57:48.269329 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:57:48.269247    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d3162ecf-6ee9-4b2b-a8d5-0ab3838f9835-tls-certs\") pod \"d3162ecf-6ee9-4b2b-a8d5-0ab3838f9835\" (UID: \"d3162ecf-6ee9-4b2b-a8d5-0ab3838f9835\") "
Sep 16 18:57:48.269329 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:57:48.269287    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4j9wl\" (UniqueName: \"kubernetes.io/projected/d3162ecf-6ee9-4b2b-a8d5-0ab3838f9835-kube-api-access-4j9wl\") pod \"d3162ecf-6ee9-4b2b-a8d5-0ab3838f9835\" (UID: \"d3162ecf-6ee9-4b2b-a8d5-0ab3838f9835\") "
Sep 16 18:57:48.271439 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:57:48.271405    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d3162ecf-6ee9-4b2b-a8d5-0ab3838f9835-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "d3162ecf-6ee9-4b2b-a8d5-0ab3838f9835" (UID: "d3162ecf-6ee9-4b2b-a8d5-0ab3838f9835"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:57:48.271561 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:57:48.271466    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d3162ecf-6ee9-4b2b-a8d5-0ab3838f9835-kube-api-access-4j9wl" (OuterVolumeSpecName: "kube-api-access-4j9wl") pod "d3162ecf-6ee9-4b2b-a8d5-0ab3838f9835" (UID: "d3162ecf-6ee9-4b2b-a8d5-0ab3838f9835"). InnerVolumeSpecName "kube-api-access-4j9wl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:57:48.320395 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:57:48.320362    2578 generic.go:356] "Generic (PLEG): container finished" podID="d3162ecf-6ee9-4b2b-a8d5-0ab3838f9835" containerID="e11ad302c00c76e2be66913aae4886b4c1ac4b346f3236c75bc1df34a1f00428" exitCode=0
Sep 16 18:57:48.320543 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:57:48.320412    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbljd5t"
Sep 16 18:57:48.320543 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:57:48.320456    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbljd5t" event={"ID":"d3162ecf-6ee9-4b2b-a8d5-0ab3838f9835","Type":"ContainerDied","Data":"e11ad302c00c76e2be66913aae4886b4c1ac4b346f3236c75bc1df34a1f00428"}
Sep 16 18:57:48.320543 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:57:48.320516    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbljd5t" event={"ID":"d3162ecf-6ee9-4b2b-a8d5-0ab3838f9835","Type":"ContainerDied","Data":"28cf21d5359d74caaf1019707a9dee23f620ce95ba29d2bc386c97dc8383de8e"}
Sep 16 18:57:48.320681 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:57:48.320544    2578 scope.go:117] "RemoveContainer" containerID="e11ad302c00c76e2be66913aae4886b4c1ac4b346f3236c75bc1df34a1f00428"
Sep 16 18:57:48.329781 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:57:48.329764    2578 scope.go:117] "RemoveContainer" containerID="e11ad302c00c76e2be66913aae4886b4c1ac4b346f3236c75bc1df34a1f00428"
Sep 16 18:57:48.330066 ip-10-0-135-119 kubenswrapper[2578]: E0916 18:57:48.330044    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e11ad302c00c76e2be66913aae4886b4c1ac4b346f3236c75bc1df34a1f00428\": container with ID starting with e11ad302c00c76e2be66913aae4886b4c1ac4b346f3236c75bc1df34a1f00428 not found: ID does not exist" containerID="e11ad302c00c76e2be66913aae4886b4c1ac4b346f3236c75bc1df34a1f00428"
Sep 16 18:57:48.330162 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:57:48.330071    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e11ad302c00c76e2be66913aae4886b4c1ac4b346f3236c75bc1df34a1f00428"} err="failed to get container status \"e11ad302c00c76e2be66913aae4886b4c1ac4b346f3236c75bc1df34a1f00428\": rpc error: code = NotFound desc = could not find container \"e11ad302c00c76e2be66913aae4886b4c1ac4b346f3236c75bc1df34a1f00428\": container with ID starting with e11ad302c00c76e2be66913aae4886b4c1ac4b346f3236c75bc1df34a1f00428 not found: ID does not exist"
Sep 16 18:57:48.344365 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:57:48.344340    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbljd5t"]
Sep 16 18:57:48.356911 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:57:48.356888    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbljd5t"]
Sep 16 18:57:48.370764 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:57:48.370741    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4j9wl\" (UniqueName: \"kubernetes.io/projected/d3162ecf-6ee9-4b2b-a8d5-0ab3838f9835-kube-api-access-4j9wl\") on node \"ip-10-0-135-119.ec2.internal\" DevicePath \"\""
Sep 16 18:57:48.370764 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:57:48.370762    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d3162ecf-6ee9-4b2b-a8d5-0ab3838f9835-tls-certs\") on node \"ip-10-0-135-119.ec2.internal\" DevicePath \"\""
Sep 16 18:57:49.434569 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:57:49.434536    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d3162ecf-6ee9-4b2b-a8d5-0ab3838f9835" path="/var/lib/kubelet/pods/d3162ecf-6ee9-4b2b-a8d5-0ab3838f9835/volumes"
Sep 16 18:57:52.518159 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:57:52.518130    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 18:57:52.613724 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:57:52.613694    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 18:57:59.128281 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:57:59.128243    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 18:58:00.291184 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:58:00.291145    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fghm5r"]
Sep 16 18:58:00.291721 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:58:00.291701    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0024e7f2-4253-481c-9709-82b0b268c238" containerName="main"
Sep 16 18:58:00.291721 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:58:00.291721    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="0024e7f2-4253-481c-9709-82b0b268c238" containerName="main"
Sep 16 18:58:00.291841 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:58:00.291750    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d3162ecf-6ee9-4b2b-a8d5-0ab3838f9835" containerName="main"
Sep 16 18:58:00.291841 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:58:00.291759    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="d3162ecf-6ee9-4b2b-a8d5-0ab3838f9835" containerName="main"
Sep 16 18:58:00.291947 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:58:00.291839    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="d3162ecf-6ee9-4b2b-a8d5-0ab3838f9835" containerName="main"
Sep 16 18:58:00.291947 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:58:00.291858    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="0024e7f2-4253-481c-9709-82b0b268c238" containerName="main"
Sep 16 18:58:00.296058 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:58:00.296037    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fghm5r"
Sep 16 18:58:00.298619 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:58:00.298600    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-aae51b37\"/\"llmisv431d7e30f13554296364403a5b6365f5-kserve-self-signed-certs\""
Sep 16 18:58:00.299313 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:58:00.299290    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-aae51b37\"/\"default-dockercfg-lvlvv\""
Sep 16 18:58:00.299410 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:58:00.299333    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-aae51b37\"/\"openshift-service-ca.crt\""
Sep 16 18:58:00.299410 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:58:00.299333    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-aae51b37\"/\"kube-root-ca.crt\""
Sep 16 18:58:00.315120 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:58:00.315087    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fghm5r"]
Sep 16 18:58:00.323520 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:58:00.323492    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 18:58:00.377024 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:58:00.376997    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/20799a18-e910-4803-9416-1a9a1698262e-tmp-dir\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fghm5r\" (UID: \"20799a18-e910-4803-9416-1a9a1698262e\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fghm5r"
Sep 16 18:58:00.377197 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:58:00.377049    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/20799a18-e910-4803-9416-1a9a1698262e-dshm\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fghm5r\" (UID: \"20799a18-e910-4803-9416-1a9a1698262e\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fghm5r"
Sep 16 18:58:00.377197 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:58:00.377079    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/20799a18-e910-4803-9416-1a9a1698262e-model-cache\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fghm5r\" (UID: \"20799a18-e910-4803-9416-1a9a1698262e\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fghm5r"
Sep 16 18:58:00.377197 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:58:00.377123    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/20799a18-e910-4803-9416-1a9a1698262e-tls-certs\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fghm5r\" (UID: \"20799a18-e910-4803-9416-1a9a1698262e\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fghm5r"
Sep 16 18:58:00.377197 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:58:00.377156    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nhh9x\" (UniqueName: \"kubernetes.io/projected/20799a18-e910-4803-9416-1a9a1698262e-kube-api-access-nhh9x\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fghm5r\" (UID: \"20799a18-e910-4803-9416-1a9a1698262e\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fghm5r"
Sep 16 18:58:00.377367 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:58:00.377207    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/20799a18-e910-4803-9416-1a9a1698262e-home\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fghm5r\" (UID: \"20799a18-e910-4803-9416-1a9a1698262e\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fghm5r"
Sep 16 18:58:00.478281 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:58:00.478251    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/20799a18-e910-4803-9416-1a9a1698262e-dshm\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fghm5r\" (UID: \"20799a18-e910-4803-9416-1a9a1698262e\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fghm5r"
Sep 16 18:58:00.478434 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:58:00.478288    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/20799a18-e910-4803-9416-1a9a1698262e-model-cache\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fghm5r\" (UID: \"20799a18-e910-4803-9416-1a9a1698262e\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fghm5r"
Sep 16 18:58:00.478434 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:58:00.478318    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/20799a18-e910-4803-9416-1a9a1698262e-tls-certs\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fghm5r\" (UID: \"20799a18-e910-4803-9416-1a9a1698262e\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fghm5r"
Sep 16 18:58:00.478434 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:58:00.478350    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nhh9x\" (UniqueName: \"kubernetes.io/projected/20799a18-e910-4803-9416-1a9a1698262e-kube-api-access-nhh9x\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fghm5r\" (UID: \"20799a18-e910-4803-9416-1a9a1698262e\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fghm5r"
Sep 16 18:58:00.478607 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:58:00.478435    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/20799a18-e910-4803-9416-1a9a1698262e-home\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fghm5r\" (UID: \"20799a18-e910-4803-9416-1a9a1698262e\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fghm5r"
Sep 16 18:58:00.478607 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:58:00.478495    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/20799a18-e910-4803-9416-1a9a1698262e-tmp-dir\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fghm5r\" (UID: \"20799a18-e910-4803-9416-1a9a1698262e\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fghm5r"
Sep 16 18:58:00.478819 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:58:00.478782    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/20799a18-e910-4803-9416-1a9a1698262e-model-cache\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fghm5r\" (UID: \"20799a18-e910-4803-9416-1a9a1698262e\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fghm5r"
Sep 16 18:58:00.478898 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:58:00.478824    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/20799a18-e910-4803-9416-1a9a1698262e-tmp-dir\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fghm5r\" (UID: \"20799a18-e910-4803-9416-1a9a1698262e\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fghm5r"
Sep 16 18:58:00.478898 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:58:00.478873    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/20799a18-e910-4803-9416-1a9a1698262e-home\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fghm5r\" (UID: \"20799a18-e910-4803-9416-1a9a1698262e\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fghm5r"
Sep 16 18:58:00.480650 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:58:00.480626    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/20799a18-e910-4803-9416-1a9a1698262e-dshm\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fghm5r\" (UID: \"20799a18-e910-4803-9416-1a9a1698262e\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fghm5r"
Sep 16 18:58:00.480917 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:58:00.480899    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/20799a18-e910-4803-9416-1a9a1698262e-tls-certs\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fghm5r\" (UID: \"20799a18-e910-4803-9416-1a9a1698262e\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fghm5r"
Sep 16 18:58:00.488021 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:58:00.488002    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nhh9x\" (UniqueName: \"kubernetes.io/projected/20799a18-e910-4803-9416-1a9a1698262e-kube-api-access-nhh9x\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fghm5r\" (UID: \"20799a18-e910-4803-9416-1a9a1698262e\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fghm5r"
Sep 16 18:58:00.557943 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:58:00.557870    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schetvtq7"]
Sep 16 18:58:00.561468 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:58:00.561446    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schetvtq7"
Sep 16 18:58:00.563621 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:58:00.563594    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-aae51b37\"/\"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-epp-sa-dockercfg-bt6bv\""
Sep 16 18:58:00.571889 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:58:00.571863    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schetvtq7"]
Sep 16 18:58:00.594146 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:58:00.594116    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 18:58:00.606692 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:58:00.606669    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fghm5r"
Sep 16 18:58:00.680714 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:58:00.680498    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-777c5\" (UniqueName: \"kubernetes.io/projected/fb723aa0-4482-49d7-870a-a16314410730-kube-api-access-777c5\") pod \"llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schetvtq7\" (UID: \"fb723aa0-4482-49d7-870a-a16314410730\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schetvtq7"
Sep 16 18:58:00.680714 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:58:00.680647    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fb723aa0-4482-49d7-870a-a16314410730-tls-certs\") pod \"llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schetvtq7\" (UID: \"fb723aa0-4482-49d7-870a-a16314410730\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schetvtq7"
Sep 16 18:58:00.781098 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:58:00.781066    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fb723aa0-4482-49d7-870a-a16314410730-tls-certs\") pod \"llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schetvtq7\" (UID: \"fb723aa0-4482-49d7-870a-a16314410730\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schetvtq7"
Sep 16 18:58:00.781276 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:58:00.781110    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-777c5\" (UniqueName: \"kubernetes.io/projected/fb723aa0-4482-49d7-870a-a16314410730-kube-api-access-777c5\") pod \"llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schetvtq7\" (UID: \"fb723aa0-4482-49d7-870a-a16314410730\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schetvtq7"
Sep 16 18:58:00.784167 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:58:00.784137    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fb723aa0-4482-49d7-870a-a16314410730-tls-certs\") pod \"llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schetvtq7\" (UID: \"fb723aa0-4482-49d7-870a-a16314410730\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schetvtq7"
Sep 16 18:58:00.790286 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:58:00.790263    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-777c5\" (UniqueName: \"kubernetes.io/projected/fb723aa0-4482-49d7-870a-a16314410730-kube-api-access-777c5\") pod \"llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schetvtq7\" (UID: \"fb723aa0-4482-49d7-870a-a16314410730\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schetvtq7"
Sep 16 18:58:00.872644 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:58:00.872563    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schetvtq7"
Sep 16 18:58:00.907392 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:58:00.907369    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fghm5r"]
Sep 16 18:58:00.910467 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:58:00.910437    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod20799a18_e910_4803_9416_1a9a1698262e.slice/crio-f8e90cde468ca2659bf112faef6c9296a45f46d5a00e171af7853a255e53362c WatchSource:0}: Error finding container f8e90cde468ca2659bf112faef6c9296a45f46d5a00e171af7853a255e53362c: Status 404 returned error can't find the container with id f8e90cde468ca2659bf112faef6c9296a45f46d5a00e171af7853a255e53362c
Sep 16 18:58:01.159424 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:58:01.159386    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schetvtq7"]
Sep 16 18:58:01.161349 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:58:01.161326    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb723aa0_4482_49d7_870a_a16314410730.slice/crio-8460c5b215d5037da71ec6450c31b7b457f628c734a3d4192d4eda2ee91649ea WatchSource:0}: Error finding container 8460c5b215d5037da71ec6450c31b7b457f628c734a3d4192d4eda2ee91649ea: Status 404 returned error can't find the container with id 8460c5b215d5037da71ec6450c31b7b457f628c734a3d4192d4eda2ee91649ea
Sep 16 18:58:01.385346 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:58:01.385302    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fghm5r" event={"ID":"20799a18-e910-4803-9416-1a9a1698262e","Type":"ContainerStarted","Data":"80cdc3c80f7b8b55ac36c6b70402e5b5dccce1e1a4e3a7e532c3c5eafd2dfb2e"}
Sep 16 18:58:01.386083 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:58:01.385373    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fghm5r" event={"ID":"20799a18-e910-4803-9416-1a9a1698262e","Type":"ContainerStarted","Data":"f8e90cde468ca2659bf112faef6c9296a45f46d5a00e171af7853a255e53362c"}
Sep 16 18:58:01.387351 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:58:01.387328    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schetvtq7" event={"ID":"fb723aa0-4482-49d7-870a-a16314410730","Type":"ContainerStarted","Data":"b7fb190a1385b1bb25754a0b65a5278693a757766ed5a8188897c045087a9104"}
Sep 16 18:58:01.387444 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:58:01.387370    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schetvtq7" event={"ID":"fb723aa0-4482-49d7-870a-a16314410730","Type":"ContainerStarted","Data":"8460c5b215d5037da71ec6450c31b7b457f628c734a3d4192d4eda2ee91649ea"}
Sep 16 18:58:01.387444 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:58:01.387406    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schetvtq7"
Sep 16 18:58:01.412165 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:58:01.412076    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fghm5r" podStartSLOduration=1.41205733 podStartE2EDuration="1.41205733s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:58:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:58:01.41061686 +0000 UTC m=+1062.477108381" watchObservedRunningTime="2026-09-16 18:58:01.41205733 +0000 UTC m=+1062.478548818"
Sep 16 18:58:01.430146 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:58:01.430095    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schetvtq7" podStartSLOduration=1.430080614 podStartE2EDuration="1.430080614s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:58:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:58:01.429074623 +0000 UTC m=+1062.495566134" watchObservedRunningTime="2026-09-16 18:58:01.430080614 +0000 UTC m=+1062.496572103"
Sep 16 18:58:01.607223 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:58:01.607188    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fghm5r"
Sep 16 18:58:05.282682 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:58:05.282640    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 18:58:05.350614 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:58:05.350584    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 18:58:10.607444 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:58:10.607354    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fghm5r"
Sep 16 18:58:10.619825 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:58:10.619791    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fghm5r"
Sep 16 18:58:11.445499 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:58:11.445472    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fghm5r"
Sep 16 18:58:13.071772 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:58:13.071735    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 18:58:32.395348 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:58:32.395316    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schetvtq7"
Sep 16 18:58:34.394806 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:58:34.394771    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fghm5r"]
Sep 16 18:58:34.401589 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:58:34.401561    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schetvtq7"]
Sep 16 18:58:34.405215 ip-10-0-135-119 kubenswrapper[2578]: E0916 18:58:34.405177    2578 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 16 18:58:34.405215 ip-10-0-135-119 kubenswrapper[2578]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 18:58:34.405215 ip-10-0-135-119 kubenswrapper[2578]:         , stdout: , stderr: , exit code -1
Sep 16 18:58:34.405215 ip-10-0-135-119 kubenswrapper[2578]:  > containerID="80cdc3c80f7b8b55ac36c6b70402e5b5dccce1e1a4e3a7e532c3c5eafd2dfb2e" cmd=["/bin/sleep","15"]
Sep 16 18:58:34.405450 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:58:34.405229    2578 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fghm5r" message="" err=<
Sep 16 18:58:34.405450 ip-10-0-135-119 kubenswrapper[2578]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 18:58:34.405450 ip-10-0-135-119 kubenswrapper[2578]:         , stdout: , stderr: , exit code -1
Sep 16 18:58:34.405450 ip-10-0-135-119 kubenswrapper[2578]:  >
Sep 16 18:58:34.405450 ip-10-0-135-119 kubenswrapper[2578]: E0916 18:58:34.405243    2578 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 16 18:58:34.405450 ip-10-0-135-119 kubenswrapper[2578]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 18:58:34.405450 ip-10-0-135-119 kubenswrapper[2578]:         , stdout: , stderr: , exit code -1
Sep 16 18:58:34.405450 ip-10-0-135-119 kubenswrapper[2578]:  > pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fghm5r" podUID="20799a18-e910-4803-9416-1a9a1698262e" containerName="main" containerID="cri-o://80cdc3c80f7b8b55ac36c6b70402e5b5dccce1e1a4e3a7e532c3c5eafd2dfb2e"
Sep 16 18:58:34.405450 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:58:34.405286    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fghm5r" podUID="20799a18-e910-4803-9416-1a9a1698262e" containerName="main" containerID="cri-o://80cdc3c80f7b8b55ac36c6b70402e5b5dccce1e1a4e3a7e532c3c5eafd2dfb2e" gracePeriod=60
Sep 16 18:58:34.434857 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:58:34.434821    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fghm5r" podUID="20799a18-e910-4803-9416-1a9a1698262e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.38:8000/health\": dial tcp 10.132.0.38:8000: connect: connection refused"
Sep 16 18:58:34.461237 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:58:34.461205    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 18:58:34.565269 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:58:34.565211    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 18:58:34.749688 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:58:34.749664    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fghm5r"
Sep 16 18:58:34.784363 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:58:34.784334    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/20799a18-e910-4803-9416-1a9a1698262e-model-cache\") pod \"20799a18-e910-4803-9416-1a9a1698262e\" (UID: \"20799a18-e910-4803-9416-1a9a1698262e\") "
Sep 16 18:58:34.784526 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:58:34.784374    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/20799a18-e910-4803-9416-1a9a1698262e-home\") pod \"20799a18-e910-4803-9416-1a9a1698262e\" (UID: \"20799a18-e910-4803-9416-1a9a1698262e\") "
Sep 16 18:58:34.784526 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:58:34.784433    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/20799a18-e910-4803-9416-1a9a1698262e-tls-certs\") pod \"20799a18-e910-4803-9416-1a9a1698262e\" (UID: \"20799a18-e910-4803-9416-1a9a1698262e\") "
Sep 16 18:58:34.784526 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:58:34.784455    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/20799a18-e910-4803-9416-1a9a1698262e-tmp-dir\") pod \"20799a18-e910-4803-9416-1a9a1698262e\" (UID: \"20799a18-e910-4803-9416-1a9a1698262e\") "
Sep 16 18:58:34.784526 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:58:34.784492    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nhh9x\" (UniqueName: \"kubernetes.io/projected/20799a18-e910-4803-9416-1a9a1698262e-kube-api-access-nhh9x\") pod \"20799a18-e910-4803-9416-1a9a1698262e\" (UID: \"20799a18-e910-4803-9416-1a9a1698262e\") "
Sep 16 18:58:34.784763 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:58:34.784543    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/20799a18-e910-4803-9416-1a9a1698262e-dshm\") pod \"20799a18-e910-4803-9416-1a9a1698262e\" (UID: \"20799a18-e910-4803-9416-1a9a1698262e\") "
Sep 16 18:58:34.784763 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:58:34.784623    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20799a18-e910-4803-9416-1a9a1698262e-model-cache" (OuterVolumeSpecName: "model-cache") pod "20799a18-e910-4803-9416-1a9a1698262e" (UID: "20799a18-e910-4803-9416-1a9a1698262e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:58:34.784763 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:58:34.784690    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20799a18-e910-4803-9416-1a9a1698262e-home" (OuterVolumeSpecName: "home") pod "20799a18-e910-4803-9416-1a9a1698262e" (UID: "20799a18-e910-4803-9416-1a9a1698262e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:58:34.784913 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:58:34.784819    2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/20799a18-e910-4803-9416-1a9a1698262e-model-cache\") on node \"ip-10-0-135-119.ec2.internal\" DevicePath \"\""
Sep 16 18:58:34.784913 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:58:34.784837    2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/20799a18-e910-4803-9416-1a9a1698262e-home\") on node \"ip-10-0-135-119.ec2.internal\" DevicePath \"\""
Sep 16 18:58:34.785188 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:58:34.785102    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20799a18-e910-4803-9416-1a9a1698262e-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "20799a18-e910-4803-9416-1a9a1698262e" (UID: "20799a18-e910-4803-9416-1a9a1698262e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:58:34.787218 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:58:34.787190    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/20799a18-e910-4803-9416-1a9a1698262e-kube-api-access-nhh9x" (OuterVolumeSpecName: "kube-api-access-nhh9x") pod "20799a18-e910-4803-9416-1a9a1698262e" (UID: "20799a18-e910-4803-9416-1a9a1698262e"). InnerVolumeSpecName "kube-api-access-nhh9x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:58:34.787218 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:58:34.787205    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/20799a18-e910-4803-9416-1a9a1698262e-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "20799a18-e910-4803-9416-1a9a1698262e" (UID: "20799a18-e910-4803-9416-1a9a1698262e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:58:34.787350 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:58:34.787225    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20799a18-e910-4803-9416-1a9a1698262e-dshm" (OuterVolumeSpecName: "dshm") pod "20799a18-e910-4803-9416-1a9a1698262e" (UID: "20799a18-e910-4803-9416-1a9a1698262e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:58:34.885487 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:58:34.885459    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/20799a18-e910-4803-9416-1a9a1698262e-tls-certs\") on node \"ip-10-0-135-119.ec2.internal\" DevicePath \"\""
Sep 16 18:58:34.885487 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:58:34.885486    2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/20799a18-e910-4803-9416-1a9a1698262e-tmp-dir\") on node \"ip-10-0-135-119.ec2.internal\" DevicePath \"\""
Sep 16 18:58:34.885686 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:58:34.885498    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nhh9x\" (UniqueName: \"kubernetes.io/projected/20799a18-e910-4803-9416-1a9a1698262e-kube-api-access-nhh9x\") on node \"ip-10-0-135-119.ec2.internal\" DevicePath \"\""
Sep 16 18:58:34.885686 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:58:34.885510    2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/20799a18-e910-4803-9416-1a9a1698262e-dshm\") on node \"ip-10-0-135-119.ec2.internal\" DevicePath \"\""
Sep 16 18:58:35.555800 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:58:35.555760    2578 generic.go:356] "Generic (PLEG): container finished" podID="20799a18-e910-4803-9416-1a9a1698262e" containerID="80cdc3c80f7b8b55ac36c6b70402e5b5dccce1e1a4e3a7e532c3c5eafd2dfb2e" exitCode=0
Sep 16 18:58:35.556285 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:58:35.555807    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fghm5r"
Sep 16 18:58:35.556285 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:58:35.555845    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fghm5r" event={"ID":"20799a18-e910-4803-9416-1a9a1698262e","Type":"ContainerDied","Data":"80cdc3c80f7b8b55ac36c6b70402e5b5dccce1e1a4e3a7e532c3c5eafd2dfb2e"}
Sep 16 18:58:35.556285 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:58:35.555898    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fghm5r" event={"ID":"20799a18-e910-4803-9416-1a9a1698262e","Type":"ContainerDied","Data":"f8e90cde468ca2659bf112faef6c9296a45f46d5a00e171af7853a255e53362c"}
Sep 16 18:58:35.556285 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:58:35.555920    2578 scope.go:117] "RemoveContainer" containerID="80cdc3c80f7b8b55ac36c6b70402e5b5dccce1e1a4e3a7e532c3c5eafd2dfb2e"
Sep 16 18:58:35.564540 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:58:35.564522    2578 scope.go:117] "RemoveContainer" containerID="80cdc3c80f7b8b55ac36c6b70402e5b5dccce1e1a4e3a7e532c3c5eafd2dfb2e"
Sep 16 18:58:35.564818 ip-10-0-135-119 kubenswrapper[2578]: E0916 18:58:35.564795    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"80cdc3c80f7b8b55ac36c6b70402e5b5dccce1e1a4e3a7e532c3c5eafd2dfb2e\": container with ID starting with 80cdc3c80f7b8b55ac36c6b70402e5b5dccce1e1a4e3a7e532c3c5eafd2dfb2e not found: ID does not exist" containerID="80cdc3c80f7b8b55ac36c6b70402e5b5dccce1e1a4e3a7e532c3c5eafd2dfb2e"
Sep 16 18:58:35.564876 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:58:35.564830    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"80cdc3c80f7b8b55ac36c6b70402e5b5dccce1e1a4e3a7e532c3c5eafd2dfb2e"} err="failed to get container status \"80cdc3c80f7b8b55ac36c6b70402e5b5dccce1e1a4e3a7e532c3c5eafd2dfb2e\": rpc error: code = NotFound desc = could not find container \"80cdc3c80f7b8b55ac36c6b70402e5b5dccce1e1a4e3a7e532c3c5eafd2dfb2e\": container with ID starting with 80cdc3c80f7b8b55ac36c6b70402e5b5dccce1e1a4e3a7e532c3c5eafd2dfb2e not found: ID does not exist"
Sep 16 18:58:35.578692 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:58:35.578668    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fghm5r"]
Sep 16 18:58:35.585016 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:58:35.584991    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fghm5r"]
Sep 16 18:58:37.434747 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:58:37.434716    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="20799a18-e910-4803-9416-1a9a1698262e" path="/var/lib/kubelet/pods/20799a18-e910-4803-9416-1a9a1698262e/volumes"
Sep 16 18:58:42.395864 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:58:42.395823    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schetvtq7" podUID="fb723aa0-4482-49d7-870a-a16314410730" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 18:58:49.421665 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:58:49.421608    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schetvtq7" podUID="fb723aa0-4482-49d7-870a-a16314410730" containerName="main" containerID="cri-o://b7fb190a1385b1bb25754a0b65a5278693a757766ed5a8188897c045087a9104" gracePeriod=45
Sep 16 18:58:52.395545 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:58:52.395498    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schetvtq7" podUID="fb723aa0-4482-49d7-870a-a16314410730" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 18:58:55.959319 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:58:55.959276    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 18:58:56.111551 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:58:56.111498    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 18:59:02.395708 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:59:02.395667    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schetvtq7" podUID="fb723aa0-4482-49d7-870a-a16314410730" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 18:59:02.396113 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:59:02.395792    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schetvtq7"
Sep 16 18:59:12.395924 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:59:12.395875    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schetvtq7" podUID="fb723aa0-4482-49d7-870a-a16314410730" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 18:59:19.683138 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:59:19.683069    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schetvtq7"
Sep 16 18:59:19.698215 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:59:19.698182    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-777c5\" (UniqueName: \"kubernetes.io/projected/fb723aa0-4482-49d7-870a-a16314410730-kube-api-access-777c5\") pod \"fb723aa0-4482-49d7-870a-a16314410730\" (UID: \"fb723aa0-4482-49d7-870a-a16314410730\") "
Sep 16 18:59:19.698351 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:59:19.698221    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fb723aa0-4482-49d7-870a-a16314410730-tls-certs\") pod \"fb723aa0-4482-49d7-870a-a16314410730\" (UID: \"fb723aa0-4482-49d7-870a-a16314410730\") "
Sep 16 18:59:19.700592 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:59:19.700554    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fb723aa0-4482-49d7-870a-a16314410730-kube-api-access-777c5" (OuterVolumeSpecName: "kube-api-access-777c5") pod "fb723aa0-4482-49d7-870a-a16314410730" (UID: "fb723aa0-4482-49d7-870a-a16314410730"). InnerVolumeSpecName "kube-api-access-777c5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:59:19.700711 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:59:19.700644    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fb723aa0-4482-49d7-870a-a16314410730-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "fb723aa0-4482-49d7-870a-a16314410730" (UID: "fb723aa0-4482-49d7-870a-a16314410730"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:59:19.780112 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:59:19.780075    2578 generic.go:356] "Generic (PLEG): container finished" podID="fb723aa0-4482-49d7-870a-a16314410730" containerID="b7fb190a1385b1bb25754a0b65a5278693a757766ed5a8188897c045087a9104" exitCode=0
Sep 16 18:59:19.780295 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:59:19.780128    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schetvtq7"
Sep 16 18:59:19.780295 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:59:19.780162    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schetvtq7" event={"ID":"fb723aa0-4482-49d7-870a-a16314410730","Type":"ContainerDied","Data":"b7fb190a1385b1bb25754a0b65a5278693a757766ed5a8188897c045087a9104"}
Sep 16 18:59:19.780295 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:59:19.780212    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schetvtq7" event={"ID":"fb723aa0-4482-49d7-870a-a16314410730","Type":"ContainerDied","Data":"8460c5b215d5037da71ec6450c31b7b457f628c734a3d4192d4eda2ee91649ea"}
Sep 16 18:59:19.780295 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:59:19.780235    2578 scope.go:117] "RemoveContainer" containerID="b7fb190a1385b1bb25754a0b65a5278693a757766ed5a8188897c045087a9104"
Sep 16 18:59:19.789625 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:59:19.789605    2578 scope.go:117] "RemoveContainer" containerID="b7fb190a1385b1bb25754a0b65a5278693a757766ed5a8188897c045087a9104"
Sep 16 18:59:19.789964 ip-10-0-135-119 kubenswrapper[2578]: E0916 18:59:19.789943    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b7fb190a1385b1bb25754a0b65a5278693a757766ed5a8188897c045087a9104\": container with ID starting with b7fb190a1385b1bb25754a0b65a5278693a757766ed5a8188897c045087a9104 not found: ID does not exist" containerID="b7fb190a1385b1bb25754a0b65a5278693a757766ed5a8188897c045087a9104"
Sep 16 18:59:19.790062 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:59:19.789993    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b7fb190a1385b1bb25754a0b65a5278693a757766ed5a8188897c045087a9104"} err="failed to get container status \"b7fb190a1385b1bb25754a0b65a5278693a757766ed5a8188897c045087a9104\": rpc error: code = NotFound desc = could not find container \"b7fb190a1385b1bb25754a0b65a5278693a757766ed5a8188897c045087a9104\": container with ID starting with b7fb190a1385b1bb25754a0b65a5278693a757766ed5a8188897c045087a9104 not found: ID does not exist"
Sep 16 18:59:19.799440 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:59:19.799414    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-777c5\" (UniqueName: \"kubernetes.io/projected/fb723aa0-4482-49d7-870a-a16314410730-kube-api-access-777c5\") on node \"ip-10-0-135-119.ec2.internal\" DevicePath \"\""
Sep 16 18:59:19.799440 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:59:19.799441    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fb723aa0-4482-49d7-870a-a16314410730-tls-certs\") on node \"ip-10-0-135-119.ec2.internal\" DevicePath \"\""
Sep 16 18:59:19.810493 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:59:19.810462    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schetvtq7"]
Sep 16 18:59:19.828684 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:59:19.828646    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schetvtq7"]
Sep 16 18:59:21.434588 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:59:21.434551    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fb723aa0-4482-49d7-870a-a16314410730" path="/var/lib/kubelet/pods/fb723aa0-4482-49d7-870a-a16314410730/volumes"
Sep 16 18:59:24.385674 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:59:24.385616    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 18:59:24.486660 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:59:24.486618    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 18:59:31.684595 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:59:31.684539    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 18:59:38.933080 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:59:38.933043    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 18:59:39.060564 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:59:39.060527    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 18:59:41.295575 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:59:41.295497    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-p4fsf"]
Sep 16 18:59:41.296038 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:59:41.295831    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="20799a18-e910-4803-9416-1a9a1698262e" containerName="main"
Sep 16 18:59:41.296038 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:59:41.295842    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="20799a18-e910-4803-9416-1a9a1698262e" containerName="main"
Sep 16 18:59:41.296038 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:59:41.295857    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fb723aa0-4482-49d7-870a-a16314410730" containerName="main"
Sep 16 18:59:41.296038 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:59:41.295862    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="fb723aa0-4482-49d7-870a-a16314410730" containerName="main"
Sep 16 18:59:41.296038 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:59:41.295922    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="fb723aa0-4482-49d7-870a-a16314410730" containerName="main"
Sep 16 18:59:41.296038 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:59:41.295932    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="20799a18-e910-4803-9416-1a9a1698262e" containerName="main"
Sep 16 18:59:41.300152 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:59:41.300127    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-p4fsf"
Sep 16 18:59:41.303234 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:59:41.303205    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-15d6126c\"/\"openshift-service-ca.crt\""
Sep 16 18:59:41.303378 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:59:41.303206    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-15d6126c\"/\"scheduler-configmap-ref-test-kserve-self-signed-certs\""
Sep 16 18:59:41.303378 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:59:41.303295    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-15d6126c\"/\"kube-root-ca.crt\""
Sep 16 18:59:41.303378 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:59:41.303327    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-15d6126c\"/\"default-dockercfg-mbrg9\""
Sep 16 18:59:41.313601 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:59:41.313581    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-p4fsf"]
Sep 16 18:59:41.328021 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:59:41.327973    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 18:59:41.394421 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:59:41.394393    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b699c7f4-1a33-4af0-a38d-d8085854563a-tmp-dir\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-p4fsf\" (UID: \"b699c7f4-1a33-4af0-a38d-d8085854563a\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-p4fsf"
Sep 16 18:59:41.394421 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:59:41.394422    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b699c7f4-1a33-4af0-a38d-d8085854563a-tls-certs\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-p4fsf\" (UID: \"b699c7f4-1a33-4af0-a38d-d8085854563a\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-p4fsf"
Sep 16 18:59:41.394607 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:59:41.394441    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b699c7f4-1a33-4af0-a38d-d8085854563a-home\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-p4fsf\" (UID: \"b699c7f4-1a33-4af0-a38d-d8085854563a\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-p4fsf"
Sep 16 18:59:41.394607 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:59:41.394547    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b699c7f4-1a33-4af0-a38d-d8085854563a-model-cache\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-p4fsf\" (UID: \"b699c7f4-1a33-4af0-a38d-d8085854563a\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-p4fsf"
Sep 16 18:59:41.394607 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:59:41.394574    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m87qn\" (UniqueName: \"kubernetes.io/projected/b699c7f4-1a33-4af0-a38d-d8085854563a-kube-api-access-m87qn\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-p4fsf\" (UID: \"b699c7f4-1a33-4af0-a38d-d8085854563a\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-p4fsf"
Sep 16 18:59:41.394746 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:59:41.394613    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b699c7f4-1a33-4af0-a38d-d8085854563a-dshm\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-p4fsf\" (UID: \"b699c7f4-1a33-4af0-a38d-d8085854563a\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-p4fsf"
Sep 16 18:59:41.495334 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:59:41.495297    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b699c7f4-1a33-4af0-a38d-d8085854563a-model-cache\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-p4fsf\" (UID: \"b699c7f4-1a33-4af0-a38d-d8085854563a\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-p4fsf"
Sep 16 18:59:41.495538 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:59:41.495341    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m87qn\" (UniqueName: \"kubernetes.io/projected/b699c7f4-1a33-4af0-a38d-d8085854563a-kube-api-access-m87qn\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-p4fsf\" (UID: \"b699c7f4-1a33-4af0-a38d-d8085854563a\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-p4fsf"
Sep 16 18:59:41.495538 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:59:41.495412    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b699c7f4-1a33-4af0-a38d-d8085854563a-dshm\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-p4fsf\" (UID: \"b699c7f4-1a33-4af0-a38d-d8085854563a\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-p4fsf"
Sep 16 18:59:41.495538 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:59:41.495507    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b699c7f4-1a33-4af0-a38d-d8085854563a-tmp-dir\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-p4fsf\" (UID: \"b699c7f4-1a33-4af0-a38d-d8085854563a\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-p4fsf"
Sep 16 18:59:41.495538 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:59:41.495533    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b699c7f4-1a33-4af0-a38d-d8085854563a-tls-certs\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-p4fsf\" (UID: \"b699c7f4-1a33-4af0-a38d-d8085854563a\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-p4fsf"
Sep 16 18:59:41.495791 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:59:41.495562    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b699c7f4-1a33-4af0-a38d-d8085854563a-home\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-p4fsf\" (UID: \"b699c7f4-1a33-4af0-a38d-d8085854563a\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-p4fsf"
Sep 16 18:59:41.495894 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:59:41.495876    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b699c7f4-1a33-4af0-a38d-d8085854563a-model-cache\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-p4fsf\" (UID: \"b699c7f4-1a33-4af0-a38d-d8085854563a\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-p4fsf"
Sep 16 18:59:41.496038 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:59:41.496013    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b699c7f4-1a33-4af0-a38d-d8085854563a-tmp-dir\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-p4fsf\" (UID: \"b699c7f4-1a33-4af0-a38d-d8085854563a\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-p4fsf"
Sep 16 18:59:41.496099 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:59:41.496035    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b699c7f4-1a33-4af0-a38d-d8085854563a-home\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-p4fsf\" (UID: \"b699c7f4-1a33-4af0-a38d-d8085854563a\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-p4fsf"
Sep 16 18:59:41.498012 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:59:41.497972    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b699c7f4-1a33-4af0-a38d-d8085854563a-dshm\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-p4fsf\" (UID: \"b699c7f4-1a33-4af0-a38d-d8085854563a\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-p4fsf"
Sep 16 18:59:41.498293 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:59:41.498276    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b699c7f4-1a33-4af0-a38d-d8085854563a-tls-certs\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-p4fsf\" (UID: \"b699c7f4-1a33-4af0-a38d-d8085854563a\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-p4fsf"
Sep 16 18:59:41.510065 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:59:41.510038    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m87qn\" (UniqueName: \"kubernetes.io/projected/b699c7f4-1a33-4af0-a38d-d8085854563a-kube-api-access-m87qn\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-p4fsf\" (UID: \"b699c7f4-1a33-4af0-a38d-d8085854563a\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-p4fsf"
Sep 16 18:59:41.610293 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:59:41.610193    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-p4fsf"
Sep 16 18:59:41.750746 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:59:41.750707    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 18:59:41.887062 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:59:41.887023    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-p4fsf"]
Sep 16 18:59:41.889715 ip-10-0-135-119 kubenswrapper[2578]: W0916 18:59:41.889689    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb699c7f4_1a33_4af0_a38d_d8085854563a.slice/crio-2e4ad687901538c6d36ce8c313d095f50cb44248a7e330b30263d7de5c851b99 WatchSource:0}: Error finding container 2e4ad687901538c6d36ce8c313d095f50cb44248a7e330b30263d7de5c851b99: Status 404 returned error can't find the container with id 2e4ad687901538c6d36ce8c313d095f50cb44248a7e330b30263d7de5c851b99
Sep 16 18:59:42.895015 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:59:42.894952    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-p4fsf" event={"ID":"b699c7f4-1a33-4af0-a38d-d8085854563a","Type":"ContainerStarted","Data":"5799da220d9d6c79f7fca98278420683d74b9cb6a4330d564e9d2fdc6f091a8c"}
Sep 16 18:59:42.895413 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:59:42.895058    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-p4fsf" event={"ID":"b699c7f4-1a33-4af0-a38d-d8085854563a","Type":"ContainerStarted","Data":"2e4ad687901538c6d36ce8c313d095f50cb44248a7e330b30263d7de5c851b99"}
Sep 16 18:59:42.930713 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:59:42.930653    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-p4fsf" podStartSLOduration=1.930637859 podStartE2EDuration="1.930637859s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:59:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:59:42.93033342 +0000 UTC m=+1163.996824907" watchObservedRunningTime="2026-09-16 18:59:42.930637859 +0000 UTC m=+1163.997129347"
Sep 16 18:59:43.611133 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:59:43.611092    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-p4fsf"
Sep 16 18:59:46.028072 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:59:46.028034    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 18:59:46.171191 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:59:46.171157    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 18:59:51.610458 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:59:51.610417    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-p4fsf"
Sep 16 18:59:51.623072 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:59:51.623039    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-p4fsf"
Sep 16 18:59:51.948201 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:59:51.948168    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-p4fsf"
Sep 16 18:59:54.392638 ip-10-0-135-119 kubenswrapper[2578]: I0916 18:59:54.392605    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:00:01.212195 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:00:01.212158    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:00:01.325192 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:00:01.325160    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:00:12.476545 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:00:12.476507    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:00:19.413327 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:00:19.413299    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-tmrkx_4c2078bb-281a-4aa1-8e38-238897865434/console-operator/2.log"
Sep 16 19:00:19.413827 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:00:19.413504    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-tmrkx_4c2078bb-281a-4aa1-8e38-238897865434/console-operator/2.log"
Sep 16 19:00:19.421435 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:00:19.421414    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-k6gvs_6012728d-aee6-4f93-ab6e-fad616ef36e8/ovn-acl-logging/0.log"
Sep 16 19:00:19.421937 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:00:19.421919    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-k6gvs_6012728d-aee6-4f93-ab6e-fad616ef36e8/ovn-acl-logging/0.log"
Sep 16 19:00:20.466657 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:00:20.466623    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:00:20.694076 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:00:20.694044    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9ckth59"]
Sep 16 19:00:20.699101 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:00:20.699085    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9ckth59"
Sep 16 19:00:20.702209 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:00:20.702178    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6\"/\"default-dockercfg-8v8vp\""
Sep 16 19:00:20.702331 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:00:20.702181    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6\"/\"kube-root-ca.crt\""
Sep 16 19:00:20.702331 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:00:20.702181    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6\"/\"auth-invalid-token-test-kserve-self-signed-certs\""
Sep 16 19:00:20.702461 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:00:20.702189    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6\"/\"openshift-service-ca.crt\""
Sep 16 19:00:20.702461 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:00:20.702265    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6\"/\"auth-invalid-token-test-epp-sa-dockercfg-znzd4\""
Sep 16 19:00:20.709349 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:00:20.709326    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9ckth59"]
Sep 16 19:00:20.725345 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:00:20.725284    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/41f68bae-75a8-4eee-a514-78afe30fff37-tls-certs\") pod \"auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9ckth59\" (UID: \"41f68bae-75a8-4eee-a514-78afe30fff37\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9ckth59"
Sep 16 19:00:20.725345 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:00:20.725318    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4fgdx\" (UniqueName: \"kubernetes.io/projected/41f68bae-75a8-4eee-a514-78afe30fff37-kube-api-access-4fgdx\") pod \"auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9ckth59\" (UID: \"41f68bae-75a8-4eee-a514-78afe30fff37\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9ckth59"
Sep 16 19:00:20.730701 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:00:20.730671    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:00:20.825839 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:00:20.825794    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/41f68bae-75a8-4eee-a514-78afe30fff37-tls-certs\") pod \"auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9ckth59\" (UID: \"41f68bae-75a8-4eee-a514-78afe30fff37\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9ckth59"
Sep 16 19:00:20.826041 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:00:20.825848    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4fgdx\" (UniqueName: \"kubernetes.io/projected/41f68bae-75a8-4eee-a514-78afe30fff37-kube-api-access-4fgdx\") pod \"auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9ckth59\" (UID: \"41f68bae-75a8-4eee-a514-78afe30fff37\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9ckth59"
Sep 16 19:00:20.828553 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:00:20.828531    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/41f68bae-75a8-4eee-a514-78afe30fff37-tls-certs\") pod \"auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9ckth59\" (UID: \"41f68bae-75a8-4eee-a514-78afe30fff37\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9ckth59"
Sep 16 19:00:20.838784 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:00:20.838762    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4fgdx\" (UniqueName: \"kubernetes.io/projected/41f68bae-75a8-4eee-a514-78afe30fff37-kube-api-access-4fgdx\") pod \"auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9ckth59\" (UID: \"41f68bae-75a8-4eee-a514-78afe30fff37\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9ckth59"
Sep 16 19:00:21.009755 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:00:21.009665    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9ckth59"
Sep 16 19:00:21.288428 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:00:21.288344    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9ckth59"]
Sep 16 19:00:21.292018 ip-10-0-135-119 kubenswrapper[2578]: W0916 19:00:21.291991    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41f68bae_75a8_4eee_a514_78afe30fff37.slice/crio-eb6a59e491a76096ebb6862af24388050c165bbe56b18ae5d1dfc129ba693281 WatchSource:0}: Error finding container eb6a59e491a76096ebb6862af24388050c165bbe56b18ae5d1dfc129ba693281: Status 404 returned error can't find the container with id eb6a59e491a76096ebb6862af24388050c165bbe56b18ae5d1dfc129ba693281
Sep 16 19:00:22.081611 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:00:22.081571    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9ckth59" event={"ID":"41f68bae-75a8-4eee-a514-78afe30fff37","Type":"ContainerStarted","Data":"79a37cb1e22cc682d0efdb0bb4f4c70a739434ddadc2d283b3a572f799d75d2d"}
Sep 16 19:00:22.081956 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:00:22.081634    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9ckth59" event={"ID":"41f68bae-75a8-4eee-a514-78afe30fff37","Type":"ContainerStarted","Data":"eb6a59e491a76096ebb6862af24388050c165bbe56b18ae5d1dfc129ba693281"}
Sep 16 19:00:22.081956 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:00:22.081680    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9ckth59"
Sep 16 19:00:22.100893 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:00:22.100829    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9ckth59" podStartSLOduration=2.100807441 podStartE2EDuration="2.100807441s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:00:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:00:22.099558945 +0000 UTC m=+1203.166050432" watchObservedRunningTime="2026-09-16 19:00:22.100807441 +0000 UTC m=+1203.167298930"
Sep 16 19:00:29.507580 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:00:29.507542    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:00:30.588104 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:00:30.588073    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:00:30.644273 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:00:30.644243    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-p4fsf"]
Sep 16 19:00:30.653327 ip-10-0-135-119 kubenswrapper[2578]: E0916 19:00:30.653295    2578 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 16 19:00:30.653327 ip-10-0-135-119 kubenswrapper[2578]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 19:00:30.653327 ip-10-0-135-119 kubenswrapper[2578]:         , stdout: , stderr: , exit code -1
Sep 16 19:00:30.653327 ip-10-0-135-119 kubenswrapper[2578]:  > containerID="5799da220d9d6c79f7fca98278420683d74b9cb6a4330d564e9d2fdc6f091a8c" cmd=["/bin/sleep","15"]
Sep 16 19:00:30.653607 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:00:30.653341    2578 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-p4fsf" message="" err=<
Sep 16 19:00:30.653607 ip-10-0-135-119 kubenswrapper[2578]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 19:00:30.653607 ip-10-0-135-119 kubenswrapper[2578]:         , stdout: , stderr: , exit code -1
Sep 16 19:00:30.653607 ip-10-0-135-119 kubenswrapper[2578]:  >
Sep 16 19:00:30.653607 ip-10-0-135-119 kubenswrapper[2578]: E0916 19:00:30.653353    2578 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 16 19:00:30.653607 ip-10-0-135-119 kubenswrapper[2578]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 19:00:30.653607 ip-10-0-135-119 kubenswrapper[2578]:         , stdout: , stderr: , exit code -1
Sep 16 19:00:30.653607 ip-10-0-135-119 kubenswrapper[2578]:  > pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-p4fsf" podUID="b699c7f4-1a33-4af0-a38d-d8085854563a" containerName="main" containerID="cri-o://5799da220d9d6c79f7fca98278420683d74b9cb6a4330d564e9d2fdc6f091a8c"
Sep 16 19:00:30.653607 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:00:30.653400    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-p4fsf" podUID="b699c7f4-1a33-4af0-a38d-d8085854563a" containerName="main" containerID="cri-o://5799da220d9d6c79f7fca98278420683d74b9cb6a4330d564e9d2fdc6f091a8c" gracePeriod=60
Sep 16 19:00:30.690417 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:00:30.690389    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:00:30.780787 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:00:30.780761    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:00:31.011481 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:00:31.011456    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-p4fsf"
Sep 16 19:00:31.114954 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:00:31.114915    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b699c7f4-1a33-4af0-a38d-d8085854563a-model-cache\") pod \"b699c7f4-1a33-4af0-a38d-d8085854563a\" (UID: \"b699c7f4-1a33-4af0-a38d-d8085854563a\") "
Sep 16 19:00:31.115155 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:00:31.115020    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b699c7f4-1a33-4af0-a38d-d8085854563a-dshm\") pod \"b699c7f4-1a33-4af0-a38d-d8085854563a\" (UID: \"b699c7f4-1a33-4af0-a38d-d8085854563a\") "
Sep 16 19:00:31.115155 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:00:31.115090    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m87qn\" (UniqueName: \"kubernetes.io/projected/b699c7f4-1a33-4af0-a38d-d8085854563a-kube-api-access-m87qn\") pod \"b699c7f4-1a33-4af0-a38d-d8085854563a\" (UID: \"b699c7f4-1a33-4af0-a38d-d8085854563a\") "
Sep 16 19:00:31.115155 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:00:31.115131    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b699c7f4-1a33-4af0-a38d-d8085854563a-tls-certs\") pod \"b699c7f4-1a33-4af0-a38d-d8085854563a\" (UID: \"b699c7f4-1a33-4af0-a38d-d8085854563a\") "
Sep 16 19:00:31.115332 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:00:31.115216    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b699c7f4-1a33-4af0-a38d-d8085854563a-model-cache" (OuterVolumeSpecName: "model-cache") pod "b699c7f4-1a33-4af0-a38d-d8085854563a" (UID: "b699c7f4-1a33-4af0-a38d-d8085854563a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:00:31.115332 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:00:31.115254    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b699c7f4-1a33-4af0-a38d-d8085854563a-home\") pod \"b699c7f4-1a33-4af0-a38d-d8085854563a\" (UID: \"b699c7f4-1a33-4af0-a38d-d8085854563a\") "
Sep 16 19:00:31.115332 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:00:31.115283    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b699c7f4-1a33-4af0-a38d-d8085854563a-tmp-dir\") pod \"b699c7f4-1a33-4af0-a38d-d8085854563a\" (UID: \"b699c7f4-1a33-4af0-a38d-d8085854563a\") "
Sep 16 19:00:31.115502 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:00:31.115474    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b699c7f4-1a33-4af0-a38d-d8085854563a-home" (OuterVolumeSpecName: "home") pod "b699c7f4-1a33-4af0-a38d-d8085854563a" (UID: "b699c7f4-1a33-4af0-a38d-d8085854563a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:00:31.115615 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:00:31.115584    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b699c7f4-1a33-4af0-a38d-d8085854563a-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "b699c7f4-1a33-4af0-a38d-d8085854563a" (UID: "b699c7f4-1a33-4af0-a38d-d8085854563a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:00:31.115615 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:00:31.115609    2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b699c7f4-1a33-4af0-a38d-d8085854563a-home\") on node \"ip-10-0-135-119.ec2.internal\" DevicePath \"\""
Sep 16 19:00:31.115724 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:00:31.115629    2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b699c7f4-1a33-4af0-a38d-d8085854563a-model-cache\") on node \"ip-10-0-135-119.ec2.internal\" DevicePath \"\""
Sep 16 19:00:31.117600 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:00:31.117578    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b699c7f4-1a33-4af0-a38d-d8085854563a-dshm" (OuterVolumeSpecName: "dshm") pod "b699c7f4-1a33-4af0-a38d-d8085854563a" (UID: "b699c7f4-1a33-4af0-a38d-d8085854563a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:00:31.117697 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:00:31.117594    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b699c7f4-1a33-4af0-a38d-d8085854563a-kube-api-access-m87qn" (OuterVolumeSpecName: "kube-api-access-m87qn") pod "b699c7f4-1a33-4af0-a38d-d8085854563a" (UID: "b699c7f4-1a33-4af0-a38d-d8085854563a"). InnerVolumeSpecName "kube-api-access-m87qn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:00:31.117697 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:00:31.117607    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b699c7f4-1a33-4af0-a38d-d8085854563a-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "b699c7f4-1a33-4af0-a38d-d8085854563a" (UID: "b699c7f4-1a33-4af0-a38d-d8085854563a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:00:31.126266 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:00:31.126245    2578 generic.go:356] "Generic (PLEG): container finished" podID="b699c7f4-1a33-4af0-a38d-d8085854563a" containerID="5799da220d9d6c79f7fca98278420683d74b9cb6a4330d564e9d2fdc6f091a8c" exitCode=0
Sep 16 19:00:31.126371 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:00:31.126291    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-p4fsf"
Sep 16 19:00:31.126371 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:00:31.126332    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-p4fsf" event={"ID":"b699c7f4-1a33-4af0-a38d-d8085854563a","Type":"ContainerDied","Data":"5799da220d9d6c79f7fca98278420683d74b9cb6a4330d564e9d2fdc6f091a8c"}
Sep 16 19:00:31.126488 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:00:31.126396    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-p4fsf" event={"ID":"b699c7f4-1a33-4af0-a38d-d8085854563a","Type":"ContainerDied","Data":"2e4ad687901538c6d36ce8c313d095f50cb44248a7e330b30263d7de5c851b99"}
Sep 16 19:00:31.126488 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:00:31.126432    2578 scope.go:117] "RemoveContainer" containerID="5799da220d9d6c79f7fca98278420683d74b9cb6a4330d564e9d2fdc6f091a8c"
Sep 16 19:00:31.137238 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:00:31.137220    2578 scope.go:117] "RemoveContainer" containerID="5799da220d9d6c79f7fca98278420683d74b9cb6a4330d564e9d2fdc6f091a8c"
Sep 16 19:00:31.137507 ip-10-0-135-119 kubenswrapper[2578]: E0916 19:00:31.137490    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5799da220d9d6c79f7fca98278420683d74b9cb6a4330d564e9d2fdc6f091a8c\": container with ID starting with 5799da220d9d6c79f7fca98278420683d74b9cb6a4330d564e9d2fdc6f091a8c not found: ID does not exist" containerID="5799da220d9d6c79f7fca98278420683d74b9cb6a4330d564e9d2fdc6f091a8c"
Sep 16 19:00:31.137563 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:00:31.137512    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5799da220d9d6c79f7fca98278420683d74b9cb6a4330d564e9d2fdc6f091a8c"} err="failed to get container status \"5799da220d9d6c79f7fca98278420683d74b9cb6a4330d564e9d2fdc6f091a8c\": rpc error: code = NotFound desc = could not find container \"5799da220d9d6c79f7fca98278420683d74b9cb6a4330d564e9d2fdc6f091a8c\": container with ID starting with 5799da220d9d6c79f7fca98278420683d74b9cb6a4330d564e9d2fdc6f091a8c not found: ID does not exist"
Sep 16 19:00:31.156685 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:00:31.156657    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-p4fsf"]
Sep 16 19:00:31.164723 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:00:31.164703    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-p4fsf"]
Sep 16 19:00:31.217002 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:00:31.216965    2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b699c7f4-1a33-4af0-a38d-d8085854563a-dshm\") on node \"ip-10-0-135-119.ec2.internal\" DevicePath \"\""
Sep 16 19:00:31.217002 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:00:31.217001    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m87qn\" (UniqueName: \"kubernetes.io/projected/b699c7f4-1a33-4af0-a38d-d8085854563a-kube-api-access-m87qn\") on node \"ip-10-0-135-119.ec2.internal\" DevicePath \"\""
Sep 16 19:00:31.217136 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:00:31.217010    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b699c7f4-1a33-4af0-a38d-d8085854563a-tls-certs\") on node \"ip-10-0-135-119.ec2.internal\" DevicePath \"\""
Sep 16 19:00:31.217136 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:00:31.217019    2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b699c7f4-1a33-4af0-a38d-d8085854563a-tmp-dir\") on node \"ip-10-0-135-119.ec2.internal\" DevicePath \"\""
Sep 16 19:00:31.434002 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:00:31.433934    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b699c7f4-1a33-4af0-a38d-d8085854563a" path="/var/lib/kubelet/pods/b699c7f4-1a33-4af0-a38d-d8085854563a/volumes"
Sep 16 19:00:31.937924 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:00:31.937880    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-p4fsf" podUID="b699c7f4-1a33-4af0-a38d-d8085854563a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.40:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Sep 16 19:00:53.089130 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:00:53.089096    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9ckth59"
Sep 16 19:00:59.798577 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:00:59.798546    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:00:59.898970 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:00:59.898930    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:01:19.822408 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:01:19.822294    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:01:20.430825 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:01:20.430781    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:01:20.529112 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:01:20.529065    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:01:26.432047 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:01:26.432009    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:01:32.452868 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:01:32.452836    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:01:32.552033 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:01:32.552002    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:01:39.752118 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:01:39.752078    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-6gmzv"]
Sep 16 19:01:39.752548 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:01:39.752530    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b699c7f4-1a33-4af0-a38d-d8085854563a" containerName="main"
Sep 16 19:01:39.752597 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:01:39.752550    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="b699c7f4-1a33-4af0-a38d-d8085854563a" containerName="main"
Sep 16 19:01:39.752639 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:01:39.752610    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="b699c7f4-1a33-4af0-a38d-d8085854563a" containerName="main"
Sep 16 19:01:39.756775 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:01:39.756754    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-6gmzv"
Sep 16 19:01:39.759587 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:01:39.759559    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-2541cd65\"/\"kube-root-ca.crt\""
Sep 16 19:01:39.759587 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:01:39.759585    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-2541cd65\"/\"openshift-service-ca.crt\""
Sep 16 19:01:39.759788 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:01:39.759595    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-2541cd65\"/\"default-dockercfg-5mr8r\""
Sep 16 19:01:39.759788 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:01:39.759561    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-2541cd65\"/\"scheduler-ha-replicas-test-kserve-self-signed-certs\""
Sep 16 19:01:39.765248 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:01:39.765169    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-6gmzv"]
Sep 16 19:01:39.790815 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:01:39.790786    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:01:39.932673 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:01:39.932636    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jk7p2\" (UniqueName: \"kubernetes.io/projected/bd046529-aaef-45b6-ad27-37e1bb1f03df-kube-api-access-jk7p2\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-6gmzv\" (UID: \"bd046529-aaef-45b6-ad27-37e1bb1f03df\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-6gmzv"
Sep 16 19:01:39.932854 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:01:39.932685    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bd046529-aaef-45b6-ad27-37e1bb1f03df-tmp-dir\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-6gmzv\" (UID: \"bd046529-aaef-45b6-ad27-37e1bb1f03df\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-6gmzv"
Sep 16 19:01:39.932854 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:01:39.932713    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bd046529-aaef-45b6-ad27-37e1bb1f03df-dshm\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-6gmzv\" (UID: \"bd046529-aaef-45b6-ad27-37e1bb1f03df\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-6gmzv"
Sep 16 19:01:39.932854 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:01:39.932785    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bd046529-aaef-45b6-ad27-37e1bb1f03df-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-6gmzv\" (UID: \"bd046529-aaef-45b6-ad27-37e1bb1f03df\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-6gmzv"
Sep 16 19:01:39.932854 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:01:39.932813    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bd046529-aaef-45b6-ad27-37e1bb1f03df-model-cache\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-6gmzv\" (UID: \"bd046529-aaef-45b6-ad27-37e1bb1f03df\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-6gmzv"
Sep 16 19:01:39.933029 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:01:39.932863    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bd046529-aaef-45b6-ad27-37e1bb1f03df-home\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-6gmzv\" (UID: \"bd046529-aaef-45b6-ad27-37e1bb1f03df\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-6gmzv"
Sep 16 19:01:40.032744 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:01:40.032663    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5p7vkl"]
Sep 16 19:01:40.034399 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:01:40.034374    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bd046529-aaef-45b6-ad27-37e1bb1f03df-home\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-6gmzv\" (UID: \"bd046529-aaef-45b6-ad27-37e1bb1f03df\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-6gmzv"
Sep 16 19:01:40.034465 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:01:40.034439    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jk7p2\" (UniqueName: \"kubernetes.io/projected/bd046529-aaef-45b6-ad27-37e1bb1f03df-kube-api-access-jk7p2\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-6gmzv\" (UID: \"bd046529-aaef-45b6-ad27-37e1bb1f03df\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-6gmzv"
Sep 16 19:01:40.034509 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:01:40.034486    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bd046529-aaef-45b6-ad27-37e1bb1f03df-tmp-dir\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-6gmzv\" (UID: \"bd046529-aaef-45b6-ad27-37e1bb1f03df\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-6gmzv"
Sep 16 19:01:40.034559 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:01:40.034530    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bd046529-aaef-45b6-ad27-37e1bb1f03df-dshm\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-6gmzv\" (UID: \"bd046529-aaef-45b6-ad27-37e1bb1f03df\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-6gmzv"
Sep 16 19:01:40.034614 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:01:40.034599    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bd046529-aaef-45b6-ad27-37e1bb1f03df-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-6gmzv\" (UID: \"bd046529-aaef-45b6-ad27-37e1bb1f03df\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-6gmzv"
Sep 16 19:01:40.034669 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:01:40.034629    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bd046529-aaef-45b6-ad27-37e1bb1f03df-model-cache\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-6gmzv\" (UID: \"bd046529-aaef-45b6-ad27-37e1bb1f03df\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-6gmzv"
Sep 16 19:01:40.034851 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:01:40.034811    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bd046529-aaef-45b6-ad27-37e1bb1f03df-home\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-6gmzv\" (UID: \"bd046529-aaef-45b6-ad27-37e1bb1f03df\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-6gmzv"
Sep 16 19:01:40.035009 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:01:40.034880    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bd046529-aaef-45b6-ad27-37e1bb1f03df-tmp-dir\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-6gmzv\" (UID: \"bd046529-aaef-45b6-ad27-37e1bb1f03df\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-6gmzv"
Sep 16 19:01:40.035070 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:01:40.035033    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bd046529-aaef-45b6-ad27-37e1bb1f03df-model-cache\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-6gmzv\" (UID: \"bd046529-aaef-45b6-ad27-37e1bb1f03df\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-6gmzv"
Sep 16 19:01:40.036223 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:01:40.036204    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5p7vkl"
Sep 16 19:01:40.037007 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:01:40.036967    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bd046529-aaef-45b6-ad27-37e1bb1f03df-dshm\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-6gmzv\" (UID: \"bd046529-aaef-45b6-ad27-37e1bb1f03df\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-6gmzv"
Sep 16 19:01:40.037403 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:01:40.037380    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bd046529-aaef-45b6-ad27-37e1bb1f03df-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-6gmzv\" (UID: \"bd046529-aaef-45b6-ad27-37e1bb1f03df\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-6gmzv"
Sep 16 19:01:40.038696 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:01:40.038674    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-2541cd65\"/\"scheduler-ha-replicas-test-epp-sa-dockercfg-7ns9g\""
Sep 16 19:01:40.050378 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:01:40.050347    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5p7vkl"]
Sep 16 19:01:40.052348 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:01:40.052323    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jk7p2\" (UniqueName: \"kubernetes.io/projected/bd046529-aaef-45b6-ad27-37e1bb1f03df-kube-api-access-jk7p2\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-6gmzv\" (UID: \"bd046529-aaef-45b6-ad27-37e1bb1f03df\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-6gmzv"
Sep 16 19:01:40.060145 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:01:40.060117    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:01:40.069008 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:01:40.068959    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-6gmzv"
Sep 16 19:01:40.136015 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:01:40.135939    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1d71f0d8-ff96-4939-8ab6-40d8a3cfa254-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5p7vkl\" (UID: \"1d71f0d8-ff96-4939-8ab6-40d8a3cfa254\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5p7vkl"
Sep 16 19:01:40.136176 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:01:40.136027    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bcsnx\" (UniqueName: \"kubernetes.io/projected/1d71f0d8-ff96-4939-8ab6-40d8a3cfa254-kube-api-access-bcsnx\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5p7vkl\" (UID: \"1d71f0d8-ff96-4939-8ab6-40d8a3cfa254\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5p7vkl"
Sep 16 19:01:40.237656 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:01:40.237616    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1d71f0d8-ff96-4939-8ab6-40d8a3cfa254-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5p7vkl\" (UID: \"1d71f0d8-ff96-4939-8ab6-40d8a3cfa254\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5p7vkl"
Sep 16 19:01:40.237656 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:01:40.237660    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bcsnx\" (UniqueName: \"kubernetes.io/projected/1d71f0d8-ff96-4939-8ab6-40d8a3cfa254-kube-api-access-bcsnx\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5p7vkl\" (UID: \"1d71f0d8-ff96-4939-8ab6-40d8a3cfa254\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5p7vkl"
Sep 16 19:01:40.240427 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:01:40.240396    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1d71f0d8-ff96-4939-8ab6-40d8a3cfa254-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5p7vkl\" (UID: \"1d71f0d8-ff96-4939-8ab6-40d8a3cfa254\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5p7vkl"
Sep 16 19:01:40.247444 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:01:40.247415    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bcsnx\" (UniqueName: \"kubernetes.io/projected/1d71f0d8-ff96-4939-8ab6-40d8a3cfa254-kube-api-access-bcsnx\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5p7vkl\" (UID: \"1d71f0d8-ff96-4939-8ab6-40d8a3cfa254\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5p7vkl"
Sep 16 19:01:40.350716 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:01:40.350625    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-6gmzv"]
Sep 16 19:01:40.353677 ip-10-0-135-119 kubenswrapper[2578]: W0916 19:01:40.353650    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbd046529_aaef_45b6_ad27_37e1bb1f03df.slice/crio-b2d34411c8738d7da085703ddda278997923fdafd38d908b16e41388f4406ac2 WatchSource:0}: Error finding container b2d34411c8738d7da085703ddda278997923fdafd38d908b16e41388f4406ac2: Status 404 returned error can't find the container with id b2d34411c8738d7da085703ddda278997923fdafd38d908b16e41388f4406ac2
Sep 16 19:01:40.366756 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:01:40.366731    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5p7vkl"
Sep 16 19:01:40.451035 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:01:40.450996    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-6gmzv" event={"ID":"bd046529-aaef-45b6-ad27-37e1bb1f03df","Type":"ContainerStarted","Data":"3e80fb1d46106064ab26f6169365f2f65680ba2c38b278ea46366747772e796d"}
Sep 16 19:01:40.451228 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:01:40.451065    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-6gmzv" event={"ID":"bd046529-aaef-45b6-ad27-37e1bb1f03df","Type":"ContainerStarted","Data":"b2d34411c8738d7da085703ddda278997923fdafd38d908b16e41388f4406ac2"}
Sep 16 19:01:40.482005 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:01:40.477756    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-6gmzv" podStartSLOduration=1.477733014 podStartE2EDuration="1.477733014s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:01:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:01:40.475116134 +0000 UTC m=+1281.541607622" watchObservedRunningTime="2026-09-16 19:01:40.477733014 +0000 UTC m=+1281.544224503"
Sep 16 19:01:40.647554 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:01:40.647526    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5p7vkl"]
Sep 16 19:01:40.649362 ip-10-0-135-119 kubenswrapper[2578]: W0916 19:01:40.649334    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1d71f0d8_ff96_4939_8ab6_40d8a3cfa254.slice/crio-074eba125a50a8bd089760afa2642332f6db01517d2c1bd9703296f6978d3cd8 WatchSource:0}: Error finding container 074eba125a50a8bd089760afa2642332f6db01517d2c1bd9703296f6978d3cd8: Status 404 returned error can't find the container with id 074eba125a50a8bd089760afa2642332f6db01517d2c1bd9703296f6978d3cd8
Sep 16 19:01:41.069589 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:01:41.069473    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-6gmzv"
Sep 16 19:01:41.457074 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:01:41.457032    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5p7vkl" event={"ID":"1d71f0d8-ff96-4939-8ab6-40d8a3cfa254","Type":"ContainerStarted","Data":"dcedc8d95a78ed4009cfa827a8a723c2b2e79a16318896034f0a2f5561a5f919"}
Sep 16 19:01:41.457268 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:01:41.457087    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5p7vkl" event={"ID":"1d71f0d8-ff96-4939-8ab6-40d8a3cfa254","Type":"ContainerStarted","Data":"074eba125a50a8bd089760afa2642332f6db01517d2c1bd9703296f6978d3cd8"}
Sep 16 19:01:41.457268 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:01:41.457225    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5p7vkl"
Sep 16 19:01:41.480162 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:01:41.480117    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5p7vkl" podStartSLOduration=2.480101979 podStartE2EDuration="2.480101979s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:01:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:01:41.477833446 +0000 UTC m=+1282.544324933" watchObservedRunningTime="2026-09-16 19:01:41.480101979 +0000 UTC m=+1282.546593466"
Sep 16 19:01:50.069472 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:01:50.069436    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-6gmzv"
Sep 16 19:01:50.081609 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:01:50.081587    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-6gmzv"
Sep 16 19:01:50.512117 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:01:50.512081    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-6gmzv"
Sep 16 19:01:58.217316 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:01:58.217221    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:02:12.463871 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:02:12.463832    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5p7vkl" podUID="1d71f0d8-ff96-4939-8ab6-40d8a3cfa254" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:02:12.540338 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:02:12.540268    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:02:12.563723 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:02:12.563696    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9ckth59"]
Sep 16 19:02:12.609854 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:02:12.609812    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:02:13.088747 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:02:13.088707    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9ckth59" podUID="41f68bae-75a8-4eee-a514-78afe30fff37" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:02:15.264794 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:02:15.264760    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-6gmzv"]
Sep 16 19:02:15.270416 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:02:15.270384    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:02:15.274722 ip-10-0-135-119 kubenswrapper[2578]: E0916 19:02:15.274690    2578 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 16 19:02:15.274722 ip-10-0-135-119 kubenswrapper[2578]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 19:02:15.274722 ip-10-0-135-119 kubenswrapper[2578]:         , stdout: , stderr: , exit code -1
Sep 16 19:02:15.274722 ip-10-0-135-119 kubenswrapper[2578]:  > containerID="3e80fb1d46106064ab26f6169365f2f65680ba2c38b278ea46366747772e796d" cmd=["/bin/sleep","15"]
Sep 16 19:02:15.275048 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:02:15.274738    2578 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-6gmzv" message="" err=<
Sep 16 19:02:15.275048 ip-10-0-135-119 kubenswrapper[2578]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 19:02:15.275048 ip-10-0-135-119 kubenswrapper[2578]:         , stdout: , stderr: , exit code -1
Sep 16 19:02:15.275048 ip-10-0-135-119 kubenswrapper[2578]:  >
Sep 16 19:02:15.275048 ip-10-0-135-119 kubenswrapper[2578]: E0916 19:02:15.274750    2578 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 16 19:02:15.275048 ip-10-0-135-119 kubenswrapper[2578]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 19:02:15.275048 ip-10-0-135-119 kubenswrapper[2578]:         , stdout: , stderr: , exit code -1
Sep 16 19:02:15.275048 ip-10-0-135-119 kubenswrapper[2578]:  > pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-6gmzv" podUID="bd046529-aaef-45b6-ad27-37e1bb1f03df" containerName="main" containerID="cri-o://3e80fb1d46106064ab26f6169365f2f65680ba2c38b278ea46366747772e796d"
Sep 16 19:02:15.275048 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:02:15.274794    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-6gmzv" podUID="bd046529-aaef-45b6-ad27-37e1bb1f03df" containerName="main" containerID="cri-o://3e80fb1d46106064ab26f6169365f2f65680ba2c38b278ea46366747772e796d" gracePeriod=60
Sep 16 19:02:15.281282 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:02:15.281252    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5p7vkl"]
Sep 16 19:02:15.283488 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:02:15.283454    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5p7vkl" podUID="1d71f0d8-ff96-4939-8ab6-40d8a3cfa254" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:02:15.356952 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:02:15.356909    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:02:15.501248 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:02:15.501198    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-6gmzv" podUID="bd046529-aaef-45b6-ad27-37e1bb1f03df" containerName="main" probeResult="failure" output="Get \"https://10.132.0.42:8000/health\": dial tcp 10.132.0.42:8000: connect: connection refused"
Sep 16 19:02:15.621028 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:02:15.621001    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-6gmzv"
Sep 16 19:02:15.622912 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:02:15.622887    2578 generic.go:356] "Generic (PLEG): container finished" podID="bd046529-aaef-45b6-ad27-37e1bb1f03df" containerID="3e80fb1d46106064ab26f6169365f2f65680ba2c38b278ea46366747772e796d" exitCode=0
Sep 16 19:02:15.623059 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:02:15.622920    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-6gmzv"
Sep 16 19:02:15.623059 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:02:15.622948    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-6gmzv" event={"ID":"bd046529-aaef-45b6-ad27-37e1bb1f03df","Type":"ContainerDied","Data":"3e80fb1d46106064ab26f6169365f2f65680ba2c38b278ea46366747772e796d"}
Sep 16 19:02:15.623059 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:02:15.623009    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-6gmzv" event={"ID":"bd046529-aaef-45b6-ad27-37e1bb1f03df","Type":"ContainerDied","Data":"b2d34411c8738d7da085703ddda278997923fdafd38d908b16e41388f4406ac2"}
Sep 16 19:02:15.623059 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:02:15.623035    2578 scope.go:117] "RemoveContainer" containerID="3e80fb1d46106064ab26f6169365f2f65680ba2c38b278ea46366747772e796d"
Sep 16 19:02:15.631362 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:02:15.631335    2578 scope.go:117] "RemoveContainer" containerID="3e80fb1d46106064ab26f6169365f2f65680ba2c38b278ea46366747772e796d"
Sep 16 19:02:15.631635 ip-10-0-135-119 kubenswrapper[2578]: E0916 19:02:15.631603    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3e80fb1d46106064ab26f6169365f2f65680ba2c38b278ea46366747772e796d\": container with ID starting with 3e80fb1d46106064ab26f6169365f2f65680ba2c38b278ea46366747772e796d not found: ID does not exist" containerID="3e80fb1d46106064ab26f6169365f2f65680ba2c38b278ea46366747772e796d"
Sep 16 19:02:15.631682 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:02:15.631648    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3e80fb1d46106064ab26f6169365f2f65680ba2c38b278ea46366747772e796d"} err="failed to get container status \"3e80fb1d46106064ab26f6169365f2f65680ba2c38b278ea46366747772e796d\": rpc error: code = NotFound desc = could not find container \"3e80fb1d46106064ab26f6169365f2f65680ba2c38b278ea46366747772e796d\": container with ID starting with 3e80fb1d46106064ab26f6169365f2f65680ba2c38b278ea46366747772e796d not found: ID does not exist"
Sep 16 19:02:15.760193 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:02:15.760150    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bd046529-aaef-45b6-ad27-37e1bb1f03df-dshm\") pod \"bd046529-aaef-45b6-ad27-37e1bb1f03df\" (UID: \"bd046529-aaef-45b6-ad27-37e1bb1f03df\") "
Sep 16 19:02:15.760377 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:02:15.760203    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bd046529-aaef-45b6-ad27-37e1bb1f03df-tmp-dir\") pod \"bd046529-aaef-45b6-ad27-37e1bb1f03df\" (UID: \"bd046529-aaef-45b6-ad27-37e1bb1f03df\") "
Sep 16 19:02:15.760377 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:02:15.760238    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bd046529-aaef-45b6-ad27-37e1bb1f03df-home\") pod \"bd046529-aaef-45b6-ad27-37e1bb1f03df\" (UID: \"bd046529-aaef-45b6-ad27-37e1bb1f03df\") "
Sep 16 19:02:15.760377 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:02:15.760279    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bd046529-aaef-45b6-ad27-37e1bb1f03df-model-cache\") pod \"bd046529-aaef-45b6-ad27-37e1bb1f03df\" (UID: \"bd046529-aaef-45b6-ad27-37e1bb1f03df\") "
Sep 16 19:02:15.760377 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:02:15.760314    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jk7p2\" (UniqueName: \"kubernetes.io/projected/bd046529-aaef-45b6-ad27-37e1bb1f03df-kube-api-access-jk7p2\") pod \"bd046529-aaef-45b6-ad27-37e1bb1f03df\" (UID: \"bd046529-aaef-45b6-ad27-37e1bb1f03df\") "
Sep 16 19:02:15.760377 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:02:15.760347    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bd046529-aaef-45b6-ad27-37e1bb1f03df-tls-certs\") pod \"bd046529-aaef-45b6-ad27-37e1bb1f03df\" (UID: \"bd046529-aaef-45b6-ad27-37e1bb1f03df\") "
Sep 16 19:02:15.760635 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:02:15.760524    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bd046529-aaef-45b6-ad27-37e1bb1f03df-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "bd046529-aaef-45b6-ad27-37e1bb1f03df" (UID: "bd046529-aaef-45b6-ad27-37e1bb1f03df"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:02:15.760635 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:02:15.760541    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bd046529-aaef-45b6-ad27-37e1bb1f03df-model-cache" (OuterVolumeSpecName: "model-cache") pod "bd046529-aaef-45b6-ad27-37e1bb1f03df" (UID: "bd046529-aaef-45b6-ad27-37e1bb1f03df"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:02:15.760635 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:02:15.760561    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bd046529-aaef-45b6-ad27-37e1bb1f03df-home" (OuterVolumeSpecName: "home") pod "bd046529-aaef-45b6-ad27-37e1bb1f03df" (UID: "bd046529-aaef-45b6-ad27-37e1bb1f03df"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:02:15.760779 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:02:15.760690    2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bd046529-aaef-45b6-ad27-37e1bb1f03df-tmp-dir\") on node \"ip-10-0-135-119.ec2.internal\" DevicePath \"\""
Sep 16 19:02:15.760779 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:02:15.760708    2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bd046529-aaef-45b6-ad27-37e1bb1f03df-home\") on node \"ip-10-0-135-119.ec2.internal\" DevicePath \"\""
Sep 16 19:02:15.760779 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:02:15.760721    2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bd046529-aaef-45b6-ad27-37e1bb1f03df-model-cache\") on node \"ip-10-0-135-119.ec2.internal\" DevicePath \"\""
Sep 16 19:02:15.762644 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:02:15.762617    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bd046529-aaef-45b6-ad27-37e1bb1f03df-dshm" (OuterVolumeSpecName: "dshm") pod "bd046529-aaef-45b6-ad27-37e1bb1f03df" (UID: "bd046529-aaef-45b6-ad27-37e1bb1f03df"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:02:15.762772 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:02:15.762643    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bd046529-aaef-45b6-ad27-37e1bb1f03df-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "bd046529-aaef-45b6-ad27-37e1bb1f03df" (UID: "bd046529-aaef-45b6-ad27-37e1bb1f03df"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:02:15.762772 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:02:15.762654    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bd046529-aaef-45b6-ad27-37e1bb1f03df-kube-api-access-jk7p2" (OuterVolumeSpecName: "kube-api-access-jk7p2") pod "bd046529-aaef-45b6-ad27-37e1bb1f03df" (UID: "bd046529-aaef-45b6-ad27-37e1bb1f03df"). InnerVolumeSpecName "kube-api-access-jk7p2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:02:15.862090 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:02:15.862007    2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bd046529-aaef-45b6-ad27-37e1bb1f03df-dshm\") on node \"ip-10-0-135-119.ec2.internal\" DevicePath \"\""
Sep 16 19:02:15.862090 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:02:15.862037    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jk7p2\" (UniqueName: \"kubernetes.io/projected/bd046529-aaef-45b6-ad27-37e1bb1f03df-kube-api-access-jk7p2\") on node \"ip-10-0-135-119.ec2.internal\" DevicePath \"\""
Sep 16 19:02:15.862090 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:02:15.862052    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bd046529-aaef-45b6-ad27-37e1bb1f03df-tls-certs\") on node \"ip-10-0-135-119.ec2.internal\" DevicePath \"\""
Sep 16 19:02:15.945494 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:02:15.945462    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-6gmzv"]
Sep 16 19:02:15.953752 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:02:15.953725    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-6gmzv"]
Sep 16 19:02:17.435873 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:02:17.435838    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bd046529-aaef-45b6-ad27-37e1bb1f03df" path="/var/lib/kubelet/pods/bd046529-aaef-45b6-ad27-37e1bb1f03df/volumes"
Sep 16 19:02:22.901677 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:02:22.901621    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:02:23.089285 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:02:23.089249    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9ckth59" podUID="41f68bae-75a8-4eee-a514-78afe30fff37" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:02:25.283785 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:02:25.283743    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5p7vkl" podUID="1d71f0d8-ff96-4939-8ab6-40d8a3cfa254" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:02:27.583582 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:02:27.583529    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9ckth59" podUID="41f68bae-75a8-4eee-a514-78afe30fff37" containerName="main" containerID="cri-o://79a37cb1e22cc682d0efdb0bb4f4c70a739434ddadc2d283b3a572f799d75d2d" gracePeriod=45
Sep 16 19:02:30.301442 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:02:30.301400    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5p7vkl" podUID="1d71f0d8-ff96-4939-8ab6-40d8a3cfa254" containerName="main" containerID="cri-o://dcedc8d95a78ed4009cfa827a8a723c2b2e79a16318896034f0a2f5561a5f919" gracePeriod=45
Sep 16 19:02:33.088599 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:02:33.088557    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9ckth59" podUID="41f68bae-75a8-4eee-a514-78afe30fff37" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:02:33.089009 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:02:33.088667    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9ckth59"
Sep 16 19:02:35.283811 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:02:35.283768    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5p7vkl" podUID="1d71f0d8-ff96-4939-8ab6-40d8a3cfa254" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:02:38.442425 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:02:38.442388    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:02:43.089227 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:02:43.089142    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9ckth59" podUID="41f68bae-75a8-4eee-a514-78afe30fff37" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:02:45.283744 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:02:45.283704    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5p7vkl" podUID="1d71f0d8-ff96-4939-8ab6-40d8a3cfa254" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:02:53.088475 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:02:53.088439    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9ckth59" podUID="41f68bae-75a8-4eee-a514-78afe30fff37" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:02:55.283567 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:02:55.283527    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5p7vkl" podUID="1d71f0d8-ff96-4939-8ab6-40d8a3cfa254" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:02:57.818539 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:02:57.818510    2578 generic.go:356] "Generic (PLEG): container finished" podID="41f68bae-75a8-4eee-a514-78afe30fff37" containerID="79a37cb1e22cc682d0efdb0bb4f4c70a739434ddadc2d283b3a572f799d75d2d" exitCode=0
Sep 16 19:02:57.818867 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:02:57.818586    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9ckth59" event={"ID":"41f68bae-75a8-4eee-a514-78afe30fff37","Type":"ContainerDied","Data":"79a37cb1e22cc682d0efdb0bb4f4c70a739434ddadc2d283b3a572f799d75d2d"}
Sep 16 19:02:57.837288 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:02:57.837258    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9ckth59"
Sep 16 19:02:57.927972 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:02:57.927936    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/41f68bae-75a8-4eee-a514-78afe30fff37-tls-certs\") pod \"41f68bae-75a8-4eee-a514-78afe30fff37\" (UID: \"41f68bae-75a8-4eee-a514-78afe30fff37\") "
Sep 16 19:02:57.928178 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:02:57.928039    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4fgdx\" (UniqueName: \"kubernetes.io/projected/41f68bae-75a8-4eee-a514-78afe30fff37-kube-api-access-4fgdx\") pod \"41f68bae-75a8-4eee-a514-78afe30fff37\" (UID: \"41f68bae-75a8-4eee-a514-78afe30fff37\") "
Sep 16 19:02:57.930241 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:02:57.930202    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/41f68bae-75a8-4eee-a514-78afe30fff37-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "41f68bae-75a8-4eee-a514-78afe30fff37" (UID: "41f68bae-75a8-4eee-a514-78afe30fff37"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:02:57.930354 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:02:57.930256    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/41f68bae-75a8-4eee-a514-78afe30fff37-kube-api-access-4fgdx" (OuterVolumeSpecName: "kube-api-access-4fgdx") pod "41f68bae-75a8-4eee-a514-78afe30fff37" (UID: "41f68bae-75a8-4eee-a514-78afe30fff37"). InnerVolumeSpecName "kube-api-access-4fgdx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:02:58.028854 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:02:58.028810    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4fgdx\" (UniqueName: \"kubernetes.io/projected/41f68bae-75a8-4eee-a514-78afe30fff37-kube-api-access-4fgdx\") on node \"ip-10-0-135-119.ec2.internal\" DevicePath \"\""
Sep 16 19:02:58.028854 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:02:58.028850    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/41f68bae-75a8-4eee-a514-78afe30fff37-tls-certs\") on node \"ip-10-0-135-119.ec2.internal\" DevicePath \"\""
Sep 16 19:02:58.824212 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:02:58.824174    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9ckth59"
Sep 16 19:02:58.824678 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:02:58.824209    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9ckth59" event={"ID":"41f68bae-75a8-4eee-a514-78afe30fff37","Type":"ContainerDied","Data":"eb6a59e491a76096ebb6862af24388050c165bbe56b18ae5d1dfc129ba693281"}
Sep 16 19:02:58.824678 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:02:58.824282    2578 scope.go:117] "RemoveContainer" containerID="79a37cb1e22cc682d0efdb0bb4f4c70a739434ddadc2d283b3a572f799d75d2d"
Sep 16 19:02:58.848935 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:02:58.848913    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9ckth59"]
Sep 16 19:02:58.853714 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:02:58.853692    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9ckth59"]
Sep 16 19:02:59.440095 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:02:59.440063    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="41f68bae-75a8-4eee-a514-78afe30fff37" path="/var/lib/kubelet/pods/41f68bae-75a8-4eee-a514-78afe30fff37/volumes"
Sep 16 19:03:00.557364 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:03:00.557335    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5p7vkl"
Sep 16 19:03:00.650619 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:03:00.650594    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1d71f0d8-ff96-4939-8ab6-40d8a3cfa254-tls-certs\") pod \"1d71f0d8-ff96-4939-8ab6-40d8a3cfa254\" (UID: \"1d71f0d8-ff96-4939-8ab6-40d8a3cfa254\") "
Sep 16 19:03:00.650820 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:03:00.650702    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bcsnx\" (UniqueName: \"kubernetes.io/projected/1d71f0d8-ff96-4939-8ab6-40d8a3cfa254-kube-api-access-bcsnx\") pod \"1d71f0d8-ff96-4939-8ab6-40d8a3cfa254\" (UID: \"1d71f0d8-ff96-4939-8ab6-40d8a3cfa254\") "
Sep 16 19:03:00.652894 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:03:00.652867    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1d71f0d8-ff96-4939-8ab6-40d8a3cfa254-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "1d71f0d8-ff96-4939-8ab6-40d8a3cfa254" (UID: "1d71f0d8-ff96-4939-8ab6-40d8a3cfa254"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:03:00.653029 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:03:00.652938    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1d71f0d8-ff96-4939-8ab6-40d8a3cfa254-kube-api-access-bcsnx" (OuterVolumeSpecName: "kube-api-access-bcsnx") pod "1d71f0d8-ff96-4939-8ab6-40d8a3cfa254" (UID: "1d71f0d8-ff96-4939-8ab6-40d8a3cfa254"). InnerVolumeSpecName "kube-api-access-bcsnx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:03:00.751658 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:03:00.751628    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bcsnx\" (UniqueName: \"kubernetes.io/projected/1d71f0d8-ff96-4939-8ab6-40d8a3cfa254-kube-api-access-bcsnx\") on node \"ip-10-0-135-119.ec2.internal\" DevicePath \"\""
Sep 16 19:03:00.751658 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:03:00.751656    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1d71f0d8-ff96-4939-8ab6-40d8a3cfa254-tls-certs\") on node \"ip-10-0-135-119.ec2.internal\" DevicePath \"\""
Sep 16 19:03:00.836680 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:03:00.836640    2578 generic.go:356] "Generic (PLEG): container finished" podID="1d71f0d8-ff96-4939-8ab6-40d8a3cfa254" containerID="dcedc8d95a78ed4009cfa827a8a723c2b2e79a16318896034f0a2f5561a5f919" exitCode=0
Sep 16 19:03:00.836816 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:03:00.836688    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5p7vkl"
Sep 16 19:03:00.836816 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:03:00.836729    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5p7vkl" event={"ID":"1d71f0d8-ff96-4939-8ab6-40d8a3cfa254","Type":"ContainerDied","Data":"dcedc8d95a78ed4009cfa827a8a723c2b2e79a16318896034f0a2f5561a5f919"}
Sep 16 19:03:00.836816 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:03:00.836785    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5p7vkl" event={"ID":"1d71f0d8-ff96-4939-8ab6-40d8a3cfa254","Type":"ContainerDied","Data":"074eba125a50a8bd089760afa2642332f6db01517d2c1bd9703296f6978d3cd8"}
Sep 16 19:03:00.836816 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:03:00.836807    2578 scope.go:117] "RemoveContainer" containerID="dcedc8d95a78ed4009cfa827a8a723c2b2e79a16318896034f0a2f5561a5f919"
Sep 16 19:03:00.846454 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:03:00.846435    2578 scope.go:117] "RemoveContainer" containerID="dcedc8d95a78ed4009cfa827a8a723c2b2e79a16318896034f0a2f5561a5f919"
Sep 16 19:03:00.846763 ip-10-0-135-119 kubenswrapper[2578]: E0916 19:03:00.846741    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dcedc8d95a78ed4009cfa827a8a723c2b2e79a16318896034f0a2f5561a5f919\": container with ID starting with dcedc8d95a78ed4009cfa827a8a723c2b2e79a16318896034f0a2f5561a5f919 not found: ID does not exist" containerID="dcedc8d95a78ed4009cfa827a8a723c2b2e79a16318896034f0a2f5561a5f919"
Sep 16 19:03:00.846853 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:03:00.846773    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dcedc8d95a78ed4009cfa827a8a723c2b2e79a16318896034f0a2f5561a5f919"} err="failed to get container status \"dcedc8d95a78ed4009cfa827a8a723c2b2e79a16318896034f0a2f5561a5f919\": rpc error: code = NotFound desc = could not find container \"dcedc8d95a78ed4009cfa827a8a723c2b2e79a16318896034f0a2f5561a5f919\": container with ID starting with dcedc8d95a78ed4009cfa827a8a723c2b2e79a16318896034f0a2f5561a5f919 not found: ID does not exist"
Sep 16 19:03:00.864218 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:03:00.864193    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5p7vkl"]
Sep 16 19:03:00.872319 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:03:00.872245    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5p7vkl"]
Sep 16 19:03:01.433905 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:03:01.433870    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1d71f0d8-ff96-4939-8ab6-40d8a3cfa254" path="/var/lib/kubelet/pods/1d71f0d8-ff96-4939-8ab6-40d8a3cfa254/volumes"
Sep 16 19:03:02.363645 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:03:02.363604    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:03:02.709035 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:03:02.708995    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:03:04.802473 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:03:04.802443    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:03:04.906349 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:03:04.906319    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:03:09.309124 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:03:09.309092    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:03:11.559318 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:03:11.559285    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:03:15.435992 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:03:15.435947    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:03:15.502578 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:03:15.502545    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:03:18.174013 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:03:18.173963    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:03:18.317403 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:03:18.317365    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:03:20.625386 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:03:20.625351    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:03:20.826716 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:03:20.826683    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-nltdp"]
Sep 16 19:03:20.827247 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:03:20.827231    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="41f68bae-75a8-4eee-a514-78afe30fff37" containerName="main"
Sep 16 19:03:20.827298 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:03:20.827251    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="41f68bae-75a8-4eee-a514-78afe30fff37" containerName="main"
Sep 16 19:03:20.827298 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:03:20.827264    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1d71f0d8-ff96-4939-8ab6-40d8a3cfa254" containerName="main"
Sep 16 19:03:20.827298 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:03:20.827273    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="1d71f0d8-ff96-4939-8ab6-40d8a3cfa254" containerName="main"
Sep 16 19:03:20.827397 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:03:20.827298    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bd046529-aaef-45b6-ad27-37e1bb1f03df" containerName="main"
Sep 16 19:03:20.827397 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:03:20.827307    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="bd046529-aaef-45b6-ad27-37e1bb1f03df" containerName="main"
Sep 16 19:03:20.827397 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:03:20.827383    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="1d71f0d8-ff96-4939-8ab6-40d8a3cfa254" containerName="main"
Sep 16 19:03:20.827397 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:03:20.827392    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="bd046529-aaef-45b6-ad27-37e1bb1f03df" containerName="main"
Sep 16 19:03:20.827518 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:03:20.827402    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="41f68bae-75a8-4eee-a514-78afe30fff37" containerName="main"
Sep 16 19:03:20.831939 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:03:20.831919    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-nltdp"
Sep 16 19:03:20.834084 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:03:20.834060    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-disabled-no-token-required-396e4821\"/\"openshift-service-ca.crt\""
Sep 16 19:03:20.834631 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:03:20.834604    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-disabled-no-token-required-396e4821\"/\"kube-root-ca.crt\""
Sep 16 19:03:20.834773 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:03:20.834730    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-disabled-no-token-required-396e4821\"/\"default-dockercfg-knkkj\""
Sep 16 19:03:20.834773 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:03:20.834610    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-disabled-no-token-required-396e4821\"/\"auth-disabled-test-kserve-self-signed-certs\""
Sep 16 19:03:20.834908 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:03:20.834793    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-disabled-no-token-required-396e4821\"/\"auth-disabled-test-epp-sa-dockercfg-zcvml\""
Sep 16 19:03:20.839756 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:03:20.839739    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-nltdp"]
Sep 16 19:03:20.859132 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:03:20.859114    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:03:21.028555 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:03:21.028514    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4czq5\" (UniqueName: \"kubernetes.io/projected/952a0911-221a-4a73-9181-231030737ed5-kube-api-access-4czq5\") pod \"auth-disabled-test-kserve-router-scheduler-596d9bb85f-nltdp\" (UID: \"952a0911-221a-4a73-9181-231030737ed5\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-nltdp"
Sep 16 19:03:21.028736 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:03:21.028568    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/952a0911-221a-4a73-9181-231030737ed5-tls-certs\") pod \"auth-disabled-test-kserve-router-scheduler-596d9bb85f-nltdp\" (UID: \"952a0911-221a-4a73-9181-231030737ed5\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-nltdp"
Sep 16 19:03:21.130073 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:03:21.130036    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4czq5\" (UniqueName: \"kubernetes.io/projected/952a0911-221a-4a73-9181-231030737ed5-kube-api-access-4czq5\") pod \"auth-disabled-test-kserve-router-scheduler-596d9bb85f-nltdp\" (UID: \"952a0911-221a-4a73-9181-231030737ed5\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-nltdp"
Sep 16 19:03:21.130250 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:03:21.130083    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/952a0911-221a-4a73-9181-231030737ed5-tls-certs\") pod \"auth-disabled-test-kserve-router-scheduler-596d9bb85f-nltdp\" (UID: \"952a0911-221a-4a73-9181-231030737ed5\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-nltdp"
Sep 16 19:03:21.133035 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:03:21.133006    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/952a0911-221a-4a73-9181-231030737ed5-tls-certs\") pod \"auth-disabled-test-kserve-router-scheduler-596d9bb85f-nltdp\" (UID: \"952a0911-221a-4a73-9181-231030737ed5\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-nltdp"
Sep 16 19:03:21.139413 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:03:21.139380    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4czq5\" (UniqueName: \"kubernetes.io/projected/952a0911-221a-4a73-9181-231030737ed5-kube-api-access-4czq5\") pod \"auth-disabled-test-kserve-router-scheduler-596d9bb85f-nltdp\" (UID: \"952a0911-221a-4a73-9181-231030737ed5\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-nltdp"
Sep 16 19:03:21.143041 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:03:21.143018    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-nltdp"
Sep 16 19:03:21.425106 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:03:21.425077    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-nltdp"]
Sep 16 19:03:21.428067 ip-10-0-135-119 kubenswrapper[2578]: W0916 19:03:21.428034    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod952a0911_221a_4a73_9181_231030737ed5.slice/crio-4a6f6993d7260a878a6dbb464ada835808e0d8ff017b1d8f77930dd091ca5d16 WatchSource:0}: Error finding container 4a6f6993d7260a878a6dbb464ada835808e0d8ff017b1d8f77930dd091ca5d16: Status 404 returned error can't find the container with id 4a6f6993d7260a878a6dbb464ada835808e0d8ff017b1d8f77930dd091ca5d16
Sep 16 19:03:21.935288 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:03:21.935257    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-nltdp" event={"ID":"952a0911-221a-4a73-9181-231030737ed5","Type":"ContainerStarted","Data":"885e9f09a926948c1cb00c2f0cb24ccef62701c2e621857a6bf3440dd955ca46"}
Sep 16 19:03:21.935682 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:03:21.935311    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-nltdp" event={"ID":"952a0911-221a-4a73-9181-231030737ed5","Type":"ContainerStarted","Data":"4a6f6993d7260a878a6dbb464ada835808e0d8ff017b1d8f77930dd091ca5d16"}
Sep 16 19:03:21.935682 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:03:21.935392    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-nltdp"
Sep 16 19:03:21.953993 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:03:21.953939    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-nltdp" podStartSLOduration=1.953924824 podStartE2EDuration="1.953924824s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:03:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:03:21.952548176 +0000 UTC m=+1383.019039679" watchObservedRunningTime="2026-09-16 19:03:21.953924824 +0000 UTC m=+1383.020416302"
Sep 16 19:03:34.078259 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:03:34.078227    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:03:52.943448 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:03:52.943416    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-nltdp"
Sep 16 19:03:53.693599 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:03:53.693540    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:03:53.952460 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:03:53.952367    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:04:16.601374 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:04:16.601333    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:04:17.973504 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:04:17.973471    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:04:27.640403 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:04:27.640357    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:04:27.711487 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:04:27.711456    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:04:35.824814 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:04:35.824774    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:04:46.130797 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:04:46.130758    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:05:15.998098 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:05:15.998049    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:05:15.998098 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:05:15.998080    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-nltdp"]
Sep 16 19:05:16.102709 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:05:16.102677    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:05:17.773035 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:05:17.772996    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:05:17.891083 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:05:17.891054    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:05:19.443647 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:05:19.443597    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-tmrkx_4c2078bb-281a-4aa1-8e38-238897865434/console-operator/2.log"
Sep 16 19:05:19.444259 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:05:19.444243    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-tmrkx_4c2078bb-281a-4aa1-8e38-238897865434/console-operator/2.log"
Sep 16 19:05:19.451400 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:05:19.451374    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-k6gvs_6012728d-aee6-4f93-ab6e-fad616ef36e8/ovn-acl-logging/0.log"
Sep 16 19:05:19.452202 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:05:19.452181    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-k6gvs_6012728d-aee6-4f93-ab6e-fad616ef36e8/ovn-acl-logging/0.log"
Sep 16 19:05:22.943063 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:05:22.943022    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-nltdp" podUID="952a0911-221a-4a73-9181-231030737ed5" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:05:24.802821 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:05:24.802784    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:05:28.491777 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:05:28.491738    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:05:28.787490 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:05:28.787419    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:05:31.017633 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:05:31.017592    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-nltdp" podUID="952a0911-221a-4a73-9181-231030737ed5" containerName="main" containerID="cri-o://885e9f09a926948c1cb00c2f0cb24ccef62701c2e621857a6bf3440dd955ca46" gracePeriod=45
Sep 16 19:05:31.417745 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:05:31.417711    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:05:31.521840 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:05:31.521810    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:05:32.943014 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:05:32.942963    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-nltdp" podUID="952a0911-221a-4a73-9181-231030737ed5" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:05:38.057676 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:05:38.057640    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:05:42.943232 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:05:42.943151    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-nltdp" podUID="952a0911-221a-4a73-9181-231030737ed5" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:05:42.943606 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:05:42.943268    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-nltdp"
Sep 16 19:05:52.943373 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:05:52.943333    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-nltdp" podUID="952a0911-221a-4a73-9181-231030737ed5" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:06:01.260873 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:06:01.260837    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-nltdp"
Sep 16 19:06:01.308063 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:06:01.307963    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4czq5\" (UniqueName: \"kubernetes.io/projected/952a0911-221a-4a73-9181-231030737ed5-kube-api-access-4czq5\") pod \"952a0911-221a-4a73-9181-231030737ed5\" (UID: \"952a0911-221a-4a73-9181-231030737ed5\") "
Sep 16 19:06:01.308063 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:06:01.308042    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/952a0911-221a-4a73-9181-231030737ed5-tls-certs\") pod \"952a0911-221a-4a73-9181-231030737ed5\" (UID: \"952a0911-221a-4a73-9181-231030737ed5\") "
Sep 16 19:06:01.310344 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:06:01.310313    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/952a0911-221a-4a73-9181-231030737ed5-kube-api-access-4czq5" (OuterVolumeSpecName: "kube-api-access-4czq5") pod "952a0911-221a-4a73-9181-231030737ed5" (UID: "952a0911-221a-4a73-9181-231030737ed5"). InnerVolumeSpecName "kube-api-access-4czq5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:06:01.310447 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:06:01.310352    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/952a0911-221a-4a73-9181-231030737ed5-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "952a0911-221a-4a73-9181-231030737ed5" (UID: "952a0911-221a-4a73-9181-231030737ed5"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:06:01.409728 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:06:01.409687    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4czq5\" (UniqueName: \"kubernetes.io/projected/952a0911-221a-4a73-9181-231030737ed5-kube-api-access-4czq5\") on node \"ip-10-0-135-119.ec2.internal\" DevicePath \"\""
Sep 16 19:06:01.409728 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:06:01.409720    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/952a0911-221a-4a73-9181-231030737ed5-tls-certs\") on node \"ip-10-0-135-119.ec2.internal\" DevicePath \"\""
Sep 16 19:06:01.671409 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:06:01.671379    2578 generic.go:356] "Generic (PLEG): container finished" podID="952a0911-221a-4a73-9181-231030737ed5" containerID="885e9f09a926948c1cb00c2f0cb24ccef62701c2e621857a6bf3440dd955ca46" exitCode=0
Sep 16 19:06:01.671577 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:06:01.671423    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-nltdp"
Sep 16 19:06:01.671577 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:06:01.671462    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-nltdp" event={"ID":"952a0911-221a-4a73-9181-231030737ed5","Type":"ContainerDied","Data":"885e9f09a926948c1cb00c2f0cb24ccef62701c2e621857a6bf3440dd955ca46"}
Sep 16 19:06:01.671577 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:06:01.671518    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-nltdp" event={"ID":"952a0911-221a-4a73-9181-231030737ed5","Type":"ContainerDied","Data":"4a6f6993d7260a878a6dbb464ada835808e0d8ff017b1d8f77930dd091ca5d16"}
Sep 16 19:06:01.671577 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:06:01.671539    2578 scope.go:117] "RemoveContainer" containerID="885e9f09a926948c1cb00c2f0cb24ccef62701c2e621857a6bf3440dd955ca46"
Sep 16 19:06:01.680277 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:06:01.680220    2578 scope.go:117] "RemoveContainer" containerID="885e9f09a926948c1cb00c2f0cb24ccef62701c2e621857a6bf3440dd955ca46"
Sep 16 19:06:01.680535 ip-10-0-135-119 kubenswrapper[2578]: E0916 19:06:01.680508    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"885e9f09a926948c1cb00c2f0cb24ccef62701c2e621857a6bf3440dd955ca46\": container with ID starting with 885e9f09a926948c1cb00c2f0cb24ccef62701c2e621857a6bf3440dd955ca46 not found: ID does not exist" containerID="885e9f09a926948c1cb00c2f0cb24ccef62701c2e621857a6bf3440dd955ca46"
Sep 16 19:06:01.680578 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:06:01.680546    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"885e9f09a926948c1cb00c2f0cb24ccef62701c2e621857a6bf3440dd955ca46"} err="failed to get container status \"885e9f09a926948c1cb00c2f0cb24ccef62701c2e621857a6bf3440dd955ca46\": rpc error: code = NotFound desc = could not find container \"885e9f09a926948c1cb00c2f0cb24ccef62701c2e621857a6bf3440dd955ca46\": container with ID starting with 885e9f09a926948c1cb00c2f0cb24ccef62701c2e621857a6bf3440dd955ca46 not found: ID does not exist"
Sep 16 19:06:01.692472 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:06:01.692443    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-nltdp"]
Sep 16 19:06:01.696877 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:06:01.696856    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-nltdp"]
Sep 16 19:06:03.434688 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:06:03.434634    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="952a0911-221a-4a73-9181-231030737ed5" path="/var/lib/kubelet/pods/952a0911-221a-4a73-9181-231030737ed5/volumes"
Sep 16 19:06:05.974921 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:06:05.974883    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:06:06.072198 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:06:06.072165    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:06:06.175229 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:06:06.175200    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:06:12.563477 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:06:12.563437    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:06:19.073543 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:06:19.073500    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:06:19.153882 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:06:19.153851    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:06:23.381801 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:06:23.381768    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-s78x4"]
Sep 16 19:06:23.382215 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:06:23.382200    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="952a0911-221a-4a73-9181-231030737ed5" containerName="main"
Sep 16 19:06:23.382215 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:06:23.382216    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="952a0911-221a-4a73-9181-231030737ed5" containerName="main"
Sep 16 19:06:23.382315 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:06:23.382278    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="952a0911-221a-4a73-9181-231030737ed5" containerName="main"
Sep 16 19:06:23.385594 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:06:23.385575    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-s78x4"
Sep 16 19:06:23.387619 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:06:23.387600    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-canary-service-backend-7b63470c\"/\"kube-root-ca.crt\""
Sep 16 19:06:23.388233 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:06:23.388214    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-canary-service-backend-7b63470c\"/\"default-dockercfg-td5gh\""
Sep 16 19:06:23.388341 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:06:23.388234    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-canary-service-backend-7b63470c\"/\"canary-v1-kserve-self-signed-certs\""
Sep 16 19:06:23.388341 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:06:23.388241    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-canary-service-backend-7b63470c\"/\"openshift-service-ca.crt\""
Sep 16 19:06:23.400842 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:06:23.400820    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-s78x4"]
Sep 16 19:06:23.504768 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:06:23.504732    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/095c76bd-faf0-4db5-ba79-3b3a8dfebc7b-tmp-dir\") pod \"canary-v1-kserve-65d58ccc9-s78x4\" (UID: \"095c76bd-faf0-4db5-ba79-3b3a8dfebc7b\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-s78x4"
Sep 16 19:06:23.504948 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:06:23.504806    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/095c76bd-faf0-4db5-ba79-3b3a8dfebc7b-dshm\") pod \"canary-v1-kserve-65d58ccc9-s78x4\" (UID: \"095c76bd-faf0-4db5-ba79-3b3a8dfebc7b\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-s78x4"
Sep 16 19:06:23.504948 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:06:23.504833    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/095c76bd-faf0-4db5-ba79-3b3a8dfebc7b-tls-certs\") pod \"canary-v1-kserve-65d58ccc9-s78x4\" (UID: \"095c76bd-faf0-4db5-ba79-3b3a8dfebc7b\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-s78x4"
Sep 16 19:06:23.504948 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:06:23.504863    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/095c76bd-faf0-4db5-ba79-3b3a8dfebc7b-home\") pod \"canary-v1-kserve-65d58ccc9-s78x4\" (UID: \"095c76bd-faf0-4db5-ba79-3b3a8dfebc7b\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-s78x4"
Sep 16 19:06:23.505115 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:06:23.504946    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/095c76bd-faf0-4db5-ba79-3b3a8dfebc7b-model-cache\") pod \"canary-v1-kserve-65d58ccc9-s78x4\" (UID: \"095c76bd-faf0-4db5-ba79-3b3a8dfebc7b\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-s78x4"
Sep 16 19:06:23.505115 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:06:23.505054    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qxvw6\" (UniqueName: \"kubernetes.io/projected/095c76bd-faf0-4db5-ba79-3b3a8dfebc7b-kube-api-access-qxvw6\") pod \"canary-v1-kserve-65d58ccc9-s78x4\" (UID: \"095c76bd-faf0-4db5-ba79-3b3a8dfebc7b\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-s78x4"
Sep 16 19:06:23.606203 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:06:23.606173    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/095c76bd-faf0-4db5-ba79-3b3a8dfebc7b-tmp-dir\") pod \"canary-v1-kserve-65d58ccc9-s78x4\" (UID: \"095c76bd-faf0-4db5-ba79-3b3a8dfebc7b\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-s78x4"
Sep 16 19:06:23.606367 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:06:23.606214    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/095c76bd-faf0-4db5-ba79-3b3a8dfebc7b-dshm\") pod \"canary-v1-kserve-65d58ccc9-s78x4\" (UID: \"095c76bd-faf0-4db5-ba79-3b3a8dfebc7b\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-s78x4"
Sep 16 19:06:23.606367 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:06:23.606233    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/095c76bd-faf0-4db5-ba79-3b3a8dfebc7b-tls-certs\") pod \"canary-v1-kserve-65d58ccc9-s78x4\" (UID: \"095c76bd-faf0-4db5-ba79-3b3a8dfebc7b\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-s78x4"
Sep 16 19:06:23.606367 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:06:23.606251    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/095c76bd-faf0-4db5-ba79-3b3a8dfebc7b-home\") pod \"canary-v1-kserve-65d58ccc9-s78x4\" (UID: \"095c76bd-faf0-4db5-ba79-3b3a8dfebc7b\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-s78x4"
Sep 16 19:06:23.606506 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:06:23.606371    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/095c76bd-faf0-4db5-ba79-3b3a8dfebc7b-model-cache\") pod \"canary-v1-kserve-65d58ccc9-s78x4\" (UID: \"095c76bd-faf0-4db5-ba79-3b3a8dfebc7b\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-s78x4"
Sep 16 19:06:23.606506 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:06:23.606418    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qxvw6\" (UniqueName: \"kubernetes.io/projected/095c76bd-faf0-4db5-ba79-3b3a8dfebc7b-kube-api-access-qxvw6\") pod \"canary-v1-kserve-65d58ccc9-s78x4\" (UID: \"095c76bd-faf0-4db5-ba79-3b3a8dfebc7b\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-s78x4"
Sep 16 19:06:23.606635 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:06:23.606607    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/095c76bd-faf0-4db5-ba79-3b3a8dfebc7b-tmp-dir\") pod \"canary-v1-kserve-65d58ccc9-s78x4\" (UID: \"095c76bd-faf0-4db5-ba79-3b3a8dfebc7b\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-s78x4"
Sep 16 19:06:23.606711 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:06:23.606641    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/095c76bd-faf0-4db5-ba79-3b3a8dfebc7b-home\") pod \"canary-v1-kserve-65d58ccc9-s78x4\" (UID: \"095c76bd-faf0-4db5-ba79-3b3a8dfebc7b\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-s78x4"
Sep 16 19:06:23.606760 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:06:23.606737    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/095c76bd-faf0-4db5-ba79-3b3a8dfebc7b-model-cache\") pod \"canary-v1-kserve-65d58ccc9-s78x4\" (UID: \"095c76bd-faf0-4db5-ba79-3b3a8dfebc7b\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-s78x4"
Sep 16 19:06:23.608599 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:06:23.608578    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/095c76bd-faf0-4db5-ba79-3b3a8dfebc7b-dshm\") pod \"canary-v1-kserve-65d58ccc9-s78x4\" (UID: \"095c76bd-faf0-4db5-ba79-3b3a8dfebc7b\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-s78x4"
Sep 16 19:06:23.608922 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:06:23.608903    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/095c76bd-faf0-4db5-ba79-3b3a8dfebc7b-tls-certs\") pod \"canary-v1-kserve-65d58ccc9-s78x4\" (UID: \"095c76bd-faf0-4db5-ba79-3b3a8dfebc7b\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-s78x4"
Sep 16 19:06:23.622352 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:06:23.622328    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qxvw6\" (UniqueName: \"kubernetes.io/projected/095c76bd-faf0-4db5-ba79-3b3a8dfebc7b-kube-api-access-qxvw6\") pod \"canary-v1-kserve-65d58ccc9-s78x4\" (UID: \"095c76bd-faf0-4db5-ba79-3b3a8dfebc7b\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-s78x4"
Sep 16 19:06:23.696470 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:06:23.696395    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-s78x4"
Sep 16 19:06:23.990303 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:06:23.990274    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-s78x4"]
Sep 16 19:06:23.992358 ip-10-0-135-119 kubenswrapper[2578]: W0916 19:06:23.992330    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod095c76bd_faf0_4db5_ba79_3b3a8dfebc7b.slice/crio-091f5efea3b0dfeda0e1f09a3e66e918d8ad026c056c5abe308922dffe368936 WatchSource:0}: Error finding container 091f5efea3b0dfeda0e1f09a3e66e918d8ad026c056c5abe308922dffe368936: Status 404 returned error can't find the container with id 091f5efea3b0dfeda0e1f09a3e66e918d8ad026c056c5abe308922dffe368936
Sep 16 19:06:23.994142 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:06:23.994127    2578 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 16 19:06:24.339004 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:06:24.338901    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:06:24.777438 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:06:24.777399    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-s78x4" event={"ID":"095c76bd-faf0-4db5-ba79-3b3a8dfebc7b","Type":"ContainerStarted","Data":"091f5efea3b0dfeda0e1f09a3e66e918d8ad026c056c5abe308922dffe368936"}
Sep 16 19:06:25.783386 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:06:25.783345    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-s78x4" event={"ID":"095c76bd-faf0-4db5-ba79-3b3a8dfebc7b","Type":"ContainerStarted","Data":"086495eea1df6f69de76f6ff89b9757e1ab083bd198ee28a9ffe119e345a68f1"}
Sep 16 19:06:25.803808 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:06:25.803755    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-s78x4" podStartSLOduration=1.786221923 podStartE2EDuration="2.803740932s" totalImagesPullingTime="1.017519009s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:06:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:06:25.802172475 +0000 UTC m=+1566.868663985" watchObservedRunningTime="2026-09-16 19:06:25.803740932 +0000 UTC m=+1566.870232420"
Sep 16 19:06:26.697142 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:06:26.697107    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-s78x4"
Sep 16 19:06:33.696834 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:06:33.696802    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-s78x4"
Sep 16 19:06:33.709204 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:06:33.709181    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-s78x4"
Sep 16 19:06:33.831683 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:06:33.831657    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-s78x4"
Sep 16 19:06:50.250061 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:06:50.250030    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-m9wwp"]
Sep 16 19:06:50.253545 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:06:50.253526    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-m9wwp"
Sep 16 19:06:50.255641 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:06:50.255618    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-canary-service-backend-7b63470c\"/\"canary-v2-kserve-self-signed-certs\""
Sep 16 19:06:50.265028 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:06:50.264969    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-m9wwp"]
Sep 16 19:06:50.294363 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:06:50.294334    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:06:50.335334 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:06:50.335293    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f3ca627c-2998-4500-bca2-c983460aff65-model-cache\") pod \"canary-v2-kserve-7f65dbbccc-m9wwp\" (UID: \"f3ca627c-2998-4500-bca2-c983460aff65\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-m9wwp"
Sep 16 19:06:50.335334 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:06:50.335330    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f3ca627c-2998-4500-bca2-c983460aff65-tmp-dir\") pod \"canary-v2-kserve-7f65dbbccc-m9wwp\" (UID: \"f3ca627c-2998-4500-bca2-c983460aff65\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-m9wwp"
Sep 16 19:06:50.335521 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:06:50.335353    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f3ca627c-2998-4500-bca2-c983460aff65-dshm\") pod \"canary-v2-kserve-7f65dbbccc-m9wwp\" (UID: \"f3ca627c-2998-4500-bca2-c983460aff65\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-m9wwp"
Sep 16 19:06:50.335521 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:06:50.335458    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f3ca627c-2998-4500-bca2-c983460aff65-tls-certs\") pod \"canary-v2-kserve-7f65dbbccc-m9wwp\" (UID: \"f3ca627c-2998-4500-bca2-c983460aff65\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-m9wwp"
Sep 16 19:06:50.335521 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:06:50.335483    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2sgqb\" (UniqueName: \"kubernetes.io/projected/f3ca627c-2998-4500-bca2-c983460aff65-kube-api-access-2sgqb\") pod \"canary-v2-kserve-7f65dbbccc-m9wwp\" (UID: \"f3ca627c-2998-4500-bca2-c983460aff65\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-m9wwp"
Sep 16 19:06:50.335521 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:06:50.335514    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f3ca627c-2998-4500-bca2-c983460aff65-home\") pod \"canary-v2-kserve-7f65dbbccc-m9wwp\" (UID: \"f3ca627c-2998-4500-bca2-c983460aff65\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-m9wwp"
Sep 16 19:06:50.436068 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:06:50.436038    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f3ca627c-2998-4500-bca2-c983460aff65-dshm\") pod \"canary-v2-kserve-7f65dbbccc-m9wwp\" (UID: \"f3ca627c-2998-4500-bca2-c983460aff65\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-m9wwp"
Sep 16 19:06:50.436235 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:06:50.436099    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f3ca627c-2998-4500-bca2-c983460aff65-tls-certs\") pod \"canary-v2-kserve-7f65dbbccc-m9wwp\" (UID: \"f3ca627c-2998-4500-bca2-c983460aff65\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-m9wwp"
Sep 16 19:06:50.436235 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:06:50.436116    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2sgqb\" (UniqueName: \"kubernetes.io/projected/f3ca627c-2998-4500-bca2-c983460aff65-kube-api-access-2sgqb\") pod \"canary-v2-kserve-7f65dbbccc-m9wwp\" (UID: \"f3ca627c-2998-4500-bca2-c983460aff65\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-m9wwp"
Sep 16 19:06:50.436235 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:06:50.436144    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f3ca627c-2998-4500-bca2-c983460aff65-home\") pod \"canary-v2-kserve-7f65dbbccc-m9wwp\" (UID: \"f3ca627c-2998-4500-bca2-c983460aff65\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-m9wwp"
Sep 16 19:06:50.436235 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:06:50.436176    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f3ca627c-2998-4500-bca2-c983460aff65-model-cache\") pod \"canary-v2-kserve-7f65dbbccc-m9wwp\" (UID: \"f3ca627c-2998-4500-bca2-c983460aff65\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-m9wwp"
Sep 16 19:06:50.436235 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:06:50.436198    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f3ca627c-2998-4500-bca2-c983460aff65-tmp-dir\") pod \"canary-v2-kserve-7f65dbbccc-m9wwp\" (UID: \"f3ca627c-2998-4500-bca2-c983460aff65\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-m9wwp"
Sep 16 19:06:50.436546 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:06:50.436523    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f3ca627c-2998-4500-bca2-c983460aff65-home\") pod \"canary-v2-kserve-7f65dbbccc-m9wwp\" (UID: \"f3ca627c-2998-4500-bca2-c983460aff65\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-m9wwp"
Sep 16 19:06:50.436616 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:06:50.436574    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f3ca627c-2998-4500-bca2-c983460aff65-model-cache\") pod \"canary-v2-kserve-7f65dbbccc-m9wwp\" (UID: \"f3ca627c-2998-4500-bca2-c983460aff65\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-m9wwp"
Sep 16 19:06:50.436672 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:06:50.436627    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f3ca627c-2998-4500-bca2-c983460aff65-tmp-dir\") pod \"canary-v2-kserve-7f65dbbccc-m9wwp\" (UID: \"f3ca627c-2998-4500-bca2-c983460aff65\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-m9wwp"
Sep 16 19:06:50.438593 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:06:50.438568    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f3ca627c-2998-4500-bca2-c983460aff65-dshm\") pod \"canary-v2-kserve-7f65dbbccc-m9wwp\" (UID: \"f3ca627c-2998-4500-bca2-c983460aff65\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-m9wwp"
Sep 16 19:06:50.438731 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:06:50.438717    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f3ca627c-2998-4500-bca2-c983460aff65-tls-certs\") pod \"canary-v2-kserve-7f65dbbccc-m9wwp\" (UID: \"f3ca627c-2998-4500-bca2-c983460aff65\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-m9wwp"
Sep 16 19:06:50.446513 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:06:50.446489    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2sgqb\" (UniqueName: \"kubernetes.io/projected/f3ca627c-2998-4500-bca2-c983460aff65-kube-api-access-2sgqb\") pod \"canary-v2-kserve-7f65dbbccc-m9wwp\" (UID: \"f3ca627c-2998-4500-bca2-c983460aff65\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-m9wwp"
Sep 16 19:06:50.563621 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:06:50.563538    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-m9wwp"
Sep 16 19:06:50.852559 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:06:50.852535    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-m9wwp"]
Sep 16 19:06:50.855260 ip-10-0-135-119 kubenswrapper[2578]: W0916 19:06:50.855227    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf3ca627c_2998_4500_bca2_c983460aff65.slice/crio-f955e6eb14aa6baab044e799592dda308a05fe2054c67c8860ed44d139183b83 WatchSource:0}: Error finding container f955e6eb14aa6baab044e799592dda308a05fe2054c67c8860ed44d139183b83: Status 404 returned error can't find the container with id f955e6eb14aa6baab044e799592dda308a05fe2054c67c8860ed44d139183b83
Sep 16 19:06:50.901329 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:06:50.901303    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-m9wwp" event={"ID":"f3ca627c-2998-4500-bca2-c983460aff65","Type":"ContainerStarted","Data":"f955e6eb14aa6baab044e799592dda308a05fe2054c67c8860ed44d139183b83"}
Sep 16 19:06:51.908699 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:06:51.908656    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-m9wwp" event={"ID":"f3ca627c-2998-4500-bca2-c983460aff65","Type":"ContainerStarted","Data":"57f42a7da362608dc6cf4a4a2b93764283c8d95fc95313875df86dcd2167f007"}
Sep 16 19:06:51.929553 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:06:51.929497    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-m9wwp" podStartSLOduration=1.929478864 podStartE2EDuration="1.929478864s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:06:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:06:51.928969149 +0000 UTC m=+1592.995460655" watchObservedRunningTime="2026-09-16 19:06:51.929478864 +0000 UTC m=+1592.995970356"
Sep 16 19:06:52.564603 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:06:52.564570    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-m9wwp"
Sep 16 19:06:52.583669 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:06:52.583624    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:06:52.674059 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:06:52.674024    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:07:00.564338 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:07:00.564296    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-m9wwp"
Sep 16 19:07:00.576856 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:07:00.576834    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-m9wwp"
Sep 16 19:07:00.962660 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:07:00.962619    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-m9wwp"
Sep 16 19:07:39.895507 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:07:39.895474    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:07:44.168953 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:07:44.168918    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:07:45.384414 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:07:45.384373    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:07:45.579309 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:07:45.579273    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedpv89s"]
Sep 16 19:07:45.582720 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:07:45.582686    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedpv89s"
Sep 16 19:07:45.585871 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:07:45.585845    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-40b85f95\"/\"scheduler-v06-threshold-migration-test-epp-sa-dockercfg-w8bqt\""
Sep 16 19:07:45.586182 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:07:45.585845    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-40b85f95\"/\"openshift-service-ca.crt\""
Sep 16 19:07:45.586333 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:07:45.585879    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-40b85f95\"/\"kube-root-ca.crt\""
Sep 16 19:07:45.586563 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:07:45.586533    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-40b85f95\"/\"scheduler-v06-threshold-migration-test-kserve-self-signed-certs\""
Sep 16 19:07:45.587133 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:07:45.585940    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-40b85f95\"/\"default-dockercfg-67jn8\""
Sep 16 19:07:45.596055 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:07:45.596032    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedpv89s"]
Sep 16 19:07:45.621249 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:07:45.621221    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a0d41aa5-98c6-412e-b955-e629b3f01619-tls-certs\") pod \"scheduler-v06-threshold-migration-test-kserve-router-schedpv89s\" (UID: \"a0d41aa5-98c6-412e-b955-e629b3f01619\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedpv89s"
Sep 16 19:07:45.621389 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:07:45.621284    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t4rgg\" (UniqueName: \"kubernetes.io/projected/a0d41aa5-98c6-412e-b955-e629b3f01619-kube-api-access-t4rgg\") pod \"scheduler-v06-threshold-migration-test-kserve-router-schedpv89s\" (UID: \"a0d41aa5-98c6-412e-b955-e629b3f01619\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedpv89s"
Sep 16 19:07:45.625203 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:07:45.625180    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:07:45.722642 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:07:45.722568    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t4rgg\" (UniqueName: \"kubernetes.io/projected/a0d41aa5-98c6-412e-b955-e629b3f01619-kube-api-access-t4rgg\") pod \"scheduler-v06-threshold-migration-test-kserve-router-schedpv89s\" (UID: \"a0d41aa5-98c6-412e-b955-e629b3f01619\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedpv89s"
Sep 16 19:07:45.722751 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:07:45.722651    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a0d41aa5-98c6-412e-b955-e629b3f01619-tls-certs\") pod \"scheduler-v06-threshold-migration-test-kserve-router-schedpv89s\" (UID: \"a0d41aa5-98c6-412e-b955-e629b3f01619\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedpv89s"
Sep 16 19:07:45.725193 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:07:45.725175    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a0d41aa5-98c6-412e-b955-e629b3f01619-tls-certs\") pod \"scheduler-v06-threshold-migration-test-kserve-router-schedpv89s\" (UID: \"a0d41aa5-98c6-412e-b955-e629b3f01619\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedpv89s"
Sep 16 19:07:45.733056 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:07:45.733035    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t4rgg\" (UniqueName: \"kubernetes.io/projected/a0d41aa5-98c6-412e-b955-e629b3f01619-kube-api-access-t4rgg\") pod \"scheduler-v06-threshold-migration-test-kserve-router-schedpv89s\" (UID: \"a0d41aa5-98c6-412e-b955-e629b3f01619\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedpv89s"
Sep 16 19:07:45.901251 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:07:45.901216    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedpv89s"
Sep 16 19:07:46.173832 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:07:46.173801    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedpv89s"]
Sep 16 19:07:46.177244 ip-10-0-135-119 kubenswrapper[2578]: W0916 19:07:46.177218    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda0d41aa5_98c6_412e_b955_e629b3f01619.slice/crio-4c8168154b2c84c70709c2c6a67f862872496e2c1312662b1d6055bcc74410c4 WatchSource:0}: Error finding container 4c8168154b2c84c70709c2c6a67f862872496e2c1312662b1d6055bcc74410c4: Status 404 returned error can't find the container with id 4c8168154b2c84c70709c2c6a67f862872496e2c1312662b1d6055bcc74410c4
Sep 16 19:07:47.170395 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:07:47.170358    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedpv89s" event={"ID":"a0d41aa5-98c6-412e-b955-e629b3f01619","Type":"ContainerStarted","Data":"767416d09137acb889c70f676c8621a29fb40a2c4ee9d41cb129031ce1fb6829"}
Sep 16 19:07:47.170776 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:07:47.170421    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedpv89s" event={"ID":"a0d41aa5-98c6-412e-b955-e629b3f01619","Type":"ContainerStarted","Data":"4c8168154b2c84c70709c2c6a67f862872496e2c1312662b1d6055bcc74410c4"}
Sep 16 19:07:47.170776 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:07:47.170463    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedpv89s"
Sep 16 19:07:47.209192 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:07:47.209119    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedpv89s" podStartSLOduration=2.209102363 podStartE2EDuration="2.209102363s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:07:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:07:47.207833343 +0000 UTC m=+1648.274324854" watchObservedRunningTime="2026-09-16 19:07:47.209102363 +0000 UTC m=+1648.275593851"
Sep 16 19:07:47.633865 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:07:47.633832    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:07:53.803038 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:07:53.802969    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:07:53.876808 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:07:53.876774    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:08:05.326904 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:08:05.326866    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:08:18.177023 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:08:18.176961    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedpv89s"
Sep 16 19:08:23.951027 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:08:23.950940    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:08:24.064995 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:08:24.064941    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedpv89s"]
Sep 16 19:08:24.109419 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:08:24.109390    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:08:28.177603 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:08:28.177567    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedpv89s" podUID="a0d41aa5-98c6-412e-b955-e629b3f01619" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:08:28.805458 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:08:28.805416    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:08:38.176701 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:08:38.176665    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedpv89s" podUID="a0d41aa5-98c6-412e-b955-e629b3f01619" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:08:39.084600 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:08:39.084526    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedpv89s" podUID="a0d41aa5-98c6-412e-b955-e629b3f01619" containerName="main" containerID="cri-o://767416d09137acb889c70f676c8621a29fb40a2c4ee9d41cb129031ce1fb6829" gracePeriod=45
Sep 16 19:08:39.472826 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:08:39.472787    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:08:43.200125 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:08:43.200041    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:08:43.315086 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:08:43.315049    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:08:43.446701 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:08:43.446669    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:08:43.676791 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:08:43.676753    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-s78x4"]
Sep 16 19:08:43.699245 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:08:43.699210    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:08:44.395857 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:08:44.395823    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-m9wwp"]
Sep 16 19:08:44.433008 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:08:44.432955    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:08:48.176541 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:08:48.176502    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedpv89s" podUID="a0d41aa5-98c6-412e-b955-e629b3f01619" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:08:48.177007 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:08:48.176616    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedpv89s"
Sep 16 19:08:48.274238 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:08:48.274206    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:08:56.092530 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:08:56.092497    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:08:58.176730 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:08:58.176691    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedpv89s" podUID="a0d41aa5-98c6-412e-b955-e629b3f01619" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:08:58.697014 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:08:58.696939    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-s78x4" podUID="095c76bd-faf0-4db5-ba79-3b3a8dfebc7b" containerName="main" containerID="cri-o://086495eea1df6f69de76f6ff89b9757e1ab083bd198ee28a9ffe119e345a68f1" gracePeriod=45
Sep 16 19:08:58.821580 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:08:58.821525    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-s78x4" podUID="095c76bd-faf0-4db5-ba79-3b3a8dfebc7b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.45:8000/health\": dial tcp 10.132.0.45:8000: connect: connection refused"
Sep 16 19:08:58.946048 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:08:58.946024    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-s78x4"
Sep 16 19:08:59.081080 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:08:59.080969    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/095c76bd-faf0-4db5-ba79-3b3a8dfebc7b-model-cache\") pod \"095c76bd-faf0-4db5-ba79-3b3a8dfebc7b\" (UID: \"095c76bd-faf0-4db5-ba79-3b3a8dfebc7b\") "
Sep 16 19:08:59.081080 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:08:59.081032    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/095c76bd-faf0-4db5-ba79-3b3a8dfebc7b-home\") pod \"095c76bd-faf0-4db5-ba79-3b3a8dfebc7b\" (UID: \"095c76bd-faf0-4db5-ba79-3b3a8dfebc7b\") "
Sep 16 19:08:59.081080 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:08:59.081066    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qxvw6\" (UniqueName: \"kubernetes.io/projected/095c76bd-faf0-4db5-ba79-3b3a8dfebc7b-kube-api-access-qxvw6\") pod \"095c76bd-faf0-4db5-ba79-3b3a8dfebc7b\" (UID: \"095c76bd-faf0-4db5-ba79-3b3a8dfebc7b\") "
Sep 16 19:08:59.081367 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:08:59.081099    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/095c76bd-faf0-4db5-ba79-3b3a8dfebc7b-tls-certs\") pod \"095c76bd-faf0-4db5-ba79-3b3a8dfebc7b\" (UID: \"095c76bd-faf0-4db5-ba79-3b3a8dfebc7b\") "
Sep 16 19:08:59.081367 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:08:59.081173    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/095c76bd-faf0-4db5-ba79-3b3a8dfebc7b-dshm\") pod \"095c76bd-faf0-4db5-ba79-3b3a8dfebc7b\" (UID: \"095c76bd-faf0-4db5-ba79-3b3a8dfebc7b\") "
Sep 16 19:08:59.081367 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:08:59.081208    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/095c76bd-faf0-4db5-ba79-3b3a8dfebc7b-tmp-dir\") pod \"095c76bd-faf0-4db5-ba79-3b3a8dfebc7b\" (UID: \"095c76bd-faf0-4db5-ba79-3b3a8dfebc7b\") "
Sep 16 19:08:59.081367 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:08:59.081254    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/095c76bd-faf0-4db5-ba79-3b3a8dfebc7b-model-cache" (OuterVolumeSpecName: "model-cache") pod "095c76bd-faf0-4db5-ba79-3b3a8dfebc7b" (UID: "095c76bd-faf0-4db5-ba79-3b3a8dfebc7b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:08:59.081367 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:08:59.081268    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/095c76bd-faf0-4db5-ba79-3b3a8dfebc7b-home" (OuterVolumeSpecName: "home") pod "095c76bd-faf0-4db5-ba79-3b3a8dfebc7b" (UID: "095c76bd-faf0-4db5-ba79-3b3a8dfebc7b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:08:59.081682 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:08:59.081468    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/095c76bd-faf0-4db5-ba79-3b3a8dfebc7b-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "095c76bd-faf0-4db5-ba79-3b3a8dfebc7b" (UID: "095c76bd-faf0-4db5-ba79-3b3a8dfebc7b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:08:59.081682 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:08:59.081552    2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/095c76bd-faf0-4db5-ba79-3b3a8dfebc7b-model-cache\") on node \"ip-10-0-135-119.ec2.internal\" DevicePath \"\""
Sep 16 19:08:59.081682 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:08:59.081573    2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/095c76bd-faf0-4db5-ba79-3b3a8dfebc7b-home\") on node \"ip-10-0-135-119.ec2.internal\" DevicePath \"\""
Sep 16 19:08:59.081682 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:08:59.081600    2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/095c76bd-faf0-4db5-ba79-3b3a8dfebc7b-tmp-dir\") on node \"ip-10-0-135-119.ec2.internal\" DevicePath \"\""
Sep 16 19:08:59.083469 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:08:59.083440    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/095c76bd-faf0-4db5-ba79-3b3a8dfebc7b-dshm" (OuterVolumeSpecName: "dshm") pod "095c76bd-faf0-4db5-ba79-3b3a8dfebc7b" (UID: "095c76bd-faf0-4db5-ba79-3b3a8dfebc7b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:08:59.083586 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:08:59.083551    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/095c76bd-faf0-4db5-ba79-3b3a8dfebc7b-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "095c76bd-faf0-4db5-ba79-3b3a8dfebc7b" (UID: "095c76bd-faf0-4db5-ba79-3b3a8dfebc7b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:08:59.083688 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:08:59.083665    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/095c76bd-faf0-4db5-ba79-3b3a8dfebc7b-kube-api-access-qxvw6" (OuterVolumeSpecName: "kube-api-access-qxvw6") pod "095c76bd-faf0-4db5-ba79-3b3a8dfebc7b" (UID: "095c76bd-faf0-4db5-ba79-3b3a8dfebc7b"). InnerVolumeSpecName "kube-api-access-qxvw6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:08:59.182700 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:08:59.182670    2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/095c76bd-faf0-4db5-ba79-3b3a8dfebc7b-dshm\") on node \"ip-10-0-135-119.ec2.internal\" DevicePath \"\""
Sep 16 19:08:59.182700 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:08:59.182697    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qxvw6\" (UniqueName: \"kubernetes.io/projected/095c76bd-faf0-4db5-ba79-3b3a8dfebc7b-kube-api-access-qxvw6\") on node \"ip-10-0-135-119.ec2.internal\" DevicePath \"\""
Sep 16 19:08:59.183160 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:08:59.182708    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/095c76bd-faf0-4db5-ba79-3b3a8dfebc7b-tls-certs\") on node \"ip-10-0-135-119.ec2.internal\" DevicePath \"\""
Sep 16 19:08:59.415733 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:08:59.415690    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-m9wwp" podUID="f3ca627c-2998-4500-bca2-c983460aff65" containerName="main" containerID="cri-o://57f42a7da362608dc6cf4a4a2b93764283c8d95fc95313875df86dcd2167f007" gracePeriod=45
Sep 16 19:08:59.508870 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:08:59.508825    2578 generic.go:356] "Generic (PLEG): container finished" podID="095c76bd-faf0-4db5-ba79-3b3a8dfebc7b" containerID="086495eea1df6f69de76f6ff89b9757e1ab083bd198ee28a9ffe119e345a68f1" exitCode=0
Sep 16 19:08:59.508999 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:08:59.508873    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-s78x4"
Sep 16 19:08:59.508999 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:08:59.508903    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-s78x4" event={"ID":"095c76bd-faf0-4db5-ba79-3b3a8dfebc7b","Type":"ContainerDied","Data":"086495eea1df6f69de76f6ff89b9757e1ab083bd198ee28a9ffe119e345a68f1"}
Sep 16 19:08:59.508999 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:08:59.508956    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-s78x4" event={"ID":"095c76bd-faf0-4db5-ba79-3b3a8dfebc7b","Type":"ContainerDied","Data":"091f5efea3b0dfeda0e1f09a3e66e918d8ad026c056c5abe308922dffe368936"}
Sep 16 19:08:59.508999 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:08:59.508992    2578 scope.go:117] "RemoveContainer" containerID="086495eea1df6f69de76f6ff89b9757e1ab083bd198ee28a9ffe119e345a68f1"
Sep 16 19:08:59.526136 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:08:59.526118    2578 scope.go:117] "RemoveContainer" containerID="086495eea1df6f69de76f6ff89b9757e1ab083bd198ee28a9ffe119e345a68f1"
Sep 16 19:08:59.526424 ip-10-0-135-119 kubenswrapper[2578]: E0916 19:08:59.526405    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"086495eea1df6f69de76f6ff89b9757e1ab083bd198ee28a9ffe119e345a68f1\": container with ID starting with 086495eea1df6f69de76f6ff89b9757e1ab083bd198ee28a9ffe119e345a68f1 not found: ID does not exist" containerID="086495eea1df6f69de76f6ff89b9757e1ab083bd198ee28a9ffe119e345a68f1"
Sep 16 19:08:59.526486 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:08:59.526437    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"086495eea1df6f69de76f6ff89b9757e1ab083bd198ee28a9ffe119e345a68f1"} err="failed to get container status \"086495eea1df6f69de76f6ff89b9757e1ab083bd198ee28a9ffe119e345a68f1\": rpc error: code = NotFound desc = could not find container \"086495eea1df6f69de76f6ff89b9757e1ab083bd198ee28a9ffe119e345a68f1\": container with ID starting with 086495eea1df6f69de76f6ff89b9757e1ab083bd198ee28a9ffe119e345a68f1 not found: ID does not exist"
Sep 16 19:08:59.533609 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:08:59.533576    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-s78x4"]
Sep 16 19:08:59.547314 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:08:59.547289    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-s78x4"]
Sep 16 19:08:59.654746 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:08:59.654720    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-m9wwp"
Sep 16 19:08:59.787134 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:08:59.787103    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f3ca627c-2998-4500-bca2-c983460aff65-home\") pod \"f3ca627c-2998-4500-bca2-c983460aff65\" (UID: \"f3ca627c-2998-4500-bca2-c983460aff65\") "
Sep 16 19:08:59.787322 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:08:59.787170    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f3ca627c-2998-4500-bca2-c983460aff65-dshm\") pod \"f3ca627c-2998-4500-bca2-c983460aff65\" (UID: \"f3ca627c-2998-4500-bca2-c983460aff65\") "
Sep 16 19:08:59.787322 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:08:59.787199    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f3ca627c-2998-4500-bca2-c983460aff65-tmp-dir\") pod \"f3ca627c-2998-4500-bca2-c983460aff65\" (UID: \"f3ca627c-2998-4500-bca2-c983460aff65\") "
Sep 16 19:08:59.787322 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:08:59.787231    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f3ca627c-2998-4500-bca2-c983460aff65-tls-certs\") pod \"f3ca627c-2998-4500-bca2-c983460aff65\" (UID: \"f3ca627c-2998-4500-bca2-c983460aff65\") "
Sep 16 19:08:59.787322 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:08:59.787257    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f3ca627c-2998-4500-bca2-c983460aff65-model-cache\") pod \"f3ca627c-2998-4500-bca2-c983460aff65\" (UID: \"f3ca627c-2998-4500-bca2-c983460aff65\") "
Sep 16 19:08:59.787551 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:08:59.787338    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3ca627c-2998-4500-bca2-c983460aff65-home" (OuterVolumeSpecName: "home") pod "f3ca627c-2998-4500-bca2-c983460aff65" (UID: "f3ca627c-2998-4500-bca2-c983460aff65"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:08:59.787551 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:08:59.787347    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2sgqb\" (UniqueName: \"kubernetes.io/projected/f3ca627c-2998-4500-bca2-c983460aff65-kube-api-access-2sgqb\") pod \"f3ca627c-2998-4500-bca2-c983460aff65\" (UID: \"f3ca627c-2998-4500-bca2-c983460aff65\") "
Sep 16 19:08:59.787551 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:08:59.787499    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3ca627c-2998-4500-bca2-c983460aff65-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "f3ca627c-2998-4500-bca2-c983460aff65" (UID: "f3ca627c-2998-4500-bca2-c983460aff65"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:08:59.787551 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:08:59.787512    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3ca627c-2998-4500-bca2-c983460aff65-model-cache" (OuterVolumeSpecName: "model-cache") pod "f3ca627c-2998-4500-bca2-c983460aff65" (UID: "f3ca627c-2998-4500-bca2-c983460aff65"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:08:59.787763 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:08:59.787677    2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f3ca627c-2998-4500-bca2-c983460aff65-home\") on node \"ip-10-0-135-119.ec2.internal\" DevicePath \"\""
Sep 16 19:08:59.787763 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:08:59.787697    2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f3ca627c-2998-4500-bca2-c983460aff65-tmp-dir\") on node \"ip-10-0-135-119.ec2.internal\" DevicePath \"\""
Sep 16 19:08:59.787763 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:08:59.787708    2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f3ca627c-2998-4500-bca2-c983460aff65-model-cache\") on node \"ip-10-0-135-119.ec2.internal\" DevicePath \"\""
Sep 16 19:08:59.789500 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:08:59.789477    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3ca627c-2998-4500-bca2-c983460aff65-dshm" (OuterVolumeSpecName: "dshm") pod "f3ca627c-2998-4500-bca2-c983460aff65" (UID: "f3ca627c-2998-4500-bca2-c983460aff65"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:08:59.789911 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:08:59.789893    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f3ca627c-2998-4500-bca2-c983460aff65-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "f3ca627c-2998-4500-bca2-c983460aff65" (UID: "f3ca627c-2998-4500-bca2-c983460aff65"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:08:59.789911 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:08:59.789896    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f3ca627c-2998-4500-bca2-c983460aff65-kube-api-access-2sgqb" (OuterVolumeSpecName: "kube-api-access-2sgqb") pod "f3ca627c-2998-4500-bca2-c983460aff65" (UID: "f3ca627c-2998-4500-bca2-c983460aff65"). InnerVolumeSpecName "kube-api-access-2sgqb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:08:59.889155 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:08:59.889127    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2sgqb\" (UniqueName: \"kubernetes.io/projected/f3ca627c-2998-4500-bca2-c983460aff65-kube-api-access-2sgqb\") on node \"ip-10-0-135-119.ec2.internal\" DevicePath \"\""
Sep 16 19:08:59.889155 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:08:59.889149    2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f3ca627c-2998-4500-bca2-c983460aff65-dshm\") on node \"ip-10-0-135-119.ec2.internal\" DevicePath \"\""
Sep 16 19:08:59.889155 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:08:59.889158    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f3ca627c-2998-4500-bca2-c983460aff65-tls-certs\") on node \"ip-10-0-135-119.ec2.internal\" DevicePath \"\""
Sep 16 19:09:00.514328 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:09:00.514287    2578 generic.go:356] "Generic (PLEG): container finished" podID="f3ca627c-2998-4500-bca2-c983460aff65" containerID="57f42a7da362608dc6cf4a4a2b93764283c8d95fc95313875df86dcd2167f007" exitCode=0
Sep 16 19:09:00.514719 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:09:00.514336    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-m9wwp"
Sep 16 19:09:00.514719 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:09:00.514382    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-m9wwp" event={"ID":"f3ca627c-2998-4500-bca2-c983460aff65","Type":"ContainerDied","Data":"57f42a7da362608dc6cf4a4a2b93764283c8d95fc95313875df86dcd2167f007"}
Sep 16 19:09:00.514719 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:09:00.514439    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-m9wwp" event={"ID":"f3ca627c-2998-4500-bca2-c983460aff65","Type":"ContainerDied","Data":"f955e6eb14aa6baab044e799592dda308a05fe2054c67c8860ed44d139183b83"}
Sep 16 19:09:00.514719 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:09:00.514463    2578 scope.go:117] "RemoveContainer" containerID="57f42a7da362608dc6cf4a4a2b93764283c8d95fc95313875df86dcd2167f007"
Sep 16 19:09:00.523950 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:09:00.523933    2578 scope.go:117] "RemoveContainer" containerID="57f42a7da362608dc6cf4a4a2b93764283c8d95fc95313875df86dcd2167f007"
Sep 16 19:09:00.524250 ip-10-0-135-119 kubenswrapper[2578]: E0916 19:09:00.524229    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"57f42a7da362608dc6cf4a4a2b93764283c8d95fc95313875df86dcd2167f007\": container with ID starting with 57f42a7da362608dc6cf4a4a2b93764283c8d95fc95313875df86dcd2167f007 not found: ID does not exist" containerID="57f42a7da362608dc6cf4a4a2b93764283c8d95fc95313875df86dcd2167f007"
Sep 16 19:09:00.524341 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:09:00.524256    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"57f42a7da362608dc6cf4a4a2b93764283c8d95fc95313875df86dcd2167f007"} err="failed to get container status \"57f42a7da362608dc6cf4a4a2b93764283c8d95fc95313875df86dcd2167f007\": rpc error: code = NotFound desc = could not find container \"57f42a7da362608dc6cf4a4a2b93764283c8d95fc95313875df86dcd2167f007\": container with ID starting with 57f42a7da362608dc6cf4a4a2b93764283c8d95fc95313875df86dcd2167f007 not found: ID does not exist"
Sep 16 19:09:00.547905 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:09:00.547882    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-m9wwp"]
Sep 16 19:09:00.556055 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:09:00.556035    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-m9wwp"]
Sep 16 19:09:01.434443 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:09:01.434410    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="095c76bd-faf0-4db5-ba79-3b3a8dfebc7b" path="/var/lib/kubelet/pods/095c76bd-faf0-4db5-ba79-3b3a8dfebc7b/volumes"
Sep 16 19:09:01.434797 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:09:01.434782    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f3ca627c-2998-4500-bca2-c983460aff65" path="/var/lib/kubelet/pods/f3ca627c-2998-4500-bca2-c983460aff65/volumes"
Sep 16 19:09:08.177013 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:09:08.176957    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedpv89s" podUID="a0d41aa5-98c6-412e-b955-e629b3f01619" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:09:09.330988 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:09:09.330947    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedpv89s"
Sep 16 19:09:09.367084 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:09:09.366966    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a0d41aa5-98c6-412e-b955-e629b3f01619-tls-certs\") pod \"a0d41aa5-98c6-412e-b955-e629b3f01619\" (UID: \"a0d41aa5-98c6-412e-b955-e629b3f01619\") "
Sep 16 19:09:09.367084 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:09:09.367059    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-t4rgg\" (UniqueName: \"kubernetes.io/projected/a0d41aa5-98c6-412e-b955-e629b3f01619-kube-api-access-t4rgg\") pod \"a0d41aa5-98c6-412e-b955-e629b3f01619\" (UID: \"a0d41aa5-98c6-412e-b955-e629b3f01619\") "
Sep 16 19:09:09.369622 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:09:09.369588    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a0d41aa5-98c6-412e-b955-e629b3f01619-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "a0d41aa5-98c6-412e-b955-e629b3f01619" (UID: "a0d41aa5-98c6-412e-b955-e629b3f01619"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:09:09.369622 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:09:09.369602    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a0d41aa5-98c6-412e-b955-e629b3f01619-kube-api-access-t4rgg" (OuterVolumeSpecName: "kube-api-access-t4rgg") pod "a0d41aa5-98c6-412e-b955-e629b3f01619" (UID: "a0d41aa5-98c6-412e-b955-e629b3f01619"). InnerVolumeSpecName "kube-api-access-t4rgg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:09:09.468722 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:09:09.468673    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t4rgg\" (UniqueName: \"kubernetes.io/projected/a0d41aa5-98c6-412e-b955-e629b3f01619-kube-api-access-t4rgg\") on node \"ip-10-0-135-119.ec2.internal\" DevicePath \"\""
Sep 16 19:09:09.468722 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:09:09.468719    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a0d41aa5-98c6-412e-b955-e629b3f01619-tls-certs\") on node \"ip-10-0-135-119.ec2.internal\" DevicePath \"\""
Sep 16 19:09:09.559696 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:09:09.559653    2578 generic.go:356] "Generic (PLEG): container finished" podID="a0d41aa5-98c6-412e-b955-e629b3f01619" containerID="767416d09137acb889c70f676c8621a29fb40a2c4ee9d41cb129031ce1fb6829" exitCode=0
Sep 16 19:09:09.559848 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:09:09.559721    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedpv89s"
Sep 16 19:09:09.559848 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:09:09.559731    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedpv89s" event={"ID":"a0d41aa5-98c6-412e-b955-e629b3f01619","Type":"ContainerDied","Data":"767416d09137acb889c70f676c8621a29fb40a2c4ee9d41cb129031ce1fb6829"}
Sep 16 19:09:09.559848 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:09:09.559778    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedpv89s" event={"ID":"a0d41aa5-98c6-412e-b955-e629b3f01619","Type":"ContainerDied","Data":"4c8168154b2c84c70709c2c6a67f862872496e2c1312662b1d6055bcc74410c4"}
Sep 16 19:09:09.559848 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:09:09.559799    2578 scope.go:117] "RemoveContainer" containerID="767416d09137acb889c70f676c8621a29fb40a2c4ee9d41cb129031ce1fb6829"
Sep 16 19:09:09.568934 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:09:09.568908    2578 scope.go:117] "RemoveContainer" containerID="767416d09137acb889c70f676c8621a29fb40a2c4ee9d41cb129031ce1fb6829"
Sep 16 19:09:09.569356 ip-10-0-135-119 kubenswrapper[2578]: E0916 19:09:09.569333    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"767416d09137acb889c70f676c8621a29fb40a2c4ee9d41cb129031ce1fb6829\": container with ID starting with 767416d09137acb889c70f676c8621a29fb40a2c4ee9d41cb129031ce1fb6829 not found: ID does not exist" containerID="767416d09137acb889c70f676c8621a29fb40a2c4ee9d41cb129031ce1fb6829"
Sep 16 19:09:09.569424 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:09:09.569366    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"767416d09137acb889c70f676c8621a29fb40a2c4ee9d41cb129031ce1fb6829"} err="failed to get container status \"767416d09137acb889c70f676c8621a29fb40a2c4ee9d41cb129031ce1fb6829\": rpc error: code = NotFound desc = could not find container \"767416d09137acb889c70f676c8621a29fb40a2c4ee9d41cb129031ce1fb6829\": container with ID starting with 767416d09137acb889c70f676c8621a29fb40a2c4ee9d41cb129031ce1fb6829 not found: ID does not exist"
Sep 16 19:09:09.579573 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:09:09.579533    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedpv89s"]
Sep 16 19:09:09.587626 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:09:09.587590    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedpv89s"]
Sep 16 19:09:11.434561 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:09:11.434530    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a0d41aa5-98c6-412e-b955-e629b3f01619" path="/var/lib/kubelet/pods/a0d41aa5-98c6-412e-b955-e629b3f01619/volumes"
Sep 16 19:09:14.016575 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:09:14.016545    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:09:14.115480 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:09:14.115442    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:09:15.034042 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:09:15.034005    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-cg8nl"]
Sep 16 19:09:15.034465 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:09:15.034443    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="095c76bd-faf0-4db5-ba79-3b3a8dfebc7b" containerName="main"
Sep 16 19:09:15.034465 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:09:15.034463    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="095c76bd-faf0-4db5-ba79-3b3a8dfebc7b" containerName="main"
Sep 16 19:09:15.034567 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:09:15.034477    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f3ca627c-2998-4500-bca2-c983460aff65" containerName="main"
Sep 16 19:09:15.034567 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:09:15.034483    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="f3ca627c-2998-4500-bca2-c983460aff65" containerName="main"
Sep 16 19:09:15.034567 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:09:15.034493    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a0d41aa5-98c6-412e-b955-e629b3f01619" containerName="main"
Sep 16 19:09:15.034567 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:09:15.034503    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="a0d41aa5-98c6-412e-b955-e629b3f01619" containerName="main"
Sep 16 19:09:15.034701 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:09:15.034571    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="a0d41aa5-98c6-412e-b955-e629b3f01619" containerName="main"
Sep 16 19:09:15.034701 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:09:15.034582    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="f3ca627c-2998-4500-bca2-c983460aff65" containerName="main"
Sep 16 19:09:15.034701 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:09:15.034588    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="095c76bd-faf0-4db5-ba79-3b3a8dfebc7b" containerName="main"
Sep 16 19:09:15.039010 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:09:15.038941    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-cg8nl"
Sep 16 19:09:15.041786 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:09:15.041760    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-model-name-divergence-75f1ef44\"/\"default-dockercfg-zvx5r\""
Sep 16 19:09:15.042054 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:09:15.042036    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-model-name-divergence-75f1ef44\"/\"openshift-service-ca.crt\""
Sep 16 19:09:15.042054 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:09:15.042049    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-model-name-divergence-75f1ef44\"/\"test-div-v2-kserve-self-signed-certs\""
Sep 16 19:09:15.042209 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:09:15.042104    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-model-name-divergence-75f1ef44\"/\"kube-root-ca.crt\""
Sep 16 19:09:15.050539 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:09:15.050517    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-cg8nl"]
Sep 16 19:09:15.074256 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:09:15.074223    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:09:15.118297 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:09:15.118266    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tqm7s\" (UniqueName: \"kubernetes.io/projected/77617eaf-a7ee-4f26-9be8-16cc9815bbdd-kube-api-access-tqm7s\") pod \"test-div-v2-kserve-5f6fdb6b77-cg8nl\" (UID: \"77617eaf-a7ee-4f26-9be8-16cc9815bbdd\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-cg8nl"
Sep 16 19:09:15.118482 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:09:15.118308    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/77617eaf-a7ee-4f26-9be8-16cc9815bbdd-tls-certs\") pod \"test-div-v2-kserve-5f6fdb6b77-cg8nl\" (UID: \"77617eaf-a7ee-4f26-9be8-16cc9815bbdd\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-cg8nl"
Sep 16 19:09:15.118482 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:09:15.118337    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/77617eaf-a7ee-4f26-9be8-16cc9815bbdd-tmp-dir\") pod \"test-div-v2-kserve-5f6fdb6b77-cg8nl\" (UID: \"77617eaf-a7ee-4f26-9be8-16cc9815bbdd\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-cg8nl"
Sep 16 19:09:15.118482 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:09:15.118422    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/77617eaf-a7ee-4f26-9be8-16cc9815bbdd-model-cache\") pod \"test-div-v2-kserve-5f6fdb6b77-cg8nl\" (UID: \"77617eaf-a7ee-4f26-9be8-16cc9815bbdd\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-cg8nl"
Sep 16 19:09:15.118659 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:09:15.118496    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/77617eaf-a7ee-4f26-9be8-16cc9815bbdd-home\") pod \"test-div-v2-kserve-5f6fdb6b77-cg8nl\" (UID: \"77617eaf-a7ee-4f26-9be8-16cc9815bbdd\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-cg8nl"
Sep 16 19:09:15.118659 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:09:15.118555    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/77617eaf-a7ee-4f26-9be8-16cc9815bbdd-dshm\") pod \"test-div-v2-kserve-5f6fdb6b77-cg8nl\" (UID: \"77617eaf-a7ee-4f26-9be8-16cc9815bbdd\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-cg8nl"
Sep 16 19:09:15.219919 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:09:15.219815    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/77617eaf-a7ee-4f26-9be8-16cc9815bbdd-model-cache\") pod \"test-div-v2-kserve-5f6fdb6b77-cg8nl\" (UID: \"77617eaf-a7ee-4f26-9be8-16cc9815bbdd\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-cg8nl"
Sep 16 19:09:15.220111 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:09:15.219958    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/77617eaf-a7ee-4f26-9be8-16cc9815bbdd-home\") pod \"test-div-v2-kserve-5f6fdb6b77-cg8nl\" (UID: \"77617eaf-a7ee-4f26-9be8-16cc9815bbdd\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-cg8nl"
Sep 16 19:09:15.220111 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:09:15.220035    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/77617eaf-a7ee-4f26-9be8-16cc9815bbdd-dshm\") pod \"test-div-v2-kserve-5f6fdb6b77-cg8nl\" (UID: \"77617eaf-a7ee-4f26-9be8-16cc9815bbdd\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-cg8nl"
Sep 16 19:09:15.220111 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:09:15.220068    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tqm7s\" (UniqueName: \"kubernetes.io/projected/77617eaf-a7ee-4f26-9be8-16cc9815bbdd-kube-api-access-tqm7s\") pod \"test-div-v2-kserve-5f6fdb6b77-cg8nl\" (UID: \"77617eaf-a7ee-4f26-9be8-16cc9815bbdd\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-cg8nl"
Sep 16 19:09:15.220111 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:09:15.220095    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/77617eaf-a7ee-4f26-9be8-16cc9815bbdd-tls-certs\") pod \"test-div-v2-kserve-5f6fdb6b77-cg8nl\" (UID: \"77617eaf-a7ee-4f26-9be8-16cc9815bbdd\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-cg8nl"
Sep 16 19:09:15.220283 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:09:15.220141    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/77617eaf-a7ee-4f26-9be8-16cc9815bbdd-tmp-dir\") pod \"test-div-v2-kserve-5f6fdb6b77-cg8nl\" (UID: \"77617eaf-a7ee-4f26-9be8-16cc9815bbdd\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-cg8nl"
Sep 16 19:09:15.220283 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:09:15.220217    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/77617eaf-a7ee-4f26-9be8-16cc9815bbdd-model-cache\") pod \"test-div-v2-kserve-5f6fdb6b77-cg8nl\" (UID: \"77617eaf-a7ee-4f26-9be8-16cc9815bbdd\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-cg8nl"
Sep 16 19:09:15.220358 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:09:15.220302    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/77617eaf-a7ee-4f26-9be8-16cc9815bbdd-home\") pod \"test-div-v2-kserve-5f6fdb6b77-cg8nl\" (UID: \"77617eaf-a7ee-4f26-9be8-16cc9815bbdd\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-cg8nl"
Sep 16 19:09:15.220469 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:09:15.220448    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/77617eaf-a7ee-4f26-9be8-16cc9815bbdd-tmp-dir\") pod \"test-div-v2-kserve-5f6fdb6b77-cg8nl\" (UID: \"77617eaf-a7ee-4f26-9be8-16cc9815bbdd\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-cg8nl"
Sep 16 19:09:15.222403 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:09:15.222384    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/77617eaf-a7ee-4f26-9be8-16cc9815bbdd-dshm\") pod \"test-div-v2-kserve-5f6fdb6b77-cg8nl\" (UID: \"77617eaf-a7ee-4f26-9be8-16cc9815bbdd\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-cg8nl"
Sep 16 19:09:15.222703 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:09:15.222683    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/77617eaf-a7ee-4f26-9be8-16cc9815bbdd-tls-certs\") pod \"test-div-v2-kserve-5f6fdb6b77-cg8nl\" (UID: \"77617eaf-a7ee-4f26-9be8-16cc9815bbdd\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-cg8nl"
Sep 16 19:09:15.232258 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:09:15.232226    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tqm7s\" (UniqueName: \"kubernetes.io/projected/77617eaf-a7ee-4f26-9be8-16cc9815bbdd-kube-api-access-tqm7s\") pod \"test-div-v2-kserve-5f6fdb6b77-cg8nl\" (UID: \"77617eaf-a7ee-4f26-9be8-16cc9815bbdd\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-cg8nl"
Sep 16 19:09:15.349644 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:09:15.349534    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-cg8nl"
Sep 16 19:09:15.627682 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:09:15.627652    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-cg8nl"]
Sep 16 19:09:15.630407 ip-10-0-135-119 kubenswrapper[2578]: W0916 19:09:15.630368    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod77617eaf_a7ee_4f26_9be8_16cc9815bbdd.slice/crio-a2df48a13282bb0e43d10b3ef9088f8e9d3418a096bede1068171e6976284fa9 WatchSource:0}: Error finding container a2df48a13282bb0e43d10b3ef9088f8e9d3418a096bede1068171e6976284fa9: Status 404 returned error can't find the container with id a2df48a13282bb0e43d10b3ef9088f8e9d3418a096bede1068171e6976284fa9
Sep 16 19:09:16.597839 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:09:16.597798    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-cg8nl" event={"ID":"77617eaf-a7ee-4f26-9be8-16cc9815bbdd","Type":"ContainerStarted","Data":"847b26c7fb6ab4df4b73c1362640f2cd79d9520000c62ff761bc0d6d5af009a4"}
Sep 16 19:09:16.598284 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:09:16.597863    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-cg8nl" event={"ID":"77617eaf-a7ee-4f26-9be8-16cc9815bbdd","Type":"ContainerStarted","Data":"a2df48a13282bb0e43d10b3ef9088f8e9d3418a096bede1068171e6976284fa9"}
Sep 16 19:09:16.623523 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:09:16.623459    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-cg8nl" podStartSLOduration=1.623440912 podStartE2EDuration="1.623440912s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:09:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:09:16.622301454 +0000 UTC m=+1737.688792964" watchObservedRunningTime="2026-09-16 19:09:16.623440912 +0000 UTC m=+1737.689932400"
Sep 16 19:09:17.349856 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:09:17.349817    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-cg8nl"
Sep 16 19:09:21.026357 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:09:21.026322    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:09:25.349770 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:09:25.349738    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-cg8nl"
Sep 16 19:09:25.362196 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:09:25.362171    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-cg8nl"
Sep 16 19:09:25.659390 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:09:25.659362    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-cg8nl"
Sep 16 19:09:26.384739 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:09:26.384697    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:09:27.592479 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:09:27.592446    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:09:27.659442 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:09:27.659401    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:09:32.447757 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:09:32.447720    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:09:32.682954 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:09:32.682902    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:09:38.120816 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:09:38.120786    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:09:38.423392 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:09:38.423274    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69lkn4w"]
Sep 16 19:09:38.426794 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:09:38.426768    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69lkn4w"
Sep 16 19:09:38.429800 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:09:38.429771    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"tokenizer-clean-path-test-kserve-self-signed-certs\""
Sep 16 19:09:38.430008 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:09:38.429771    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"openshift-service-ca.crt\""
Sep 16 19:09:38.430008 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:09:38.429774    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"tokenizer-clean-path-test-epp-sa-dockercfg-vrt7t\""
Sep 16 19:09:38.430008 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:09:38.429912    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"kube-root-ca.crt\""
Sep 16 19:09:38.430008 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:09:38.429932    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"default-dockercfg-zjzwv\""
Sep 16 19:09:38.430443 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:09:38.430420    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:09:38.436667 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:09:38.436640    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69lkn4w"]
Sep 16 19:09:38.529499 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:09:38.529466    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:09:38.532723 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:09:38.532688    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/80ec9a19-99f6-4bf0-90cb-593f6456c285-tls-certs\") pod \"tokenizer-clean-path-test-kserve-router-scheduler-58494b69lkn4w\" (UID: \"80ec9a19-99f6-4bf0-90cb-593f6456c285\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69lkn4w"
Sep 16 19:09:38.532874 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:09:38.532755    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-27vqx\" (UniqueName: \"kubernetes.io/projected/80ec9a19-99f6-4bf0-90cb-593f6456c285-kube-api-access-27vqx\") pod \"tokenizer-clean-path-test-kserve-router-scheduler-58494b69lkn4w\" (UID: \"80ec9a19-99f6-4bf0-90cb-593f6456c285\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69lkn4w"
Sep 16 19:09:38.633497 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:09:38.633460    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/80ec9a19-99f6-4bf0-90cb-593f6456c285-tls-certs\") pod \"tokenizer-clean-path-test-kserve-router-scheduler-58494b69lkn4w\" (UID: \"80ec9a19-99f6-4bf0-90cb-593f6456c285\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69lkn4w"
Sep 16 19:09:38.633676 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:09:38.633508    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-27vqx\" (UniqueName: \"kubernetes.io/projected/80ec9a19-99f6-4bf0-90cb-593f6456c285-kube-api-access-27vqx\") pod \"tokenizer-clean-path-test-kserve-router-scheduler-58494b69lkn4w\" (UID: \"80ec9a19-99f6-4bf0-90cb-593f6456c285\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69lkn4w"
Sep 16 19:09:38.636409 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:09:38.636382    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/80ec9a19-99f6-4bf0-90cb-593f6456c285-tls-certs\") pod \"tokenizer-clean-path-test-kserve-router-scheduler-58494b69lkn4w\" (UID: \"80ec9a19-99f6-4bf0-90cb-593f6456c285\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69lkn4w"
Sep 16 19:09:38.642781 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:09:38.642747    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-27vqx\" (UniqueName: \"kubernetes.io/projected/80ec9a19-99f6-4bf0-90cb-593f6456c285-kube-api-access-27vqx\") pod \"tokenizer-clean-path-test-kserve-router-scheduler-58494b69lkn4w\" (UID: \"80ec9a19-99f6-4bf0-90cb-593f6456c285\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69lkn4w"
Sep 16 19:09:38.738873 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:09:38.738762    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69lkn4w"
Sep 16 19:09:39.017837 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:09:39.017809    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69lkn4w"]
Sep 16 19:09:39.020721 ip-10-0-135-119 kubenswrapper[2578]: W0916 19:09:39.020689    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod80ec9a19_99f6_4bf0_90cb_593f6456c285.slice/crio-747280f2248058f4360a88be976d3ee4cf4abb7d465b19eab03a792f00838c12 WatchSource:0}: Error finding container 747280f2248058f4360a88be976d3ee4cf4abb7d465b19eab03a792f00838c12: Status 404 returned error can't find the container with id 747280f2248058f4360a88be976d3ee4cf4abb7d465b19eab03a792f00838c12
Sep 16 19:09:39.718497 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:09:39.718456    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69lkn4w" event={"ID":"80ec9a19-99f6-4bf0-90cb-593f6456c285","Type":"ContainerStarted","Data":"28f3963cced1832e2edc22686aa1ad57b6572f4299a57f26d196be23343da438"}
Sep 16 19:09:39.718935 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:09:39.718520    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69lkn4w"
Sep 16 19:09:39.718935 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:09:39.718537    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69lkn4w" event={"ID":"80ec9a19-99f6-4bf0-90cb-593f6456c285","Type":"ContainerStarted","Data":"747280f2248058f4360a88be976d3ee4cf4abb7d465b19eab03a792f00838c12"}
Sep 16 19:09:39.740218 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:09:39.740157    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69lkn4w" podStartSLOduration=1.7401385839999999 podStartE2EDuration="1.740138584s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:09:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:09:39.739410573 +0000 UTC m=+1760.805902085" watchObservedRunningTime="2026-09-16 19:09:39.740138584 +0000 UTC m=+1760.806630072"
Sep 16 19:09:42.370227 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:09:42.370186    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:09:42.474744 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:09:42.474708    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:09:42.571571 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:09:42.571536    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:09:47.363535 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:09:47.363491    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-cg8nl"]
Sep 16 19:09:47.445420 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:09:47.445384    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:09:55.791080 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:09:55.791039    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:09:55.887267 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:09:55.887227    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:09:57.765135 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:09:57.765102    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:10:00.160137 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:10:00.160090    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:10:02.384169 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:10:02.384127    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-cg8nl" podUID="77617eaf-a7ee-4f26-9be8-16cc9815bbdd" containerName="main" containerID="cri-o://847b26c7fb6ab4df4b73c1362640f2cd79d9520000c62ff761bc0d6d5af009a4" gracePeriod=45
Sep 16 19:10:02.646609 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:10:02.646543    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-cg8nl"
Sep 16 19:10:02.752650 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:10:02.752610    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/77617eaf-a7ee-4f26-9be8-16cc9815bbdd-model-cache\") pod \"77617eaf-a7ee-4f26-9be8-16cc9815bbdd\" (UID: \"77617eaf-a7ee-4f26-9be8-16cc9815bbdd\") "
Sep 16 19:10:02.752823 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:10:02.752675    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/77617eaf-a7ee-4f26-9be8-16cc9815bbdd-home\") pod \"77617eaf-a7ee-4f26-9be8-16cc9815bbdd\" (UID: \"77617eaf-a7ee-4f26-9be8-16cc9815bbdd\") "
Sep 16 19:10:02.752823 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:10:02.752754    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/77617eaf-a7ee-4f26-9be8-16cc9815bbdd-tmp-dir\") pod \"77617eaf-a7ee-4f26-9be8-16cc9815bbdd\" (UID: \"77617eaf-a7ee-4f26-9be8-16cc9815bbdd\") "
Sep 16 19:10:02.752823 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:10:02.752784    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/77617eaf-a7ee-4f26-9be8-16cc9815bbdd-tls-certs\") pod \"77617eaf-a7ee-4f26-9be8-16cc9815bbdd\" (UID: \"77617eaf-a7ee-4f26-9be8-16cc9815bbdd\") "
Sep 16 19:10:02.752823 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:10:02.752789    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/77617eaf-a7ee-4f26-9be8-16cc9815bbdd-model-cache" (OuterVolumeSpecName: "model-cache") pod "77617eaf-a7ee-4f26-9be8-16cc9815bbdd" (UID: "77617eaf-a7ee-4f26-9be8-16cc9815bbdd"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:10:02.752967 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:10:02.752826    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tqm7s\" (UniqueName: \"kubernetes.io/projected/77617eaf-a7ee-4f26-9be8-16cc9815bbdd-kube-api-access-tqm7s\") pod \"77617eaf-a7ee-4f26-9be8-16cc9815bbdd\" (UID: \"77617eaf-a7ee-4f26-9be8-16cc9815bbdd\") "
Sep 16 19:10:02.752967 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:10:02.752871    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/77617eaf-a7ee-4f26-9be8-16cc9815bbdd-dshm\") pod \"77617eaf-a7ee-4f26-9be8-16cc9815bbdd\" (UID: \"77617eaf-a7ee-4f26-9be8-16cc9815bbdd\") "
Sep 16 19:10:02.753101 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:10:02.752964    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/77617eaf-a7ee-4f26-9be8-16cc9815bbdd-home" (OuterVolumeSpecName: "home") pod "77617eaf-a7ee-4f26-9be8-16cc9815bbdd" (UID: "77617eaf-a7ee-4f26-9be8-16cc9815bbdd"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:10:02.753101 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:10:02.753066    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/77617eaf-a7ee-4f26-9be8-16cc9815bbdd-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "77617eaf-a7ee-4f26-9be8-16cc9815bbdd" (UID: "77617eaf-a7ee-4f26-9be8-16cc9815bbdd"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:10:02.753217 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:10:02.753204    2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/77617eaf-a7ee-4f26-9be8-16cc9815bbdd-tmp-dir\") on node \"ip-10-0-135-119.ec2.internal\" DevicePath \"\""
Sep 16 19:10:02.753276 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:10:02.753221    2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/77617eaf-a7ee-4f26-9be8-16cc9815bbdd-model-cache\") on node \"ip-10-0-135-119.ec2.internal\" DevicePath \"\""
Sep 16 19:10:02.753276 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:10:02.753233    2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/77617eaf-a7ee-4f26-9be8-16cc9815bbdd-home\") on node \"ip-10-0-135-119.ec2.internal\" DevicePath \"\""
Sep 16 19:10:02.755315 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:10:02.755291    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/77617eaf-a7ee-4f26-9be8-16cc9815bbdd-dshm" (OuterVolumeSpecName: "dshm") pod "77617eaf-a7ee-4f26-9be8-16cc9815bbdd" (UID: "77617eaf-a7ee-4f26-9be8-16cc9815bbdd"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:10:02.755419 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:10:02.755353    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/77617eaf-a7ee-4f26-9be8-16cc9815bbdd-kube-api-access-tqm7s" (OuterVolumeSpecName: "kube-api-access-tqm7s") pod "77617eaf-a7ee-4f26-9be8-16cc9815bbdd" (UID: "77617eaf-a7ee-4f26-9be8-16cc9815bbdd"). InnerVolumeSpecName "kube-api-access-tqm7s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:10:02.755419 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:10:02.755383    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/77617eaf-a7ee-4f26-9be8-16cc9815bbdd-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "77617eaf-a7ee-4f26-9be8-16cc9815bbdd" (UID: "77617eaf-a7ee-4f26-9be8-16cc9815bbdd"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:10:02.841939 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:10:02.841902    2578 generic.go:356] "Generic (PLEG): container finished" podID="77617eaf-a7ee-4f26-9be8-16cc9815bbdd" containerID="847b26c7fb6ab4df4b73c1362640f2cd79d9520000c62ff761bc0d6d5af009a4" exitCode=0
Sep 16 19:10:02.842127 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:10:02.841945    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-cg8nl"
Sep 16 19:10:02.842127 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:10:02.841961    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-cg8nl" event={"ID":"77617eaf-a7ee-4f26-9be8-16cc9815bbdd","Type":"ContainerDied","Data":"847b26c7fb6ab4df4b73c1362640f2cd79d9520000c62ff761bc0d6d5af009a4"}
Sep 16 19:10:02.842127 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:10:02.842026    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-cg8nl" event={"ID":"77617eaf-a7ee-4f26-9be8-16cc9815bbdd","Type":"ContainerDied","Data":"a2df48a13282bb0e43d10b3ef9088f8e9d3418a096bede1068171e6976284fa9"}
Sep 16 19:10:02.842127 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:10:02.842049    2578 scope.go:117] "RemoveContainer" containerID="847b26c7fb6ab4df4b73c1362640f2cd79d9520000c62ff761bc0d6d5af009a4"
Sep 16 19:10:02.851272 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:10:02.851248    2578 scope.go:117] "RemoveContainer" containerID="847b26c7fb6ab4df4b73c1362640f2cd79d9520000c62ff761bc0d6d5af009a4"
Sep 16 19:10:02.851643 ip-10-0-135-119 kubenswrapper[2578]: E0916 19:10:02.851624    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"847b26c7fb6ab4df4b73c1362640f2cd79d9520000c62ff761bc0d6d5af009a4\": container with ID starting with 847b26c7fb6ab4df4b73c1362640f2cd79d9520000c62ff761bc0d6d5af009a4 not found: ID does not exist" containerID="847b26c7fb6ab4df4b73c1362640f2cd79d9520000c62ff761bc0d6d5af009a4"
Sep 16 19:10:02.851692 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:10:02.851652    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"847b26c7fb6ab4df4b73c1362640f2cd79d9520000c62ff761bc0d6d5af009a4"} err="failed to get container status \"847b26c7fb6ab4df4b73c1362640f2cd79d9520000c62ff761bc0d6d5af009a4\": rpc error: code = NotFound desc = could not find container \"847b26c7fb6ab4df4b73c1362640f2cd79d9520000c62ff761bc0d6d5af009a4\": container with ID starting with 847b26c7fb6ab4df4b73c1362640f2cd79d9520000c62ff761bc0d6d5af009a4 not found: ID does not exist"
Sep 16 19:10:02.853696 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:10:02.853676    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/77617eaf-a7ee-4f26-9be8-16cc9815bbdd-tls-certs\") on node \"ip-10-0-135-119.ec2.internal\" DevicePath \"\""
Sep 16 19:10:02.853769 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:10:02.853698    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tqm7s\" (UniqueName: \"kubernetes.io/projected/77617eaf-a7ee-4f26-9be8-16cc9815bbdd-kube-api-access-tqm7s\") on node \"ip-10-0-135-119.ec2.internal\" DevicePath \"\""
Sep 16 19:10:02.853769 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:10:02.853707    2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/77617eaf-a7ee-4f26-9be8-16cc9815bbdd-dshm\") on node \"ip-10-0-135-119.ec2.internal\" DevicePath \"\""
Sep 16 19:10:02.865013 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:10:02.864962    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-cg8nl"]
Sep 16 19:10:02.867722 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:10:02.867696    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-cg8nl"]
Sep 16 19:10:03.433970 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:10:03.433927    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="77617eaf-a7ee-4f26-9be8-16cc9815bbdd" path="/var/lib/kubelet/pods/77617eaf-a7ee-4f26-9be8-16cc9815bbdd/volumes"
Sep 16 19:10:10.736409 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:10:10.736336    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69lkn4w"
Sep 16 19:10:19.472372 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:10:19.472337    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-tmrkx_4c2078bb-281a-4aa1-8e38-238897865434/console-operator/2.log"
Sep 16 19:10:19.474732 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:10:19.474709    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-tmrkx_4c2078bb-281a-4aa1-8e38-238897865434/console-operator/2.log"
Sep 16 19:10:19.483682 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:10:19.483652    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-k6gvs_6012728d-aee6-4f93-ab6e-fad616ef36e8/ovn-acl-logging/0.log"
Sep 16 19:10:19.486554 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:10:19.486523    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-k6gvs_6012728d-aee6-4f93-ab6e-fad616ef36e8/ovn-acl-logging/0.log"
Sep 16 19:10:25.471557 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:10:25.471515    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:10:27.762473 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:10:27.762441    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:10:32.051083 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:10:32.051050    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:10:32.149175 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:10:32.149137    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:10:34.478561 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:10:34.478517    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:10:34.580191 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:10:34.580154    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:10:43.393728 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:10:43.393692    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:11:10.391670 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:11:10.391622    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:11:10.421542 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:11:10.421508    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69lkn4w"]
Sep 16 19:11:10.473956 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:11:10.473926    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:11:10.735918 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:11:10.735828    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69lkn4w" podUID="80ec9a19-99f6-4bf0-90cb-593f6456c285" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:11:17.279167 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:11:17.279125    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:11:17.389408 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:11:17.389371    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:11:17.497637 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:11:17.497606    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:11:17.641418 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:11:17.641384    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:11:17.738617 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:11:17.738587    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:11:17.816576 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:11:17.816542    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:11:18.207231 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:11:18.207190    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:11:20.735866 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:11:20.735826    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69lkn4w" podUID="80ec9a19-99f6-4bf0-90cb-593f6456c285" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:11:22.320748 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:11:22.320714    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:11:24.194523 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:11:24.194481    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:11:25.442434 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:11:25.442393    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69lkn4w" podUID="80ec9a19-99f6-4bf0-90cb-593f6456c285" containerName="main" containerID="cri-o://28f3963cced1832e2edc22686aa1ad57b6572f4299a57f26d196be23343da438" gracePeriod=45
Sep 16 19:11:28.262227 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:11:28.262174    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:11:30.476868 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:11:30.476824    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:11:30.575604 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:11:30.575570    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:11:30.736165 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:11:30.736080    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69lkn4w" podUID="80ec9a19-99f6-4bf0-90cb-593f6456c285" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:11:30.736329 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:11:30.736199    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69lkn4w"
Sep 16 19:11:40.735926 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:11:40.735837    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69lkn4w" podUID="80ec9a19-99f6-4bf0-90cb-593f6456c285" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:11:50.736259 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:11:50.736219    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69lkn4w" podUID="80ec9a19-99f6-4bf0-90cb-593f6456c285" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:11:55.694621 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:11:55.694588    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69lkn4w"
Sep 16 19:11:55.791605 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:11:55.791508    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/80ec9a19-99f6-4bf0-90cb-593f6456c285-tls-certs\") pod \"80ec9a19-99f6-4bf0-90cb-593f6456c285\" (UID: \"80ec9a19-99f6-4bf0-90cb-593f6456c285\") "
Sep 16 19:11:55.791771 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:11:55.791643    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-27vqx\" (UniqueName: \"kubernetes.io/projected/80ec9a19-99f6-4bf0-90cb-593f6456c285-kube-api-access-27vqx\") pod \"80ec9a19-99f6-4bf0-90cb-593f6456c285\" (UID: \"80ec9a19-99f6-4bf0-90cb-593f6456c285\") "
Sep 16 19:11:55.794059 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:11:55.794017    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/80ec9a19-99f6-4bf0-90cb-593f6456c285-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "80ec9a19-99f6-4bf0-90cb-593f6456c285" (UID: "80ec9a19-99f6-4bf0-90cb-593f6456c285"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:11:55.794182 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:11:55.794030    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/80ec9a19-99f6-4bf0-90cb-593f6456c285-kube-api-access-27vqx" (OuterVolumeSpecName: "kube-api-access-27vqx") pod "80ec9a19-99f6-4bf0-90cb-593f6456c285" (UID: "80ec9a19-99f6-4bf0-90cb-593f6456c285"). InnerVolumeSpecName "kube-api-access-27vqx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:11:55.893186 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:11:55.893145    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-27vqx\" (UniqueName: \"kubernetes.io/projected/80ec9a19-99f6-4bf0-90cb-593f6456c285-kube-api-access-27vqx\") on node \"ip-10-0-135-119.ec2.internal\" DevicePath \"\""
Sep 16 19:11:55.893186 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:11:55.893178    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/80ec9a19-99f6-4bf0-90cb-593f6456c285-tls-certs\") on node \"ip-10-0-135-119.ec2.internal\" DevicePath \"\""
Sep 16 19:11:56.386686 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:11:56.386640    2578 generic.go:356] "Generic (PLEG): container finished" podID="80ec9a19-99f6-4bf0-90cb-593f6456c285" containerID="28f3963cced1832e2edc22686aa1ad57b6572f4299a57f26d196be23343da438" exitCode=0
Sep 16 19:11:56.386880 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:11:56.386701    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69lkn4w"
Sep 16 19:11:56.386880 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:11:56.386726    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69lkn4w" event={"ID":"80ec9a19-99f6-4bf0-90cb-593f6456c285","Type":"ContainerDied","Data":"28f3963cced1832e2edc22686aa1ad57b6572f4299a57f26d196be23343da438"}
Sep 16 19:11:56.386880 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:11:56.386786    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69lkn4w" event={"ID":"80ec9a19-99f6-4bf0-90cb-593f6456c285","Type":"ContainerDied","Data":"747280f2248058f4360a88be976d3ee4cf4abb7d465b19eab03a792f00838c12"}
Sep 16 19:11:56.386880 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:11:56.386808    2578 scope.go:117] "RemoveContainer" containerID="28f3963cced1832e2edc22686aa1ad57b6572f4299a57f26d196be23343da438"
Sep 16 19:11:56.396459 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:11:56.396438    2578 scope.go:117] "RemoveContainer" containerID="28f3963cced1832e2edc22686aa1ad57b6572f4299a57f26d196be23343da438"
Sep 16 19:11:56.396743 ip-10-0-135-119 kubenswrapper[2578]: E0916 19:11:56.396724    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"28f3963cced1832e2edc22686aa1ad57b6572f4299a57f26d196be23343da438\": container with ID starting with 28f3963cced1832e2edc22686aa1ad57b6572f4299a57f26d196be23343da438 not found: ID does not exist" containerID="28f3963cced1832e2edc22686aa1ad57b6572f4299a57f26d196be23343da438"
Sep 16 19:11:56.396790 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:11:56.396752    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"28f3963cced1832e2edc22686aa1ad57b6572f4299a57f26d196be23343da438"} err="failed to get container status \"28f3963cced1832e2edc22686aa1ad57b6572f4299a57f26d196be23343da438\": rpc error: code = NotFound desc = could not find container \"28f3963cced1832e2edc22686aa1ad57b6572f4299a57f26d196be23343da438\": container with ID starting with 28f3963cced1832e2edc22686aa1ad57b6572f4299a57f26d196be23343da438 not found: ID does not exist"
Sep 16 19:11:56.409370 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:11:56.409335    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69lkn4w"]
Sep 16 19:11:56.414785 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:11:56.414756    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69lkn4w"]
Sep 16 19:11:57.434212 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:11:57.434176    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="80ec9a19-99f6-4bf0-90cb-593f6456c285" path="/var/lib/kubelet/pods/80ec9a19-99f6-4bf0-90cb-593f6456c285/volumes"
Sep 16 19:11:59.859694 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:11:59.859663    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-f4nkv"]
Sep 16 19:11:59.860119 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:11:59.860038    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="80ec9a19-99f6-4bf0-90cb-593f6456c285" containerName="main"
Sep 16 19:11:59.860119 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:11:59.860051    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="80ec9a19-99f6-4bf0-90cb-593f6456c285" containerName="main"
Sep 16 19:11:59.860119 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:11:59.860061    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="77617eaf-a7ee-4f26-9be8-16cc9815bbdd" containerName="main"
Sep 16 19:11:59.860119 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:11:59.860067    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="77617eaf-a7ee-4f26-9be8-16cc9815bbdd" containerName="main"
Sep 16 19:11:59.860273 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:11:59.860138    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="80ec9a19-99f6-4bf0-90cb-593f6456c285" containerName="main"
Sep 16 19:11:59.860273 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:11:59.860148    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="77617eaf-a7ee-4f26-9be8-16cc9815bbdd" containerName="main"
Sep 16 19:11:59.864328 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:11:59.864310    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-f4nkv"
Sep 16 19:11:59.866791 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:11:59.866765    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-force-stop-3e394e04\"/\"kube-root-ca.crt\""
Sep 16 19:11:59.867224 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:11:59.867206    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-force-stop-3e394e04\"/\"stop-v2-kserve-self-signed-certs\""
Sep 16 19:11:59.867528 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:11:59.867510    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-force-stop-3e394e04\"/\"default-dockercfg-bggzx\""
Sep 16 19:11:59.867643 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:11:59.867516    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-force-stop-3e394e04\"/\"openshift-service-ca.crt\""
Sep 16 19:11:59.873881 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:11:59.873855    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-f4nkv"]
Sep 16 19:11:59.901585 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:11:59.901550    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:11:59.930726 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:11:59.930688    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e6c1f6cc-4179-4c6e-b2c6-f7a78922854e-tmp-dir\") pod \"stop-v2-kserve-7f68747c45-f4nkv\" (UID: \"e6c1f6cc-4179-4c6e-b2c6-f7a78922854e\") " pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-f4nkv"
Sep 16 19:11:59.930887 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:11:59.930747    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e6c1f6cc-4179-4c6e-b2c6-f7a78922854e-home\") pod \"stop-v2-kserve-7f68747c45-f4nkv\" (UID: \"e6c1f6cc-4179-4c6e-b2c6-f7a78922854e\") " pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-f4nkv"
Sep 16 19:11:59.930887 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:11:59.930781    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e6c1f6cc-4179-4c6e-b2c6-f7a78922854e-tls-certs\") pod \"stop-v2-kserve-7f68747c45-f4nkv\" (UID: \"e6c1f6cc-4179-4c6e-b2c6-f7a78922854e\") " pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-f4nkv"
Sep 16 19:11:59.930887 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:11:59.930809    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e6c1f6cc-4179-4c6e-b2c6-f7a78922854e-model-cache\") pod \"stop-v2-kserve-7f68747c45-f4nkv\" (UID: \"e6c1f6cc-4179-4c6e-b2c6-f7a78922854e\") " pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-f4nkv"
Sep 16 19:11:59.930887 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:11:59.930832    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2zxn4\" (UniqueName: \"kubernetes.io/projected/e6c1f6cc-4179-4c6e-b2c6-f7a78922854e-kube-api-access-2zxn4\") pod \"stop-v2-kserve-7f68747c45-f4nkv\" (UID: \"e6c1f6cc-4179-4c6e-b2c6-f7a78922854e\") " pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-f4nkv"
Sep 16 19:11:59.930887 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:11:59.930878    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e6c1f6cc-4179-4c6e-b2c6-f7a78922854e-dshm\") pod \"stop-v2-kserve-7f68747c45-f4nkv\" (UID: \"e6c1f6cc-4179-4c6e-b2c6-f7a78922854e\") " pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-f4nkv"
Sep 16 19:12:00.014682 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:12:00.014639    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:12:00.032137 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:12:00.032091    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e6c1f6cc-4179-4c6e-b2c6-f7a78922854e-dshm\") pod \"stop-v2-kserve-7f68747c45-f4nkv\" (UID: \"e6c1f6cc-4179-4c6e-b2c6-f7a78922854e\") " pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-f4nkv"
Sep 16 19:12:00.032313 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:12:00.032179    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e6c1f6cc-4179-4c6e-b2c6-f7a78922854e-tmp-dir\") pod \"stop-v2-kserve-7f68747c45-f4nkv\" (UID: \"e6c1f6cc-4179-4c6e-b2c6-f7a78922854e\") " pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-f4nkv"
Sep 16 19:12:00.032313 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:12:00.032235    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e6c1f6cc-4179-4c6e-b2c6-f7a78922854e-home\") pod \"stop-v2-kserve-7f68747c45-f4nkv\" (UID: \"e6c1f6cc-4179-4c6e-b2c6-f7a78922854e\") " pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-f4nkv"
Sep 16 19:12:00.032313 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:12:00.032274    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e6c1f6cc-4179-4c6e-b2c6-f7a78922854e-tls-certs\") pod \"stop-v2-kserve-7f68747c45-f4nkv\" (UID: \"e6c1f6cc-4179-4c6e-b2c6-f7a78922854e\") " pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-f4nkv"
Sep 16 19:12:00.032497 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:12:00.032311    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e6c1f6cc-4179-4c6e-b2c6-f7a78922854e-model-cache\") pod \"stop-v2-kserve-7f68747c45-f4nkv\" (UID: \"e6c1f6cc-4179-4c6e-b2c6-f7a78922854e\") " pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-f4nkv"
Sep 16 19:12:00.032497 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:12:00.032338    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2zxn4\" (UniqueName: \"kubernetes.io/projected/e6c1f6cc-4179-4c6e-b2c6-f7a78922854e-kube-api-access-2zxn4\") pod \"stop-v2-kserve-7f68747c45-f4nkv\" (UID: \"e6c1f6cc-4179-4c6e-b2c6-f7a78922854e\") " pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-f4nkv"
Sep 16 19:12:00.032660 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:12:00.032637    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e6c1f6cc-4179-4c6e-b2c6-f7a78922854e-tmp-dir\") pod \"stop-v2-kserve-7f68747c45-f4nkv\" (UID: \"e6c1f6cc-4179-4c6e-b2c6-f7a78922854e\") " pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-f4nkv"
Sep 16 19:12:00.032744 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:12:00.032675    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e6c1f6cc-4179-4c6e-b2c6-f7a78922854e-model-cache\") pod \"stop-v2-kserve-7f68747c45-f4nkv\" (UID: \"e6c1f6cc-4179-4c6e-b2c6-f7a78922854e\") " pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-f4nkv"
Sep 16 19:12:00.032744 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:12:00.032691    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e6c1f6cc-4179-4c6e-b2c6-f7a78922854e-home\") pod \"stop-v2-kserve-7f68747c45-f4nkv\" (UID: \"e6c1f6cc-4179-4c6e-b2c6-f7a78922854e\") " pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-f4nkv"
Sep 16 19:12:00.034630 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:12:00.034607    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e6c1f6cc-4179-4c6e-b2c6-f7a78922854e-dshm\") pod \"stop-v2-kserve-7f68747c45-f4nkv\" (UID: \"e6c1f6cc-4179-4c6e-b2c6-f7a78922854e\") " pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-f4nkv"
Sep 16 19:12:00.034889 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:12:00.034872    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e6c1f6cc-4179-4c6e-b2c6-f7a78922854e-tls-certs\") pod \"stop-v2-kserve-7f68747c45-f4nkv\" (UID: \"e6c1f6cc-4179-4c6e-b2c6-f7a78922854e\") " pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-f4nkv"
Sep 16 19:12:00.047200 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:12:00.047170    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2zxn4\" (UniqueName: \"kubernetes.io/projected/e6c1f6cc-4179-4c6e-b2c6-f7a78922854e-kube-api-access-2zxn4\") pod \"stop-v2-kserve-7f68747c45-f4nkv\" (UID: \"e6c1f6cc-4179-4c6e-b2c6-f7a78922854e\") " pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-f4nkv"
Sep 16 19:12:00.176059 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:12:00.176021    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-f4nkv"
Sep 16 19:12:00.253506 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:12:00.253473    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:12:00.395532 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:12:00.395503    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:12:00.469324 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:12:00.469294    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-f4nkv"]
Sep 16 19:12:00.471410 ip-10-0-135-119 kubenswrapper[2578]: W0916 19:12:00.471381    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode6c1f6cc_4179_4c6e_b2c6_f7a78922854e.slice/crio-8a8bc581f705f5e0275b8b5c0eb8249edea4188781e99101a2055239a07f8a6b WatchSource:0}: Error finding container 8a8bc581f705f5e0275b8b5c0eb8249edea4188781e99101a2055239a07f8a6b: Status 404 returned error can't find the container with id 8a8bc581f705f5e0275b8b5c0eb8249edea4188781e99101a2055239a07f8a6b
Sep 16 19:12:01.414162 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:12:01.414113    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-f4nkv" event={"ID":"e6c1f6cc-4179-4c6e-b2c6-f7a78922854e","Type":"ContainerStarted","Data":"110e2b5d6d68380646070d46e1bf19e3b939c021f51896d7d15d54b323a246c2"}
Sep 16 19:12:01.414555 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:12:01.414180    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-f4nkv" event={"ID":"e6c1f6cc-4179-4c6e-b2c6-f7a78922854e","Type":"ContainerStarted","Data":"8a8bc581f705f5e0275b8b5c0eb8249edea4188781e99101a2055239a07f8a6b"}
Sep 16 19:12:01.436596 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:12:01.436540    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-f4nkv" podStartSLOduration=2.436522454 podStartE2EDuration="2.436522454s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:11:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:12:01.435020667 +0000 UTC m=+1902.501512175" watchObservedRunningTime="2026-09-16 19:12:01.436522454 +0000 UTC m=+1902.503013942"
Sep 16 19:12:02.177230 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:12:02.177187    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-f4nkv"
Sep 16 19:12:06.577078 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:12:06.577038    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:12:06.672260 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:12:06.672230    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:12:06.998790 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:12:06.998741    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:12:10.176634 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:12:10.176597    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-f4nkv"
Sep 16 19:12:10.189041 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:12:10.189014    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-f4nkv"
Sep 16 19:12:10.471076 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:12:10.470994    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-f4nkv"
Sep 16 19:12:13.054292 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:12:13.054257    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:12:13.130405 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:12:13.130363    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:12:45.685350 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:12:45.685292    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:12:45.949617 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:12:45.949518    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-57857cgn7"]
Sep 16 19:12:45.953994 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:12:45.953945    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-57857cgn7"
Sep 16 19:12:45.957563 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:12:45.957528    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"default-dockercfg-tl57x\""
Sep 16 19:12:45.957721 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:12:45.957599    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"openshift-service-ca.crt\""
Sep 16 19:12:45.957721 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:12:45.957528    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"tokenizer-migration-path-test-epp-sa-dockercfg-z7rhw\""
Sep 16 19:12:45.957837 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:12:45.957528    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"tokenizer-migration-path-test-kserve-self-signed-certs\""
Sep 16 19:12:45.957873 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:12:45.957530    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"kube-root-ca.crt\""
Sep 16 19:12:45.964323 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:12:45.964294    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-57857cgn7"]
Sep 16 19:12:46.048532 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:12:46.048491    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e5612738-0c09-4b4c-90f3-e1490ad9e4ea-tls-certs\") pod \"tokenizer-migration-path-test-kserve-router-scheduler-57857cgn7\" (UID: \"e5612738-0c09-4b4c-90f3-e1490ad9e4ea\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-57857cgn7"
Sep 16 19:12:46.048741 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:12:46.048615    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4kxss\" (UniqueName: \"kubernetes.io/projected/e5612738-0c09-4b4c-90f3-e1490ad9e4ea-kube-api-access-4kxss\") pod \"tokenizer-migration-path-test-kserve-router-scheduler-57857cgn7\" (UID: \"e5612738-0c09-4b4c-90f3-e1490ad9e4ea\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-57857cgn7"
Sep 16 19:12:46.150076 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:12:46.150034    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e5612738-0c09-4b4c-90f3-e1490ad9e4ea-tls-certs\") pod \"tokenizer-migration-path-test-kserve-router-scheduler-57857cgn7\" (UID: \"e5612738-0c09-4b4c-90f3-e1490ad9e4ea\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-57857cgn7"
Sep 16 19:12:46.150283 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:12:46.150105    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4kxss\" (UniqueName: \"kubernetes.io/projected/e5612738-0c09-4b4c-90f3-e1490ad9e4ea-kube-api-access-4kxss\") pod \"tokenizer-migration-path-test-kserve-router-scheduler-57857cgn7\" (UID: \"e5612738-0c09-4b4c-90f3-e1490ad9e4ea\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-57857cgn7"
Sep 16 19:12:46.152868 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:12:46.152838    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e5612738-0c09-4b4c-90f3-e1490ad9e4ea-tls-certs\") pod \"tokenizer-migration-path-test-kserve-router-scheduler-57857cgn7\" (UID: \"e5612738-0c09-4b4c-90f3-e1490ad9e4ea\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-57857cgn7"
Sep 16 19:12:46.163055 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:12:46.163020    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4kxss\" (UniqueName: \"kubernetes.io/projected/e5612738-0c09-4b4c-90f3-e1490ad9e4ea-kube-api-access-4kxss\") pod \"tokenizer-migration-path-test-kserve-router-scheduler-57857cgn7\" (UID: \"e5612738-0c09-4b4c-90f3-e1490ad9e4ea\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-57857cgn7"
Sep 16 19:12:46.265708 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:12:46.265586    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-57857cgn7"
Sep 16 19:12:46.545857 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:12:46.545828    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-57857cgn7"]
Sep 16 19:12:46.548505 ip-10-0-135-119 kubenswrapper[2578]: W0916 19:12:46.548474    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode5612738_0c09_4b4c_90f3_e1490ad9e4ea.slice/crio-6dce08154c7992f395a75798b54e340ea9541a8a41564bc9744387d855af12b8 WatchSource:0}: Error finding container 6dce08154c7992f395a75798b54e340ea9541a8a41564bc9744387d855af12b8: Status 404 returned error can't find the container with id 6dce08154c7992f395a75798b54e340ea9541a8a41564bc9744387d855af12b8
Sep 16 19:12:46.639355 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:12:46.639318    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-57857cgn7" event={"ID":"e5612738-0c09-4b4c-90f3-e1490ad9e4ea","Type":"ContainerStarted","Data":"f7bc2d4ee9a65500e8aa371bc64b010f76015ff32e09a25cdc6a9e0ed2608352"}
Sep 16 19:12:46.639529 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:12:46.639384    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-57857cgn7" event={"ID":"e5612738-0c09-4b4c-90f3-e1490ad9e4ea","Type":"ContainerStarted","Data":"6dce08154c7992f395a75798b54e340ea9541a8a41564bc9744387d855af12b8"}
Sep 16 19:12:46.639529 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:12:46.639411    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-57857cgn7"
Sep 16 19:12:46.662267 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:12:46.662189    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-57857cgn7" podStartSLOduration=1.662166691 podStartE2EDuration="1.662166691s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:12:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:12:46.661362172 +0000 UTC m=+1947.727853697" watchObservedRunningTime="2026-09-16 19:12:46.662166691 +0000 UTC m=+1947.728658180"
Sep 16 19:12:47.598773 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:12:47.598711    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:13:09.278124 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:13:09.278079    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:13:12.028504 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:13:12.028417    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-f4nkv"]
Sep 16 19:13:12.052973 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:13:12.052927    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:13:17.647045 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:13:17.647014    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-57857cgn7"
Sep 16 19:13:27.048594 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:13:27.048501    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-f4nkv" podUID="e6c1f6cc-4179-4c6e-b2c6-f7a78922854e" containerName="main" containerID="cri-o://110e2b5d6d68380646070d46e1bf19e3b939c021f51896d7d15d54b323a246c2" gracePeriod=45
Sep 16 19:13:27.309119 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:13:27.309052    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-f4nkv"
Sep 16 19:13:27.416735 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:13:27.416695    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e6c1f6cc-4179-4c6e-b2c6-f7a78922854e-tmp-dir\") pod \"e6c1f6cc-4179-4c6e-b2c6-f7a78922854e\" (UID: \"e6c1f6cc-4179-4c6e-b2c6-f7a78922854e\") "
Sep 16 19:13:27.416941 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:13:27.416748    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e6c1f6cc-4179-4c6e-b2c6-f7a78922854e-dshm\") pod \"e6c1f6cc-4179-4c6e-b2c6-f7a78922854e\" (UID: \"e6c1f6cc-4179-4c6e-b2c6-f7a78922854e\") "
Sep 16 19:13:27.416941 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:13:27.416778    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e6c1f6cc-4179-4c6e-b2c6-f7a78922854e-model-cache\") pod \"e6c1f6cc-4179-4c6e-b2c6-f7a78922854e\" (UID: \"e6c1f6cc-4179-4c6e-b2c6-f7a78922854e\") "
Sep 16 19:13:27.416941 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:13:27.416850    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2zxn4\" (UniqueName: \"kubernetes.io/projected/e6c1f6cc-4179-4c6e-b2c6-f7a78922854e-kube-api-access-2zxn4\") pod \"e6c1f6cc-4179-4c6e-b2c6-f7a78922854e\" (UID: \"e6c1f6cc-4179-4c6e-b2c6-f7a78922854e\") "
Sep 16 19:13:27.416941 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:13:27.416894    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e6c1f6cc-4179-4c6e-b2c6-f7a78922854e-tls-certs\") pod \"e6c1f6cc-4179-4c6e-b2c6-f7a78922854e\" (UID: \"e6c1f6cc-4179-4c6e-b2c6-f7a78922854e\") "
Sep 16 19:13:27.417199 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:13:27.416963    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e6c1f6cc-4179-4c6e-b2c6-f7a78922854e-home\") pod \"e6c1f6cc-4179-4c6e-b2c6-f7a78922854e\" (UID: \"e6c1f6cc-4179-4c6e-b2c6-f7a78922854e\") "
Sep 16 19:13:27.417199 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:13:27.417092    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e6c1f6cc-4179-4c6e-b2c6-f7a78922854e-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "e6c1f6cc-4179-4c6e-b2c6-f7a78922854e" (UID: "e6c1f6cc-4179-4c6e-b2c6-f7a78922854e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:13:27.417199 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:13:27.417108    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e6c1f6cc-4179-4c6e-b2c6-f7a78922854e-model-cache" (OuterVolumeSpecName: "model-cache") pod "e6c1f6cc-4179-4c6e-b2c6-f7a78922854e" (UID: "e6c1f6cc-4179-4c6e-b2c6-f7a78922854e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:13:27.417323 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:13:27.417299    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e6c1f6cc-4179-4c6e-b2c6-f7a78922854e-home" (OuterVolumeSpecName: "home") pod "e6c1f6cc-4179-4c6e-b2c6-f7a78922854e" (UID: "e6c1f6cc-4179-4c6e-b2c6-f7a78922854e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:13:27.417388 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:13:27.417355    2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e6c1f6cc-4179-4c6e-b2c6-f7a78922854e-tmp-dir\") on node \"ip-10-0-135-119.ec2.internal\" DevicePath \"\""
Sep 16 19:13:27.417388 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:13:27.417370    2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e6c1f6cc-4179-4c6e-b2c6-f7a78922854e-model-cache\") on node \"ip-10-0-135-119.ec2.internal\" DevicePath \"\""
Sep 16 19:13:27.419314 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:13:27.419287    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e6c1f6cc-4179-4c6e-b2c6-f7a78922854e-kube-api-access-2zxn4" (OuterVolumeSpecName: "kube-api-access-2zxn4") pod "e6c1f6cc-4179-4c6e-b2c6-f7a78922854e" (UID: "e6c1f6cc-4179-4c6e-b2c6-f7a78922854e"). InnerVolumeSpecName "kube-api-access-2zxn4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:13:27.419419 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:13:27.419399    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e6c1f6cc-4179-4c6e-b2c6-f7a78922854e-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "e6c1f6cc-4179-4c6e-b2c6-f7a78922854e" (UID: "e6c1f6cc-4179-4c6e-b2c6-f7a78922854e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:13:27.419419 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:13:27.419397    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e6c1f6cc-4179-4c6e-b2c6-f7a78922854e-dshm" (OuterVolumeSpecName: "dshm") pod "e6c1f6cc-4179-4c6e-b2c6-f7a78922854e" (UID: "e6c1f6cc-4179-4c6e-b2c6-f7a78922854e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:13:27.517865 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:13:27.517822    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2zxn4\" (UniqueName: \"kubernetes.io/projected/e6c1f6cc-4179-4c6e-b2c6-f7a78922854e-kube-api-access-2zxn4\") on node \"ip-10-0-135-119.ec2.internal\" DevicePath \"\""
Sep 16 19:13:27.517865 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:13:27.517857    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e6c1f6cc-4179-4c6e-b2c6-f7a78922854e-tls-certs\") on node \"ip-10-0-135-119.ec2.internal\" DevicePath \"\""
Sep 16 19:13:27.517865 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:13:27.517867    2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e6c1f6cc-4179-4c6e-b2c6-f7a78922854e-home\") on node \"ip-10-0-135-119.ec2.internal\" DevicePath \"\""
Sep 16 19:13:27.517865 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:13:27.517874    2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e6c1f6cc-4179-4c6e-b2c6-f7a78922854e-dshm\") on node \"ip-10-0-135-119.ec2.internal\" DevicePath \"\""
Sep 16 19:13:27.835482 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:13:27.835449    2578 generic.go:356] "Generic (PLEG): container finished" podID="e6c1f6cc-4179-4c6e-b2c6-f7a78922854e" containerID="110e2b5d6d68380646070d46e1bf19e3b939c021f51896d7d15d54b323a246c2" exitCode=0
Sep 16 19:13:27.835674 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:13:27.835510    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-f4nkv"
Sep 16 19:13:27.835674 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:13:27.835546    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-f4nkv" event={"ID":"e6c1f6cc-4179-4c6e-b2c6-f7a78922854e","Type":"ContainerDied","Data":"110e2b5d6d68380646070d46e1bf19e3b939c021f51896d7d15d54b323a246c2"}
Sep 16 19:13:27.835674 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:13:27.835600    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-f4nkv" event={"ID":"e6c1f6cc-4179-4c6e-b2c6-f7a78922854e","Type":"ContainerDied","Data":"8a8bc581f705f5e0275b8b5c0eb8249edea4188781e99101a2055239a07f8a6b"}
Sep 16 19:13:27.835674 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:13:27.835624    2578 scope.go:117] "RemoveContainer" containerID="110e2b5d6d68380646070d46e1bf19e3b939c021f51896d7d15d54b323a246c2"
Sep 16 19:13:27.844507 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:13:27.844487    2578 scope.go:117] "RemoveContainer" containerID="110e2b5d6d68380646070d46e1bf19e3b939c021f51896d7d15d54b323a246c2"
Sep 16 19:13:27.844836 ip-10-0-135-119 kubenswrapper[2578]: E0916 19:13:27.844815    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"110e2b5d6d68380646070d46e1bf19e3b939c021f51896d7d15d54b323a246c2\": container with ID starting with 110e2b5d6d68380646070d46e1bf19e3b939c021f51896d7d15d54b323a246c2 not found: ID does not exist" containerID="110e2b5d6d68380646070d46e1bf19e3b939c021f51896d7d15d54b323a246c2"
Sep 16 19:13:27.844885 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:13:27.844845    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"110e2b5d6d68380646070d46e1bf19e3b939c021f51896d7d15d54b323a246c2"} err="failed to get container status \"110e2b5d6d68380646070d46e1bf19e3b939c021f51896d7d15d54b323a246c2\": rpc error: code = NotFound desc = could not find container \"110e2b5d6d68380646070d46e1bf19e3b939c021f51896d7d15d54b323a246c2\": container with ID starting with 110e2b5d6d68380646070d46e1bf19e3b939c021f51896d7d15d54b323a246c2 not found: ID does not exist"
Sep 16 19:13:27.853310 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:13:27.853281    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-f4nkv"]
Sep 16 19:13:27.857917 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:13:27.857889    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-f4nkv"]
Sep 16 19:13:29.434338 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:13:29.434300    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e6c1f6cc-4179-4c6e-b2c6-f7a78922854e" path="/var/lib/kubelet/pods/e6c1f6cc-4179-4c6e-b2c6-f7a78922854e/volumes"
Sep 16 19:13:32.077961 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:13:32.077925    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:13:32.197902 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:13:32.197851    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:13:32.297024 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:13:32.296963    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:13:36.511480 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:13:36.511437    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:13:36.546824 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:13:36.546786    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-57857cgn7"]
Sep 16 19:13:36.605621 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:13:36.605585    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:13:37.250546 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:13:37.250513    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:13:37.646907 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:13:37.646867    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-57857cgn7" podUID="e5612738-0c09-4b4c-90f3-e1490ad9e4ea" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:13:37.757095 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:13:37.757058    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:13:45.779135 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:13:45.779093    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:13:47.646814 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:13:47.646759    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-57857cgn7" podUID="e5612738-0c09-4b4c-90f3-e1490ad9e4ea" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:13:51.567379 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:13:51.567317    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-57857cgn7" podUID="e5612738-0c09-4b4c-90f3-e1490ad9e4ea" containerName="main" containerID="cri-o://f7bc2d4ee9a65500e8aa371bc64b010f76015ff32e09a25cdc6a9e0ed2608352" gracePeriod=45
Sep 16 19:13:57.647105 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:13:57.647064    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-57857cgn7" podUID="e5612738-0c09-4b4c-90f3-e1490ad9e4ea" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:13:57.647577 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:13:57.647169    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-57857cgn7"
Sep 16 19:14:00.820841 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:14:00.820803    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:14:07.646760 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:14:07.646714    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-57857cgn7" podUID="e5612738-0c09-4b4c-90f3-e1490ad9e4ea" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:14:14.711113 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:14:14.711057    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:14:17.646626 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:14:17.646583    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-57857cgn7" podUID="e5612738-0c09-4b4c-90f3-e1490ad9e4ea" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:14:21.069716 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:14:21.069685    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:14:21.201032 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:14:21.200991    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:14:21.823752 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:14:21.823725    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-57857cgn7"
Sep 16 19:14:22.014810 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:14:22.014774    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e5612738-0c09-4b4c-90f3-e1490ad9e4ea-tls-certs\") pod \"e5612738-0c09-4b4c-90f3-e1490ad9e4ea\" (UID: \"e5612738-0c09-4b4c-90f3-e1490ad9e4ea\") "
Sep 16 19:14:22.015032 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:14:22.014820    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4kxss\" (UniqueName: \"kubernetes.io/projected/e5612738-0c09-4b4c-90f3-e1490ad9e4ea-kube-api-access-4kxss\") pod \"e5612738-0c09-4b4c-90f3-e1490ad9e4ea\" (UID: \"e5612738-0c09-4b4c-90f3-e1490ad9e4ea\") "
Sep 16 19:14:22.017270 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:14:22.017238    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e5612738-0c09-4b4c-90f3-e1490ad9e4ea-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "e5612738-0c09-4b4c-90f3-e1490ad9e4ea" (UID: "e5612738-0c09-4b4c-90f3-e1490ad9e4ea"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:14:22.017383 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:14:22.017337    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e5612738-0c09-4b4c-90f3-e1490ad9e4ea-kube-api-access-4kxss" (OuterVolumeSpecName: "kube-api-access-4kxss") pod "e5612738-0c09-4b4c-90f3-e1490ad9e4ea" (UID: "e5612738-0c09-4b4c-90f3-e1490ad9e4ea"). InnerVolumeSpecName "kube-api-access-4kxss". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:14:22.100965 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:14:22.100924    2578 generic.go:356] "Generic (PLEG): container finished" podID="e5612738-0c09-4b4c-90f3-e1490ad9e4ea" containerID="f7bc2d4ee9a65500e8aa371bc64b010f76015ff32e09a25cdc6a9e0ed2608352" exitCode=0
Sep 16 19:14:22.100965 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:14:22.100991    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-57857cgn7"
Sep 16 19:14:22.101465 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:14:22.101003    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-57857cgn7" event={"ID":"e5612738-0c09-4b4c-90f3-e1490ad9e4ea","Type":"ContainerDied","Data":"f7bc2d4ee9a65500e8aa371bc64b010f76015ff32e09a25cdc6a9e0ed2608352"}
Sep 16 19:14:22.101465 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:14:22.101058    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-57857cgn7" event={"ID":"e5612738-0c09-4b4c-90f3-e1490ad9e4ea","Type":"ContainerDied","Data":"6dce08154c7992f395a75798b54e340ea9541a8a41564bc9744387d855af12b8"}
Sep 16 19:14:22.101465 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:14:22.101088    2578 scope.go:117] "RemoveContainer" containerID="f7bc2d4ee9a65500e8aa371bc64b010f76015ff32e09a25cdc6a9e0ed2608352"
Sep 16 19:14:22.110295 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:14:22.110278    2578 scope.go:117] "RemoveContainer" containerID="f7bc2d4ee9a65500e8aa371bc64b010f76015ff32e09a25cdc6a9e0ed2608352"
Sep 16 19:14:22.110609 ip-10-0-135-119 kubenswrapper[2578]: E0916 19:14:22.110585    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f7bc2d4ee9a65500e8aa371bc64b010f76015ff32e09a25cdc6a9e0ed2608352\": container with ID starting with f7bc2d4ee9a65500e8aa371bc64b010f76015ff32e09a25cdc6a9e0ed2608352 not found: ID does not exist" containerID="f7bc2d4ee9a65500e8aa371bc64b010f76015ff32e09a25cdc6a9e0ed2608352"
Sep 16 19:14:22.110707 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:14:22.110616    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f7bc2d4ee9a65500e8aa371bc64b010f76015ff32e09a25cdc6a9e0ed2608352"} err="failed to get container status \"f7bc2d4ee9a65500e8aa371bc64b010f76015ff32e09a25cdc6a9e0ed2608352\": rpc error: code = NotFound desc = could not find container \"f7bc2d4ee9a65500e8aa371bc64b010f76015ff32e09a25cdc6a9e0ed2608352\": container with ID starting with f7bc2d4ee9a65500e8aa371bc64b010f76015ff32e09a25cdc6a9e0ed2608352 not found: ID does not exist"
Sep 16 19:14:22.116105 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:14:22.116084    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e5612738-0c09-4b4c-90f3-e1490ad9e4ea-tls-certs\") on node \"ip-10-0-135-119.ec2.internal\" DevicePath \"\""
Sep 16 19:14:22.116105 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:14:22.116105    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4kxss\" (UniqueName: \"kubernetes.io/projected/e5612738-0c09-4b4c-90f3-e1490ad9e4ea-kube-api-access-4kxss\") on node \"ip-10-0-135-119.ec2.internal\" DevicePath \"\""
Sep 16 19:14:22.124830 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:14:22.124807    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-57857cgn7"]
Sep 16 19:14:22.129910 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:14:22.129888    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-57857cgn7"]
Sep 16 19:14:23.434276 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:14:23.434238    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e5612738-0c09-4b4c-90f3-e1490ad9e4ea" path="/var/lib/kubelet/pods/e5612738-0c09-4b4c-90f3-e1490ad9e4ea/volumes"
Sep 16 19:14:24.232863 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:14:24.232831    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:14:24.329214 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:14:24.329174    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:14:31.921534 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:14:31.921500    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:14:36.172776 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:14:36.172741    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:14:36.419334 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:14:36.419297    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7ddzshp"]
Sep 16 19:14:36.419728 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:14:36.419709    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e6c1f6cc-4179-4c6e-b2c6-f7a78922854e" containerName="main"
Sep 16 19:14:36.419815 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:14:36.419730    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="e6c1f6cc-4179-4c6e-b2c6-f7a78922854e" containerName="main"
Sep 16 19:14:36.419815 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:14:36.419790    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e5612738-0c09-4b4c-90f3-e1490ad9e4ea" containerName="main"
Sep 16 19:14:36.419815 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:14:36.419801    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="e5612738-0c09-4b4c-90f3-e1490ad9e4ea" containerName="main"
Sep 16 19:14:36.419969 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:14:36.419894    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="e6c1f6cc-4179-4c6e-b2c6-f7a78922854e" containerName="main"
Sep 16 19:14:36.419969 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:14:36.419911    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="e5612738-0c09-4b4c-90f3-e1490ad9e4ea" containerName="main"
Sep 16 19:14:36.423303 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:14:36.423241    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7ddzshp"
Sep 16 19:14:36.426538 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:14:36.426513    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-0eb24714\"/\"openshift-service-ca.crt\""
Sep 16 19:14:36.426538 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:14:36.426529    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-0eb24714\"/\"tokenizer-template-args-test-kserve-self-signed-certs\""
Sep 16 19:14:36.426755 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:14:36.426587    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-0eb24714\"/\"kube-root-ca.crt\""
Sep 16 19:14:36.426755 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:14:36.426531    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-0eb24714\"/\"default-dockercfg-x7z49\""
Sep 16 19:14:36.426755 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:14:36.426598    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-0eb24714\"/\"tokenizer-template-args-test-epp-sa-dockercfg-wrmjd\""
Sep 16 19:14:36.435250 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:14:36.435225    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7ddzshp"]
Sep 16 19:14:36.443415 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:14:36.443389    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:14:36.535039 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:14:36.535000    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3efb0c8c-a9be-4997-97f1-4c435faf9ad4-tls-certs\") pod \"tokenizer-template-args-test-kserve-router-scheduler-8cd7ddzshp\" (UID: \"3efb0c8c-a9be-4997-97f1-4c435faf9ad4\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7ddzshp"
Sep 16 19:14:36.535039 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:14:36.535036    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s8scz\" (UniqueName: \"kubernetes.io/projected/3efb0c8c-a9be-4997-97f1-4c435faf9ad4-kube-api-access-s8scz\") pod \"tokenizer-template-args-test-kserve-router-scheduler-8cd7ddzshp\" (UID: \"3efb0c8c-a9be-4997-97f1-4c435faf9ad4\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7ddzshp"
Sep 16 19:14:36.539104 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:14:36.539076    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:14:36.635943 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:14:36.635900    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3efb0c8c-a9be-4997-97f1-4c435faf9ad4-tls-certs\") pod \"tokenizer-template-args-test-kserve-router-scheduler-8cd7ddzshp\" (UID: \"3efb0c8c-a9be-4997-97f1-4c435faf9ad4\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7ddzshp"
Sep 16 19:14:36.635943 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:14:36.635944    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s8scz\" (UniqueName: \"kubernetes.io/projected/3efb0c8c-a9be-4997-97f1-4c435faf9ad4-kube-api-access-s8scz\") pod \"tokenizer-template-args-test-kserve-router-scheduler-8cd7ddzshp\" (UID: \"3efb0c8c-a9be-4997-97f1-4c435faf9ad4\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7ddzshp"
Sep 16 19:14:36.638770 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:14:36.638740    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3efb0c8c-a9be-4997-97f1-4c435faf9ad4-tls-certs\") pod \"tokenizer-template-args-test-kserve-router-scheduler-8cd7ddzshp\" (UID: \"3efb0c8c-a9be-4997-97f1-4c435faf9ad4\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7ddzshp"
Sep 16 19:14:36.645278 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:14:36.645247    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s8scz\" (UniqueName: \"kubernetes.io/projected/3efb0c8c-a9be-4997-97f1-4c435faf9ad4-kube-api-access-s8scz\") pod \"tokenizer-template-args-test-kserve-router-scheduler-8cd7ddzshp\" (UID: \"3efb0c8c-a9be-4997-97f1-4c435faf9ad4\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7ddzshp"
Sep 16 19:14:36.735363 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:14:36.735274    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7ddzshp"
Sep 16 19:14:37.010923 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:14:37.010892    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7ddzshp"]
Sep 16 19:14:37.013872 ip-10-0-135-119 kubenswrapper[2578]: W0916 19:14:37.013839    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3efb0c8c_a9be_4997_97f1_4c435faf9ad4.slice/crio-1b6b5b30720d0b9ba39e5d27afd8b9f2fb7cab308b6aa27256ec0ced515e7e28 WatchSource:0}: Error finding container 1b6b5b30720d0b9ba39e5d27afd8b9f2fb7cab308b6aa27256ec0ced515e7e28: Status 404 returned error can't find the container with id 1b6b5b30720d0b9ba39e5d27afd8b9f2fb7cab308b6aa27256ec0ced515e7e28
Sep 16 19:14:37.177091 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:14:37.177052    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7ddzshp" event={"ID":"3efb0c8c-a9be-4997-97f1-4c435faf9ad4","Type":"ContainerStarted","Data":"71224d83b1daa1049f39bc050304ee122b5c916c43e87f80228e794c61e1a3a0"}
Sep 16 19:14:37.177580 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:14:37.177105    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7ddzshp" event={"ID":"3efb0c8c-a9be-4997-97f1-4c435faf9ad4","Type":"ContainerStarted","Data":"1b6b5b30720d0b9ba39e5d27afd8b9f2fb7cab308b6aa27256ec0ced515e7e28"}
Sep 16 19:14:37.177580 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:14:37.177185    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7ddzshp"
Sep 16 19:14:37.209071 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:14:37.209010    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7ddzshp" podStartSLOduration=1.208992507 podStartE2EDuration="1.208992507s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:14:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:14:37.206854905 +0000 UTC m=+2058.273346414" watchObservedRunningTime="2026-09-16 19:14:37.208992507 +0000 UTC m=+2058.275483986"
Sep 16 19:14:38.742044 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:14:38.741958    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:14:38.841120 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:14:38.841080    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:14:40.590764 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:14:40.590620    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:14:47.615215 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:14:47.615175    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:14:57.616559 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:14:57.616521    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:14:57.743784 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:14:57.743751    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:14:57.865123 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:14:57.865063    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:15:02.690065 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:15:02.690022    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:15:08.184217 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:15:08.184186    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7ddzshp"
Sep 16 19:15:14.473490 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:15:14.473456    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-zjsmx"]
Sep 16 19:15:14.477069 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:15:14.477051    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-zjsmx"
Sep 16 19:15:14.480676 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:15:14.480641    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-leave-group-5b69d57a\"/\"openshift-service-ca.crt\""
Sep 16 19:15:14.480944 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:15:14.480922    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-leave-group-5b69d57a\"/\"leave-v1-kserve-self-signed-certs\""
Sep 16 19:15:14.481267 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:15:14.481068    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-leave-group-5b69d57a\"/\"kube-root-ca.crt\""
Sep 16 19:15:14.481267 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:15:14.481190    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-leave-group-5b69d57a\"/\"default-dockercfg-lmnrm\""
Sep 16 19:15:14.488258 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:15:14.488233    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-zjsmx"]
Sep 16 19:15:14.557182 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:15:14.557130    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:15:14.586473 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:15:14.586432    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1067ca9f-7d6b-4d46-98b0-cce3cf2a9ef9-home\") pod \"leave-v1-kserve-65684d4f5-zjsmx\" (UID: \"1067ca9f-7d6b-4d46-98b0-cce3cf2a9ef9\") " pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-zjsmx"
Sep 16 19:15:14.586657 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:15:14.586516    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1067ca9f-7d6b-4d46-98b0-cce3cf2a9ef9-dshm\") pod \"leave-v1-kserve-65684d4f5-zjsmx\" (UID: \"1067ca9f-7d6b-4d46-98b0-cce3cf2a9ef9\") " pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-zjsmx"
Sep 16 19:15:14.586657 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:15:14.586571    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1067ca9f-7d6b-4d46-98b0-cce3cf2a9ef9-tls-certs\") pod \"leave-v1-kserve-65684d4f5-zjsmx\" (UID: \"1067ca9f-7d6b-4d46-98b0-cce3cf2a9ef9\") " pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-zjsmx"
Sep 16 19:15:14.586657 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:15:14.586593    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1067ca9f-7d6b-4d46-98b0-cce3cf2a9ef9-model-cache\") pod \"leave-v1-kserve-65684d4f5-zjsmx\" (UID: \"1067ca9f-7d6b-4d46-98b0-cce3cf2a9ef9\") " pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-zjsmx"
Sep 16 19:15:14.586657 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:15:14.586621    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7f7lj\" (UniqueName: \"kubernetes.io/projected/1067ca9f-7d6b-4d46-98b0-cce3cf2a9ef9-kube-api-access-7f7lj\") pod \"leave-v1-kserve-65684d4f5-zjsmx\" (UID: \"1067ca9f-7d6b-4d46-98b0-cce3cf2a9ef9\") " pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-zjsmx"
Sep 16 19:15:14.586895 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:15:14.586691    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1067ca9f-7d6b-4d46-98b0-cce3cf2a9ef9-tmp-dir\") pod \"leave-v1-kserve-65684d4f5-zjsmx\" (UID: \"1067ca9f-7d6b-4d46-98b0-cce3cf2a9ef9\") " pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-zjsmx"
Sep 16 19:15:14.687613 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:15:14.687581    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1067ca9f-7d6b-4d46-98b0-cce3cf2a9ef9-tls-certs\") pod \"leave-v1-kserve-65684d4f5-zjsmx\" (UID: \"1067ca9f-7d6b-4d46-98b0-cce3cf2a9ef9\") " pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-zjsmx"
Sep 16 19:15:14.687785 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:15:14.687630    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1067ca9f-7d6b-4d46-98b0-cce3cf2a9ef9-model-cache\") pod \"leave-v1-kserve-65684d4f5-zjsmx\" (UID: \"1067ca9f-7d6b-4d46-98b0-cce3cf2a9ef9\") " pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-zjsmx"
Sep 16 19:15:14.687785 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:15:14.687672    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7f7lj\" (UniqueName: \"kubernetes.io/projected/1067ca9f-7d6b-4d46-98b0-cce3cf2a9ef9-kube-api-access-7f7lj\") pod \"leave-v1-kserve-65684d4f5-zjsmx\" (UID: \"1067ca9f-7d6b-4d46-98b0-cce3cf2a9ef9\") " pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-zjsmx"
Sep 16 19:15:14.687785 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:15:14.687710    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1067ca9f-7d6b-4d46-98b0-cce3cf2a9ef9-tmp-dir\") pod \"leave-v1-kserve-65684d4f5-zjsmx\" (UID: \"1067ca9f-7d6b-4d46-98b0-cce3cf2a9ef9\") " pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-zjsmx"
Sep 16 19:15:14.687785 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:15:14.687740    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1067ca9f-7d6b-4d46-98b0-cce3cf2a9ef9-home\") pod \"leave-v1-kserve-65684d4f5-zjsmx\" (UID: \"1067ca9f-7d6b-4d46-98b0-cce3cf2a9ef9\") " pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-zjsmx"
Sep 16 19:15:14.687967 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:15:14.687800    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1067ca9f-7d6b-4d46-98b0-cce3cf2a9ef9-dshm\") pod \"leave-v1-kserve-65684d4f5-zjsmx\" (UID: \"1067ca9f-7d6b-4d46-98b0-cce3cf2a9ef9\") " pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-zjsmx"
Sep 16 19:15:14.688236 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:15:14.688211    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1067ca9f-7d6b-4d46-98b0-cce3cf2a9ef9-tmp-dir\") pod \"leave-v1-kserve-65684d4f5-zjsmx\" (UID: \"1067ca9f-7d6b-4d46-98b0-cce3cf2a9ef9\") " pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-zjsmx"
Sep 16 19:15:14.688369 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:15:14.688269    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1067ca9f-7d6b-4d46-98b0-cce3cf2a9ef9-model-cache\") pod \"leave-v1-kserve-65684d4f5-zjsmx\" (UID: \"1067ca9f-7d6b-4d46-98b0-cce3cf2a9ef9\") " pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-zjsmx"
Sep 16 19:15:14.688369 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:15:14.688316    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1067ca9f-7d6b-4d46-98b0-cce3cf2a9ef9-home\") pod \"leave-v1-kserve-65684d4f5-zjsmx\" (UID: \"1067ca9f-7d6b-4d46-98b0-cce3cf2a9ef9\") " pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-zjsmx"
Sep 16 19:15:14.690159 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:15:14.690134    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1067ca9f-7d6b-4d46-98b0-cce3cf2a9ef9-dshm\") pod \"leave-v1-kserve-65684d4f5-zjsmx\" (UID: \"1067ca9f-7d6b-4d46-98b0-cce3cf2a9ef9\") " pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-zjsmx"
Sep 16 19:15:14.690439 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:15:14.690419    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1067ca9f-7d6b-4d46-98b0-cce3cf2a9ef9-tls-certs\") pod \"leave-v1-kserve-65684d4f5-zjsmx\" (UID: \"1067ca9f-7d6b-4d46-98b0-cce3cf2a9ef9\") " pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-zjsmx"
Sep 16 19:15:14.697179 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:15:14.697160    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7f7lj\" (UniqueName: \"kubernetes.io/projected/1067ca9f-7d6b-4d46-98b0-cce3cf2a9ef9-kube-api-access-7f7lj\") pod \"leave-v1-kserve-65684d4f5-zjsmx\" (UID: \"1067ca9f-7d6b-4d46-98b0-cce3cf2a9ef9\") " pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-zjsmx"
Sep 16 19:15:14.791483 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:15:14.791414    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-zjsmx"
Sep 16 19:15:15.078793 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:15:15.078628    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-zjsmx"]
Sep 16 19:15:15.081547 ip-10-0-135-119 kubenswrapper[2578]: W0916 19:15:15.081511    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1067ca9f_7d6b_4d46_98b0_cce3cf2a9ef9.slice/crio-342e0b06bddce18ad9e48abeb4d4902ed136b75b2c41284541cd21196704414e WatchSource:0}: Error finding container 342e0b06bddce18ad9e48abeb4d4902ed136b75b2c41284541cd21196704414e: Status 404 returned error can't find the container with id 342e0b06bddce18ad9e48abeb4d4902ed136b75b2c41284541cd21196704414e
Sep 16 19:15:15.368637 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:15:15.368536    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-zjsmx" event={"ID":"1067ca9f-7d6b-4d46-98b0-cce3cf2a9ef9","Type":"ContainerStarted","Data":"1b312ef2d8cce0f8b5a7d583d29604ffa3e201c903d192fff81c6c14c72daa07"}
Sep 16 19:15:15.368637 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:15:15.368597    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-zjsmx" event={"ID":"1067ca9f-7d6b-4d46-98b0-cce3cf2a9ef9","Type":"ContainerStarted","Data":"342e0b06bddce18ad9e48abeb4d4902ed136b75b2c41284541cd21196704414e"}
Sep 16 19:15:15.387089 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:15:15.387027    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-zjsmx" podStartSLOduration=1.38700573 podStartE2EDuration="1.38700573s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:15:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:15:15.386091325 +0000 UTC m=+2096.452582828" watchObservedRunningTime="2026-09-16 19:15:15.38700573 +0000 UTC m=+2096.453497213"
Sep 16 19:15:15.792219 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:15:15.792184    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-zjsmx"
Sep 16 19:15:19.340041 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:15:19.340001    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-9n8lj"]
Sep 16 19:15:19.343547 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:15:19.343524    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-9n8lj"
Sep 16 19:15:19.345530 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:15:19.345507    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-leave-group-5b69d57a\"/\"leave-v2-kserve-self-signed-certs\""
Sep 16 19:15:19.353970 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:15:19.353936    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-9n8lj"]
Sep 16 19:15:19.431422 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:15:19.431363    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/55e41cd7-3842-401f-b88a-3382deabe6b0-tmp-dir\") pod \"leave-v2-kserve-556b4dddf9-9n8lj\" (UID: \"55e41cd7-3842-401f-b88a-3382deabe6b0\") " pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-9n8lj"
Sep 16 19:15:19.431422 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:15:19.431409    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/55e41cd7-3842-401f-b88a-3382deabe6b0-dshm\") pod \"leave-v2-kserve-556b4dddf9-9n8lj\" (UID: \"55e41cd7-3842-401f-b88a-3382deabe6b0\") " pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-9n8lj"
Sep 16 19:15:19.431711 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:15:19.431461    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/55e41cd7-3842-401f-b88a-3382deabe6b0-model-cache\") pod \"leave-v2-kserve-556b4dddf9-9n8lj\" (UID: \"55e41cd7-3842-401f-b88a-3382deabe6b0\") " pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-9n8lj"
Sep 16 19:15:19.431711 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:15:19.431523    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n7qmv\" (UniqueName: \"kubernetes.io/projected/55e41cd7-3842-401f-b88a-3382deabe6b0-kube-api-access-n7qmv\") pod \"leave-v2-kserve-556b4dddf9-9n8lj\" (UID: \"55e41cd7-3842-401f-b88a-3382deabe6b0\") " pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-9n8lj"
Sep 16 19:15:19.431711 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:15:19.431563    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/55e41cd7-3842-401f-b88a-3382deabe6b0-home\") pod \"leave-v2-kserve-556b4dddf9-9n8lj\" (UID: \"55e41cd7-3842-401f-b88a-3382deabe6b0\") " pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-9n8lj"
Sep 16 19:15:19.431711 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:15:19.431652    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/55e41cd7-3842-401f-b88a-3382deabe6b0-tls-certs\") pod \"leave-v2-kserve-556b4dddf9-9n8lj\" (UID: \"55e41cd7-3842-401f-b88a-3382deabe6b0\") " pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-9n8lj"
Sep 16 19:15:19.512160 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:15:19.512125    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-tmrkx_4c2078bb-281a-4aa1-8e38-238897865434/console-operator/2.log"
Sep 16 19:15:19.514489 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:15:19.514463    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-tmrkx_4c2078bb-281a-4aa1-8e38-238897865434/console-operator/2.log"
Sep 16 19:15:19.522140 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:15:19.522113    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-k6gvs_6012728d-aee6-4f93-ab6e-fad616ef36e8/ovn-acl-logging/0.log"
Sep 16 19:15:19.524549 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:15:19.524521    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-k6gvs_6012728d-aee6-4f93-ab6e-fad616ef36e8/ovn-acl-logging/0.log"
Sep 16 19:15:19.532361 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:15:19.532328    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/55e41cd7-3842-401f-b88a-3382deabe6b0-tmp-dir\") pod \"leave-v2-kserve-556b4dddf9-9n8lj\" (UID: \"55e41cd7-3842-401f-b88a-3382deabe6b0\") " pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-9n8lj"
Sep 16 19:15:19.532527 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:15:19.532368    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/55e41cd7-3842-401f-b88a-3382deabe6b0-dshm\") pod \"leave-v2-kserve-556b4dddf9-9n8lj\" (UID: \"55e41cd7-3842-401f-b88a-3382deabe6b0\") " pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-9n8lj"
Sep 16 19:15:19.532527 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:15:19.532428    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/55e41cd7-3842-401f-b88a-3382deabe6b0-model-cache\") pod \"leave-v2-kserve-556b4dddf9-9n8lj\" (UID: \"55e41cd7-3842-401f-b88a-3382deabe6b0\") " pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-9n8lj"
Sep 16 19:15:19.532527 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:15:19.532472    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n7qmv\" (UniqueName: \"kubernetes.io/projected/55e41cd7-3842-401f-b88a-3382deabe6b0-kube-api-access-n7qmv\") pod \"leave-v2-kserve-556b4dddf9-9n8lj\" (UID: \"55e41cd7-3842-401f-b88a-3382deabe6b0\") " pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-9n8lj"
Sep 16 19:15:19.532710 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:15:19.532596    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/55e41cd7-3842-401f-b88a-3382deabe6b0-home\") pod \"leave-v2-kserve-556b4dddf9-9n8lj\" (UID: \"55e41cd7-3842-401f-b88a-3382deabe6b0\") " pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-9n8lj"
Sep 16 19:15:19.532766 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:15:19.532724    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/55e41cd7-3842-401f-b88a-3382deabe6b0-tls-certs\") pod \"leave-v2-kserve-556b4dddf9-9n8lj\" (UID: \"55e41cd7-3842-401f-b88a-3382deabe6b0\") " pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-9n8lj"
Sep 16 19:15:19.532848 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:15:19.532816    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/55e41cd7-3842-401f-b88a-3382deabe6b0-tmp-dir\") pod \"leave-v2-kserve-556b4dddf9-9n8lj\" (UID: \"55e41cd7-3842-401f-b88a-3382deabe6b0\") " pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-9n8lj"
Sep 16 19:15:19.533396 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:15:19.533365    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/55e41cd7-3842-401f-b88a-3382deabe6b0-home\") pod \"leave-v2-kserve-556b4dddf9-9n8lj\" (UID: \"55e41cd7-3842-401f-b88a-3382deabe6b0\") " pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-9n8lj"
Sep 16 19:15:19.533518 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:15:19.533420    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/55e41cd7-3842-401f-b88a-3382deabe6b0-model-cache\") pod \"leave-v2-kserve-556b4dddf9-9n8lj\" (UID: \"55e41cd7-3842-401f-b88a-3382deabe6b0\") " pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-9n8lj"
Sep 16 19:15:19.535237 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:15:19.535214    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/55e41cd7-3842-401f-b88a-3382deabe6b0-dshm\") pod \"leave-v2-kserve-556b4dddf9-9n8lj\" (UID: \"55e41cd7-3842-401f-b88a-3382deabe6b0\") " pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-9n8lj"
Sep 16 19:15:19.536122 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:15:19.536102    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/55e41cd7-3842-401f-b88a-3382deabe6b0-tls-certs\") pod \"leave-v2-kserve-556b4dddf9-9n8lj\" (UID: \"55e41cd7-3842-401f-b88a-3382deabe6b0\") " pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-9n8lj"
Sep 16 19:15:19.541780 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:15:19.541750    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:15:19.542543 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:15:19.542523    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n7qmv\" (UniqueName: \"kubernetes.io/projected/55e41cd7-3842-401f-b88a-3382deabe6b0-kube-api-access-n7qmv\") pod \"leave-v2-kserve-556b4dddf9-9n8lj\" (UID: \"55e41cd7-3842-401f-b88a-3382deabe6b0\") " pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-9n8lj"
Sep 16 19:15:19.654921 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:15:19.654884    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-9n8lj"
Sep 16 19:15:19.936254 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:15:19.936156    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-9n8lj"]
Sep 16 19:15:19.938536 ip-10-0-135-119 kubenswrapper[2578]: W0916 19:15:19.938501    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod55e41cd7_3842_401f_b88a_3382deabe6b0.slice/crio-df3f07da4c62b03d29efd08d8c94a64f708dccba3e2ce8ae01e721eb76b73855 WatchSource:0}: Error finding container df3f07da4c62b03d29efd08d8c94a64f708dccba3e2ce8ae01e721eb76b73855: Status 404 returned error can't find the container with id df3f07da4c62b03d29efd08d8c94a64f708dccba3e2ce8ae01e721eb76b73855
Sep 16 19:15:20.395030 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:15:20.394965    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-9n8lj" event={"ID":"55e41cd7-3842-401f-b88a-3382deabe6b0","Type":"ContainerStarted","Data":"77e1002717bf10b1a74443e7aab861da614cd7c178d0561503ee49171aafeba2"}
Sep 16 19:15:20.395475 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:15:20.395053    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-9n8lj" event={"ID":"55e41cd7-3842-401f-b88a-3382deabe6b0","Type":"ContainerStarted","Data":"df3f07da4c62b03d29efd08d8c94a64f708dccba3e2ce8ae01e721eb76b73855"}
Sep 16 19:15:20.414852 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:15:20.414798    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-9n8lj" podStartSLOduration=1.414781147 podStartE2EDuration="1.414781147s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:15:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:15:20.413713123 +0000 UTC m=+2101.480204624" watchObservedRunningTime="2026-09-16 19:15:20.414781147 +0000 UTC m=+2101.481272635"
Sep 16 19:15:20.656118 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:15:20.656015    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-9n8lj"
Sep 16 19:15:24.792429 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:15:24.792389    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-zjsmx"
Sep 16 19:15:24.805233 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:15:24.805197    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-zjsmx"
Sep 16 19:15:25.433995 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:15:25.433949    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-zjsmx"
Sep 16 19:15:29.655614 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:15:29.655569    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-9n8lj"
Sep 16 19:15:29.668409 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:15:29.668380    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-9n8lj"
Sep 16 19:15:30.457600 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:15:30.457563    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-9n8lj"
Sep 16 19:15:41.076727 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:15:41.076687    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:15:47.927027 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:15:47.926953    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:15:48.029006 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:15:48.028939    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:15:50.093086 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:15:50.093049    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:15:50.097936 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:15:50.097909    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7ddzshp"]
Sep 16 19:15:50.164488 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:15:50.164449    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:15:58.183829 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:15:58.183789    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7ddzshp" podUID="3efb0c8c-a9be-4997-97f1-4c435faf9ad4" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:16:01.137106 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:16:01.137076    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:16:01.554075 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:16:01.553968    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:16:01.657509 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:16:01.657469    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:16:02.152600 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:16:02.152556    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-zjsmx"]
Sep 16 19:16:02.158742 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:16:02.158707    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:16:03.138556 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:16:03.138522    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-9n8lj"]
Sep 16 19:16:03.167374 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:16:03.167343    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:16:04.351323 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:16:04.351290    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:16:04.452503 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:16:04.452462    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:16:05.120103 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:16:05.120042    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7ddzshp" podUID="3efb0c8c-a9be-4997-97f1-4c435faf9ad4" containerName="main" containerID="cri-o://71224d83b1daa1049f39bc050304ee122b5c916c43e87f80228e794c61e1a3a0" gracePeriod=45
Sep 16 19:16:06.504085 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:16:06.504050    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:16:06.602419 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:16:06.602377    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:16:08.184745 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:16:08.184696    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7ddzshp" podUID="3efb0c8c-a9be-4997-97f1-4c435faf9ad4" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:16:17.173600 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:16:17.173536    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-zjsmx" podUID="1067ca9f-7d6b-4d46-98b0-cce3cf2a9ef9" containerName="main" containerID="cri-o://1b312ef2d8cce0f8b5a7d583d29604ffa3e201c903d192fff81c6c14c72daa07" gracePeriod=45
Sep 16 19:16:17.436863 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:16:17.436792    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-zjsmx"
Sep 16 19:16:17.557345 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:16:17.557303    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1067ca9f-7d6b-4d46-98b0-cce3cf2a9ef9-home\") pod \"1067ca9f-7d6b-4d46-98b0-cce3cf2a9ef9\" (UID: \"1067ca9f-7d6b-4d46-98b0-cce3cf2a9ef9\") "
Sep 16 19:16:17.557550 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:16:17.557356    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1067ca9f-7d6b-4d46-98b0-cce3cf2a9ef9-dshm\") pod \"1067ca9f-7d6b-4d46-98b0-cce3cf2a9ef9\" (UID: \"1067ca9f-7d6b-4d46-98b0-cce3cf2a9ef9\") "
Sep 16 19:16:17.557550 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:16:17.557427    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1067ca9f-7d6b-4d46-98b0-cce3cf2a9ef9-tmp-dir\") pod \"1067ca9f-7d6b-4d46-98b0-cce3cf2a9ef9\" (UID: \"1067ca9f-7d6b-4d46-98b0-cce3cf2a9ef9\") "
Sep 16 19:16:17.557550 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:16:17.557447    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1067ca9f-7d6b-4d46-98b0-cce3cf2a9ef9-model-cache\") pod \"1067ca9f-7d6b-4d46-98b0-cce3cf2a9ef9\" (UID: \"1067ca9f-7d6b-4d46-98b0-cce3cf2a9ef9\") "
Sep 16 19:16:17.557550 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:16:17.557477    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1067ca9f-7d6b-4d46-98b0-cce3cf2a9ef9-tls-certs\") pod \"1067ca9f-7d6b-4d46-98b0-cce3cf2a9ef9\" (UID: \"1067ca9f-7d6b-4d46-98b0-cce3cf2a9ef9\") "
Sep 16 19:16:17.557550 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:16:17.557499    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7f7lj\" (UniqueName: \"kubernetes.io/projected/1067ca9f-7d6b-4d46-98b0-cce3cf2a9ef9-kube-api-access-7f7lj\") pod \"1067ca9f-7d6b-4d46-98b0-cce3cf2a9ef9\" (UID: \"1067ca9f-7d6b-4d46-98b0-cce3cf2a9ef9\") "
Sep 16 19:16:17.557831 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:16:17.557649    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1067ca9f-7d6b-4d46-98b0-cce3cf2a9ef9-home" (OuterVolumeSpecName: "home") pod "1067ca9f-7d6b-4d46-98b0-cce3cf2a9ef9" (UID: "1067ca9f-7d6b-4d46-98b0-cce3cf2a9ef9"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:16:17.557831 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:16:17.557660    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1067ca9f-7d6b-4d46-98b0-cce3cf2a9ef9-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "1067ca9f-7d6b-4d46-98b0-cce3cf2a9ef9" (UID: "1067ca9f-7d6b-4d46-98b0-cce3cf2a9ef9"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:16:17.557831 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:16:17.557758    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1067ca9f-7d6b-4d46-98b0-cce3cf2a9ef9-model-cache" (OuterVolumeSpecName: "model-cache") pod "1067ca9f-7d6b-4d46-98b0-cce3cf2a9ef9" (UID: "1067ca9f-7d6b-4d46-98b0-cce3cf2a9ef9"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:16:17.557831 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:16:17.557799    2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1067ca9f-7d6b-4d46-98b0-cce3cf2a9ef9-tmp-dir\") on node \"ip-10-0-135-119.ec2.internal\" DevicePath \"\""
Sep 16 19:16:17.557831 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:16:17.557816    2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1067ca9f-7d6b-4d46-98b0-cce3cf2a9ef9-home\") on node \"ip-10-0-135-119.ec2.internal\" DevicePath \"\""
Sep 16 19:16:17.559959 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:16:17.559928    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1067ca9f-7d6b-4d46-98b0-cce3cf2a9ef9-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "1067ca9f-7d6b-4d46-98b0-cce3cf2a9ef9" (UID: "1067ca9f-7d6b-4d46-98b0-cce3cf2a9ef9"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:16:17.560110 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:16:17.559966    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1067ca9f-7d6b-4d46-98b0-cce3cf2a9ef9-dshm" (OuterVolumeSpecName: "dshm") pod "1067ca9f-7d6b-4d46-98b0-cce3cf2a9ef9" (UID: "1067ca9f-7d6b-4d46-98b0-cce3cf2a9ef9"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:16:17.560110 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:16:17.559972    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1067ca9f-7d6b-4d46-98b0-cce3cf2a9ef9-kube-api-access-7f7lj" (OuterVolumeSpecName: "kube-api-access-7f7lj") pod "1067ca9f-7d6b-4d46-98b0-cce3cf2a9ef9" (UID: "1067ca9f-7d6b-4d46-98b0-cce3cf2a9ef9"). InnerVolumeSpecName "kube-api-access-7f7lj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:16:17.659278 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:16:17.659238    2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1067ca9f-7d6b-4d46-98b0-cce3cf2a9ef9-model-cache\") on node \"ip-10-0-135-119.ec2.internal\" DevicePath \"\""
Sep 16 19:16:17.659278 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:16:17.659271    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1067ca9f-7d6b-4d46-98b0-cce3cf2a9ef9-tls-certs\") on node \"ip-10-0-135-119.ec2.internal\" DevicePath \"\""
Sep 16 19:16:17.659278 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:16:17.659280    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7f7lj\" (UniqueName: \"kubernetes.io/projected/1067ca9f-7d6b-4d46-98b0-cce3cf2a9ef9-kube-api-access-7f7lj\") on node \"ip-10-0-135-119.ec2.internal\" DevicePath \"\""
Sep 16 19:16:17.659528 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:16:17.659292    2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1067ca9f-7d6b-4d46-98b0-cce3cf2a9ef9-dshm\") on node \"ip-10-0-135-119.ec2.internal\" DevicePath \"\""
Sep 16 19:16:17.677521 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:16:17.677481    2578 generic.go:356] "Generic (PLEG): container finished" podID="1067ca9f-7d6b-4d46-98b0-cce3cf2a9ef9" containerID="1b312ef2d8cce0f8b5a7d583d29604ffa3e201c903d192fff81c6c14c72daa07" exitCode=0
Sep 16 19:16:17.677717 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:16:17.677534    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-zjsmx"
Sep 16 19:16:17.677717 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:16:17.677553    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-zjsmx" event={"ID":"1067ca9f-7d6b-4d46-98b0-cce3cf2a9ef9","Type":"ContainerDied","Data":"1b312ef2d8cce0f8b5a7d583d29604ffa3e201c903d192fff81c6c14c72daa07"}
Sep 16 19:16:17.677717 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:16:17.677609    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-zjsmx" event={"ID":"1067ca9f-7d6b-4d46-98b0-cce3cf2a9ef9","Type":"ContainerDied","Data":"342e0b06bddce18ad9e48abeb4d4902ed136b75b2c41284541cd21196704414e"}
Sep 16 19:16:17.677717 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:16:17.677640    2578 scope.go:117] "RemoveContainer" containerID="1b312ef2d8cce0f8b5a7d583d29604ffa3e201c903d192fff81c6c14c72daa07"
Sep 16 19:16:17.688333 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:16:17.688311    2578 scope.go:117] "RemoveContainer" containerID="1b312ef2d8cce0f8b5a7d583d29604ffa3e201c903d192fff81c6c14c72daa07"
Sep 16 19:16:17.688659 ip-10-0-135-119 kubenswrapper[2578]: E0916 19:16:17.688642    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1b312ef2d8cce0f8b5a7d583d29604ffa3e201c903d192fff81c6c14c72daa07\": container with ID starting with 1b312ef2d8cce0f8b5a7d583d29604ffa3e201c903d192fff81c6c14c72daa07 not found: ID does not exist" containerID="1b312ef2d8cce0f8b5a7d583d29604ffa3e201c903d192fff81c6c14c72daa07"
Sep 16 19:16:17.688727 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:16:17.688666    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1b312ef2d8cce0f8b5a7d583d29604ffa3e201c903d192fff81c6c14c72daa07"} err="failed to get container status \"1b312ef2d8cce0f8b5a7d583d29604ffa3e201c903d192fff81c6c14c72daa07\": rpc error: code = NotFound desc = could not find container \"1b312ef2d8cce0f8b5a7d583d29604ffa3e201c903d192fff81c6c14c72daa07\": container with ID starting with 1b312ef2d8cce0f8b5a7d583d29604ffa3e201c903d192fff81c6c14c72daa07 not found: ID does not exist"
Sep 16 19:16:17.700888 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:16:17.700854    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-zjsmx"]
Sep 16 19:16:17.706669 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:16:17.706640    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-zjsmx"]
Sep 16 19:16:18.158269 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:16:18.158228    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-9n8lj" podUID="55e41cd7-3842-401f-b88a-3382deabe6b0" containerName="main" containerID="cri-o://77e1002717bf10b1a74443e7aab861da614cd7c178d0561503ee49171aafeba2" gracePeriod=45
Sep 16 19:16:18.184108 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:16:18.184067    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7ddzshp" podUID="3efb0c8c-a9be-4997-97f1-4c435faf9ad4" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:16:18.184499 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:16:18.184206    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7ddzshp"
Sep 16 19:16:18.418617 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:16:18.418543    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-9n8lj"
Sep 16 19:16:18.421457 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:16:18.421426    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-zjsmx" podUID="1067ca9f-7d6b-4d46-98b0-cce3cf2a9ef9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Sep 16 19:16:18.465455 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:16:18.465422    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/55e41cd7-3842-401f-b88a-3382deabe6b0-tls-certs\") pod \"55e41cd7-3842-401f-b88a-3382deabe6b0\" (UID: \"55e41cd7-3842-401f-b88a-3382deabe6b0\") "
Sep 16 19:16:18.465600 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:16:18.465486    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/55e41cd7-3842-401f-b88a-3382deabe6b0-tmp-dir\") pod \"55e41cd7-3842-401f-b88a-3382deabe6b0\" (UID: \"55e41cd7-3842-401f-b88a-3382deabe6b0\") "
Sep 16 19:16:18.465600 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:16:18.465524    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/55e41cd7-3842-401f-b88a-3382deabe6b0-dshm\") pod \"55e41cd7-3842-401f-b88a-3382deabe6b0\" (UID: \"55e41cd7-3842-401f-b88a-3382deabe6b0\") "
Sep 16 19:16:18.465600 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:16:18.465543    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/55e41cd7-3842-401f-b88a-3382deabe6b0-model-cache\") pod \"55e41cd7-3842-401f-b88a-3382deabe6b0\" (UID: \"55e41cd7-3842-401f-b88a-3382deabe6b0\") "
Sep 16 19:16:18.465600 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:16:18.465581    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-n7qmv\" (UniqueName: \"kubernetes.io/projected/55e41cd7-3842-401f-b88a-3382deabe6b0-kube-api-access-n7qmv\") pod \"55e41cd7-3842-401f-b88a-3382deabe6b0\" (UID: \"55e41cd7-3842-401f-b88a-3382deabe6b0\") "
Sep 16 19:16:18.465812 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:16:18.465613    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/55e41cd7-3842-401f-b88a-3382deabe6b0-home\") pod \"55e41cd7-3842-401f-b88a-3382deabe6b0\" (UID: \"55e41cd7-3842-401f-b88a-3382deabe6b0\") "
Sep 16 19:16:18.465899 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:16:18.465869    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55e41cd7-3842-401f-b88a-3382deabe6b0-model-cache" (OuterVolumeSpecName: "model-cache") pod "55e41cd7-3842-401f-b88a-3382deabe6b0" (UID: "55e41cd7-3842-401f-b88a-3382deabe6b0"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:16:18.465964 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:16:18.465879    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55e41cd7-3842-401f-b88a-3382deabe6b0-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "55e41cd7-3842-401f-b88a-3382deabe6b0" (UID: "55e41cd7-3842-401f-b88a-3382deabe6b0"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:16:18.466052 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:16:18.466003    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55e41cd7-3842-401f-b88a-3382deabe6b0-home" (OuterVolumeSpecName: "home") pod "55e41cd7-3842-401f-b88a-3382deabe6b0" (UID: "55e41cd7-3842-401f-b88a-3382deabe6b0"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:16:18.468003 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:16:18.467950    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55e41cd7-3842-401f-b88a-3382deabe6b0-dshm" (OuterVolumeSpecName: "dshm") pod "55e41cd7-3842-401f-b88a-3382deabe6b0" (UID: "55e41cd7-3842-401f-b88a-3382deabe6b0"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:16:18.468127 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:16:18.468047    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/55e41cd7-3842-401f-b88a-3382deabe6b0-kube-api-access-n7qmv" (OuterVolumeSpecName: "kube-api-access-n7qmv") pod "55e41cd7-3842-401f-b88a-3382deabe6b0" (UID: "55e41cd7-3842-401f-b88a-3382deabe6b0"). InnerVolumeSpecName "kube-api-access-n7qmv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:16:18.468127 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:16:18.468109    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/55e41cd7-3842-401f-b88a-3382deabe6b0-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "55e41cd7-3842-401f-b88a-3382deabe6b0" (UID: "55e41cd7-3842-401f-b88a-3382deabe6b0"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:16:18.567355 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:16:18.567312    2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/55e41cd7-3842-401f-b88a-3382deabe6b0-tmp-dir\") on node \"ip-10-0-135-119.ec2.internal\" DevicePath \"\""
Sep 16 19:16:18.567355 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:16:18.567350    2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/55e41cd7-3842-401f-b88a-3382deabe6b0-dshm\") on node \"ip-10-0-135-119.ec2.internal\" DevicePath \"\""
Sep 16 19:16:18.567355 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:16:18.567362    2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/55e41cd7-3842-401f-b88a-3382deabe6b0-model-cache\") on node \"ip-10-0-135-119.ec2.internal\" DevicePath \"\""
Sep 16 19:16:18.567355 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:16:18.567370    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n7qmv\" (UniqueName: \"kubernetes.io/projected/55e41cd7-3842-401f-b88a-3382deabe6b0-kube-api-access-n7qmv\") on node \"ip-10-0-135-119.ec2.internal\" DevicePath \"\""
Sep 16 19:16:18.567648 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:16:18.567379    2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/55e41cd7-3842-401f-b88a-3382deabe6b0-home\") on node \"ip-10-0-135-119.ec2.internal\" DevicePath \"\""
Sep 16 19:16:18.567648 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:16:18.567387    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/55e41cd7-3842-401f-b88a-3382deabe6b0-tls-certs\") on node \"ip-10-0-135-119.ec2.internal\" DevicePath \"\""
Sep 16 19:16:18.683784 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:16:18.683687    2578 generic.go:356] "Generic (PLEG): container finished" podID="55e41cd7-3842-401f-b88a-3382deabe6b0" containerID="77e1002717bf10b1a74443e7aab861da614cd7c178d0561503ee49171aafeba2" exitCode=0
Sep 16 19:16:18.683784 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:16:18.683734    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-9n8lj"
Sep 16 19:16:18.683784 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:16:18.683776    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-9n8lj" event={"ID":"55e41cd7-3842-401f-b88a-3382deabe6b0","Type":"ContainerDied","Data":"77e1002717bf10b1a74443e7aab861da614cd7c178d0561503ee49171aafeba2"}
Sep 16 19:16:18.684125 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:16:18.683827    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-9n8lj" event={"ID":"55e41cd7-3842-401f-b88a-3382deabe6b0","Type":"ContainerDied","Data":"df3f07da4c62b03d29efd08d8c94a64f708dccba3e2ce8ae01e721eb76b73855"}
Sep 16 19:16:18.684125 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:16:18.683853    2578 scope.go:117] "RemoveContainer" containerID="77e1002717bf10b1a74443e7aab861da614cd7c178d0561503ee49171aafeba2"
Sep 16 19:16:18.693796 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:16:18.693776    2578 scope.go:117] "RemoveContainer" containerID="77e1002717bf10b1a74443e7aab861da614cd7c178d0561503ee49171aafeba2"
Sep 16 19:16:18.694153 ip-10-0-135-119 kubenswrapper[2578]: E0916 19:16:18.694130    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"77e1002717bf10b1a74443e7aab861da614cd7c178d0561503ee49171aafeba2\": container with ID starting with 77e1002717bf10b1a74443e7aab861da614cd7c178d0561503ee49171aafeba2 not found: ID does not exist" containerID="77e1002717bf10b1a74443e7aab861da614cd7c178d0561503ee49171aafeba2"
Sep 16 19:16:18.694208 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:16:18.694165    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"77e1002717bf10b1a74443e7aab861da614cd7c178d0561503ee49171aafeba2"} err="failed to get container status \"77e1002717bf10b1a74443e7aab861da614cd7c178d0561503ee49171aafeba2\": rpc error: code = NotFound desc = could not find container \"77e1002717bf10b1a74443e7aab861da614cd7c178d0561503ee49171aafeba2\": container with ID starting with 77e1002717bf10b1a74443e7aab861da614cd7c178d0561503ee49171aafeba2 not found: ID does not exist"
Sep 16 19:16:18.711312 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:16:18.711274    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-9n8lj"]
Sep 16 19:16:18.723190 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:16:18.723160    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-9n8lj"]
Sep 16 19:16:19.440853 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:16:19.440820    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1067ca9f-7d6b-4d46-98b0-cce3cf2a9ef9" path="/var/lib/kubelet/pods/1067ca9f-7d6b-4d46-98b0-cce3cf2a9ef9/volumes"
Sep 16 19:16:19.441248 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:16:19.441234    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="55e41cd7-3842-401f-b88a-3382deabe6b0" path="/var/lib/kubelet/pods/55e41cd7-3842-401f-b88a-3382deabe6b0/volumes"
Sep 16 19:16:21.077007 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:16:21.076947    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:16:28.184642 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:16:28.184599    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7ddzshp" podUID="3efb0c8c-a9be-4997-97f1-4c435faf9ad4" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:16:28.948552 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:16:28.948511    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:16:35.366331 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:16:35.366300    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7ddzshp"
Sep 16 19:16:35.521805 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:16:35.521697    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3efb0c8c-a9be-4997-97f1-4c435faf9ad4-tls-certs\") pod \"3efb0c8c-a9be-4997-97f1-4c435faf9ad4\" (UID: \"3efb0c8c-a9be-4997-97f1-4c435faf9ad4\") "
Sep 16 19:16:35.521967 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:16:35.521808    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-s8scz\" (UniqueName: \"kubernetes.io/projected/3efb0c8c-a9be-4997-97f1-4c435faf9ad4-kube-api-access-s8scz\") pod \"3efb0c8c-a9be-4997-97f1-4c435faf9ad4\" (UID: \"3efb0c8c-a9be-4997-97f1-4c435faf9ad4\") "
Sep 16 19:16:35.524853 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:16:35.524819    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3efb0c8c-a9be-4997-97f1-4c435faf9ad4-kube-api-access-s8scz" (OuterVolumeSpecName: "kube-api-access-s8scz") pod "3efb0c8c-a9be-4997-97f1-4c435faf9ad4" (UID: "3efb0c8c-a9be-4997-97f1-4c435faf9ad4"). InnerVolumeSpecName "kube-api-access-s8scz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:16:35.524853 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:16:35.524828    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3efb0c8c-a9be-4997-97f1-4c435faf9ad4-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "3efb0c8c-a9be-4997-97f1-4c435faf9ad4" (UID: "3efb0c8c-a9be-4997-97f1-4c435faf9ad4"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:16:35.622739 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:16:35.622694    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s8scz\" (UniqueName: \"kubernetes.io/projected/3efb0c8c-a9be-4997-97f1-4c435faf9ad4-kube-api-access-s8scz\") on node \"ip-10-0-135-119.ec2.internal\" DevicePath \"\""
Sep 16 19:16:35.622739 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:16:35.622732    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3efb0c8c-a9be-4997-97f1-4c435faf9ad4-tls-certs\") on node \"ip-10-0-135-119.ec2.internal\" DevicePath \"\""
Sep 16 19:16:35.769714 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:16:35.769680    2578 generic.go:356] "Generic (PLEG): container finished" podID="3efb0c8c-a9be-4997-97f1-4c435faf9ad4" containerID="71224d83b1daa1049f39bc050304ee122b5c916c43e87f80228e794c61e1a3a0" exitCode=0
Sep 16 19:16:35.769889 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:16:35.769725    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7ddzshp"
Sep 16 19:16:35.769889 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:16:35.769762    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7ddzshp" event={"ID":"3efb0c8c-a9be-4997-97f1-4c435faf9ad4","Type":"ContainerDied","Data":"71224d83b1daa1049f39bc050304ee122b5c916c43e87f80228e794c61e1a3a0"}
Sep 16 19:16:35.769889 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:16:35.769814    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7ddzshp" event={"ID":"3efb0c8c-a9be-4997-97f1-4c435faf9ad4","Type":"ContainerDied","Data":"1b6b5b30720d0b9ba39e5d27afd8b9f2fb7cab308b6aa27256ec0ced515e7e28"}
Sep 16 19:16:35.769889 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:16:35.769837    2578 scope.go:117] "RemoveContainer" containerID="71224d83b1daa1049f39bc050304ee122b5c916c43e87f80228e794c61e1a3a0"
Sep 16 19:16:35.779011 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:16:35.778990    2578 scope.go:117] "RemoveContainer" containerID="71224d83b1daa1049f39bc050304ee122b5c916c43e87f80228e794c61e1a3a0"
Sep 16 19:16:35.779378 ip-10-0-135-119 kubenswrapper[2578]: E0916 19:16:35.779355    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"71224d83b1daa1049f39bc050304ee122b5c916c43e87f80228e794c61e1a3a0\": container with ID starting with 71224d83b1daa1049f39bc050304ee122b5c916c43e87f80228e794c61e1a3a0 not found: ID does not exist" containerID="71224d83b1daa1049f39bc050304ee122b5c916c43e87f80228e794c61e1a3a0"
Sep 16 19:16:35.779460 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:16:35.779391    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"71224d83b1daa1049f39bc050304ee122b5c916c43e87f80228e794c61e1a3a0"} err="failed to get container status \"71224d83b1daa1049f39bc050304ee122b5c916c43e87f80228e794c61e1a3a0\": rpc error: code = NotFound desc = could not find container \"71224d83b1daa1049f39bc050304ee122b5c916c43e87f80228e794c61e1a3a0\": container with ID starting with 71224d83b1daa1049f39bc050304ee122b5c916c43e87f80228e794c61e1a3a0 not found: ID does not exist"
Sep 16 19:16:35.797035 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:16:35.796999    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7ddzshp"]
Sep 16 19:16:35.805870 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:16:35.805838    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7ddzshp"]
Sep 16 19:16:37.434477 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:16:37.434442    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3efb0c8c-a9be-4997-97f1-4c435faf9ad4" path="/var/lib/kubelet/pods/3efb0c8c-a9be-4997-97f1-4c435faf9ad4/volumes"
Sep 16 19:16:38.031597 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:16:38.031560    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:16:38.126654 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:16:38.126624    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:16:44.524998 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:16:44.524944    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:16:45.800309 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:16:45.800274    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:16:52.688553 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:16:52.688512    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:16:52.780658 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:16:52.780626    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:17:07.738867 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:17:07.738799    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-z6d52"]
Sep 16 19:17:07.739349 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:17:07.739195    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="55e41cd7-3842-401f-b88a-3382deabe6b0" containerName="main"
Sep 16 19:17:07.739349 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:17:07.739208    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="55e41cd7-3842-401f-b88a-3382deabe6b0" containerName="main"
Sep 16 19:17:07.739349 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:17:07.739223    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1067ca9f-7d6b-4d46-98b0-cce3cf2a9ef9" containerName="main"
Sep 16 19:17:07.739349 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:17:07.739229    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="1067ca9f-7d6b-4d46-98b0-cce3cf2a9ef9" containerName="main"
Sep 16 19:17:07.739349 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:17:07.739243    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3efb0c8c-a9be-4997-97f1-4c435faf9ad4" containerName="main"
Sep 16 19:17:07.739349 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:17:07.739249    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="3efb0c8c-a9be-4997-97f1-4c435faf9ad4" containerName="main"
Sep 16 19:17:07.739349 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:17:07.739308    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="3efb0c8c-a9be-4997-97f1-4c435faf9ad4" containerName="main"
Sep 16 19:17:07.739349 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:17:07.739314    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="1067ca9f-7d6b-4d46-98b0-cce3cf2a9ef9" containerName="main"
Sep 16 19:17:07.739349 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:17:07.739320    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="55e41cd7-3842-401f-b88a-3382deabe6b0" containerName="main"
Sep 16 19:17:07.742189 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:17:07.742172    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-z6d52"
Sep 16 19:17:07.745323 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:17:07.745284    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-350756ca\"/\"kube-root-ca.crt\""
Sep 16 19:17:07.745455 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:17:07.745419    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-350756ca\"/\"default-dockercfg-gzcj4\""
Sep 16 19:17:07.745518 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:17:07.745454    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-350756ca\"/\"openshift-service-ca.crt\""
Sep 16 19:17:07.745518 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:17:07.745469    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-350756ca\"/\"llmisv5c7e67b6c51568d1d6d13829a9337f2a-kserve-self-signed-certs\""
Sep 16 19:17:07.756497 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:17:07.756474    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-z6d52"]
Sep 16 19:17:07.767381 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:17:07.767357    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:17:07.809629 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:17:07.809588    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fe520c9f-09b4-4465-8150-0b1a62c30917-model-cache\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-z6d52\" (UID: \"fe520c9f-09b4-4465-8150-0b1a62c30917\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-z6d52"
Sep 16 19:17:07.809783 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:17:07.809644    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fe520c9f-09b4-4465-8150-0b1a62c30917-dshm\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-z6d52\" (UID: \"fe520c9f-09b4-4465-8150-0b1a62c30917\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-z6d52"
Sep 16 19:17:07.809783 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:17:07.809692    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fe520c9f-09b4-4465-8150-0b1a62c30917-home\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-z6d52\" (UID: \"fe520c9f-09b4-4465-8150-0b1a62c30917\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-z6d52"
Sep 16 19:17:07.809783 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:17:07.809768    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-55tkt\" (UniqueName: \"kubernetes.io/projected/fe520c9f-09b4-4465-8150-0b1a62c30917-kube-api-access-55tkt\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-z6d52\" (UID: \"fe520c9f-09b4-4465-8150-0b1a62c30917\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-z6d52"
Sep 16 19:17:07.809926 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:17:07.809810    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fe520c9f-09b4-4465-8150-0b1a62c30917-tls-certs\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-z6d52\" (UID: \"fe520c9f-09b4-4465-8150-0b1a62c30917\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-z6d52"
Sep 16 19:17:07.809926 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:17:07.809892    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fe520c9f-09b4-4465-8150-0b1a62c30917-tmp-dir\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-z6d52\" (UID: \"fe520c9f-09b4-4465-8150-0b1a62c30917\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-z6d52"
Sep 16 19:17:07.910625 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:17:07.910586    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-55tkt\" (UniqueName: \"kubernetes.io/projected/fe520c9f-09b4-4465-8150-0b1a62c30917-kube-api-access-55tkt\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-z6d52\" (UID: \"fe520c9f-09b4-4465-8150-0b1a62c30917\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-z6d52"
Sep 16 19:17:07.910625 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:17:07.910633    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fe520c9f-09b4-4465-8150-0b1a62c30917-tls-certs\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-z6d52\" (UID: \"fe520c9f-09b4-4465-8150-0b1a62c30917\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-z6d52"
Sep 16 19:17:07.910849 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:17:07.910669    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fe520c9f-09b4-4465-8150-0b1a62c30917-tmp-dir\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-z6d52\" (UID: \"fe520c9f-09b4-4465-8150-0b1a62c30917\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-z6d52"
Sep 16 19:17:07.910849 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:17:07.910704    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fe520c9f-09b4-4465-8150-0b1a62c30917-model-cache\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-z6d52\" (UID: \"fe520c9f-09b4-4465-8150-0b1a62c30917\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-z6d52"
Sep 16 19:17:07.910849 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:17:07.910726    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fe520c9f-09b4-4465-8150-0b1a62c30917-dshm\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-z6d52\" (UID: \"fe520c9f-09b4-4465-8150-0b1a62c30917\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-z6d52"
Sep 16 19:17:07.910849 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:17:07.910761    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fe520c9f-09b4-4465-8150-0b1a62c30917-home\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-z6d52\" (UID: \"fe520c9f-09b4-4465-8150-0b1a62c30917\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-z6d52"
Sep 16 19:17:07.911135 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:17:07.911117    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fe520c9f-09b4-4465-8150-0b1a62c30917-tmp-dir\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-z6d52\" (UID: \"fe520c9f-09b4-4465-8150-0b1a62c30917\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-z6d52"
Sep 16 19:17:07.911192 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:17:07.911171    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fe520c9f-09b4-4465-8150-0b1a62c30917-model-cache\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-z6d52\" (UID: \"fe520c9f-09b4-4465-8150-0b1a62c30917\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-z6d52"
Sep 16 19:17:07.911239 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:17:07.911219    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fe520c9f-09b4-4465-8150-0b1a62c30917-home\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-z6d52\" (UID: \"fe520c9f-09b4-4465-8150-0b1a62c30917\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-z6d52"
Sep 16 19:17:07.913293 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:17:07.913269    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fe520c9f-09b4-4465-8150-0b1a62c30917-dshm\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-z6d52\" (UID: \"fe520c9f-09b4-4465-8150-0b1a62c30917\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-z6d52"
Sep 16 19:17:07.913468 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:17:07.913450    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fe520c9f-09b4-4465-8150-0b1a62c30917-tls-certs\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-z6d52\" (UID: \"fe520c9f-09b4-4465-8150-0b1a62c30917\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-z6d52"
Sep 16 19:17:07.920803 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:17:07.920778    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-55tkt\" (UniqueName: \"kubernetes.io/projected/fe520c9f-09b4-4465-8150-0b1a62c30917-kube-api-access-55tkt\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-z6d52\" (UID: \"fe520c9f-09b4-4465-8150-0b1a62c30917\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-z6d52"
Sep 16 19:17:08.047581 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:17:08.047501    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:17:08.052529 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:17:08.052506    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-z6d52"
Sep 16 19:17:08.328436 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:17:08.328412    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-z6d52"]
Sep 16 19:17:08.330716 ip-10-0-135-119 kubenswrapper[2578]: W0916 19:17:08.330678    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfe520c9f_09b4_4465_8150_0b1a62c30917.slice/crio-5e12bf6ece00bca182e69f2bc9c8dc165fb9710e9db92b799913ac5e8d5c45ca WatchSource:0}: Error finding container 5e12bf6ece00bca182e69f2bc9c8dc165fb9710e9db92b799913ac5e8d5c45ca: Status 404 returned error can't find the container with id 5e12bf6ece00bca182e69f2bc9c8dc165fb9710e9db92b799913ac5e8d5c45ca
Sep 16 19:17:08.931423 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:17:08.931383    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-z6d52" event={"ID":"fe520c9f-09b4-4465-8150-0b1a62c30917","Type":"ContainerStarted","Data":"dee159bab15681c6664a8211898843e542552426156cbcc1fe7d8c446c59b291"}
Sep 16 19:17:08.931826 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:17:08.931442    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-z6d52" event={"ID":"fe520c9f-09b4-4465-8150-0b1a62c30917","Type":"ContainerStarted","Data":"5e12bf6ece00bca182e69f2bc9c8dc165fb9710e9db92b799913ac5e8d5c45ca"}
Sep 16 19:17:08.957002 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:17:08.956930    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-z6d52" podStartSLOduration=1.956915753 podStartE2EDuration="1.956915753s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:17:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:17:08.956618313 +0000 UTC m=+2210.023109862" watchObservedRunningTime="2026-09-16 19:17:08.956915753 +0000 UTC m=+2210.023407241"
Sep 16 19:17:09.053380 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:17:09.053354    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-z6d52"
Sep 16 19:17:17.133655 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:17:17.133619    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:17:18.053356 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:17:18.053309    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-z6d52"
Sep 16 19:17:18.066160 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:17:18.066124    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-z6d52"
Sep 16 19:17:18.992706 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:17:18.992671    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-z6d52"
Sep 16 19:17:37.332248 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:17:37.332206    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:17:40.912259 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:17:40.912171    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:17:47.218669 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:17:47.218629    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-z6d52"]
Sep 16 19:17:47.228663 ip-10-0-135-119 kubenswrapper[2578]: E0916 19:17:47.228620    2578 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 16 19:17:47.228663 ip-10-0-135-119 kubenswrapper[2578]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 19:17:47.228663 ip-10-0-135-119 kubenswrapper[2578]:         , stdout: , stderr: , exit code -1
Sep 16 19:17:47.228663 ip-10-0-135-119 kubenswrapper[2578]:  > containerID="dee159bab15681c6664a8211898843e542552426156cbcc1fe7d8c446c59b291" cmd=["/bin/sleep","15"]
Sep 16 19:17:47.229031 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:17:47.228681    2578 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-z6d52" message="" err=<
Sep 16 19:17:47.229031 ip-10-0-135-119 kubenswrapper[2578]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 19:17:47.229031 ip-10-0-135-119 kubenswrapper[2578]:         , stdout: , stderr: , exit code -1
Sep 16 19:17:47.229031 ip-10-0-135-119 kubenswrapper[2578]:  >
Sep 16 19:17:47.229031 ip-10-0-135-119 kubenswrapper[2578]: E0916 19:17:47.228693    2578 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 16 19:17:47.229031 ip-10-0-135-119 kubenswrapper[2578]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 19:17:47.229031 ip-10-0-135-119 kubenswrapper[2578]:         , stdout: , stderr: , exit code -1
Sep 16 19:17:47.229031 ip-10-0-135-119 kubenswrapper[2578]:  > pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-z6d52" podUID="fe520c9f-09b4-4465-8150-0b1a62c30917" containerName="main" containerID="cri-o://dee159bab15681c6664a8211898843e542552426156cbcc1fe7d8c446c59b291"
Sep 16 19:17:47.229031 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:17:47.228742    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-z6d52" podUID="fe520c9f-09b4-4465-8150-0b1a62c30917" containerName="main" containerID="cri-o://dee159bab15681c6664a8211898843e542552426156cbcc1fe7d8c446c59b291" gracePeriod=60
Sep 16 19:17:47.230806 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:17:47.230779    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:17:47.328859 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:17:47.328824    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:17:47.979322 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:17:47.979294    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-z6d52"
Sep 16 19:17:48.080814 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:17:48.080775    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fe520c9f-09b4-4465-8150-0b1a62c30917-model-cache\") pod \"fe520c9f-09b4-4465-8150-0b1a62c30917\" (UID: \"fe520c9f-09b4-4465-8150-0b1a62c30917\") "
Sep 16 19:17:48.080814 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:17:48.080822    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-55tkt\" (UniqueName: \"kubernetes.io/projected/fe520c9f-09b4-4465-8150-0b1a62c30917-kube-api-access-55tkt\") pod \"fe520c9f-09b4-4465-8150-0b1a62c30917\" (UID: \"fe520c9f-09b4-4465-8150-0b1a62c30917\") "
Sep 16 19:17:48.081138 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:17:48.080846    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fe520c9f-09b4-4465-8150-0b1a62c30917-home\") pod \"fe520c9f-09b4-4465-8150-0b1a62c30917\" (UID: \"fe520c9f-09b4-4465-8150-0b1a62c30917\") "
Sep 16 19:17:48.081138 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:17:48.080878    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fe520c9f-09b4-4465-8150-0b1a62c30917-tmp-dir\") pod \"fe520c9f-09b4-4465-8150-0b1a62c30917\" (UID: \"fe520c9f-09b4-4465-8150-0b1a62c30917\") "
Sep 16 19:17:48.081138 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:17:48.080928    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fe520c9f-09b4-4465-8150-0b1a62c30917-tls-certs\") pod \"fe520c9f-09b4-4465-8150-0b1a62c30917\" (UID: \"fe520c9f-09b4-4465-8150-0b1a62c30917\") "
Sep 16 19:17:48.081138 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:17:48.080951    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fe520c9f-09b4-4465-8150-0b1a62c30917-dshm\") pod \"fe520c9f-09b4-4465-8150-0b1a62c30917\" (UID: \"fe520c9f-09b4-4465-8150-0b1a62c30917\") "
Sep 16 19:17:48.081320 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:17:48.081193    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe520c9f-09b4-4465-8150-0b1a62c30917-home" (OuterVolumeSpecName: "home") pod "fe520c9f-09b4-4465-8150-0b1a62c30917" (UID: "fe520c9f-09b4-4465-8150-0b1a62c30917"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:17:48.081320 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:17:48.081205    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe520c9f-09b4-4465-8150-0b1a62c30917-model-cache" (OuterVolumeSpecName: "model-cache") pod "fe520c9f-09b4-4465-8150-0b1a62c30917" (UID: "fe520c9f-09b4-4465-8150-0b1a62c30917"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:17:48.081320 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:17:48.081217    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe520c9f-09b4-4465-8150-0b1a62c30917-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "fe520c9f-09b4-4465-8150-0b1a62c30917" (UID: "fe520c9f-09b4-4465-8150-0b1a62c30917"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:17:48.081477 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:17:48.081328    2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fe520c9f-09b4-4465-8150-0b1a62c30917-model-cache\") on node \"ip-10-0-135-119.ec2.internal\" DevicePath \"\""
Sep 16 19:17:48.081477 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:17:48.081347    2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fe520c9f-09b4-4465-8150-0b1a62c30917-home\") on node \"ip-10-0-135-119.ec2.internal\" DevicePath \"\""
Sep 16 19:17:48.081477 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:17:48.081359    2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fe520c9f-09b4-4465-8150-0b1a62c30917-tmp-dir\") on node \"ip-10-0-135-119.ec2.internal\" DevicePath \"\""
Sep 16 19:17:48.083445 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:17:48.083416    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fe520c9f-09b4-4465-8150-0b1a62c30917-kube-api-access-55tkt" (OuterVolumeSpecName: "kube-api-access-55tkt") pod "fe520c9f-09b4-4465-8150-0b1a62c30917" (UID: "fe520c9f-09b4-4465-8150-0b1a62c30917"). InnerVolumeSpecName "kube-api-access-55tkt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:17:48.083614 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:17:48.083593    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fe520c9f-09b4-4465-8150-0b1a62c30917-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "fe520c9f-09b4-4465-8150-0b1a62c30917" (UID: "fe520c9f-09b4-4465-8150-0b1a62c30917"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:17:48.083685 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:17:48.083626    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe520c9f-09b4-4465-8150-0b1a62c30917-dshm" (OuterVolumeSpecName: "dshm") pod "fe520c9f-09b4-4465-8150-0b1a62c30917" (UID: "fe520c9f-09b4-4465-8150-0b1a62c30917"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:17:48.120398 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:17:48.120310    2578 generic.go:356] "Generic (PLEG): container finished" podID="fe520c9f-09b4-4465-8150-0b1a62c30917" containerID="dee159bab15681c6664a8211898843e542552426156cbcc1fe7d8c446c59b291" exitCode=0
Sep 16 19:17:48.120398 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:17:48.120359    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-z6d52"
Sep 16 19:17:48.120398 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:17:48.120376    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-z6d52" event={"ID":"fe520c9f-09b4-4465-8150-0b1a62c30917","Type":"ContainerDied","Data":"dee159bab15681c6664a8211898843e542552426156cbcc1fe7d8c446c59b291"}
Sep 16 19:17:48.120647 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:17:48.120420    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-z6d52" event={"ID":"fe520c9f-09b4-4465-8150-0b1a62c30917","Type":"ContainerDied","Data":"5e12bf6ece00bca182e69f2bc9c8dc165fb9710e9db92b799913ac5e8d5c45ca"}
Sep 16 19:17:48.120647 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:17:48.120441    2578 scope.go:117] "RemoveContainer" containerID="dee159bab15681c6664a8211898843e542552426156cbcc1fe7d8c446c59b291"
Sep 16 19:17:48.131286 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:17:48.131258    2578 scope.go:117] "RemoveContainer" containerID="dee159bab15681c6664a8211898843e542552426156cbcc1fe7d8c446c59b291"
Sep 16 19:17:48.131657 ip-10-0-135-119 kubenswrapper[2578]: E0916 19:17:48.131633    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dee159bab15681c6664a8211898843e542552426156cbcc1fe7d8c446c59b291\": container with ID starting with dee159bab15681c6664a8211898843e542552426156cbcc1fe7d8c446c59b291 not found: ID does not exist" containerID="dee159bab15681c6664a8211898843e542552426156cbcc1fe7d8c446c59b291"
Sep 16 19:17:48.131753 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:17:48.131666    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dee159bab15681c6664a8211898843e542552426156cbcc1fe7d8c446c59b291"} err="failed to get container status \"dee159bab15681c6664a8211898843e542552426156cbcc1fe7d8c446c59b291\": rpc error: code = NotFound desc = could not find container \"dee159bab15681c6664a8211898843e542552426156cbcc1fe7d8c446c59b291\": container with ID starting with dee159bab15681c6664a8211898843e542552426156cbcc1fe7d8c446c59b291 not found: ID does not exist"
Sep 16 19:17:48.147683 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:17:48.147646    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-z6d52"]
Sep 16 19:17:48.155805 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:17:48.155772    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-z6d52"]
Sep 16 19:17:48.182781 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:17:48.182741    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fe520c9f-09b4-4465-8150-0b1a62c30917-tls-certs\") on node \"ip-10-0-135-119.ec2.internal\" DevicePath \"\""
Sep 16 19:17:48.182781 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:17:48.182775    2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fe520c9f-09b4-4465-8150-0b1a62c30917-dshm\") on node \"ip-10-0-135-119.ec2.internal\" DevicePath \"\""
Sep 16 19:17:48.183064 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:17:48.182805    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-55tkt\" (UniqueName: \"kubernetes.io/projected/fe520c9f-09b4-4465-8150-0b1a62c30917-kube-api-access-55tkt\") on node \"ip-10-0-135-119.ec2.internal\" DevicePath \"\""
Sep 16 19:17:49.436530 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:17:49.436486    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fe520c9f-09b4-4465-8150-0b1a62c30917" path="/var/lib/kubelet/pods/fe520c9f-09b4-4465-8150-0b1a62c30917/volumes"
Sep 16 19:18:21.927551 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:18:21.927508    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:18:25.338997 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:18:25.338955    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:18:35.051813 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:18:35.051774    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:18:35.149702 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:18:35.149662    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:18:42.123010 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:18:42.122945    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:18:43.187721 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:18:43.187684    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:18:43.296152 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:18:43.296118    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:18:43.397110 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:18:43.397075    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:18:48.281617 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:18:48.281577    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:18:48.355388 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:18:48.355350    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:18:53.751020 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:18:53.750962    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-bp97h"]
Sep 16 19:18:53.751594 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:18:53.751506    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fe520c9f-09b4-4465-8150-0b1a62c30917" containerName="main"
Sep 16 19:18:53.751594 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:18:53.751525    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="fe520c9f-09b4-4465-8150-0b1a62c30917" containerName="main"
Sep 16 19:18:53.751716 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:18:53.751605    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="fe520c9f-09b4-4465-8150-0b1a62c30917" containerName="main"
Sep 16 19:18:53.754418 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:18:53.754396    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-bp97h"
Sep 16 19:18:53.759963 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:18:53.759939    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"kube-root-ca.crt\""
Sep 16 19:18:53.760478 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:18:53.760456    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"default-dockercfg-bjrrr\""
Sep 16 19:18:53.761228 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:18:53.761038    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"openshift-service-ca.crt\""
Sep 16 19:18:53.761398 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:18:53.761381    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"llmisve55ae740357a3a31a27cdb8b66ffe20f-kserve-self-signed-certs\""
Sep 16 19:18:53.767940 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:18:53.767919    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-bp97h"]
Sep 16 19:18:53.786952 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:18:53.786923    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:18:53.872651 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:18:53.872616    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9222c29c-0338-48b4-8a56-d059b44509db-model-cache\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-bp97h\" (UID: \"9222c29c-0338-48b4-8a56-d059b44509db\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-bp97h"
Sep 16 19:18:53.872804 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:18:53.872666    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9222c29c-0338-48b4-8a56-d059b44509db-home\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-bp97h\" (UID: \"9222c29c-0338-48b4-8a56-d059b44509db\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-bp97h"
Sep 16 19:18:53.872804 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:18:53.872700    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9222c29c-0338-48b4-8a56-d059b44509db-tls-certs\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-bp97h\" (UID: \"9222c29c-0338-48b4-8a56-d059b44509db\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-bp97h"
Sep 16 19:18:53.872804 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:18:53.872723    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9222c29c-0338-48b4-8a56-d059b44509db-tmp-dir\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-bp97h\" (UID: \"9222c29c-0338-48b4-8a56-d059b44509db\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-bp97h"
Sep 16 19:18:53.872804 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:18:53.872741    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9222c29c-0338-48b4-8a56-d059b44509db-dshm\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-bp97h\" (UID: \"9222c29c-0338-48b4-8a56-d059b44509db\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-bp97h"
Sep 16 19:18:53.872804 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:18:53.872768    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-87d9j\" (UniqueName: \"kubernetes.io/projected/9222c29c-0338-48b4-8a56-d059b44509db-kube-api-access-87d9j\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-bp97h\" (UID: \"9222c29c-0338-48b4-8a56-d059b44509db\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-bp97h"
Sep 16 19:18:53.974096 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:18:53.974064    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-87d9j\" (UniqueName: \"kubernetes.io/projected/9222c29c-0338-48b4-8a56-d059b44509db-kube-api-access-87d9j\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-bp97h\" (UID: \"9222c29c-0338-48b4-8a56-d059b44509db\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-bp97h"
Sep 16 19:18:53.974275 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:18:53.974156    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9222c29c-0338-48b4-8a56-d059b44509db-model-cache\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-bp97h\" (UID: \"9222c29c-0338-48b4-8a56-d059b44509db\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-bp97h"
Sep 16 19:18:53.974275 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:18:53.974204    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9222c29c-0338-48b4-8a56-d059b44509db-home\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-bp97h\" (UID: \"9222c29c-0338-48b4-8a56-d059b44509db\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-bp97h"
Sep 16 19:18:53.974275 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:18:53.974237    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9222c29c-0338-48b4-8a56-d059b44509db-tls-certs\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-bp97h\" (UID: \"9222c29c-0338-48b4-8a56-d059b44509db\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-bp97h"
Sep 16 19:18:53.974436 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:18:53.974275    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9222c29c-0338-48b4-8a56-d059b44509db-tmp-dir\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-bp97h\" (UID: \"9222c29c-0338-48b4-8a56-d059b44509db\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-bp97h"
Sep 16 19:18:53.974436 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:18:53.974309    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9222c29c-0338-48b4-8a56-d059b44509db-dshm\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-bp97h\" (UID: \"9222c29c-0338-48b4-8a56-d059b44509db\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-bp97h"
Sep 16 19:18:53.974626 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:18:53.974602    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9222c29c-0338-48b4-8a56-d059b44509db-model-cache\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-bp97h\" (UID: \"9222c29c-0338-48b4-8a56-d059b44509db\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-bp97h"
Sep 16 19:18:53.974733 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:18:53.974657    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9222c29c-0338-48b4-8a56-d059b44509db-home\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-bp97h\" (UID: \"9222c29c-0338-48b4-8a56-d059b44509db\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-bp97h"
Sep 16 19:18:53.974797 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:18:53.974738    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9222c29c-0338-48b4-8a56-d059b44509db-tmp-dir\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-bp97h\" (UID: \"9222c29c-0338-48b4-8a56-d059b44509db\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-bp97h"
Sep 16 19:18:53.976710 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:18:53.976678    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9222c29c-0338-48b4-8a56-d059b44509db-dshm\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-bp97h\" (UID: \"9222c29c-0338-48b4-8a56-d059b44509db\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-bp97h"
Sep 16 19:18:53.977492 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:18:53.977471    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9222c29c-0338-48b4-8a56-d059b44509db-tls-certs\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-bp97h\" (UID: \"9222c29c-0338-48b4-8a56-d059b44509db\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-bp97h"
Sep 16 19:18:53.983551 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:18:53.983532    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-87d9j\" (UniqueName: \"kubernetes.io/projected/9222c29c-0338-48b4-8a56-d059b44509db-kube-api-access-87d9j\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-bp97h\" (UID: \"9222c29c-0338-48b4-8a56-d059b44509db\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-bp97h"
Sep 16 19:18:54.025526 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:18:54.025449    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:18:54.064189 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:18:54.064156    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-bp97h"
Sep 16 19:18:54.337656 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:18:54.337461    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-bp97h"]
Sep 16 19:18:54.340250 ip-10-0-135-119 kubenswrapper[2578]: W0916 19:18:54.340222    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9222c29c_0338_48b4_8a56_d059b44509db.slice/crio-8ce51d4e2de03a2591c7538e2e49d49e22e0a01ce92e6580751bdf6d8e00f9ba WatchSource:0}: Error finding container 8ce51d4e2de03a2591c7538e2e49d49e22e0a01ce92e6580751bdf6d8e00f9ba: Status 404 returned error can't find the container with id 8ce51d4e2de03a2591c7538e2e49d49e22e0a01ce92e6580751bdf6d8e00f9ba
Sep 16 19:18:54.451743 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:18:54.451707    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-bp97h" event={"ID":"9222c29c-0338-48b4-8a56-d059b44509db","Type":"ContainerStarted","Data":"c1c33b668f913f3fe56bbdc4fa39667c9a28c06eb6af7e34c797ecfeb2699050"}
Sep 16 19:18:54.451905 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:18:54.451760    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-bp97h" event={"ID":"9222c29c-0338-48b4-8a56-d059b44509db","Type":"ContainerStarted","Data":"8ce51d4e2de03a2591c7538e2e49d49e22e0a01ce92e6580751bdf6d8e00f9ba"}
Sep 16 19:18:54.472625 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:18:54.472572    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-bp97h" podStartSLOduration=1.47255355 podStartE2EDuration="1.47255355s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:18:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:18:54.471756519 +0000 UTC m=+2315.538248013" watchObservedRunningTime="2026-09-16 19:18:54.47255355 +0000 UTC m=+2315.539045040"
Sep 16 19:18:55.065107 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:18:55.065051    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-bp97h"
Sep 16 19:18:55.866526 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:18:55.866475    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:18:56.344313 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:18:56.344276    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:18:57.155511 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:18:57.155474    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:19:04.064968 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:19:04.064867    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-bp97h"
Sep 16 19:19:04.077751 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:19:04.077719    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-bp97h"
Sep 16 19:19:04.514722 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:19:04.514693    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-bp97h"
Sep 16 19:19:09.073724 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:19:09.073678    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:19:12.251526 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:19:12.251488    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:19:26.272627 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:19:26.272589    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:19:32.637639 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:19:32.637596    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:19:32.743224 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:19:32.743185    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:19:37.492754 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:19:37.492713    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:19:37.726636 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:19:37.726601    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche8kw8c"]
Sep 16 19:19:37.730417 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:19:37.730391    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche8kw8c"
Sep 16 19:19:37.733132 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:19:37.733100    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-epp-sa-dockercfg-v2xkn\""
Sep 16 19:19:37.733132 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:19:37.733119    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"llmisvb11a9c9832b99b016bc8f8e0ea095712-kserve-self-signed-certs\""
Sep 16 19:19:37.741497 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:19:37.741471    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche8kw8c"]
Sep 16 19:19:37.796042 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:19:37.795905    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:19:37.888522 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:19:37.888484    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pgngx\" (UniqueName: \"kubernetes.io/projected/2e6a6a03-2d63-40b8-bd6f-90f8f1d13991-kube-api-access-pgngx\") pod \"llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche8kw8c\" (UID: \"2e6a6a03-2d63-40b8-bd6f-90f8f1d13991\") " pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche8kw8c"
Sep 16 19:19:37.888771 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:19:37.888553    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2e6a6a03-2d63-40b8-bd6f-90f8f1d13991-tls-certs\") pod \"llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche8kw8c\" (UID: \"2e6a6a03-2d63-40b8-bd6f-90f8f1d13991\") " pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche8kw8c"
Sep 16 19:19:37.989358 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:19:37.989318    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pgngx\" (UniqueName: \"kubernetes.io/projected/2e6a6a03-2d63-40b8-bd6f-90f8f1d13991-kube-api-access-pgngx\") pod \"llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche8kw8c\" (UID: \"2e6a6a03-2d63-40b8-bd6f-90f8f1d13991\") " pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche8kw8c"
Sep 16 19:19:37.989535 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:19:37.989379    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2e6a6a03-2d63-40b8-bd6f-90f8f1d13991-tls-certs\") pod \"llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche8kw8c\" (UID: \"2e6a6a03-2d63-40b8-bd6f-90f8f1d13991\") " pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche8kw8c"
Sep 16 19:19:37.992358 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:19:37.992319    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2e6a6a03-2d63-40b8-bd6f-90f8f1d13991-tls-certs\") pod \"llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche8kw8c\" (UID: \"2e6a6a03-2d63-40b8-bd6f-90f8f1d13991\") " pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche8kw8c"
Sep 16 19:19:38.002339 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:19:38.002299    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pgngx\" (UniqueName: \"kubernetes.io/projected/2e6a6a03-2d63-40b8-bd6f-90f8f1d13991-kube-api-access-pgngx\") pod \"llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche8kw8c\" (UID: \"2e6a6a03-2d63-40b8-bd6f-90f8f1d13991\") " pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche8kw8c"
Sep 16 19:19:38.042341 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:19:38.042302    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche8kw8c"
Sep 16 19:19:38.320470 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:19:38.320439    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche8kw8c"]
Sep 16 19:19:38.323219 ip-10-0-135-119 kubenswrapper[2578]: W0916 19:19:38.323194    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e6a6a03_2d63_40b8_bd6f_90f8f1d13991.slice/crio-d4b6ccb46df86c8719a5a1aab71ee9cca50f2804dd679f4f8584d8aa5dd81dbe WatchSource:0}: Error finding container d4b6ccb46df86c8719a5a1aab71ee9cca50f2804dd679f4f8584d8aa5dd81dbe: Status 404 returned error can't find the container with id d4b6ccb46df86c8719a5a1aab71ee9cca50f2804dd679f4f8584d8aa5dd81dbe
Sep 16 19:19:38.667144 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:19:38.667101    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche8kw8c" event={"ID":"2e6a6a03-2d63-40b8-bd6f-90f8f1d13991","Type":"ContainerStarted","Data":"7fa488ae936a0370b75e7ea1bd8f3045d003a7ec5fb855c95dd4e38cc571cdf0"}
Sep 16 19:19:38.667541 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:19:38.667159    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche8kw8c" event={"ID":"2e6a6a03-2d63-40b8-bd6f-90f8f1d13991","Type":"ContainerStarted","Data":"d4b6ccb46df86c8719a5a1aab71ee9cca50f2804dd679f4f8584d8aa5dd81dbe"}
Sep 16 19:19:38.667541 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:19:38.667221    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche8kw8c"
Sep 16 19:20:09.673951 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:20:09.673914    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche8kw8c"
Sep 16 19:20:09.696476 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:20:09.696417    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche8kw8c" podStartSLOduration=32.696400686 podStartE2EDuration="32.696400686s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:19:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:19:38.692336726 +0000 UTC m=+2359.758828235" watchObservedRunningTime="2026-09-16 19:20:09.696400686 +0000 UTC m=+2390.762892174"
Sep 16 19:20:10.941762 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:20:10.941728    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:20:19.543887 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:20:19.543850    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-tmrkx_4c2078bb-281a-4aa1-8e38-238897865434/console-operator/2.log"
Sep 16 19:20:19.548643 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:20:19.548622    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-tmrkx_4c2078bb-281a-4aa1-8e38-238897865434/console-operator/2.log"
Sep 16 19:20:19.552445 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:20:19.552415    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-k6gvs_6012728d-aee6-4f93-ab6e-fad616ef36e8/ovn-acl-logging/0.log"
Sep 16 19:20:19.557274 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:20:19.557255    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-k6gvs_6012728d-aee6-4f93-ab6e-fad616ef36e8/ovn-acl-logging/0.log"
Sep 16 19:20:29.366395 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:20:29.366359    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-late-join-483adbac/late-v2-kserve-86766546f-slt4b"]
Sep 16 19:20:29.370100 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:20:29.370072    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-slt4b"
Sep 16 19:20:29.372903 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:20:29.372882    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-late-join-483adbac\"/\"default-dockercfg-rjgp5\""
Sep 16 19:20:29.373560 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:20:29.373540    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-late-join-483adbac\"/\"late-v2-kserve-self-signed-certs\""
Sep 16 19:20:29.373618 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:20:29.373557    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-late-join-483adbac\"/\"openshift-service-ca.crt\""
Sep 16 19:20:29.373618 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:20:29.373593    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-late-join-483adbac\"/\"kube-root-ca.crt\""
Sep 16 19:20:29.388163 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:20:29.388130    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-late-join-483adbac/late-v2-kserve-86766546f-slt4b"]
Sep 16 19:20:29.415264 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:20:29.415235    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:20:29.455041 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:20:29.455008    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7729df72-406a-4c10-a523-0efbe5ba6097-dshm\") pod \"late-v2-kserve-86766546f-slt4b\" (UID: \"7729df72-406a-4c10-a523-0efbe5ba6097\") " pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-slt4b"
Sep 16 19:20:29.455190 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:20:29.455045    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7729df72-406a-4c10-a523-0efbe5ba6097-home\") pod \"late-v2-kserve-86766546f-slt4b\" (UID: \"7729df72-406a-4c10-a523-0efbe5ba6097\") " pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-slt4b"
Sep 16 19:20:29.455190 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:20:29.455098    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7729df72-406a-4c10-a523-0efbe5ba6097-tls-certs\") pod \"late-v2-kserve-86766546f-slt4b\" (UID: \"7729df72-406a-4c10-a523-0efbe5ba6097\") " pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-slt4b"
Sep 16 19:20:29.455190 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:20:29.455122    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vxpjg\" (UniqueName: \"kubernetes.io/projected/7729df72-406a-4c10-a523-0efbe5ba6097-kube-api-access-vxpjg\") pod \"late-v2-kserve-86766546f-slt4b\" (UID: \"7729df72-406a-4c10-a523-0efbe5ba6097\") " pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-slt4b"
Sep 16 19:20:29.455316 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:20:29.455211    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7729df72-406a-4c10-a523-0efbe5ba6097-model-cache\") pod \"late-v2-kserve-86766546f-slt4b\" (UID: \"7729df72-406a-4c10-a523-0efbe5ba6097\") " pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-slt4b"
Sep 16 19:20:29.455316 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:20:29.455271    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7729df72-406a-4c10-a523-0efbe5ba6097-tmp-dir\") pod \"late-v2-kserve-86766546f-slt4b\" (UID: \"7729df72-406a-4c10-a523-0efbe5ba6097\") " pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-slt4b"
Sep 16 19:20:29.556472 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:20:29.556439    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7729df72-406a-4c10-a523-0efbe5ba6097-dshm\") pod \"late-v2-kserve-86766546f-slt4b\" (UID: \"7729df72-406a-4c10-a523-0efbe5ba6097\") " pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-slt4b"
Sep 16 19:20:29.556634 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:20:29.556479    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7729df72-406a-4c10-a523-0efbe5ba6097-home\") pod \"late-v2-kserve-86766546f-slt4b\" (UID: \"7729df72-406a-4c10-a523-0efbe5ba6097\") " pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-slt4b"
Sep 16 19:20:29.556634 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:20:29.556608    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7729df72-406a-4c10-a523-0efbe5ba6097-tls-certs\") pod \"late-v2-kserve-86766546f-slt4b\" (UID: \"7729df72-406a-4c10-a523-0efbe5ba6097\") " pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-slt4b"
Sep 16 19:20:29.556734 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:20:29.556642    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vxpjg\" (UniqueName: \"kubernetes.io/projected/7729df72-406a-4c10-a523-0efbe5ba6097-kube-api-access-vxpjg\") pod \"late-v2-kserve-86766546f-slt4b\" (UID: \"7729df72-406a-4c10-a523-0efbe5ba6097\") " pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-slt4b"
Sep 16 19:20:29.556734 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:20:29.556697    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7729df72-406a-4c10-a523-0efbe5ba6097-model-cache\") pod \"late-v2-kserve-86766546f-slt4b\" (UID: \"7729df72-406a-4c10-a523-0efbe5ba6097\") " pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-slt4b"
Sep 16 19:20:29.556842 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:20:29.556744    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7729df72-406a-4c10-a523-0efbe5ba6097-tmp-dir\") pod \"late-v2-kserve-86766546f-slt4b\" (UID: \"7729df72-406a-4c10-a523-0efbe5ba6097\") " pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-slt4b"
Sep 16 19:20:29.556902 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:20:29.556872    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7729df72-406a-4c10-a523-0efbe5ba6097-home\") pod \"late-v2-kserve-86766546f-slt4b\" (UID: \"7729df72-406a-4c10-a523-0efbe5ba6097\") " pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-slt4b"
Sep 16 19:20:29.557071 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:20:29.557042    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7729df72-406a-4c10-a523-0efbe5ba6097-model-cache\") pod \"late-v2-kserve-86766546f-slt4b\" (UID: \"7729df72-406a-4c10-a523-0efbe5ba6097\") " pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-slt4b"
Sep 16 19:20:29.557173 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:20:29.557101    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7729df72-406a-4c10-a523-0efbe5ba6097-tmp-dir\") pod \"late-v2-kserve-86766546f-slt4b\" (UID: \"7729df72-406a-4c10-a523-0efbe5ba6097\") " pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-slt4b"
Sep 16 19:20:29.559097 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:20:29.559077    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7729df72-406a-4c10-a523-0efbe5ba6097-dshm\") pod \"late-v2-kserve-86766546f-slt4b\" (UID: \"7729df72-406a-4c10-a523-0efbe5ba6097\") " pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-slt4b"
Sep 16 19:20:29.559289 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:20:29.559272    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7729df72-406a-4c10-a523-0efbe5ba6097-tls-certs\") pod \"late-v2-kserve-86766546f-slt4b\" (UID: \"7729df72-406a-4c10-a523-0efbe5ba6097\") " pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-slt4b"
Sep 16 19:20:29.569400 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:20:29.569378    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vxpjg\" (UniqueName: \"kubernetes.io/projected/7729df72-406a-4c10-a523-0efbe5ba6097-kube-api-access-vxpjg\") pod \"late-v2-kserve-86766546f-slt4b\" (UID: \"7729df72-406a-4c10-a523-0efbe5ba6097\") " pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-slt4b"
Sep 16 19:20:29.681520 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:20:29.681495    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-slt4b"
Sep 16 19:20:29.989391 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:20:29.989359    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-late-join-483adbac/late-v2-kserve-86766546f-slt4b"]
Sep 16 19:20:29.991555 ip-10-0-135-119 kubenswrapper[2578]: W0916 19:20:29.991525    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7729df72_406a_4c10_a523_0efbe5ba6097.slice/crio-f2325a9d45a37b0b9f7b58d19df36a6c0c379bf85d019d2ce84f6c166279f844 WatchSource:0}: Error finding container f2325a9d45a37b0b9f7b58d19df36a6c0c379bf85d019d2ce84f6c166279f844: Status 404 returned error can't find the container with id f2325a9d45a37b0b9f7b58d19df36a6c0c379bf85d019d2ce84f6c166279f844
Sep 16 19:20:30.915374 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:20:30.915337    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-slt4b" event={"ID":"7729df72-406a-4c10-a523-0efbe5ba6097","Type":"ContainerStarted","Data":"8a6319c517c34ee937893cd990ff975780f4072f9c72732fd2eb6a5e81c20a4c"}
Sep 16 19:20:30.915771 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:20:30.915399    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-slt4b" event={"ID":"7729df72-406a-4c10-a523-0efbe5ba6097","Type":"ContainerStarted","Data":"f2325a9d45a37b0b9f7b58d19df36a6c0c379bf85d019d2ce84f6c166279f844"}
Sep 16 19:20:30.951944 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:20:30.951891    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-slt4b" podStartSLOduration=1.951873511 podStartE2EDuration="1.951873511s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:20:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:20:30.95049334 +0000 UTC m=+2412.016984872" watchObservedRunningTime="2026-09-16 19:20:30.951873511 +0000 UTC m=+2412.018364999"
Sep 16 19:20:31.178463 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:20:31.178388    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche8kw8c"]
Sep 16 19:20:31.182732 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:20:31.182679    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:20:31.281358 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:20:31.281328    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:20:31.682565 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:20:31.682526    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-slt4b"
Sep 16 19:20:39.673746 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:20:39.673709    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche8kw8c" podUID="2e6a6a03-2d63-40b8-bd6f-90f8f1d13991" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:20:39.681742 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:20:39.681719    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-slt4b"
Sep 16 19:20:39.693970 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:20:39.693945    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-slt4b"
Sep 16 19:20:39.969917 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:20:39.969838    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-slt4b"
Sep 16 19:20:46.198568 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:20:46.198481    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche8kw8c" podUID="2e6a6a03-2d63-40b8-bd6f-90f8f1d13991" containerName="main" containerID="cri-o://7fa488ae936a0370b75e7ea1bd8f3045d003a7ec5fb855c95dd4e38cc571cdf0" gracePeriod=45
Sep 16 19:20:46.850456 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:20:46.850421    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:20:46.950572 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:20:46.950536    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:20:49.673863 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:20:49.673824    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche8kw8c" podUID="2e6a6a03-2d63-40b8-bd6f-90f8f1d13991" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:20:59.673301 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:20:59.673257    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche8kw8c" podUID="2e6a6a03-2d63-40b8-bd6f-90f8f1d13991" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:20:59.675664 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:20:59.673352    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche8kw8c"
Sep 16 19:21:09.673021 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:21:09.672965    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche8kw8c" podUID="2e6a6a03-2d63-40b8-bd6f-90f8f1d13991" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:21:09.709384 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:21:09.709350    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:21:09.806137 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:21:09.806103    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-late-join-483adbac/late-v2-kserve-86766546f-slt4b"]
Sep 16 19:21:09.901816 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:21:09.901786    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:21:10.074323 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:21:10.074246    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:21:10.723272 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:21:10.723240    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:21:14.751394 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:21:14.751359    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:21:16.446010 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:21:16.445969    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche8kw8c"
Sep 16 19:21:16.586845 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:21:16.586749    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2e6a6a03-2d63-40b8-bd6f-90f8f1d13991-tls-certs\") pod \"2e6a6a03-2d63-40b8-bd6f-90f8f1d13991\" (UID: \"2e6a6a03-2d63-40b8-bd6f-90f8f1d13991\") "
Sep 16 19:21:16.586845 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:21:16.586820    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pgngx\" (UniqueName: \"kubernetes.io/projected/2e6a6a03-2d63-40b8-bd6f-90f8f1d13991-kube-api-access-pgngx\") pod \"2e6a6a03-2d63-40b8-bd6f-90f8f1d13991\" (UID: \"2e6a6a03-2d63-40b8-bd6f-90f8f1d13991\") "
Sep 16 19:21:16.589221 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:21:16.589185    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2e6a6a03-2d63-40b8-bd6f-90f8f1d13991-kube-api-access-pgngx" (OuterVolumeSpecName: "kube-api-access-pgngx") pod "2e6a6a03-2d63-40b8-bd6f-90f8f1d13991" (UID: "2e6a6a03-2d63-40b8-bd6f-90f8f1d13991"). InnerVolumeSpecName "kube-api-access-pgngx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:21:16.589361 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:21:16.589246    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2e6a6a03-2d63-40b8-bd6f-90f8f1d13991-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "2e6a6a03-2d63-40b8-bd6f-90f8f1d13991" (UID: "2e6a6a03-2d63-40b8-bd6f-90f8f1d13991"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:21:16.688624 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:21:16.688587    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pgngx\" (UniqueName: \"kubernetes.io/projected/2e6a6a03-2d63-40b8-bd6f-90f8f1d13991-kube-api-access-pgngx\") on node \"ip-10-0-135-119.ec2.internal\" DevicePath \"\""
Sep 16 19:21:16.688624 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:21:16.688623    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2e6a6a03-2d63-40b8-bd6f-90f8f1d13991-tls-certs\") on node \"ip-10-0-135-119.ec2.internal\" DevicePath \"\""
Sep 16 19:21:17.136562 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:21:17.136532    2578 generic.go:356] "Generic (PLEG): container finished" podID="2e6a6a03-2d63-40b8-bd6f-90f8f1d13991" containerID="7fa488ae936a0370b75e7ea1bd8f3045d003a7ec5fb855c95dd4e38cc571cdf0" exitCode=0
Sep 16 19:21:17.136726 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:21:17.136581    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche8kw8c"
Sep 16 19:21:17.136726 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:21:17.136618    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche8kw8c" event={"ID":"2e6a6a03-2d63-40b8-bd6f-90f8f1d13991","Type":"ContainerDied","Data":"7fa488ae936a0370b75e7ea1bd8f3045d003a7ec5fb855c95dd4e38cc571cdf0"}
Sep 16 19:21:17.136726 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:21:17.136687    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche8kw8c" event={"ID":"2e6a6a03-2d63-40b8-bd6f-90f8f1d13991","Type":"ContainerDied","Data":"d4b6ccb46df86c8719a5a1aab71ee9cca50f2804dd679f4f8584d8aa5dd81dbe"}
Sep 16 19:21:17.136726 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:21:17.136720    2578 scope.go:117] "RemoveContainer" containerID="7fa488ae936a0370b75e7ea1bd8f3045d003a7ec5fb855c95dd4e38cc571cdf0"
Sep 16 19:21:17.147266 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:21:17.147248    2578 scope.go:117] "RemoveContainer" containerID="7fa488ae936a0370b75e7ea1bd8f3045d003a7ec5fb855c95dd4e38cc571cdf0"
Sep 16 19:21:17.147598 ip-10-0-135-119 kubenswrapper[2578]: E0916 19:21:17.147576    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7fa488ae936a0370b75e7ea1bd8f3045d003a7ec5fb855c95dd4e38cc571cdf0\": container with ID starting with 7fa488ae936a0370b75e7ea1bd8f3045d003a7ec5fb855c95dd4e38cc571cdf0 not found: ID does not exist" containerID="7fa488ae936a0370b75e7ea1bd8f3045d003a7ec5fb855c95dd4e38cc571cdf0"
Sep 16 19:21:17.147684 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:21:17.147608    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7fa488ae936a0370b75e7ea1bd8f3045d003a7ec5fb855c95dd4e38cc571cdf0"} err="failed to get container status \"7fa488ae936a0370b75e7ea1bd8f3045d003a7ec5fb855c95dd4e38cc571cdf0\": rpc error: code = NotFound desc = could not find container \"7fa488ae936a0370b75e7ea1bd8f3045d003a7ec5fb855c95dd4e38cc571cdf0\": container with ID starting with 7fa488ae936a0370b75e7ea1bd8f3045d003a7ec5fb855c95dd4e38cc571cdf0 not found: ID does not exist"
Sep 16 19:21:17.161486 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:21:17.161459    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche8kw8c"]
Sep 16 19:21:17.167594 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:21:17.167571    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche8kw8c"]
Sep 16 19:21:17.434900 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:21:17.434820    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2e6a6a03-2d63-40b8-bd6f-90f8f1d13991" path="/var/lib/kubelet/pods/2e6a6a03-2d63-40b8-bd6f-90f8f1d13991/volumes"
Sep 16 19:21:24.826474 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:21:24.826425    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-slt4b" podUID="7729df72-406a-4c10-a523-0efbe5ba6097" containerName="main" containerID="cri-o://8a6319c517c34ee937893cd990ff975780f4072f9c72732fd2eb6a5e81c20a4c" gracePeriod=45
Sep 16 19:21:24.960041 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:21:24.960003    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-slt4b" podUID="7729df72-406a-4c10-a523-0efbe5ba6097" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/health\": dial tcp 10.132.0.58:8000: connect: connection refused"
Sep 16 19:21:25.075278 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:21:25.075252    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-slt4b"
Sep 16 19:21:25.161501 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:21:25.161471    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7729df72-406a-4c10-a523-0efbe5ba6097-tls-certs\") pod \"7729df72-406a-4c10-a523-0efbe5ba6097\" (UID: \"7729df72-406a-4c10-a523-0efbe5ba6097\") "
Sep 16 19:21:25.161779 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:21:25.161513    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7729df72-406a-4c10-a523-0efbe5ba6097-model-cache\") pod \"7729df72-406a-4c10-a523-0efbe5ba6097\" (UID: \"7729df72-406a-4c10-a523-0efbe5ba6097\") "
Sep 16 19:21:25.161779 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:21:25.161535    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7729df72-406a-4c10-a523-0efbe5ba6097-tmp-dir\") pod \"7729df72-406a-4c10-a523-0efbe5ba6097\" (UID: \"7729df72-406a-4c10-a523-0efbe5ba6097\") "
Sep 16 19:21:25.161779 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:21:25.161563    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7729df72-406a-4c10-a523-0efbe5ba6097-dshm\") pod \"7729df72-406a-4c10-a523-0efbe5ba6097\" (UID: \"7729df72-406a-4c10-a523-0efbe5ba6097\") "
Sep 16 19:21:25.161779 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:21:25.161632    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7729df72-406a-4c10-a523-0efbe5ba6097-home\") pod \"7729df72-406a-4c10-a523-0efbe5ba6097\" (UID: \"7729df72-406a-4c10-a523-0efbe5ba6097\") "
Sep 16 19:21:25.161779 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:21:25.161695    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vxpjg\" (UniqueName: \"kubernetes.io/projected/7729df72-406a-4c10-a523-0efbe5ba6097-kube-api-access-vxpjg\") pod \"7729df72-406a-4c10-a523-0efbe5ba6097\" (UID: \"7729df72-406a-4c10-a523-0efbe5ba6097\") "
Sep 16 19:21:25.162083 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:21:25.161803    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7729df72-406a-4c10-a523-0efbe5ba6097-model-cache" (OuterVolumeSpecName: "model-cache") pod "7729df72-406a-4c10-a523-0efbe5ba6097" (UID: "7729df72-406a-4c10-a523-0efbe5ba6097"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:21:25.162083 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:21:25.161811    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7729df72-406a-4c10-a523-0efbe5ba6097-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "7729df72-406a-4c10-a523-0efbe5ba6097" (UID: "7729df72-406a-4c10-a523-0efbe5ba6097"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:21:25.162083 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:21:25.161943    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7729df72-406a-4c10-a523-0efbe5ba6097-home" (OuterVolumeSpecName: "home") pod "7729df72-406a-4c10-a523-0efbe5ba6097" (UID: "7729df72-406a-4c10-a523-0efbe5ba6097"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:21:25.162083 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:21:25.162023    2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7729df72-406a-4c10-a523-0efbe5ba6097-model-cache\") on node \"ip-10-0-135-119.ec2.internal\" DevicePath \"\""
Sep 16 19:21:25.162083 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:21:25.162038    2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7729df72-406a-4c10-a523-0efbe5ba6097-tmp-dir\") on node \"ip-10-0-135-119.ec2.internal\" DevicePath \"\""
Sep 16 19:21:25.163988 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:21:25.163951    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7729df72-406a-4c10-a523-0efbe5ba6097-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "7729df72-406a-4c10-a523-0efbe5ba6097" (UID: "7729df72-406a-4c10-a523-0efbe5ba6097"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:21:25.164080 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:21:25.164023    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7729df72-406a-4c10-a523-0efbe5ba6097-dshm" (OuterVolumeSpecName: "dshm") pod "7729df72-406a-4c10-a523-0efbe5ba6097" (UID: "7729df72-406a-4c10-a523-0efbe5ba6097"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:21:25.164080 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:21:25.164038    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7729df72-406a-4c10-a523-0efbe5ba6097-kube-api-access-vxpjg" (OuterVolumeSpecName: "kube-api-access-vxpjg") pod "7729df72-406a-4c10-a523-0efbe5ba6097" (UID: "7729df72-406a-4c10-a523-0efbe5ba6097"). InnerVolumeSpecName "kube-api-access-vxpjg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:21:25.176443 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:21:25.176420    2578 generic.go:356] "Generic (PLEG): container finished" podID="7729df72-406a-4c10-a523-0efbe5ba6097" containerID="8a6319c517c34ee937893cd990ff975780f4072f9c72732fd2eb6a5e81c20a4c" exitCode=0
Sep 16 19:21:25.176544 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:21:25.176469    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-slt4b"
Sep 16 19:21:25.176544 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:21:25.176499    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-slt4b" event={"ID":"7729df72-406a-4c10-a523-0efbe5ba6097","Type":"ContainerDied","Data":"8a6319c517c34ee937893cd990ff975780f4072f9c72732fd2eb6a5e81c20a4c"}
Sep 16 19:21:25.176637 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:21:25.176556    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-slt4b" event={"ID":"7729df72-406a-4c10-a523-0efbe5ba6097","Type":"ContainerDied","Data":"f2325a9d45a37b0b9f7b58d19df36a6c0c379bf85d019d2ce84f6c166279f844"}
Sep 16 19:21:25.176637 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:21:25.176587    2578 scope.go:117] "RemoveContainer" containerID="8a6319c517c34ee937893cd990ff975780f4072f9c72732fd2eb6a5e81c20a4c"
Sep 16 19:21:25.185884 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:21:25.185867    2578 scope.go:117] "RemoveContainer" containerID="8a6319c517c34ee937893cd990ff975780f4072f9c72732fd2eb6a5e81c20a4c"
Sep 16 19:21:25.186199 ip-10-0-135-119 kubenswrapper[2578]: E0916 19:21:25.186173    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8a6319c517c34ee937893cd990ff975780f4072f9c72732fd2eb6a5e81c20a4c\": container with ID starting with 8a6319c517c34ee937893cd990ff975780f4072f9c72732fd2eb6a5e81c20a4c not found: ID does not exist" containerID="8a6319c517c34ee937893cd990ff975780f4072f9c72732fd2eb6a5e81c20a4c"
Sep 16 19:21:25.186266 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:21:25.186212    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8a6319c517c34ee937893cd990ff975780f4072f9c72732fd2eb6a5e81c20a4c"} err="failed to get container status \"8a6319c517c34ee937893cd990ff975780f4072f9c72732fd2eb6a5e81c20a4c\": rpc error: code = NotFound desc = could not find container \"8a6319c517c34ee937893cd990ff975780f4072f9c72732fd2eb6a5e81c20a4c\": container with ID starting with 8a6319c517c34ee937893cd990ff975780f4072f9c72732fd2eb6a5e81c20a4c not found: ID does not exist"
Sep 16 19:21:25.201610 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:21:25.201591    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-late-join-483adbac/late-v2-kserve-86766546f-slt4b"]
Sep 16 19:21:25.210099 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:21:25.210080    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-late-join-483adbac/late-v2-kserve-86766546f-slt4b"]
Sep 16 19:21:25.263087 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:21:25.263053    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vxpjg\" (UniqueName: \"kubernetes.io/projected/7729df72-406a-4c10-a523-0efbe5ba6097-kube-api-access-vxpjg\") on node \"ip-10-0-135-119.ec2.internal\" DevicePath \"\""
Sep 16 19:21:25.263087 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:21:25.263082    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7729df72-406a-4c10-a523-0efbe5ba6097-tls-certs\") on node \"ip-10-0-135-119.ec2.internal\" DevicePath \"\""
Sep 16 19:21:25.263087 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:21:25.263090    2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7729df72-406a-4c10-a523-0efbe5ba6097-dshm\") on node \"ip-10-0-135-119.ec2.internal\" DevicePath \"\""
Sep 16 19:21:25.263310 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:21:25.263098    2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7729df72-406a-4c10-a523-0efbe5ba6097-home\") on node \"ip-10-0-135-119.ec2.internal\" DevicePath \"\""
Sep 16 19:21:25.433953 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:21:25.433876    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7729df72-406a-4c10-a523-0efbe5ba6097" path="/var/lib/kubelet/pods/7729df72-406a-4c10-a523-0efbe5ba6097/volumes"
Sep 16 19:21:29.599672 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:21:29.599622    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:21:31.488272 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:21:31.488227    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:21:31.521037 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:21:31.521007    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-bp97h"]
Sep 16 19:21:31.529879 ip-10-0-135-119 kubenswrapper[2578]: E0916 19:21:31.529847    2578 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 16 19:21:31.529879 ip-10-0-135-119 kubenswrapper[2578]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 19:21:31.529879 ip-10-0-135-119 kubenswrapper[2578]:         , stdout: , stderr: , exit code -1
Sep 16 19:21:31.529879 ip-10-0-135-119 kubenswrapper[2578]:  > containerID="c1c33b668f913f3fe56bbdc4fa39667c9a28c06eb6af7e34c797ecfeb2699050" cmd=["/bin/sleep","15"]
Sep 16 19:21:31.530112 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:21:31.529894    2578 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-bp97h" message="" err=<
Sep 16 19:21:31.530112 ip-10-0-135-119 kubenswrapper[2578]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 19:21:31.530112 ip-10-0-135-119 kubenswrapper[2578]:         , stdout: , stderr: , exit code -1
Sep 16 19:21:31.530112 ip-10-0-135-119 kubenswrapper[2578]:  >
Sep 16 19:21:31.530112 ip-10-0-135-119 kubenswrapper[2578]: E0916 19:21:31.529907    2578 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 16 19:21:31.530112 ip-10-0-135-119 kubenswrapper[2578]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 19:21:31.530112 ip-10-0-135-119 kubenswrapper[2578]:         , stdout: , stderr: , exit code -1
Sep 16 19:21:31.530112 ip-10-0-135-119 kubenswrapper[2578]:  > pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-bp97h" podUID="9222c29c-0338-48b4-8a56-d059b44509db" containerName="main" containerID="cri-o://c1c33b668f913f3fe56bbdc4fa39667c9a28c06eb6af7e34c797ecfeb2699050"
Sep 16 19:21:31.530112 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:21:31.529951    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-bp97h" podUID="9222c29c-0338-48b4-8a56-d059b44509db" containerName="main" containerID="cri-o://c1c33b668f913f3fe56bbdc4fa39667c9a28c06eb6af7e34c797ecfeb2699050" gracePeriod=60
Sep 16 19:21:31.606275 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:21:31.606243    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:21:31.878243 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:21:31.878219    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-bp97h"
Sep 16 19:21:31.921300 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:21:31.921274    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9222c29c-0338-48b4-8a56-d059b44509db-tmp-dir\") pod \"9222c29c-0338-48b4-8a56-d059b44509db\" (UID: \"9222c29c-0338-48b4-8a56-d059b44509db\") "
Sep 16 19:21:31.921460 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:21:31.921310    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9222c29c-0338-48b4-8a56-d059b44509db-tls-certs\") pod \"9222c29c-0338-48b4-8a56-d059b44509db\" (UID: \"9222c29c-0338-48b4-8a56-d059b44509db\") "
Sep 16 19:21:31.921460 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:21:31.921346    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9222c29c-0338-48b4-8a56-d059b44509db-home\") pod \"9222c29c-0338-48b4-8a56-d059b44509db\" (UID: \"9222c29c-0338-48b4-8a56-d059b44509db\") "
Sep 16 19:21:31.921460 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:21:31.921404    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9222c29c-0338-48b4-8a56-d059b44509db-model-cache\") pod \"9222c29c-0338-48b4-8a56-d059b44509db\" (UID: \"9222c29c-0338-48b4-8a56-d059b44509db\") "
Sep 16 19:21:31.921460 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:21:31.921434    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9222c29c-0338-48b4-8a56-d059b44509db-dshm\") pod \"9222c29c-0338-48b4-8a56-d059b44509db\" (UID: \"9222c29c-0338-48b4-8a56-d059b44509db\") "
Sep 16 19:21:31.921697 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:21:31.921470    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-87d9j\" (UniqueName: \"kubernetes.io/projected/9222c29c-0338-48b4-8a56-d059b44509db-kube-api-access-87d9j\") pod \"9222c29c-0338-48b4-8a56-d059b44509db\" (UID: \"9222c29c-0338-48b4-8a56-d059b44509db\") "
Sep 16 19:21:31.921697 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:21:31.921535    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9222c29c-0338-48b4-8a56-d059b44509db-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "9222c29c-0338-48b4-8a56-d059b44509db" (UID: "9222c29c-0338-48b4-8a56-d059b44509db"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:21:31.921697 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:21:31.921669    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9222c29c-0338-48b4-8a56-d059b44509db-home" (OuterVolumeSpecName: "home") pod "9222c29c-0338-48b4-8a56-d059b44509db" (UID: "9222c29c-0338-48b4-8a56-d059b44509db"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:21:31.921697 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:21:31.921689    2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9222c29c-0338-48b4-8a56-d059b44509db-tmp-dir\") on node \"ip-10-0-135-119.ec2.internal\" DevicePath \"\""
Sep 16 19:21:31.921966 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:21:31.921938    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9222c29c-0338-48b4-8a56-d059b44509db-model-cache" (OuterVolumeSpecName: "model-cache") pod "9222c29c-0338-48b4-8a56-d059b44509db" (UID: "9222c29c-0338-48b4-8a56-d059b44509db"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:21:31.923693 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:21:31.923660    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9222c29c-0338-48b4-8a56-d059b44509db-dshm" (OuterVolumeSpecName: "dshm") pod "9222c29c-0338-48b4-8a56-d059b44509db" (UID: "9222c29c-0338-48b4-8a56-d059b44509db"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:21:31.924406 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:21:31.924382    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9222c29c-0338-48b4-8a56-d059b44509db-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "9222c29c-0338-48b4-8a56-d059b44509db" (UID: "9222c29c-0338-48b4-8a56-d059b44509db"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:21:31.924507 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:21:31.924382    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9222c29c-0338-48b4-8a56-d059b44509db-kube-api-access-87d9j" (OuterVolumeSpecName: "kube-api-access-87d9j") pod "9222c29c-0338-48b4-8a56-d059b44509db" (UID: "9222c29c-0338-48b4-8a56-d059b44509db"). InnerVolumeSpecName "kube-api-access-87d9j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:21:32.022830 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:21:32.022758    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-87d9j\" (UniqueName: \"kubernetes.io/projected/9222c29c-0338-48b4-8a56-d059b44509db-kube-api-access-87d9j\") on node \"ip-10-0-135-119.ec2.internal\" DevicePath \"\""
Sep 16 19:21:32.022830 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:21:32.022786    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9222c29c-0338-48b4-8a56-d059b44509db-tls-certs\") on node \"ip-10-0-135-119.ec2.internal\" DevicePath \"\""
Sep 16 19:21:32.022830 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:21:32.022796    2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9222c29c-0338-48b4-8a56-d059b44509db-home\") on node \"ip-10-0-135-119.ec2.internal\" DevicePath \"\""
Sep 16 19:21:32.022830 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:21:32.022803    2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9222c29c-0338-48b4-8a56-d059b44509db-model-cache\") on node \"ip-10-0-135-119.ec2.internal\" DevicePath \"\""
Sep 16 19:21:32.022830 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:21:32.022812    2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9222c29c-0338-48b4-8a56-d059b44509db-dshm\") on node \"ip-10-0-135-119.ec2.internal\" DevicePath \"\""
Sep 16 19:21:32.220195 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:21:32.220163    2578 generic.go:356] "Generic (PLEG): container finished" podID="9222c29c-0338-48b4-8a56-d059b44509db" containerID="c1c33b668f913f3fe56bbdc4fa39667c9a28c06eb6af7e34c797ecfeb2699050" exitCode=0
Sep 16 19:21:32.220361 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:21:32.220200    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-bp97h" event={"ID":"9222c29c-0338-48b4-8a56-d059b44509db","Type":"ContainerDied","Data":"c1c33b668f913f3fe56bbdc4fa39667c9a28c06eb6af7e34c797ecfeb2699050"}
Sep 16 19:21:32.220361 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:21:32.220238    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-bp97h"
Sep 16 19:21:32.220361 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:21:32.220304    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-bp97h" event={"ID":"9222c29c-0338-48b4-8a56-d059b44509db","Type":"ContainerDied","Data":"8ce51d4e2de03a2591c7538e2e49d49e22e0a01ce92e6580751bdf6d8e00f9ba"}
Sep 16 19:21:32.220361 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:21:32.220334    2578 scope.go:117] "RemoveContainer" containerID="c1c33b668f913f3fe56bbdc4fa39667c9a28c06eb6af7e34c797ecfeb2699050"
Sep 16 19:21:32.229948 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:21:32.229932    2578 scope.go:117] "RemoveContainer" containerID="c1c33b668f913f3fe56bbdc4fa39667c9a28c06eb6af7e34c797ecfeb2699050"
Sep 16 19:21:32.230237 ip-10-0-135-119 kubenswrapper[2578]: E0916 19:21:32.230217    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c1c33b668f913f3fe56bbdc4fa39667c9a28c06eb6af7e34c797ecfeb2699050\": container with ID starting with c1c33b668f913f3fe56bbdc4fa39667c9a28c06eb6af7e34c797ecfeb2699050 not found: ID does not exist" containerID="c1c33b668f913f3fe56bbdc4fa39667c9a28c06eb6af7e34c797ecfeb2699050"
Sep 16 19:21:32.230304 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:21:32.230245    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c1c33b668f913f3fe56bbdc4fa39667c9a28c06eb6af7e34c797ecfeb2699050"} err="failed to get container status \"c1c33b668f913f3fe56bbdc4fa39667c9a28c06eb6af7e34c797ecfeb2699050\": rpc error: code = NotFound desc = could not find container \"c1c33b668f913f3fe56bbdc4fa39667c9a28c06eb6af7e34c797ecfeb2699050\": container with ID starting with c1c33b668f913f3fe56bbdc4fa39667c9a28c06eb6af7e34c797ecfeb2699050 not found: ID does not exist"
Sep 16 19:21:32.250923 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:21:32.250890    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-bp97h"]
Sep 16 19:21:32.275731 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:21:32.275667    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-bp97h"]
Sep 16 19:21:33.434619 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:21:33.434578    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9222c29c-0338-48b4-8a56-d059b44509db" path="/var/lib/kubelet/pods/9222c29c-0338-48b4-8a56-d059b44509db/volumes"
Sep 16 19:21:44.573845 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:21:44.573812    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-s44gn"]
Sep 16 19:21:44.574265 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:21:44.574163    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2e6a6a03-2d63-40b8-bd6f-90f8f1d13991" containerName="main"
Sep 16 19:21:44.574265 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:21:44.574176    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="2e6a6a03-2d63-40b8-bd6f-90f8f1d13991" containerName="main"
Sep 16 19:21:44.574265 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:21:44.574195    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9222c29c-0338-48b4-8a56-d059b44509db" containerName="main"
Sep 16 19:21:44.574265 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:21:44.574200    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="9222c29c-0338-48b4-8a56-d059b44509db" containerName="main"
Sep 16 19:21:44.574265 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:21:44.574215    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7729df72-406a-4c10-a523-0efbe5ba6097" containerName="main"
Sep 16 19:21:44.574265 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:21:44.574221    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="7729df72-406a-4c10-a523-0efbe5ba6097" containerName="main"
Sep 16 19:21:44.574483 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:21:44.574273    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="2e6a6a03-2d63-40b8-bd6f-90f8f1d13991" containerName="main"
Sep 16 19:21:44.574483 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:21:44.574280    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="9222c29c-0338-48b4-8a56-d059b44509db" containerName="main"
Sep 16 19:21:44.574483 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:21:44.574290    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="7729df72-406a-4c10-a523-0efbe5ba6097" containerName="main"
Sep 16 19:21:44.578546 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:21:44.578529    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-s44gn"
Sep 16 19:21:44.581020 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:21:44.581002    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-delete-at-nonzero-weight-6b4017d2\"/\"kube-root-ca.crt\""
Sep 16 19:21:44.581100 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:21:44.581003    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-delete-at-nonzero-weight-6b4017d2\"/\"openshift-service-ca.crt\""
Sep 16 19:21:44.582014 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:21:44.581991    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-delete-at-nonzero-weight-6b4017d2\"/\"delw-v2-kserve-self-signed-certs\""
Sep 16 19:21:44.582170 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:21:44.581995    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-delete-at-nonzero-weight-6b4017d2\"/\"default-dockercfg-b8ppm\""
Sep 16 19:21:44.591805 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:21:44.591785    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-s44gn"]
Sep 16 19:21:44.622874 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:21:44.622839    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:21:44.631923 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:21:44.631895    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8f5085f5-78ad-4357-8979-376130311a24-tls-certs\") pod \"delw-v2-kserve-5b84cc7ddf-s44gn\" (UID: \"8f5085f5-78ad-4357-8979-376130311a24\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-s44gn"
Sep 16 19:21:44.632235 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:21:44.631938    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-256nn\" (UniqueName: \"kubernetes.io/projected/8f5085f5-78ad-4357-8979-376130311a24-kube-api-access-256nn\") pod \"delw-v2-kserve-5b84cc7ddf-s44gn\" (UID: \"8f5085f5-78ad-4357-8979-376130311a24\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-s44gn"
Sep 16 19:21:44.632235 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:21:44.631956    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8f5085f5-78ad-4357-8979-376130311a24-model-cache\") pod \"delw-v2-kserve-5b84cc7ddf-s44gn\" (UID: \"8f5085f5-78ad-4357-8979-376130311a24\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-s44gn"
Sep 16 19:21:44.632235 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:21:44.632081    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8f5085f5-78ad-4357-8979-376130311a24-dshm\") pod \"delw-v2-kserve-5b84cc7ddf-s44gn\" (UID: \"8f5085f5-78ad-4357-8979-376130311a24\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-s44gn"
Sep 16 19:21:44.632235 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:21:44.632121    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8f5085f5-78ad-4357-8979-376130311a24-tmp-dir\") pod \"delw-v2-kserve-5b84cc7ddf-s44gn\" (UID: \"8f5085f5-78ad-4357-8979-376130311a24\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-s44gn"
Sep 16 19:21:44.632235 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:21:44.632184    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8f5085f5-78ad-4357-8979-376130311a24-home\") pod \"delw-v2-kserve-5b84cc7ddf-s44gn\" (UID: \"8f5085f5-78ad-4357-8979-376130311a24\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-s44gn"
Sep 16 19:21:44.733276 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:21:44.733245    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8f5085f5-78ad-4357-8979-376130311a24-tmp-dir\") pod \"delw-v2-kserve-5b84cc7ddf-s44gn\" (UID: \"8f5085f5-78ad-4357-8979-376130311a24\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-s44gn"
Sep 16 19:21:44.733457 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:21:44.733305    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8f5085f5-78ad-4357-8979-376130311a24-home\") pod \"delw-v2-kserve-5b84cc7ddf-s44gn\" (UID: \"8f5085f5-78ad-4357-8979-376130311a24\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-s44gn"
Sep 16 19:21:44.733457 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:21:44.733346    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8f5085f5-78ad-4357-8979-376130311a24-tls-certs\") pod \"delw-v2-kserve-5b84cc7ddf-s44gn\" (UID: \"8f5085f5-78ad-4357-8979-376130311a24\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-s44gn"
Sep 16 19:21:44.733457 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:21:44.733393    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-256nn\" (UniqueName: \"kubernetes.io/projected/8f5085f5-78ad-4357-8979-376130311a24-kube-api-access-256nn\") pod \"delw-v2-kserve-5b84cc7ddf-s44gn\" (UID: \"8f5085f5-78ad-4357-8979-376130311a24\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-s44gn"
Sep 16 19:21:44.733457 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:21:44.733419    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8f5085f5-78ad-4357-8979-376130311a24-model-cache\") pod \"delw-v2-kserve-5b84cc7ddf-s44gn\" (UID: \"8f5085f5-78ad-4357-8979-376130311a24\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-s44gn"
Sep 16 19:21:44.733672 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:21:44.733467    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8f5085f5-78ad-4357-8979-376130311a24-dshm\") pod \"delw-v2-kserve-5b84cc7ddf-s44gn\" (UID: \"8f5085f5-78ad-4357-8979-376130311a24\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-s44gn"
Sep 16 19:21:44.733730 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:21:44.733713    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8f5085f5-78ad-4357-8979-376130311a24-tmp-dir\") pod \"delw-v2-kserve-5b84cc7ddf-s44gn\" (UID: \"8f5085f5-78ad-4357-8979-376130311a24\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-s44gn"
Sep 16 19:21:44.733793 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:21:44.733721    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8f5085f5-78ad-4357-8979-376130311a24-home\") pod \"delw-v2-kserve-5b84cc7ddf-s44gn\" (UID: \"8f5085f5-78ad-4357-8979-376130311a24\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-s44gn"
Sep 16 19:21:44.733839 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:21:44.733787    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8f5085f5-78ad-4357-8979-376130311a24-model-cache\") pod \"delw-v2-kserve-5b84cc7ddf-s44gn\" (UID: \"8f5085f5-78ad-4357-8979-376130311a24\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-s44gn"
Sep 16 19:21:44.735834 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:21:44.735814    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8f5085f5-78ad-4357-8979-376130311a24-dshm\") pod \"delw-v2-kserve-5b84cc7ddf-s44gn\" (UID: \"8f5085f5-78ad-4357-8979-376130311a24\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-s44gn"
Sep 16 19:21:44.736248 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:21:44.736228    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8f5085f5-78ad-4357-8979-376130311a24-tls-certs\") pod \"delw-v2-kserve-5b84cc7ddf-s44gn\" (UID: \"8f5085f5-78ad-4357-8979-376130311a24\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-s44gn"
Sep 16 19:21:44.743479 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:21:44.743456    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-256nn\" (UniqueName: \"kubernetes.io/projected/8f5085f5-78ad-4357-8979-376130311a24-kube-api-access-256nn\") pod \"delw-v2-kserve-5b84cc7ddf-s44gn\" (UID: \"8f5085f5-78ad-4357-8979-376130311a24\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-s44gn"
Sep 16 19:21:44.889210 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:21:44.889182    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-s44gn"
Sep 16 19:21:45.167510 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:21:45.167483    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-s44gn"]
Sep 16 19:21:45.169463 ip-10-0-135-119 kubenswrapper[2578]: W0916 19:21:45.169438    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8f5085f5_78ad_4357_8979_376130311a24.slice/crio-7e8e6dca636e8f7473504042d710bab4f44f7bf827128908f455da8ca0952e36 WatchSource:0}: Error finding container 7e8e6dca636e8f7473504042d710bab4f44f7bf827128908f455da8ca0952e36: Status 404 returned error can't find the container with id 7e8e6dca636e8f7473504042d710bab4f44f7bf827128908f455da8ca0952e36
Sep 16 19:21:45.284761 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:21:45.284726    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-s44gn" event={"ID":"8f5085f5-78ad-4357-8979-376130311a24","Type":"ContainerStarted","Data":"de357a252bb35cd72835c2f32f270e58f4dc650ea11b422050b26517b44b9b12"}
Sep 16 19:21:45.284925 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:21:45.284781    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-s44gn" event={"ID":"8f5085f5-78ad-4357-8979-376130311a24","Type":"ContainerStarted","Data":"7e8e6dca636e8f7473504042d710bab4f44f7bf827128908f455da8ca0952e36"}
Sep 16 19:21:45.308783 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:21:45.308735    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-s44gn" podStartSLOduration=1.30871831 podStartE2EDuration="1.30871831s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:21:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:21:45.306142767 +0000 UTC m=+2486.372634266" watchObservedRunningTime="2026-09-16 19:21:45.30871831 +0000 UTC m=+2486.375209798"
Sep 16 19:21:45.889329 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:21:45.889293    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-s44gn"
Sep 16 19:21:53.189462 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:21:53.189423    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:21:54.889523 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:21:54.889478    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-s44gn"
Sep 16 19:21:54.901863 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:21:54.901835    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-s44gn"
Sep 16 19:21:55.342301 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:21:55.342218    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-s44gn"
Sep 16 19:21:59.646421 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:21:59.646389    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:21:59.762159 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:21:59.762124    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:22:06.741809 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:22:06.741776    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-s44gn"]
Sep 16 19:22:06.758215 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:22:06.758182    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:22:20.992024 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:22:20.991971    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:22:21.085314 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:22:21.085278    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:22:21.761448 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:22:21.761405    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-s44gn" podUID="8f5085f5-78ad-4357-8979-376130311a24" containerName="main" containerID="cri-o://de357a252bb35cd72835c2f32f270e58f4dc650ea11b422050b26517b44b9b12" gracePeriod=45
Sep 16 19:22:22.015189 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:22:22.015121    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-s44gn"
Sep 16 19:22:22.064213 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:22:22.064178    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8f5085f5-78ad-4357-8979-376130311a24-tmp-dir\") pod \"8f5085f5-78ad-4357-8979-376130311a24\" (UID: \"8f5085f5-78ad-4357-8979-376130311a24\") "
Sep 16 19:22:22.064213 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:22:22.064215    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8f5085f5-78ad-4357-8979-376130311a24-tls-certs\") pod \"8f5085f5-78ad-4357-8979-376130311a24\" (UID: \"8f5085f5-78ad-4357-8979-376130311a24\") "
Sep 16 19:22:22.064429 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:22:22.064233    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8f5085f5-78ad-4357-8979-376130311a24-dshm\") pod \"8f5085f5-78ad-4357-8979-376130311a24\" (UID: \"8f5085f5-78ad-4357-8979-376130311a24\") "
Sep 16 19:22:22.064429 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:22:22.064258    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8f5085f5-78ad-4357-8979-376130311a24-model-cache\") pod \"8f5085f5-78ad-4357-8979-376130311a24\" (UID: \"8f5085f5-78ad-4357-8979-376130311a24\") "
Sep 16 19:22:22.064429 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:22:22.064283    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8f5085f5-78ad-4357-8979-376130311a24-home\") pod \"8f5085f5-78ad-4357-8979-376130311a24\" (UID: \"8f5085f5-78ad-4357-8979-376130311a24\") "
Sep 16 19:22:22.064429 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:22:22.064347    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-256nn\" (UniqueName: \"kubernetes.io/projected/8f5085f5-78ad-4357-8979-376130311a24-kube-api-access-256nn\") pod \"8f5085f5-78ad-4357-8979-376130311a24\" (UID: \"8f5085f5-78ad-4357-8979-376130311a24\") "
Sep 16 19:22:22.064615 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:22:22.064498    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f5085f5-78ad-4357-8979-376130311a24-model-cache" (OuterVolumeSpecName: "model-cache") pod "8f5085f5-78ad-4357-8979-376130311a24" (UID: "8f5085f5-78ad-4357-8979-376130311a24"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:22:22.064615 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:22:22.064512    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f5085f5-78ad-4357-8979-376130311a24-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "8f5085f5-78ad-4357-8979-376130311a24" (UID: "8f5085f5-78ad-4357-8979-376130311a24"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:22:22.064698 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:22:22.064617    2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8f5085f5-78ad-4357-8979-376130311a24-tmp-dir\") on node \"ip-10-0-135-119.ec2.internal\" DevicePath \"\""
Sep 16 19:22:22.064698 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:22:22.064641    2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8f5085f5-78ad-4357-8979-376130311a24-model-cache\") on node \"ip-10-0-135-119.ec2.internal\" DevicePath \"\""
Sep 16 19:22:22.064840 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:22:22.064818    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f5085f5-78ad-4357-8979-376130311a24-home" (OuterVolumeSpecName: "home") pod "8f5085f5-78ad-4357-8979-376130311a24" (UID: "8f5085f5-78ad-4357-8979-376130311a24"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:22:22.066743 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:22:22.066716    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8f5085f5-78ad-4357-8979-376130311a24-kube-api-access-256nn" (OuterVolumeSpecName: "kube-api-access-256nn") pod "8f5085f5-78ad-4357-8979-376130311a24" (UID: "8f5085f5-78ad-4357-8979-376130311a24"). InnerVolumeSpecName "kube-api-access-256nn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:22:22.066851 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:22:22.066748    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8f5085f5-78ad-4357-8979-376130311a24-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "8f5085f5-78ad-4357-8979-376130311a24" (UID: "8f5085f5-78ad-4357-8979-376130311a24"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:22:22.066851 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:22:22.066766    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f5085f5-78ad-4357-8979-376130311a24-dshm" (OuterVolumeSpecName: "dshm") pod "8f5085f5-78ad-4357-8979-376130311a24" (UID: "8f5085f5-78ad-4357-8979-376130311a24"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:22:22.165969 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:22:22.165927    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-256nn\" (UniqueName: \"kubernetes.io/projected/8f5085f5-78ad-4357-8979-376130311a24-kube-api-access-256nn\") on node \"ip-10-0-135-119.ec2.internal\" DevicePath \"\""
Sep 16 19:22:22.165969 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:22:22.165966    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8f5085f5-78ad-4357-8979-376130311a24-tls-certs\") on node \"ip-10-0-135-119.ec2.internal\" DevicePath \"\""
Sep 16 19:22:22.165969 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:22:22.166007    2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8f5085f5-78ad-4357-8979-376130311a24-dshm\") on node \"ip-10-0-135-119.ec2.internal\" DevicePath \"\""
Sep 16 19:22:22.166239 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:22:22.166016    2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8f5085f5-78ad-4357-8979-376130311a24-home\") on node \"ip-10-0-135-119.ec2.internal\" DevicePath \"\""
Sep 16 19:22:22.458320 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:22:22.458288    2578 generic.go:356] "Generic (PLEG): container finished" podID="8f5085f5-78ad-4357-8979-376130311a24" containerID="de357a252bb35cd72835c2f32f270e58f4dc650ea11b422050b26517b44b9b12" exitCode=0
Sep 16 19:22:22.458512 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:22:22.458335    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-s44gn"
Sep 16 19:22:22.458512 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:22:22.458358    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-s44gn" event={"ID":"8f5085f5-78ad-4357-8979-376130311a24","Type":"ContainerDied","Data":"de357a252bb35cd72835c2f32f270e58f4dc650ea11b422050b26517b44b9b12"}
Sep 16 19:22:22.458512 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:22:22.458400    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-s44gn" event={"ID":"8f5085f5-78ad-4357-8979-376130311a24","Type":"ContainerDied","Data":"7e8e6dca636e8f7473504042d710bab4f44f7bf827128908f455da8ca0952e36"}
Sep 16 19:22:22.458512 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:22:22.458421    2578 scope.go:117] "RemoveContainer" containerID="de357a252bb35cd72835c2f32f270e58f4dc650ea11b422050b26517b44b9b12"
Sep 16 19:22:22.467716 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:22:22.467698    2578 scope.go:117] "RemoveContainer" containerID="de357a252bb35cd72835c2f32f270e58f4dc650ea11b422050b26517b44b9b12"
Sep 16 19:22:22.467949 ip-10-0-135-119 kubenswrapper[2578]: E0916 19:22:22.467928    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"de357a252bb35cd72835c2f32f270e58f4dc650ea11b422050b26517b44b9b12\": container with ID starting with de357a252bb35cd72835c2f32f270e58f4dc650ea11b422050b26517b44b9b12 not found: ID does not exist" containerID="de357a252bb35cd72835c2f32f270e58f4dc650ea11b422050b26517b44b9b12"
Sep 16 19:22:22.468028 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:22:22.467962    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"de357a252bb35cd72835c2f32f270e58f4dc650ea11b422050b26517b44b9b12"} err="failed to get container status \"de357a252bb35cd72835c2f32f270e58f4dc650ea11b422050b26517b44b9b12\": rpc error: code = NotFound desc = could not find container \"de357a252bb35cd72835c2f32f270e58f4dc650ea11b422050b26517b44b9b12\": container with ID starting with de357a252bb35cd72835c2f32f270e58f4dc650ea11b422050b26517b44b9b12 not found: ID does not exist"
Sep 16 19:22:22.481632 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:22:22.481609    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-s44gn"]
Sep 16 19:22:22.487435 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:22:22.487407    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-s44gn"]
Sep 16 19:22:23.212412 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:22:23.212377    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:22:23.318620 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:22:23.318544    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:22:23.413583 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:22:23.413548    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:22:23.433853 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:22:23.433824    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8f5085f5-78ad-4357-8979-376130311a24" path="/var/lib/kubelet/pods/8f5085f5-78ad-4357-8979-376130311a24/volumes"
Sep 16 19:22:28.280090 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:22:28.280061    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:22:28.810416 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:22:28.810370    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:22:33.180040 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:22:33.179972    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf8679699mzkz"]
Sep 16 19:22:33.180526 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:22:33.180358    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8f5085f5-78ad-4357-8979-376130311a24" containerName="main"
Sep 16 19:22:33.180526 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:22:33.180371    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="8f5085f5-78ad-4357-8979-376130311a24" containerName="main"
Sep 16 19:22:33.180526 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:22:33.180429    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="8f5085f5-78ad-4357-8979-376130311a24" containerName="main"
Sep 16 19:22:33.182861 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:22:33.182844    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf8679699mzkz"
Sep 16 19:22:33.184968 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:22:33.184949    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"default-dockercfg-6t5rj\""
Sep 16 19:22:33.185173 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:22:33.185159    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"openshift-service-ca.crt\""
Sep 16 19:22:33.185709 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:22:33.185694    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"llmisvca2d2d7d499abb359505529ebe02c136-kserve-self-signed-certs\""
Sep 16 19:22:33.185953 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:22:33.185933    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"kube-root-ca.crt\""
Sep 16 19:22:33.197523 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:22:33.197500    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf8679699mzkz"]
Sep 16 19:22:33.220727 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:22:33.220689    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:22:33.265941 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:22:33.265918    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0dc6ee5d-81cc-418c-a6a4-6aa59740840a-tmp-dir\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf8679699mzkz\" (UID: \"0dc6ee5d-81cc-418c-a6a4-6aa59740840a\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf8679699mzkz"
Sep 16 19:22:33.266081 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:22:33.265958    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0dc6ee5d-81cc-418c-a6a4-6aa59740840a-home\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf8679699mzkz\" (UID: \"0dc6ee5d-81cc-418c-a6a4-6aa59740840a\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf8679699mzkz"
Sep 16 19:22:33.266081 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:22:33.266011    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gbtf9\" (UniqueName: \"kubernetes.io/projected/0dc6ee5d-81cc-418c-a6a4-6aa59740840a-kube-api-access-gbtf9\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf8679699mzkz\" (UID: \"0dc6ee5d-81cc-418c-a6a4-6aa59740840a\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf8679699mzkz"
Sep 16 19:22:33.266156 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:22:33.266077    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0dc6ee5d-81cc-418c-a6a4-6aa59740840a-dshm\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf8679699mzkz\" (UID: \"0dc6ee5d-81cc-418c-a6a4-6aa59740840a\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf8679699mzkz"
Sep 16 19:22:33.266156 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:22:33.266102    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0dc6ee5d-81cc-418c-a6a4-6aa59740840a-tls-certs\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf8679699mzkz\" (UID: \"0dc6ee5d-81cc-418c-a6a4-6aa59740840a\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf8679699mzkz"
Sep 16 19:22:33.266156 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:22:33.266127    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0dc6ee5d-81cc-418c-a6a4-6aa59740840a-model-cache\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf8679699mzkz\" (UID: \"0dc6ee5d-81cc-418c-a6a4-6aa59740840a\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf8679699mzkz"
Sep 16 19:22:33.367524 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:22:33.367495    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0dc6ee5d-81cc-418c-a6a4-6aa59740840a-tmp-dir\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf8679699mzkz\" (UID: \"0dc6ee5d-81cc-418c-a6a4-6aa59740840a\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf8679699mzkz"
Sep 16 19:22:33.367659 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:22:33.367537    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0dc6ee5d-81cc-418c-a6a4-6aa59740840a-home\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf8679699mzkz\" (UID: \"0dc6ee5d-81cc-418c-a6a4-6aa59740840a\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf8679699mzkz"
Sep 16 19:22:33.367659 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:22:33.367554    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gbtf9\" (UniqueName: \"kubernetes.io/projected/0dc6ee5d-81cc-418c-a6a4-6aa59740840a-kube-api-access-gbtf9\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf8679699mzkz\" (UID: \"0dc6ee5d-81cc-418c-a6a4-6aa59740840a\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf8679699mzkz"
Sep 16 19:22:33.367659 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:22:33.367588    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0dc6ee5d-81cc-418c-a6a4-6aa59740840a-dshm\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf8679699mzkz\" (UID: \"0dc6ee5d-81cc-418c-a6a4-6aa59740840a\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf8679699mzkz"
Sep 16 19:22:33.367659 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:22:33.367613    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0dc6ee5d-81cc-418c-a6a4-6aa59740840a-tls-certs\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf8679699mzkz\" (UID: \"0dc6ee5d-81cc-418c-a6a4-6aa59740840a\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf8679699mzkz"
Sep 16 19:22:33.367659 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:22:33.367646    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0dc6ee5d-81cc-418c-a6a4-6aa59740840a-model-cache\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf8679699mzkz\" (UID: \"0dc6ee5d-81cc-418c-a6a4-6aa59740840a\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf8679699mzkz"
Sep 16 19:22:33.367963 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:22:33.367939    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0dc6ee5d-81cc-418c-a6a4-6aa59740840a-tmp-dir\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf8679699mzkz\" (UID: \"0dc6ee5d-81cc-418c-a6a4-6aa59740840a\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf8679699mzkz"
Sep 16 19:22:33.368073 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:22:33.368005    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0dc6ee5d-81cc-418c-a6a4-6aa59740840a-home\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf8679699mzkz\" (UID: \"0dc6ee5d-81cc-418c-a6a4-6aa59740840a\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf8679699mzkz"
Sep 16 19:22:33.368073 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:22:33.368030    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0dc6ee5d-81cc-418c-a6a4-6aa59740840a-model-cache\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf8679699mzkz\" (UID: \"0dc6ee5d-81cc-418c-a6a4-6aa59740840a\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf8679699mzkz"
Sep 16 19:22:33.370032 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:22:33.370010    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0dc6ee5d-81cc-418c-a6a4-6aa59740840a-dshm\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf8679699mzkz\" (UID: \"0dc6ee5d-81cc-418c-a6a4-6aa59740840a\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf8679699mzkz"
Sep 16 19:22:33.370407 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:22:33.370388    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0dc6ee5d-81cc-418c-a6a4-6aa59740840a-tls-certs\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf8679699mzkz\" (UID: \"0dc6ee5d-81cc-418c-a6a4-6aa59740840a\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf8679699mzkz"
Sep 16 19:22:33.376936 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:22:33.376910    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gbtf9\" (UniqueName: \"kubernetes.io/projected/0dc6ee5d-81cc-418c-a6a4-6aa59740840a-kube-api-access-gbtf9\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf8679699mzkz\" (UID: \"0dc6ee5d-81cc-418c-a6a4-6aa59740840a\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf8679699mzkz"
Sep 16 19:22:33.493217 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:22:33.493133    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf8679699mzkz"
Sep 16 19:22:33.551766 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:22:33.551736    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:22:33.769134 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:22:33.769081    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf8679699mzkz"]
Sep 16 19:22:33.771445 ip-10-0-135-119 kubenswrapper[2578]: W0916 19:22:33.771410    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0dc6ee5d_81cc_418c_a6a4_6aa59740840a.slice/crio-e618649ba3ac40c15b0f1ef1e77dfe68f0a7b47f8af5b9add4bbe96c409c1127 WatchSource:0}: Error finding container e618649ba3ac40c15b0f1ef1e77dfe68f0a7b47f8af5b9add4bbe96c409c1127: Status 404 returned error can't find the container with id e618649ba3ac40c15b0f1ef1e77dfe68f0a7b47f8af5b9add4bbe96c409c1127
Sep 16 19:22:34.515517 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:22:34.515480    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf8679699mzkz" event={"ID":"0dc6ee5d-81cc-418c-a6a4-6aa59740840a","Type":"ContainerStarted","Data":"2a4ac5f6fd7d1930d10a048180152885e01337c57b729f4c19003348dcaebe8a"}
Sep 16 19:22:34.515953 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:22:34.515543    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf8679699mzkz" event={"ID":"0dc6ee5d-81cc-418c-a6a4-6aa59740840a","Type":"ContainerStarted","Data":"e618649ba3ac40c15b0f1ef1e77dfe68f0a7b47f8af5b9add4bbe96c409c1127"}
Sep 16 19:22:34.541893 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:22:34.541841    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf8679699mzkz" podStartSLOduration=1.541825018 podStartE2EDuration="1.541825018s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:22:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:22:34.540144325 +0000 UTC m=+2535.606635824" watchObservedRunningTime="2026-09-16 19:22:34.541825018 +0000 UTC m=+2535.608316542"
Sep 16 19:22:35.494130 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:22:35.494090    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf8679699mzkz"
Sep 16 19:22:35.851751 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:22:35.851669    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:22:35.959103 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:22:35.959066    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:22:43.493947 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:22:43.493913    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf8679699mzkz"
Sep 16 19:22:43.506393 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:22:43.506358    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf8679699mzkz"
Sep 16 19:22:43.569589 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:22:43.569554    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf8679699mzkz"
Sep 16 19:22:49.843135 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:22:49.843101    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:22:54.130403 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:22:54.130369    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:22:55.868928 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:22:55.868877    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:23:06.262176 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:23:06.262141    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:23:11.949665 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:23:11.949626    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:23:13.216941 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:23:13.216904    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:23:13.312556 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:23:13.312523    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:23:14.777296 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:23:14.777256    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:23:25.740971 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:23:25.740930    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:23:27.146740 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:23:27.146708    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:23:29.887396 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:23:29.887361    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:24:01.718357 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:24:01.718273    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:24:08.088420 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:24:08.088381    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:24:08.500667 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:24:08.500629    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-sche7g5ps"]
Sep 16 19:24:08.504147 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:24:08.504129    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-sche7g5ps"
Sep 16 19:24:08.507154 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:24:08.507121    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-epp-sa-dockercfg-gt7sj\""
Sep 16 19:24:08.507608 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:24:08.507592    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"llmisv08544b88a8d963ffd553cc1f3ed82d16-kserve-self-signed-certs\""
Sep 16 19:24:08.516006 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:24:08.515964    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-sche7g5ps"]
Sep 16 19:24:08.545825 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:24:08.545800    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:24:08.601721 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:24:08.601685    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vrlkl\" (UniqueName: \"kubernetes.io/projected/94b68800-923a-477b-afdf-9f17ca024d69-kube-api-access-vrlkl\") pod \"llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-sche7g5ps\" (UID: \"94b68800-923a-477b-afdf-9f17ca024d69\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-sche7g5ps"
Sep 16 19:24:08.601860 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:24:08.601742    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/94b68800-923a-477b-afdf-9f17ca024d69-tls-certs\") pod \"llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-sche7g5ps\" (UID: \"94b68800-923a-477b-afdf-9f17ca024d69\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-sche7g5ps"
Sep 16 19:24:08.702771 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:24:08.702731    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vrlkl\" (UniqueName: \"kubernetes.io/projected/94b68800-923a-477b-afdf-9f17ca024d69-kube-api-access-vrlkl\") pod \"llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-sche7g5ps\" (UID: \"94b68800-923a-477b-afdf-9f17ca024d69\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-sche7g5ps"
Sep 16 19:24:08.703000 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:24:08.702789    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/94b68800-923a-477b-afdf-9f17ca024d69-tls-certs\") pod \"llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-sche7g5ps\" (UID: \"94b68800-923a-477b-afdf-9f17ca024d69\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-sche7g5ps"
Sep 16 19:24:08.705557 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:24:08.705533    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/94b68800-923a-477b-afdf-9f17ca024d69-tls-certs\") pod \"llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-sche7g5ps\" (UID: \"94b68800-923a-477b-afdf-9f17ca024d69\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-sche7g5ps"
Sep 16 19:24:08.730040 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:24:08.730015    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vrlkl\" (UniqueName: \"kubernetes.io/projected/94b68800-923a-477b-afdf-9f17ca024d69-kube-api-access-vrlkl\") pod \"llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-sche7g5ps\" (UID: \"94b68800-923a-477b-afdf-9f17ca024d69\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-sche7g5ps"
Sep 16 19:24:08.814736 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:24:08.814643    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-sche7g5ps"
Sep 16 19:24:09.085638 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:24:09.085557    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-sche7g5ps"]
Sep 16 19:24:09.085900 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:24:09.085871    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:24:09.088732 ip-10-0-135-119 kubenswrapper[2578]: W0916 19:24:09.088702    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod94b68800_923a_477b_afdf_9f17ca024d69.slice/crio-4ba94b63fa661fc7ed3c62d6a4ab919a464dff878bb28cb29838d50577701689 WatchSource:0}: Error finding container 4ba94b63fa661fc7ed3c62d6a4ab919a464dff878bb28cb29838d50577701689: Status 404 returned error can't find the container with id 4ba94b63fa661fc7ed3c62d6a4ab919a464dff878bb28cb29838d50577701689
Sep 16 19:24:09.186342 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:24:09.186314    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:24:09.956072 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:24:09.956033    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-sche7g5ps" event={"ID":"94b68800-923a-477b-afdf-9f17ca024d69","Type":"ContainerStarted","Data":"2b739e1d638509599a44db482145b729674809c119cbf2e9d84f83db9748819d"}
Sep 16 19:24:09.956237 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:24:09.956098    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-sche7g5ps" event={"ID":"94b68800-923a-477b-afdf-9f17ca024d69","Type":"ContainerStarted","Data":"4ba94b63fa661fc7ed3c62d6a4ab919a464dff878bb28cb29838d50577701689"}
Sep 16 19:24:09.956299 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:24:09.956229    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-sche7g5ps"
Sep 16 19:24:09.984113 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:24:09.984070    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-sche7g5ps" podStartSLOduration=1.984056506 podStartE2EDuration="1.984056506s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:24:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:24:09.98240511 +0000 UTC m=+2631.048896623" watchObservedRunningTime="2026-09-16 19:24:09.984056506 +0000 UTC m=+2631.050548031"
Sep 16 19:24:21.181733 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:24:21.181697    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:24:40.964220 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:24:40.964189    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-sche7g5ps"
Sep 16 19:24:42.616671 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:24:42.616639    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-sche7g5ps"]
Sep 16 19:24:42.619848 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:24:42.619817    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:24:42.687396 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:24:42.687360    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:24:50.964182 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:24:50.964142    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-sche7g5ps" podUID="94b68800-923a-477b-afdf-9f17ca024d69" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:24:57.637107 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:24:57.637046    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-sche7g5ps" podUID="94b68800-923a-477b-afdf-9f17ca024d69" containerName="main" containerID="cri-o://2b739e1d638509599a44db482145b729674809c119cbf2e9d84f83db9748819d" gracePeriod=45
Sep 16 19:25:00.964016 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:25:00.963960    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-sche7g5ps" podUID="94b68800-923a-477b-afdf-9f17ca024d69" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:25:10.963719 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:25:10.963633    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-sche7g5ps" podUID="94b68800-923a-477b-afdf-9f17ca024d69" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:25:10.964134 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:25:10.963742    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-sche7g5ps"
Sep 16 19:25:14.986349 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:25:14.986311    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:25:15.099541 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:25:15.099505    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:25:15.201931 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:25:15.201894    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:25:15.318594 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:25:15.318474    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:25:15.593070 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:25:15.592969    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:25:19.573562 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:25:19.573534    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-tmrkx_4c2078bb-281a-4aa1-8e38-238897865434/console-operator/2.log"
Sep 16 19:25:19.580245 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:25:19.580220    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-tmrkx_4c2078bb-281a-4aa1-8e38-238897865434/console-operator/2.log"
Sep 16 19:25:19.581787 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:25:19.581768    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-k6gvs_6012728d-aee6-4f93-ab6e-fad616ef36e8/ovn-acl-logging/0.log"
Sep 16 19:25:19.592221 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:25:19.592202    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-k6gvs_6012728d-aee6-4f93-ab6e-fad616ef36e8/ovn-acl-logging/0.log"
Sep 16 19:25:20.051012 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:25:20.050960    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:25:20.964504 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:25:20.964465    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-sche7g5ps" podUID="94b68800-923a-477b-afdf-9f17ca024d69" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:25:27.888402 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:25:27.888373    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-sche7g5ps"
Sep 16 19:25:28.005853 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:25:28.005775    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/94b68800-923a-477b-afdf-9f17ca024d69-tls-certs\") pod \"94b68800-923a-477b-afdf-9f17ca024d69\" (UID: \"94b68800-923a-477b-afdf-9f17ca024d69\") "
Sep 16 19:25:28.006021 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:25:28.005859    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vrlkl\" (UniqueName: \"kubernetes.io/projected/94b68800-923a-477b-afdf-9f17ca024d69-kube-api-access-vrlkl\") pod \"94b68800-923a-477b-afdf-9f17ca024d69\" (UID: \"94b68800-923a-477b-afdf-9f17ca024d69\") "
Sep 16 19:25:28.008134 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:25:28.008101    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/94b68800-923a-477b-afdf-9f17ca024d69-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "94b68800-923a-477b-afdf-9f17ca024d69" (UID: "94b68800-923a-477b-afdf-9f17ca024d69"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:25:28.008246 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:25:28.008159    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/94b68800-923a-477b-afdf-9f17ca024d69-kube-api-access-vrlkl" (OuterVolumeSpecName: "kube-api-access-vrlkl") pod "94b68800-923a-477b-afdf-9f17ca024d69" (UID: "94b68800-923a-477b-afdf-9f17ca024d69"). InnerVolumeSpecName "kube-api-access-vrlkl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:25:28.107047 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:25:28.107011    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vrlkl\" (UniqueName: \"kubernetes.io/projected/94b68800-923a-477b-afdf-9f17ca024d69-kube-api-access-vrlkl\") on node \"ip-10-0-135-119.ec2.internal\" DevicePath \"\""
Sep 16 19:25:28.107047 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:25:28.107041    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/94b68800-923a-477b-afdf-9f17ca024d69-tls-certs\") on node \"ip-10-0-135-119.ec2.internal\" DevicePath \"\""
Sep 16 19:25:28.328643 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:25:28.328558    2578 generic.go:356] "Generic (PLEG): container finished" podID="94b68800-923a-477b-afdf-9f17ca024d69" containerID="2b739e1d638509599a44db482145b729674809c119cbf2e9d84f83db9748819d" exitCode=0
Sep 16 19:25:28.328643 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:25:28.328602    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-sche7g5ps"
Sep 16 19:25:28.328819 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:25:28.328643    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-sche7g5ps" event={"ID":"94b68800-923a-477b-afdf-9f17ca024d69","Type":"ContainerDied","Data":"2b739e1d638509599a44db482145b729674809c119cbf2e9d84f83db9748819d"}
Sep 16 19:25:28.328819 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:25:28.328714    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-sche7g5ps" event={"ID":"94b68800-923a-477b-afdf-9f17ca024d69","Type":"ContainerDied","Data":"4ba94b63fa661fc7ed3c62d6a4ab919a464dff878bb28cb29838d50577701689"}
Sep 16 19:25:28.328819 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:25:28.328736    2578 scope.go:117] "RemoveContainer" containerID="2b739e1d638509599a44db482145b729674809c119cbf2e9d84f83db9748819d"
Sep 16 19:25:28.337801 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:25:28.337776    2578 scope.go:117] "RemoveContainer" containerID="2b739e1d638509599a44db482145b729674809c119cbf2e9d84f83db9748819d"
Sep 16 19:25:28.338134 ip-10-0-135-119 kubenswrapper[2578]: E0916 19:25:28.338118    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2b739e1d638509599a44db482145b729674809c119cbf2e9d84f83db9748819d\": container with ID starting with 2b739e1d638509599a44db482145b729674809c119cbf2e9d84f83db9748819d not found: ID does not exist" containerID="2b739e1d638509599a44db482145b729674809c119cbf2e9d84f83db9748819d"
Sep 16 19:25:28.338194 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:25:28.338141    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2b739e1d638509599a44db482145b729674809c119cbf2e9d84f83db9748819d"} err="failed to get container status \"2b739e1d638509599a44db482145b729674809c119cbf2e9d84f83db9748819d\": rpc error: code = NotFound desc = could not find container \"2b739e1d638509599a44db482145b729674809c119cbf2e9d84f83db9748819d\": container with ID starting with 2b739e1d638509599a44db482145b729674809c119cbf2e9d84f83db9748819d not found: ID does not exist"
Sep 16 19:25:28.350751 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:25:28.350725    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-sche7g5ps"]
Sep 16 19:25:28.355666 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:25:28.355643    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-sche7g5ps"]
Sep 16 19:25:29.434019 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:25:29.433967    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="94b68800-923a-477b-afdf-9f17ca024d69" path="/var/lib/kubelet/pods/94b68800-923a-477b-afdf-9f17ca024d69/volumes"
Sep 16 19:25:35.565546 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:25:35.565514    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:25:47.774454 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:25:47.774418    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:25:48.994009 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:25:48.993953    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:25:48.994386 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:25:48.994349    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf8679699mzkz"]
Sep 16 19:25:49.004413 ip-10-0-135-119 kubenswrapper[2578]: E0916 19:25:49.004368    2578 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 16 19:25:49.004413 ip-10-0-135-119 kubenswrapper[2578]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 19:25:49.004413 ip-10-0-135-119 kubenswrapper[2578]:         , stdout: , stderr: , exit code -1
Sep 16 19:25:49.004413 ip-10-0-135-119 kubenswrapper[2578]:  > containerID="2a4ac5f6fd7d1930d10a048180152885e01337c57b729f4c19003348dcaebe8a" cmd=["/bin/sleep","15"]
Sep 16 19:25:49.004719 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:25:49.004435    2578 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf8679699mzkz" message="" err=<
Sep 16 19:25:49.004719 ip-10-0-135-119 kubenswrapper[2578]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 19:25:49.004719 ip-10-0-135-119 kubenswrapper[2578]:         , stdout: , stderr: , exit code -1
Sep 16 19:25:49.004719 ip-10-0-135-119 kubenswrapper[2578]:  >
Sep 16 19:25:49.004719 ip-10-0-135-119 kubenswrapper[2578]: E0916 19:25:49.004451    2578 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 16 19:25:49.004719 ip-10-0-135-119 kubenswrapper[2578]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 19:25:49.004719 ip-10-0-135-119 kubenswrapper[2578]:         , stdout: , stderr: , exit code -1
Sep 16 19:25:49.004719 ip-10-0-135-119 kubenswrapper[2578]:  > pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf8679699mzkz" podUID="0dc6ee5d-81cc-418c-a6a4-6aa59740840a" containerName="main" containerID="cri-o://2a4ac5f6fd7d1930d10a048180152885e01337c57b729f4c19003348dcaebe8a"
Sep 16 19:25:49.004719 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:25:49.004512    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf8679699mzkz" podUID="0dc6ee5d-81cc-418c-a6a4-6aa59740840a" containerName="main" containerID="cri-o://2a4ac5f6fd7d1930d10a048180152885e01337c57b729f4c19003348dcaebe8a" gracePeriod=60
Sep 16 19:25:49.085355 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:25:49.085316    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:25:49.359023 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:25:49.358998    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf8679699mzkz"
Sep 16 19:25:49.386754 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:25:49.386721    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0dc6ee5d-81cc-418c-a6a4-6aa59740840a-home\") pod \"0dc6ee5d-81cc-418c-a6a4-6aa59740840a\" (UID: \"0dc6ee5d-81cc-418c-a6a4-6aa59740840a\") "
Sep 16 19:25:49.386919 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:25:49.386795    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0dc6ee5d-81cc-418c-a6a4-6aa59740840a-dshm\") pod \"0dc6ee5d-81cc-418c-a6a4-6aa59740840a\" (UID: \"0dc6ee5d-81cc-418c-a6a4-6aa59740840a\") "
Sep 16 19:25:49.386919 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:25:49.386825    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0dc6ee5d-81cc-418c-a6a4-6aa59740840a-model-cache\") pod \"0dc6ee5d-81cc-418c-a6a4-6aa59740840a\" (UID: \"0dc6ee5d-81cc-418c-a6a4-6aa59740840a\") "
Sep 16 19:25:49.386919 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:25:49.386880    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gbtf9\" (UniqueName: \"kubernetes.io/projected/0dc6ee5d-81cc-418c-a6a4-6aa59740840a-kube-api-access-gbtf9\") pod \"0dc6ee5d-81cc-418c-a6a4-6aa59740840a\" (UID: \"0dc6ee5d-81cc-418c-a6a4-6aa59740840a\") "
Sep 16 19:25:49.387436 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:25:49.386954    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0dc6ee5d-81cc-418c-a6a4-6aa59740840a-tls-certs\") pod \"0dc6ee5d-81cc-418c-a6a4-6aa59740840a\" (UID: \"0dc6ee5d-81cc-418c-a6a4-6aa59740840a\") "
Sep 16 19:25:49.387436 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:25:49.386991    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0dc6ee5d-81cc-418c-a6a4-6aa59740840a-home" (OuterVolumeSpecName: "home") pod "0dc6ee5d-81cc-418c-a6a4-6aa59740840a" (UID: "0dc6ee5d-81cc-418c-a6a4-6aa59740840a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:25:49.387436 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:25:49.387025    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0dc6ee5d-81cc-418c-a6a4-6aa59740840a-tmp-dir\") pod \"0dc6ee5d-81cc-418c-a6a4-6aa59740840a\" (UID: \"0dc6ee5d-81cc-418c-a6a4-6aa59740840a\") "
Sep 16 19:25:49.387436 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:25:49.387122    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0dc6ee5d-81cc-418c-a6a4-6aa59740840a-model-cache" (OuterVolumeSpecName: "model-cache") pod "0dc6ee5d-81cc-418c-a6a4-6aa59740840a" (UID: "0dc6ee5d-81cc-418c-a6a4-6aa59740840a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:25:49.387436 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:25:49.387317    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0dc6ee5d-81cc-418c-a6a4-6aa59740840a-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "0dc6ee5d-81cc-418c-a6a4-6aa59740840a" (UID: "0dc6ee5d-81cc-418c-a6a4-6aa59740840a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:25:49.387436 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:25:49.387358    2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0dc6ee5d-81cc-418c-a6a4-6aa59740840a-home\") on node \"ip-10-0-135-119.ec2.internal\" DevicePath \"\""
Sep 16 19:25:49.387436 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:25:49.387377    2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0dc6ee5d-81cc-418c-a6a4-6aa59740840a-model-cache\") on node \"ip-10-0-135-119.ec2.internal\" DevicePath \"\""
Sep 16 19:25:49.389340 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:25:49.389309    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0dc6ee5d-81cc-418c-a6a4-6aa59740840a-dshm" (OuterVolumeSpecName: "dshm") pod "0dc6ee5d-81cc-418c-a6a4-6aa59740840a" (UID: "0dc6ee5d-81cc-418c-a6a4-6aa59740840a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:25:49.389454 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:25:49.389376    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0dc6ee5d-81cc-418c-a6a4-6aa59740840a-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "0dc6ee5d-81cc-418c-a6a4-6aa59740840a" (UID: "0dc6ee5d-81cc-418c-a6a4-6aa59740840a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:25:49.389546 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:25:49.389527    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0dc6ee5d-81cc-418c-a6a4-6aa59740840a-kube-api-access-gbtf9" (OuterVolumeSpecName: "kube-api-access-gbtf9") pod "0dc6ee5d-81cc-418c-a6a4-6aa59740840a" (UID: "0dc6ee5d-81cc-418c-a6a4-6aa59740840a"). InnerVolumeSpecName "kube-api-access-gbtf9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:25:49.426994 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:25:49.426948    2578 generic.go:356] "Generic (PLEG): container finished" podID="0dc6ee5d-81cc-418c-a6a4-6aa59740840a" containerID="2a4ac5f6fd7d1930d10a048180152885e01337c57b729f4c19003348dcaebe8a" exitCode=0
Sep 16 19:25:49.427165 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:25:49.427018    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf8679699mzkz"
Sep 16 19:25:49.427165 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:25:49.427035    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf8679699mzkz" event={"ID":"0dc6ee5d-81cc-418c-a6a4-6aa59740840a","Type":"ContainerDied","Data":"2a4ac5f6fd7d1930d10a048180152885e01337c57b729f4c19003348dcaebe8a"}
Sep 16 19:25:49.427165 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:25:49.427091    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf8679699mzkz" event={"ID":"0dc6ee5d-81cc-418c-a6a4-6aa59740840a","Type":"ContainerDied","Data":"e618649ba3ac40c15b0f1ef1e77dfe68f0a7b47f8af5b9add4bbe96c409c1127"}
Sep 16 19:25:49.427165 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:25:49.427111    2578 scope.go:117] "RemoveContainer" containerID="2a4ac5f6fd7d1930d10a048180152885e01337c57b729f4c19003348dcaebe8a"
Sep 16 19:25:49.438631 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:25:49.438595    2578 scope.go:117] "RemoveContainer" containerID="2a4ac5f6fd7d1930d10a048180152885e01337c57b729f4c19003348dcaebe8a"
Sep 16 19:25:49.439032 ip-10-0-135-119 kubenswrapper[2578]: E0916 19:25:49.438972    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2a4ac5f6fd7d1930d10a048180152885e01337c57b729f4c19003348dcaebe8a\": container with ID starting with 2a4ac5f6fd7d1930d10a048180152885e01337c57b729f4c19003348dcaebe8a not found: ID does not exist" containerID="2a4ac5f6fd7d1930d10a048180152885e01337c57b729f4c19003348dcaebe8a"
Sep 16 19:25:49.439159 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:25:49.439033    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2a4ac5f6fd7d1930d10a048180152885e01337c57b729f4c19003348dcaebe8a"} err="failed to get container status \"2a4ac5f6fd7d1930d10a048180152885e01337c57b729f4c19003348dcaebe8a\": rpc error: code = NotFound desc = could not find container \"2a4ac5f6fd7d1930d10a048180152885e01337c57b729f4c19003348dcaebe8a\": container with ID starting with 2a4ac5f6fd7d1930d10a048180152885e01337c57b729f4c19003348dcaebe8a not found: ID does not exist"
Sep 16 19:25:49.449911 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:25:49.449890    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf8679699mzkz"]
Sep 16 19:25:49.454563 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:25:49.454540    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf8679699mzkz"]
Sep 16 19:25:49.488330 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:25:49.488306    2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0dc6ee5d-81cc-418c-a6a4-6aa59740840a-dshm\") on node \"ip-10-0-135-119.ec2.internal\" DevicePath \"\""
Sep 16 19:25:49.488421 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:25:49.488396    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gbtf9\" (UniqueName: \"kubernetes.io/projected/0dc6ee5d-81cc-418c-a6a4-6aa59740840a-kube-api-access-gbtf9\") on node \"ip-10-0-135-119.ec2.internal\" DevicePath \"\""
Sep 16 19:25:49.488421 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:25:49.488406    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0dc6ee5d-81cc-418c-a6a4-6aa59740840a-tls-certs\") on node \"ip-10-0-135-119.ec2.internal\" DevicePath \"\""
Sep 16 19:25:49.488421 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:25:49.488417    2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0dc6ee5d-81cc-418c-a6a4-6aa59740840a-tmp-dir\") on node \"ip-10-0-135-119.ec2.internal\" DevicePath \"\""
Sep 16 19:25:51.433595 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:25:51.433558    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0dc6ee5d-81cc-418c-a6a4-6aa59740840a" path="/var/lib/kubelet/pods/0dc6ee5d-81cc-418c-a6a4-6aa59740840a/volumes"
Sep 16 19:25:57.343477 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:25:57.343448    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:26:03.373772 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:26:03.373724    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:26:03.534664 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:26:03.534611    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:26:37.805375 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:26:37.805340    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:26:37.897189 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:26:37.897137    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:26:44.789201 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:26:44.789113    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:26:50.755507 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:26:50.755473    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:26:50.856330 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:26:50.856297    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:27:00.795358 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:27:00.795319    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:27:01.131576 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:27:01.131543    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-sche2g74f"]
Sep 16 19:27:01.131989 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:27:01.131955    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0dc6ee5d-81cc-418c-a6a4-6aa59740840a" containerName="main"
Sep 16 19:27:01.132089 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:27:01.131993    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="0dc6ee5d-81cc-418c-a6a4-6aa59740840a" containerName="main"
Sep 16 19:27:01.132089 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:27:01.132029    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="94b68800-923a-477b-afdf-9f17ca024d69" containerName="main"
Sep 16 19:27:01.132089 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:27:01.132038    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="94b68800-923a-477b-afdf-9f17ca024d69" containerName="main"
Sep 16 19:27:01.132246 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:27:01.132143    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="0dc6ee5d-81cc-418c-a6a4-6aa59740840a" containerName="main"
Sep 16 19:27:01.132246 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:27:01.132158    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="94b68800-923a-477b-afdf-9f17ca024d69" containerName="main"
Sep 16 19:27:01.135240 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:27:01.135217    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-sche2g74f"
Sep 16 19:27:01.138080 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:27:01.138057    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-5bc7dccd\"/\"default-dockercfg-9279j\""
Sep 16 19:27:01.138080 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:27:01.138080    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-5bc7dccd\"/\"llmisv3e414c2ba058a022dfd694dbcbac5b51-kserve-self-signed-certs\""
Sep 16 19:27:01.138268 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:27:01.138061    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-5bc7dccd\"/\"openshift-service-ca.crt\""
Sep 16 19:27:01.138268 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:27:01.138141    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-5bc7dccd\"/\"kube-root-ca.crt\""
Sep 16 19:27:01.138268 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:27:01.138191    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-5bc7dccd\"/\"llmisvc-model-fb-opt-125m-with-7ca60146-epp-sa-dockercfg-t45wk\""
Sep 16 19:27:01.151830 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:27:01.151804    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-sche2g74f"]
Sep 16 19:27:01.181048 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:27:01.181021    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:27:01.199263 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:27:01.199229    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d394e9e0-590e-4bf4-992a-cb530338552e-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-sche2g74f\" (UID: \"d394e9e0-590e-4bf4-992a-cb530338552e\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-sche2g74f"
Sep 16 19:27:01.199410 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:27:01.199280    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xdmm2\" (UniqueName: \"kubernetes.io/projected/d394e9e0-590e-4bf4-992a-cb530338552e-kube-api-access-xdmm2\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-sche2g74f\" (UID: \"d394e9e0-590e-4bf4-992a-cb530338552e\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-sche2g74f"
Sep 16 19:27:01.300113 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:27:01.300075    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d394e9e0-590e-4bf4-992a-cb530338552e-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-sche2g74f\" (UID: \"d394e9e0-590e-4bf4-992a-cb530338552e\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-sche2g74f"
Sep 16 19:27:01.300293 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:27:01.300122    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xdmm2\" (UniqueName: \"kubernetes.io/projected/d394e9e0-590e-4bf4-992a-cb530338552e-kube-api-access-xdmm2\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-sche2g74f\" (UID: \"d394e9e0-590e-4bf4-992a-cb530338552e\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-sche2g74f"
Sep 16 19:27:01.302838 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:27:01.302815    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d394e9e0-590e-4bf4-992a-cb530338552e-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-sche2g74f\" (UID: \"d394e9e0-590e-4bf4-992a-cb530338552e\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-sche2g74f"
Sep 16 19:27:01.309628 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:27:01.309599    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xdmm2\" (UniqueName: \"kubernetes.io/projected/d394e9e0-590e-4bf4-992a-cb530338552e-kube-api-access-xdmm2\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-sche2g74f\" (UID: \"d394e9e0-590e-4bf4-992a-cb530338552e\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-sche2g74f"
Sep 16 19:27:01.446788 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:27:01.446702    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-sche2g74f"
Sep 16 19:27:01.719852 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:27:01.719778    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-sche2g74f"]
Sep 16 19:27:01.723444 ip-10-0-135-119 kubenswrapper[2578]: W0916 19:27:01.723415    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd394e9e0_590e_4bf4_992a_cb530338552e.slice/crio-9ad4bcdc38f6df492719b455665e2822fd0690f73d62b4a22183653caf9668b1 WatchSource:0}: Error finding container 9ad4bcdc38f6df492719b455665e2822fd0690f73d62b4a22183653caf9668b1: Status 404 returned error can't find the container with id 9ad4bcdc38f6df492719b455665e2822fd0690f73d62b4a22183653caf9668b1
Sep 16 19:27:01.757334 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:27:01.757305    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-sche2g74f" event={"ID":"d394e9e0-590e-4bf4-992a-cb530338552e","Type":"ContainerStarted","Data":"9ad4bcdc38f6df492719b455665e2822fd0690f73d62b4a22183653caf9668b1"}
Sep 16 19:27:02.763855 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:27:02.763816    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-sche2g74f" event={"ID":"d394e9e0-590e-4bf4-992a-cb530338552e","Type":"ContainerStarted","Data":"1ef36570a9da83c9e5c080600845f0bf8aca2b1955ade0d774d5de2265302dbd"}
Sep 16 19:27:02.764356 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:27:02.763950    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-sche2g74f"
Sep 16 19:27:02.783517 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:27:02.783465    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-sche2g74f" podStartSLOduration=1.7834504089999998 podStartE2EDuration="1.783450409s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:27:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:27:02.782024893 +0000 UTC m=+2803.848516411" watchObservedRunningTime="2026-09-16 19:27:02.783450409 +0000 UTC m=+2803.849941897"
Sep 16 19:27:04.642803 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:27:04.642768    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:27:15.797999 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:27:15.797946    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:27:24.474572 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:27:24.474539    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:27:24.657044 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:27:24.657008    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:27:24.755402 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:27:24.755329    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:27:24.937586 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:27:24.937554    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:27:25.038336 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:27:25.038269    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:27:27.953401 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:27:27.953366    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:27:28.865510 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:27:28.865478    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:27:29.725845 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:27:29.725812    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:27:33.771689 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:27:33.771657    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-sche2g74f"
Sep 16 19:27:35.617541 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:27:35.617504    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:27:36.472278 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:27:36.472245    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:28:07.224870 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:28:07.224835    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:28:13.016489 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:28:13.016454    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:28:13.082760 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:28:13.082705    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:28:46.794386 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:28:46.794334    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-sche2g74f"]
Sep 16 19:28:46.815568 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:28:46.815540    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:28:48.107020 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:28:48.106961    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:28:53.771804 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:28:53.771760    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-sche2g74f" podUID="d394e9e0-590e-4bf4-992a-cb530338552e" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:29:01.815135 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:29:01.815093    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-sche2g74f" podUID="d394e9e0-590e-4bf4-992a-cb530338552e" containerName="main" containerID="cri-o://1ef36570a9da83c9e5c080600845f0bf8aca2b1955ade0d774d5de2265302dbd" gracePeriod=45
Sep 16 19:29:03.771819 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:29:03.771778    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-sche2g74f" podUID="d394e9e0-590e-4bf4-992a-cb530338552e" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:29:13.770907 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:29:13.770860    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-sche2g74f" podUID="d394e9e0-590e-4bf4-992a-cb530338552e" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:29:13.771352 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:29:13.771018    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-sche2g74f"
Sep 16 19:29:23.771598 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:29:23.771555    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-sche2g74f" podUID="d394e9e0-590e-4bf4-992a-cb530338552e" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:29:32.065435 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:29:32.065410    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-sche2g74f"
Sep 16 19:29:32.195965 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:29:32.195930    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xdmm2\" (UniqueName: \"kubernetes.io/projected/d394e9e0-590e-4bf4-992a-cb530338552e-kube-api-access-xdmm2\") pod \"d394e9e0-590e-4bf4-992a-cb530338552e\" (UID: \"d394e9e0-590e-4bf4-992a-cb530338552e\") "
Sep 16 19:29:32.196169 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:29:32.196082    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d394e9e0-590e-4bf4-992a-cb530338552e-tls-certs\") pod \"d394e9e0-590e-4bf4-992a-cb530338552e\" (UID: \"d394e9e0-590e-4bf4-992a-cb530338552e\") "
Sep 16 19:29:32.198255 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:29:32.198223    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d394e9e0-590e-4bf4-992a-cb530338552e-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "d394e9e0-590e-4bf4-992a-cb530338552e" (UID: "d394e9e0-590e-4bf4-992a-cb530338552e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:29:32.198391 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:29:32.198334    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d394e9e0-590e-4bf4-992a-cb530338552e-kube-api-access-xdmm2" (OuterVolumeSpecName: "kube-api-access-xdmm2") pod "d394e9e0-590e-4bf4-992a-cb530338552e" (UID: "d394e9e0-590e-4bf4-992a-cb530338552e"). InnerVolumeSpecName "kube-api-access-xdmm2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:29:32.297605 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:29:32.297572    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d394e9e0-590e-4bf4-992a-cb530338552e-tls-certs\") on node \"ip-10-0-135-119.ec2.internal\" DevicePath \"\""
Sep 16 19:29:32.297605 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:29:32.297601    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xdmm2\" (UniqueName: \"kubernetes.io/projected/d394e9e0-590e-4bf4-992a-cb530338552e-kube-api-access-xdmm2\") on node \"ip-10-0-135-119.ec2.internal\" DevicePath \"\""
Sep 16 19:29:32.461364 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:29:32.461278    2578 generic.go:356] "Generic (PLEG): container finished" podID="d394e9e0-590e-4bf4-992a-cb530338552e" containerID="1ef36570a9da83c9e5c080600845f0bf8aca2b1955ade0d774d5de2265302dbd" exitCode=0
Sep 16 19:29:32.461364 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:29:32.461318    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-sche2g74f"
Sep 16 19:29:32.461559 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:29:32.461364    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-sche2g74f" event={"ID":"d394e9e0-590e-4bf4-992a-cb530338552e","Type":"ContainerDied","Data":"1ef36570a9da83c9e5c080600845f0bf8aca2b1955ade0d774d5de2265302dbd"}
Sep 16 19:29:32.461559 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:29:32.461413    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-sche2g74f" event={"ID":"d394e9e0-590e-4bf4-992a-cb530338552e","Type":"ContainerDied","Data":"9ad4bcdc38f6df492719b455665e2822fd0690f73d62b4a22183653caf9668b1"}
Sep 16 19:29:32.461559 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:29:32.461436    2578 scope.go:117] "RemoveContainer" containerID="1ef36570a9da83c9e5c080600845f0bf8aca2b1955ade0d774d5de2265302dbd"
Sep 16 19:29:32.470765 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:29:32.470750    2578 scope.go:117] "RemoveContainer" containerID="1ef36570a9da83c9e5c080600845f0bf8aca2b1955ade0d774d5de2265302dbd"
Sep 16 19:29:32.471014 ip-10-0-135-119 kubenswrapper[2578]: E0916 19:29:32.470992    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1ef36570a9da83c9e5c080600845f0bf8aca2b1955ade0d774d5de2265302dbd\": container with ID starting with 1ef36570a9da83c9e5c080600845f0bf8aca2b1955ade0d774d5de2265302dbd not found: ID does not exist" containerID="1ef36570a9da83c9e5c080600845f0bf8aca2b1955ade0d774d5de2265302dbd"
Sep 16 19:29:32.471074 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:29:32.471022    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1ef36570a9da83c9e5c080600845f0bf8aca2b1955ade0d774d5de2265302dbd"} err="failed to get container status \"1ef36570a9da83c9e5c080600845f0bf8aca2b1955ade0d774d5de2265302dbd\": rpc error: code = NotFound desc = could not find container \"1ef36570a9da83c9e5c080600845f0bf8aca2b1955ade0d774d5de2265302dbd\": container with ID starting with 1ef36570a9da83c9e5c080600845f0bf8aca2b1955ade0d774d5de2265302dbd not found: ID does not exist"
Sep 16 19:29:32.484080 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:29:32.484059    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-sche2g74f"]
Sep 16 19:29:32.488492 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:29:32.488473    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-sche2g74f"]
Sep 16 19:29:33.434513 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:29:33.434474    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d394e9e0-590e-4bf4-992a-cb530338552e" path="/var/lib/kubelet/pods/d394e9e0-590e-4bf4-992a-cb530338552e/volumes"
Sep 16 19:29:36.900111 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:29:36.900080    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:29:36.999542 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:29:36.999488    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:29:43.408783 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:29:43.408694    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:29:45.516118 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:29:45.516081    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:29:45.718172 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:29:45.718142    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schel58gb"]
Sep 16 19:29:45.718499 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:29:45.718487    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d394e9e0-590e-4bf4-992a-cb530338552e" containerName="main"
Sep 16 19:29:45.718543 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:29:45.718500    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="d394e9e0-590e-4bf4-992a-cb530338552e" containerName="main"
Sep 16 19:29:45.718578 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:29:45.718555    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="d394e9e0-590e-4bf4-992a-cb530338552e" containerName="main"
Sep 16 19:29:45.722966 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:29:45.722943    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schel58gb"
Sep 16 19:29:45.725671 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:29:45.725638    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-733055c7\"/\"openshift-service-ca.crt\""
Sep 16 19:29:45.725671 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:29:45.725657    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-733055c7\"/\"llmisv77ff2528d3e9b4972cd9335229fce9f0-kserve-self-signed-certs\""
Sep 16 19:29:45.725913 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:29:45.725675    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-733055c7\"/\"default-dockercfg-jwfgz\""
Sep 16 19:29:45.725913 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:29:45.725762    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-733055c7\"/\"llmisvc-model-fb-opt-125m-with-ba4d693a-epp-sa-dockercfg-7rw2p\""
Sep 16 19:29:45.725913 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:29:45.725769    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-733055c7\"/\"kube-root-ca.crt\""
Sep 16 19:29:45.730064 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:29:45.730043    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schel58gb"]
Sep 16 19:29:45.747654 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:29:45.747625    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:29:45.813077 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:29:45.812962    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/44134c47-d2d2-46ca-95a2-fe25c509b99b-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schel58gb\" (UID: \"44134c47-d2d2-46ca-95a2-fe25c509b99b\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schel58gb"
Sep 16 19:29:45.813077 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:29:45.813036    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gdwtb\" (UniqueName: \"kubernetes.io/projected/44134c47-d2d2-46ca-95a2-fe25c509b99b-kube-api-access-gdwtb\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schel58gb\" (UID: \"44134c47-d2d2-46ca-95a2-fe25c509b99b\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schel58gb"
Sep 16 19:29:45.913629 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:29:45.913592    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/44134c47-d2d2-46ca-95a2-fe25c509b99b-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schel58gb\" (UID: \"44134c47-d2d2-46ca-95a2-fe25c509b99b\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schel58gb"
Sep 16 19:29:45.913800 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:29:45.913633    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gdwtb\" (UniqueName: \"kubernetes.io/projected/44134c47-d2d2-46ca-95a2-fe25c509b99b-kube-api-access-gdwtb\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schel58gb\" (UID: \"44134c47-d2d2-46ca-95a2-fe25c509b99b\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schel58gb"
Sep 16 19:29:45.916307 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:29:45.916276    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/44134c47-d2d2-46ca-95a2-fe25c509b99b-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schel58gb\" (UID: \"44134c47-d2d2-46ca-95a2-fe25c509b99b\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schel58gb"
Sep 16 19:29:45.922859 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:29:45.922832    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gdwtb\" (UniqueName: \"kubernetes.io/projected/44134c47-d2d2-46ca-95a2-fe25c509b99b-kube-api-access-gdwtb\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schel58gb\" (UID: \"44134c47-d2d2-46ca-95a2-fe25c509b99b\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schel58gb"
Sep 16 19:29:46.035184 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:29:46.035146    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schel58gb"
Sep 16 19:29:46.311443 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:29:46.311414    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schel58gb"]
Sep 16 19:29:46.313085 ip-10-0-135-119 kubenswrapper[2578]: W0916 19:29:46.313057    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod44134c47_d2d2_46ca_95a2_fe25c509b99b.slice/crio-61716d56dc0688c4e9287753272627d1ed620f8df735542874cab22e9a425a12 WatchSource:0}: Error finding container 61716d56dc0688c4e9287753272627d1ed620f8df735542874cab22e9a425a12: Status 404 returned error can't find the container with id 61716d56dc0688c4e9287753272627d1ed620f8df735542874cab22e9a425a12
Sep 16 19:29:46.528193 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:29:46.528160    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schel58gb" event={"ID":"44134c47-d2d2-46ca-95a2-fe25c509b99b","Type":"ContainerStarted","Data":"a733f6bbb355ae35f380dbe420dfc1c4cf50dda699294ca2dd4ce0ea50159af7"}
Sep 16 19:29:46.528572 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:29:46.528212    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schel58gb" event={"ID":"44134c47-d2d2-46ca-95a2-fe25c509b99b","Type":"ContainerStarted","Data":"61716d56dc0688c4e9287753272627d1ed620f8df735542874cab22e9a425a12"}
Sep 16 19:29:46.528572 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:29:46.528272    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schel58gb"
Sep 16 19:29:46.548000 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:29:46.547929    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schel58gb" podStartSLOduration=1.547911982 podStartE2EDuration="1.547911982s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:29:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:29:46.545822154 +0000 UTC m=+2967.612313651" watchObservedRunningTime="2026-09-16 19:29:46.547911982 +0000 UTC m=+2967.614403473"
Sep 16 19:29:50.935550 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:29:50.935511    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:29:51.012423 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:29:51.012387    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:29:52.882559 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:29:52.882527    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:29:59.539391 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:29:59.539352    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:30:01.053139 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:30:01.053096    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:30:03.155498 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:30:03.155459    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:30:03.250588 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:30:03.250534    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:30:08.208082 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:30:08.208048    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:30:08.494782 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:30:08.494691    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:30:08.724584 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:30:08.724550    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-sche8r26n"]
Sep 16 19:30:08.727757 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:30:08.727740    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-sche8r26n"
Sep 16 19:30:08.731964 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:30:08.731942    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-079cb970\"/\"llmisva690bbc929faec8bc98c767f16c003c1-kserve-self-signed-certs\""
Sep 16 19:30:08.732451 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:30:08.732432    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-079cb970\"/\"llmisvc-model-fb-opt-125m-route-87882a8e-epp-sa-dockercfg-q2l4n\""
Sep 16 19:30:08.732703 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:30:08.732689    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-079cb970\"/\"default-dockercfg-przv5\""
Sep 16 19:30:08.733467 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:30:08.733451    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-079cb970\"/\"openshift-service-ca.crt\""
Sep 16 19:30:08.733695 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:30:08.733680    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-079cb970\"/\"kube-root-ca.crt\""
Sep 16 19:30:08.743799 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:30:08.743778    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-sche8r26n"]
Sep 16 19:30:08.809565 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:30:08.809489    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:30:08.811714 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:30:08.811691    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rb5sf\" (UniqueName: \"kubernetes.io/projected/4203c019-736e-4c9e-902a-6a9944a89577-kube-api-access-rb5sf\") pod \"llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-sche8r26n\" (UID: \"4203c019-736e-4c9e-902a-6a9944a89577\") " pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-sche8r26n"
Sep 16 19:30:08.811790 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:30:08.811750    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4203c019-736e-4c9e-902a-6a9944a89577-tls-certs\") pod \"llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-sche8r26n\" (UID: \"4203c019-736e-4c9e-902a-6a9944a89577\") " pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-sche8r26n"
Sep 16 19:30:08.913156 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:30:08.913122    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rb5sf\" (UniqueName: \"kubernetes.io/projected/4203c019-736e-4c9e-902a-6a9944a89577-kube-api-access-rb5sf\") pod \"llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-sche8r26n\" (UID: \"4203c019-736e-4c9e-902a-6a9944a89577\") " pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-sche8r26n"
Sep 16 19:30:08.913353 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:30:08.913190    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4203c019-736e-4c9e-902a-6a9944a89577-tls-certs\") pod \"llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-sche8r26n\" (UID: \"4203c019-736e-4c9e-902a-6a9944a89577\") " pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-sche8r26n"
Sep 16 19:30:08.915863 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:30:08.915842    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4203c019-736e-4c9e-902a-6a9944a89577-tls-certs\") pod \"llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-sche8r26n\" (UID: \"4203c019-736e-4c9e-902a-6a9944a89577\") " pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-sche8r26n"
Sep 16 19:30:08.925860 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:30:08.925839    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rb5sf\" (UniqueName: \"kubernetes.io/projected/4203c019-736e-4c9e-902a-6a9944a89577-kube-api-access-rb5sf\") pod \"llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-sche8r26n\" (UID: \"4203c019-736e-4c9e-902a-6a9944a89577\") " pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-sche8r26n"
Sep 16 19:30:09.039188 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:30:09.039155    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-sche8r26n"
Sep 16 19:30:09.315938 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:30:09.315913    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-sche8r26n"]
Sep 16 19:30:09.317970 ip-10-0-135-119 kubenswrapper[2578]: W0916 19:30:09.317945    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4203c019_736e_4c9e_902a_6a9944a89577.slice/crio-9b3e5339835c11732f912b15372ea97100fe7c223c9ad50d870738391e095840 WatchSource:0}: Error finding container 9b3e5339835c11732f912b15372ea97100fe7c223c9ad50d870738391e095840: Status 404 returned error can't find the container with id 9b3e5339835c11732f912b15372ea97100fe7c223c9ad50d870738391e095840
Sep 16 19:30:09.644397 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:30:09.644363    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-sche8r26n" event={"ID":"4203c019-736e-4c9e-902a-6a9944a89577","Type":"ContainerStarted","Data":"43aaa1fccb206c99cc9339686b63740a590167ca2f06f4acb648830acffd1c31"}
Sep 16 19:30:09.644592 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:30:09.644417    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-sche8r26n" event={"ID":"4203c019-736e-4c9e-902a-6a9944a89577","Type":"ContainerStarted","Data":"9b3e5339835c11732f912b15372ea97100fe7c223c9ad50d870738391e095840"}
Sep 16 19:30:09.644592 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:30:09.644445    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-sche8r26n"
Sep 16 19:30:09.663147 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:30:09.663103    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-sche8r26n" podStartSLOduration=1.663085557 podStartE2EDuration="1.663085557s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:30:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:30:09.662559298 +0000 UTC m=+2990.729050806" watchObservedRunningTime="2026-09-16 19:30:09.663085557 +0000 UTC m=+2990.729577046"
Sep 16 19:30:13.500824 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:30:13.500790    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:30:17.535252 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:30:17.535216    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schel58gb"
Sep 16 19:30:19.601972 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:30:19.601946    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-tmrkx_4c2078bb-281a-4aa1-8e38-238897865434/console-operator/2.log"
Sep 16 19:30:19.609406 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:30:19.609381    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-k6gvs_6012728d-aee6-4f93-ab6e-fad616ef36e8/ovn-acl-logging/0.log"
Sep 16 19:30:19.614081 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:30:19.614061    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-tmrkx_4c2078bb-281a-4aa1-8e38-238897865434/console-operator/2.log"
Sep 16 19:30:19.621482 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:30:19.621465    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-k6gvs_6012728d-aee6-4f93-ab6e-fad616ef36e8/ovn-acl-logging/0.log"
Sep 16 19:30:29.705456 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:30:29.705417    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:30:30.928440 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:30:30.928408    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:30:37.068043 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:30:37.068009    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:30:40.650491 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:30:40.650458    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-sche8r26n"
Sep 16 19:30:52.088562 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:30:52.088518    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:31:01.742150 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:31:01.742111    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:31:01.844423 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:31:01.844394    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:31:07.740595 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:31:07.740556    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schel58gb"]
Sep 16 19:31:07.744322 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:31:07.744287    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:31:07.826025 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:31:07.825941    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:31:17.535972 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:31:17.535933    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schel58gb" podUID="44134c47-d2d2-46ca-95a2-fe25c509b99b" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:31:22.761862 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:31:22.761793    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schel58gb" podUID="44134c47-d2d2-46ca-95a2-fe25c509b99b" containerName="main" containerID="cri-o://a733f6bbb355ae35f380dbe420dfc1c4cf50dda699294ca2dd4ce0ea50159af7" gracePeriod=45
Sep 16 19:31:27.535509 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:31:27.535475    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schel58gb" podUID="44134c47-d2d2-46ca-95a2-fe25c509b99b" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:31:37.535023 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:31:37.534956    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schel58gb" podUID="44134c47-d2d2-46ca-95a2-fe25c509b99b" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:31:37.535450 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:31:37.535091    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schel58gb"
Sep 16 19:31:47.535109 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:31:47.535071    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schel58gb" podUID="44134c47-d2d2-46ca-95a2-fe25c509b99b" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:31:53.017850 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:31:53.017820    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schel58gb"
Sep 16 19:31:53.130386 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:31:53.130354    2578 generic.go:356] "Generic (PLEG): container finished" podID="44134c47-d2d2-46ca-95a2-fe25c509b99b" containerID="a733f6bbb355ae35f380dbe420dfc1c4cf50dda699294ca2dd4ce0ea50159af7" exitCode=0
Sep 16 19:31:53.130548 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:31:53.130395    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schel58gb"
Sep 16 19:31:53.130548 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:31:53.130418    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schel58gb" event={"ID":"44134c47-d2d2-46ca-95a2-fe25c509b99b","Type":"ContainerDied","Data":"a733f6bbb355ae35f380dbe420dfc1c4cf50dda699294ca2dd4ce0ea50159af7"}
Sep 16 19:31:53.130548 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:31:53.130459    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schel58gb" event={"ID":"44134c47-d2d2-46ca-95a2-fe25c509b99b","Type":"ContainerDied","Data":"61716d56dc0688c4e9287753272627d1ed620f8df735542874cab22e9a425a12"}
Sep 16 19:31:53.130548 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:31:53.130481    2578 scope.go:117] "RemoveContainer" containerID="a733f6bbb355ae35f380dbe420dfc1c4cf50dda699294ca2dd4ce0ea50159af7"
Sep 16 19:31:53.139001 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:31:53.138964    2578 scope.go:117] "RemoveContainer" containerID="a733f6bbb355ae35f380dbe420dfc1c4cf50dda699294ca2dd4ce0ea50159af7"
Sep 16 19:31:53.139290 ip-10-0-135-119 kubenswrapper[2578]: E0916 19:31:53.139262    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a733f6bbb355ae35f380dbe420dfc1c4cf50dda699294ca2dd4ce0ea50159af7\": container with ID starting with a733f6bbb355ae35f380dbe420dfc1c4cf50dda699294ca2dd4ce0ea50159af7 not found: ID does not exist" containerID="a733f6bbb355ae35f380dbe420dfc1c4cf50dda699294ca2dd4ce0ea50159af7"
Sep 16 19:31:53.139381 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:31:53.139291    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a733f6bbb355ae35f380dbe420dfc1c4cf50dda699294ca2dd4ce0ea50159af7"} err="failed to get container status \"a733f6bbb355ae35f380dbe420dfc1c4cf50dda699294ca2dd4ce0ea50159af7\": rpc error: code = NotFound desc = could not find container \"a733f6bbb355ae35f380dbe420dfc1c4cf50dda699294ca2dd4ce0ea50159af7\": container with ID starting with a733f6bbb355ae35f380dbe420dfc1c4cf50dda699294ca2dd4ce0ea50159af7 not found: ID does not exist"
Sep 16 19:31:53.140496 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:31:53.140479    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gdwtb\" (UniqueName: \"kubernetes.io/projected/44134c47-d2d2-46ca-95a2-fe25c509b99b-kube-api-access-gdwtb\") pod \"44134c47-d2d2-46ca-95a2-fe25c509b99b\" (UID: \"44134c47-d2d2-46ca-95a2-fe25c509b99b\") "
Sep 16 19:31:53.140561 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:31:53.140550    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/44134c47-d2d2-46ca-95a2-fe25c509b99b-tls-certs\") pod \"44134c47-d2d2-46ca-95a2-fe25c509b99b\" (UID: \"44134c47-d2d2-46ca-95a2-fe25c509b99b\") "
Sep 16 19:31:53.142773 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:31:53.142749    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/44134c47-d2d2-46ca-95a2-fe25c509b99b-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "44134c47-d2d2-46ca-95a2-fe25c509b99b" (UID: "44134c47-d2d2-46ca-95a2-fe25c509b99b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:31:53.142773 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:31:53.142749    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/44134c47-d2d2-46ca-95a2-fe25c509b99b-kube-api-access-gdwtb" (OuterVolumeSpecName: "kube-api-access-gdwtb") pod "44134c47-d2d2-46ca-95a2-fe25c509b99b" (UID: "44134c47-d2d2-46ca-95a2-fe25c509b99b"). InnerVolumeSpecName "kube-api-access-gdwtb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:31:53.241418 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:31:53.241390    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gdwtb\" (UniqueName: \"kubernetes.io/projected/44134c47-d2d2-46ca-95a2-fe25c509b99b-kube-api-access-gdwtb\") on node \"ip-10-0-135-119.ec2.internal\" DevicePath \"\""
Sep 16 19:31:53.241418 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:31:53.241416    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/44134c47-d2d2-46ca-95a2-fe25c509b99b-tls-certs\") on node \"ip-10-0-135-119.ec2.internal\" DevicePath \"\""
Sep 16 19:31:53.456229 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:31:53.456201    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schel58gb"]
Sep 16 19:31:53.461126 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:31:53.461101    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schel58gb"]
Sep 16 19:31:55.433737 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:31:55.433704    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="44134c47-d2d2-46ca-95a2-fe25c509b99b" path="/var/lib/kubelet/pods/44134c47-d2d2-46ca-95a2-fe25c509b99b/volumes"
Sep 16 19:31:57.827498 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:31:57.827465    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:31:57.930923 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:31:57.930888    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:31:58.032995 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:31:58.032952    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:32:00.125709 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:32:00.125672    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:32:00.227703 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:32:00.227668    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:32:00.328113 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:32:00.328078    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:32:05.305911 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:32:05.305877    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:32:06.323918 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:32:06.323883    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:32:06.424266 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:32:06.424234    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:32:10.043259 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:32:10.043227    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:32:10.253120 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:32:10.253086    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-router-scheduler-86dbc9cdb7-rnvr2"]
Sep 16 19:32:10.253443 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:32:10.253428    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="44134c47-d2d2-46ca-95a2-fe25c509b99b" containerName="main"
Sep 16 19:32:10.253443 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:32:10.253442    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="44134c47-d2d2-46ca-95a2-fe25c509b99b" containerName="main"
Sep 16 19:32:10.253588 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:32:10.253503    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="44134c47-d2d2-46ca-95a2-fe25c509b99b" containerName="main"
Sep 16 19:32:10.260115 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:32:10.257925    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-router-scheduler-86dbc9cdb7-rnvr2"
Sep 16 19:32:10.261608 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:32:10.261577    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-config-deletion-blocked-when-referenced-0f029b85\"/\"e2e-del-blocked-svc-kserve-self-signed-certs\""
Sep 16 19:32:10.261608 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:32:10.261594    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-config-deletion-blocked-when-referenced-0f029b85\"/\"e2e-del-blocked-svc-epp-sa-dockercfg-2mcnv\""
Sep 16 19:32:10.261838 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:32:10.261751    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-config-deletion-blocked-when-referenced-0f029b85\"/\"default-dockercfg-c6n9x\""
Sep 16 19:32:10.261838 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:32:10.261791    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-config-deletion-blocked-when-referenced-0f029b85\"/\"kube-root-ca.crt\""
Sep 16 19:32:10.261838 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:32:10.261825    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-config-deletion-blocked-when-referenced-0f029b85\"/\"openshift-service-ca.crt\""
Sep 16 19:32:10.266553 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:32:10.266528    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-router-scheduler-86dbc9cdb7-rnvr2"]
Sep 16 19:32:10.285019 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:32:10.284999    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:32:10.300360 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:32:10.300299    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c00d90f3-a45e-4431-8bcc-5bf39798e43f-tls-certs\") pod \"e2e-del-blocked-svc-kserve-router-scheduler-86dbc9cdb7-rnvr2\" (UID: \"c00d90f3-a45e-4431-8bcc-5bf39798e43f\") " pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-router-scheduler-86dbc9cdb7-rnvr2"
Sep 16 19:32:10.300473 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:32:10.300399    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cqwlw\" (UniqueName: \"kubernetes.io/projected/c00d90f3-a45e-4431-8bcc-5bf39798e43f-kube-api-access-cqwlw\") pod \"e2e-del-blocked-svc-kserve-router-scheduler-86dbc9cdb7-rnvr2\" (UID: \"c00d90f3-a45e-4431-8bcc-5bf39798e43f\") " pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-router-scheduler-86dbc9cdb7-rnvr2"
Sep 16 19:32:10.401203 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:32:10.401172    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cqwlw\" (UniqueName: \"kubernetes.io/projected/c00d90f3-a45e-4431-8bcc-5bf39798e43f-kube-api-access-cqwlw\") pod \"e2e-del-blocked-svc-kserve-router-scheduler-86dbc9cdb7-rnvr2\" (UID: \"c00d90f3-a45e-4431-8bcc-5bf39798e43f\") " pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-router-scheduler-86dbc9cdb7-rnvr2"
Sep 16 19:32:10.401370 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:32:10.401231    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c00d90f3-a45e-4431-8bcc-5bf39798e43f-tls-certs\") pod \"e2e-del-blocked-svc-kserve-router-scheduler-86dbc9cdb7-rnvr2\" (UID: \"c00d90f3-a45e-4431-8bcc-5bf39798e43f\") " pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-router-scheduler-86dbc9cdb7-rnvr2"
Sep 16 19:32:10.403955 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:32:10.403930    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c00d90f3-a45e-4431-8bcc-5bf39798e43f-tls-certs\") pod \"e2e-del-blocked-svc-kserve-router-scheduler-86dbc9cdb7-rnvr2\" (UID: \"c00d90f3-a45e-4431-8bcc-5bf39798e43f\") " pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-router-scheduler-86dbc9cdb7-rnvr2"
Sep 16 19:32:10.410318 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:32:10.410294    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cqwlw\" (UniqueName: \"kubernetes.io/projected/c00d90f3-a45e-4431-8bcc-5bf39798e43f-kube-api-access-cqwlw\") pod \"e2e-del-blocked-svc-kserve-router-scheduler-86dbc9cdb7-rnvr2\" (UID: \"c00d90f3-a45e-4431-8bcc-5bf39798e43f\") " pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-router-scheduler-86dbc9cdb7-rnvr2"
Sep 16 19:32:10.571422 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:32:10.571326    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-router-scheduler-86dbc9cdb7-rnvr2"
Sep 16 19:32:10.846493 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:32:10.846466    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-router-scheduler-86dbc9cdb7-rnvr2"]
Sep 16 19:32:10.848614 ip-10-0-135-119 kubenswrapper[2578]: W0916 19:32:10.848587    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc00d90f3_a45e_4431_8bcc_5bf39798e43f.slice/crio-21d454cf1fc57f44ef06091e8406f14de5c4d5454262e8e8cefe2823f7c8ff42 WatchSource:0}: Error finding container 21d454cf1fc57f44ef06091e8406f14de5c4d5454262e8e8cefe2823f7c8ff42: Status 404 returned error can't find the container with id 21d454cf1fc57f44ef06091e8406f14de5c4d5454262e8e8cefe2823f7c8ff42
Sep 16 19:32:11.220272 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:32:11.220228    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-router-scheduler-86dbc9cdb7-rnvr2" event={"ID":"c00d90f3-a45e-4431-8bcc-5bf39798e43f","Type":"ContainerStarted","Data":"940df3ec55efdb5f6341b3ebfaf2907cf663d74e7bac82a2c0c0fab06949f347"}
Sep 16 19:32:11.220649 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:32:11.220295    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-router-scheduler-86dbc9cdb7-rnvr2" event={"ID":"c00d90f3-a45e-4431-8bcc-5bf39798e43f","Type":"ContainerStarted","Data":"21d454cf1fc57f44ef06091e8406f14de5c4d5454262e8e8cefe2823f7c8ff42"}
Sep 16 19:32:11.220649 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:32:11.220337    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-router-scheduler-86dbc9cdb7-rnvr2"
Sep 16 19:32:11.237764 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:32:11.237691    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-router-scheduler-86dbc9cdb7-rnvr2" podStartSLOduration=1.237672837 podStartE2EDuration="1.237672837s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:32:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:32:11.237248589 +0000 UTC m=+3112.303740078" watchObservedRunningTime="2026-09-16 19:32:11.237672837 +0000 UTC m=+3112.304164326"
Sep 16 19:32:12.234689 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:32:12.234659    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:32:12.335247 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:32:12.335213    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:32:12.709942 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:32:12.709871    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:32:12.711257 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:32:12.711233    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-sche8r26n"]
Sep 16 19:32:12.776139 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:32:12.776109    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:32:13.219908 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:32:13.219875    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-router-scheduler-86dbc9cdb7-rnvr2"]
Sep 16 19:32:13.242582 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:32:13.242552    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:32:13.342070 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:32:13.342032    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:32:14.862924 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:32:14.862884    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:32:14.964957 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:32:14.964927    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:32:15.075089 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:32:15.075051    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:32:19.223598 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:32:19.223499    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:32:19.367099 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:32:19.367061    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:32:19.958481 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:32:19.958448    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:32:20.651193 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:32:20.651157    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-sche8r26n" podUID="4203c019-736e-4c9e-902a-6a9944a89577" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:32:23.671242 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:32:23.671197    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:32:23.853355 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:32:23.853300    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:32:23.951547 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:32:23.951472    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:32:25.886966 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:32:25.886927    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:32:25.966267 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:32:25.966236    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:32:27.730166 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:32:27.730108    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-sche8r26n" podUID="4203c019-736e-4c9e-902a-6a9944a89577" containerName="main" containerID="cri-o://43aaa1fccb206c99cc9339686b63740a590167ca2f06f4acb648830acffd1c31" gracePeriod=45
Sep 16 19:32:28.249803 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:32:28.249761    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-router-scheduler-86dbc9cdb7-rnvr2" podUID="c00d90f3-a45e-4431-8bcc-5bf39798e43f" containerName="main" containerID="cri-o://940df3ec55efdb5f6341b3ebfaf2907cf663d74e7bac82a2c0c0fab06949f347" gracePeriod=45
Sep 16 19:32:29.895194 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:32:29.895152    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:32:29.996199 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:32:29.996152    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:32:30.651104 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:32:30.651063    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-sche8r26n" podUID="4203c019-736e-4c9e-902a-6a9944a89577" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:32:33.606173 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:32:33.606135    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:32:33.848826 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:32:33.848798    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:32:35.148480 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:32:35.148433    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:32:35.244164 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:32:35.244132    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:32:36.301120 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:32:36.301087    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:32:36.394537 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:32:36.394489    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:32:40.651214 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:32:40.651134    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-sche8r26n" podUID="4203c019-736e-4c9e-902a-6a9944a89577" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:32:40.651624 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:32:40.651239    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-sche8r26n"
Sep 16 19:32:41.107343 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:32:41.107230    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:32:41.333424 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:32:41.333382    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:32:41.395586 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:32:41.395552    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:32:43.233172 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:32:43.233137    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-router-scheduler-86dbc9cdb7-rnvr2" podUID="c00d90f3-a45e-4431-8bcc-5bf39798e43f" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:32:50.651255 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:32:50.651215    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-sche8r26n" podUID="4203c019-736e-4c9e-902a-6a9944a89577" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:32:53.232537 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:32:53.232496    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-router-scheduler-86dbc9cdb7-rnvr2" podUID="c00d90f3-a45e-4431-8bcc-5bf39798e43f" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:32:57.967668 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:32:57.967641    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-sche8r26n"
Sep 16 19:32:58.068290 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:32:58.068207    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:32:58.114331 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:32:58.114302    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rb5sf\" (UniqueName: \"kubernetes.io/projected/4203c019-736e-4c9e-902a-6a9944a89577-kube-api-access-rb5sf\") pod \"4203c019-736e-4c9e-902a-6a9944a89577\" (UID: \"4203c019-736e-4c9e-902a-6a9944a89577\") "
Sep 16 19:32:58.114463 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:32:58.114342    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4203c019-736e-4c9e-902a-6a9944a89577-tls-certs\") pod \"4203c019-736e-4c9e-902a-6a9944a89577\" (UID: \"4203c019-736e-4c9e-902a-6a9944a89577\") "
Sep 16 19:32:58.116589 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:32:58.116558    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4203c019-736e-4c9e-902a-6a9944a89577-kube-api-access-rb5sf" (OuterVolumeSpecName: "kube-api-access-rb5sf") pod "4203c019-736e-4c9e-902a-6a9944a89577" (UID: "4203c019-736e-4c9e-902a-6a9944a89577"). InnerVolumeSpecName "kube-api-access-rb5sf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:32:58.116696 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:32:58.116580    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4203c019-736e-4c9e-902a-6a9944a89577-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "4203c019-736e-4c9e-902a-6a9944a89577" (UID: "4203c019-736e-4c9e-902a-6a9944a89577"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:32:58.215246 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:32:58.215217    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rb5sf\" (UniqueName: \"kubernetes.io/projected/4203c019-736e-4c9e-902a-6a9944a89577-kube-api-access-rb5sf\") on node \"ip-10-0-135-119.ec2.internal\" DevicePath \"\""
Sep 16 19:32:58.215246 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:32:58.215243    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4203c019-736e-4c9e-902a-6a9944a89577-tls-certs\") on node \"ip-10-0-135-119.ec2.internal\" DevicePath \"\""
Sep 16 19:32:58.260602 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:32:58.260576    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:32:58.441819 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:32:58.441784    2578 generic.go:356] "Generic (PLEG): container finished" podID="c00d90f3-a45e-4431-8bcc-5bf39798e43f" containerID="940df3ec55efdb5f6341b3ebfaf2907cf663d74e7bac82a2c0c0fab06949f347" exitCode=0
Sep 16 19:32:58.441967 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:32:58.441854    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-router-scheduler-86dbc9cdb7-rnvr2" event={"ID":"c00d90f3-a45e-4431-8bcc-5bf39798e43f","Type":"ContainerDied","Data":"940df3ec55efdb5f6341b3ebfaf2907cf663d74e7bac82a2c0c0fab06949f347"}
Sep 16 19:32:58.443572 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:32:58.443548    2578 generic.go:356] "Generic (PLEG): container finished" podID="4203c019-736e-4c9e-902a-6a9944a89577" containerID="43aaa1fccb206c99cc9339686b63740a590167ca2f06f4acb648830acffd1c31" exitCode=0
Sep 16 19:32:58.443691 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:32:58.443580    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-sche8r26n" event={"ID":"4203c019-736e-4c9e-902a-6a9944a89577","Type":"ContainerDied","Data":"43aaa1fccb206c99cc9339686b63740a590167ca2f06f4acb648830acffd1c31"}
Sep 16 19:32:58.443691 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:32:58.443612    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-sche8r26n" event={"ID":"4203c019-736e-4c9e-902a-6a9944a89577","Type":"ContainerDied","Data":"9b3e5339835c11732f912b15372ea97100fe7c223c9ad50d870738391e095840"}
Sep 16 19:32:58.443691 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:32:58.443627    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-sche8r26n"
Sep 16 19:32:58.443860 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:32:58.443634    2578 scope.go:117] "RemoveContainer" containerID="43aaa1fccb206c99cc9339686b63740a590167ca2f06f4acb648830acffd1c31"
Sep 16 19:32:58.454031 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:32:58.454011    2578 scope.go:117] "RemoveContainer" containerID="43aaa1fccb206c99cc9339686b63740a590167ca2f06f4acb648830acffd1c31"
Sep 16 19:32:58.454331 ip-10-0-135-119 kubenswrapper[2578]: E0916 19:32:58.454311    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"43aaa1fccb206c99cc9339686b63740a590167ca2f06f4acb648830acffd1c31\": container with ID starting with 43aaa1fccb206c99cc9339686b63740a590167ca2f06f4acb648830acffd1c31 not found: ID does not exist" containerID="43aaa1fccb206c99cc9339686b63740a590167ca2f06f4acb648830acffd1c31"
Sep 16 19:32:58.454413 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:32:58.454341    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"43aaa1fccb206c99cc9339686b63740a590167ca2f06f4acb648830acffd1c31"} err="failed to get container status \"43aaa1fccb206c99cc9339686b63740a590167ca2f06f4acb648830acffd1c31\": rpc error: code = NotFound desc = could not find container \"43aaa1fccb206c99cc9339686b63740a590167ca2f06f4acb648830acffd1c31\": container with ID starting with 43aaa1fccb206c99cc9339686b63740a590167ca2f06f4acb648830acffd1c31 not found: ID does not exist"
Sep 16 19:32:58.467832 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:32:58.467809    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-sche8r26n"]
Sep 16 19:32:58.472700 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:32:58.472674    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-sche8r26n"]
Sep 16 19:32:58.489806 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:32:58.489785    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-router-scheduler-86dbc9cdb7-rnvr2"
Sep 16 19:32:58.618198 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:32:58.618113    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cqwlw\" (UniqueName: \"kubernetes.io/projected/c00d90f3-a45e-4431-8bcc-5bf39798e43f-kube-api-access-cqwlw\") pod \"c00d90f3-a45e-4431-8bcc-5bf39798e43f\" (UID: \"c00d90f3-a45e-4431-8bcc-5bf39798e43f\") "
Sep 16 19:32:58.618369 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:32:58.618209    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c00d90f3-a45e-4431-8bcc-5bf39798e43f-tls-certs\") pod \"c00d90f3-a45e-4431-8bcc-5bf39798e43f\" (UID: \"c00d90f3-a45e-4431-8bcc-5bf39798e43f\") "
Sep 16 19:32:58.620554 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:32:58.620519    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c00d90f3-a45e-4431-8bcc-5bf39798e43f-kube-api-access-cqwlw" (OuterVolumeSpecName: "kube-api-access-cqwlw") pod "c00d90f3-a45e-4431-8bcc-5bf39798e43f" (UID: "c00d90f3-a45e-4431-8bcc-5bf39798e43f"). InnerVolumeSpecName "kube-api-access-cqwlw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:32:58.620554 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:32:58.620539    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c00d90f3-a45e-4431-8bcc-5bf39798e43f-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "c00d90f3-a45e-4431-8bcc-5bf39798e43f" (UID: "c00d90f3-a45e-4431-8bcc-5bf39798e43f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:32:58.719298 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:32:58.719267    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c00d90f3-a45e-4431-8bcc-5bf39798e43f-tls-certs\") on node \"ip-10-0-135-119.ec2.internal\" DevicePath \"\""
Sep 16 19:32:58.719298 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:32:58.719293    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cqwlw\" (UniqueName: \"kubernetes.io/projected/c00d90f3-a45e-4431-8bcc-5bf39798e43f-kube-api-access-cqwlw\") on node \"ip-10-0-135-119.ec2.internal\" DevicePath \"\""
Sep 16 19:32:59.433694 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:32:59.433662    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4203c019-736e-4c9e-902a-6a9944a89577" path="/var/lib/kubelet/pods/4203c019-736e-4c9e-902a-6a9944a89577/volumes"
Sep 16 19:32:59.449939 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:32:59.449916    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-router-scheduler-86dbc9cdb7-rnvr2"
Sep 16 19:32:59.449939 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:32:59.449930    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-router-scheduler-86dbc9cdb7-rnvr2" event={"ID":"c00d90f3-a45e-4431-8bcc-5bf39798e43f","Type":"ContainerDied","Data":"21d454cf1fc57f44ef06091e8406f14de5c4d5454262e8e8cefe2823f7c8ff42"}
Sep 16 19:32:59.450160 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:32:59.450000    2578 scope.go:117] "RemoveContainer" containerID="940df3ec55efdb5f6341b3ebfaf2907cf663d74e7bac82a2c0c0fab06949f347"
Sep 16 19:32:59.467712 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:32:59.467686    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-router-scheduler-86dbc9cdb7-rnvr2"]
Sep 16 19:32:59.472436 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:32:59.472416    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-router-scheduler-86dbc9cdb7-rnvr2"]
Sep 16 19:33:01.434587 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:33:01.434555    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c00d90f3-a45e-4431-8bcc-5bf39798e43f" path="/var/lib/kubelet/pods/c00d90f3-a45e-4431-8bcc-5bf39798e43f/volumes"
Sep 16 19:33:03.336359 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:33:03.336325    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:33:03.426470 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:33:03.426432    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:33:05.460216 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:33:05.460172    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:33:08.329793 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:33:08.329750    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:33:10.099945 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:33:10.099912    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:33:14.970845 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:33:14.970813    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dt4qpr"]
Sep 16 19:33:14.971254 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:33:14.971217    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4203c019-736e-4c9e-902a-6a9944a89577" containerName="main"
Sep 16 19:33:14.971254 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:33:14.971230    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="4203c019-736e-4c9e-902a-6a9944a89577" containerName="main"
Sep 16 19:33:14.971254 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:33:14.971255    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c00d90f3-a45e-4431-8bcc-5bf39798e43f" containerName="main"
Sep 16 19:33:14.971358 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:33:14.971260    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="c00d90f3-a45e-4431-8bcc-5bf39798e43f" containerName="main"
Sep 16 19:33:14.971358 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:33:14.971326    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="c00d90f3-a45e-4431-8bcc-5bf39798e43f" containerName="main"
Sep 16 19:33:14.971358 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:33:14.971335    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="4203c019-736e-4c9e-902a-6a9944a89577" containerName="main"
Sep 16 19:33:14.974029 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:33:14.974012    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dt4qpr"
Sep 16 19:33:14.976235 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:33:14.976211    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-7bc4bd10\"/\"default-dockercfg-bsldc\""
Sep 16 19:33:14.976371 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:33:14.976212    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-7bc4bd10\"/\"openshift-service-ca.crt\""
Sep 16 19:33:14.976371 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:33:14.976281    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-7bc4bd10\"/\"llmisv20d1e8ef181cac7effb0fab7b8590b15-kserve-self-signed-certs\""
Sep 16 19:33:14.976910 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:33:14.976888    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-7bc4bd10\"/\"kube-root-ca.crt\""
Sep 16 19:33:14.992965 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:33:14.992935    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dt4qpr"]
Sep 16 19:33:14.997689 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:33:14.997664    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:33:15.064654 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:33:15.064620    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/46db97e1-daa2-4264-8a82-e6b465ff4905-tls-certs\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dt4qpr\" (UID: \"46db97e1-daa2-4264-8a82-e6b465ff4905\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dt4qpr"
Sep 16 19:33:15.064654 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:33:15.064656    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/46db97e1-daa2-4264-8a82-e6b465ff4905-dshm\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dt4qpr\" (UID: \"46db97e1-daa2-4264-8a82-e6b465ff4905\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dt4qpr"
Sep 16 19:33:15.064885 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:33:15.064678    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wm2kn\" (UniqueName: \"kubernetes.io/projected/46db97e1-daa2-4264-8a82-e6b465ff4905-kube-api-access-wm2kn\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dt4qpr\" (UID: \"46db97e1-daa2-4264-8a82-e6b465ff4905\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dt4qpr"
Sep 16 19:33:15.064885 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:33:15.064772    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/46db97e1-daa2-4264-8a82-e6b465ff4905-home\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dt4qpr\" (UID: \"46db97e1-daa2-4264-8a82-e6b465ff4905\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dt4qpr"
Sep 16 19:33:15.064885 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:33:15.064844    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/46db97e1-daa2-4264-8a82-e6b465ff4905-model-cache\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dt4qpr\" (UID: \"46db97e1-daa2-4264-8a82-e6b465ff4905\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dt4qpr"
Sep 16 19:33:15.064885 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:33:15.064876    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/46db97e1-daa2-4264-8a82-e6b465ff4905-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dt4qpr\" (UID: \"46db97e1-daa2-4264-8a82-e6b465ff4905\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dt4qpr"
Sep 16 19:33:15.065067 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:33:15.064947    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/46db97e1-daa2-4264-8a82-e6b465ff4905-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dt4qpr\" (UID: \"46db97e1-daa2-4264-8a82-e6b465ff4905\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dt4qpr"
Sep 16 19:33:15.166130 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:33:15.166090    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/46db97e1-daa2-4264-8a82-e6b465ff4905-tls-certs\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dt4qpr\" (UID: \"46db97e1-daa2-4264-8a82-e6b465ff4905\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dt4qpr"
Sep 16 19:33:15.166335 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:33:15.166136    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/46db97e1-daa2-4264-8a82-e6b465ff4905-dshm\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dt4qpr\" (UID: \"46db97e1-daa2-4264-8a82-e6b465ff4905\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dt4qpr"
Sep 16 19:33:15.166335 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:33:15.166167    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wm2kn\" (UniqueName: \"kubernetes.io/projected/46db97e1-daa2-4264-8a82-e6b465ff4905-kube-api-access-wm2kn\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dt4qpr\" (UID: \"46db97e1-daa2-4264-8a82-e6b465ff4905\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dt4qpr"
Sep 16 19:33:15.166335 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:33:15.166206    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/46db97e1-daa2-4264-8a82-e6b465ff4905-home\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dt4qpr\" (UID: \"46db97e1-daa2-4264-8a82-e6b465ff4905\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dt4qpr"
Sep 16 19:33:15.166335 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:33:15.166262    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/46db97e1-daa2-4264-8a82-e6b465ff4905-model-cache\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dt4qpr\" (UID: \"46db97e1-daa2-4264-8a82-e6b465ff4905\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dt4qpr"
Sep 16 19:33:15.166335 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:33:15.166288    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/46db97e1-daa2-4264-8a82-e6b465ff4905-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dt4qpr\" (UID: \"46db97e1-daa2-4264-8a82-e6b465ff4905\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dt4qpr"
Sep 16 19:33:15.166335 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:33:15.166329    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/46db97e1-daa2-4264-8a82-e6b465ff4905-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dt4qpr\" (UID: \"46db97e1-daa2-4264-8a82-e6b465ff4905\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dt4qpr"
Sep 16 19:33:15.166755 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:33:15.166727    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/46db97e1-daa2-4264-8a82-e6b465ff4905-model-cache\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dt4qpr\" (UID: \"46db97e1-daa2-4264-8a82-e6b465ff4905\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dt4qpr"
Sep 16 19:33:15.166834 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:33:15.166738    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/46db97e1-daa2-4264-8a82-e6b465ff4905-home\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dt4qpr\" (UID: \"46db97e1-daa2-4264-8a82-e6b465ff4905\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dt4qpr"
Sep 16 19:33:15.166834 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:33:15.166793    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/46db97e1-daa2-4264-8a82-e6b465ff4905-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dt4qpr\" (UID: \"46db97e1-daa2-4264-8a82-e6b465ff4905\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dt4qpr"
Sep 16 19:33:15.167136 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:33:15.166838    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/46db97e1-daa2-4264-8a82-e6b465ff4905-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dt4qpr\" (UID: \"46db97e1-daa2-4264-8a82-e6b465ff4905\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dt4qpr"
Sep 16 19:33:15.168748 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:33:15.168728    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/46db97e1-daa2-4264-8a82-e6b465ff4905-dshm\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dt4qpr\" (UID: \"46db97e1-daa2-4264-8a82-e6b465ff4905\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dt4qpr"
Sep 16 19:33:15.168904 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:33:15.168889    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/46db97e1-daa2-4264-8a82-e6b465ff4905-tls-certs\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dt4qpr\" (UID: \"46db97e1-daa2-4264-8a82-e6b465ff4905\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dt4qpr"
Sep 16 19:33:15.175235 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:33:15.175216    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wm2kn\" (UniqueName: \"kubernetes.io/projected/46db97e1-daa2-4264-8a82-e6b465ff4905-kube-api-access-wm2kn\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dt4qpr\" (UID: \"46db97e1-daa2-4264-8a82-e6b465ff4905\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dt4qpr"
Sep 16 19:33:15.215061 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:33:15.215028    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-sche7v8gn"]
Sep 16 19:33:15.218668 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:33:15.218651    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-sche7v8gn"
Sep 16 19:33:15.221819 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:33:15.221763    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-7bc4bd10\"/\"llmisvc-model-fb-opt-125m-oci-r-0fba19de-epp-sa-dockercfg-5nkwr\""
Sep 16 19:33:15.235202 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:33:15.235179    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-sche7v8gn"]
Sep 16 19:33:15.251564 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:33:15.251538    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:33:15.267739 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:33:15.267707    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b0a0cbe7-40bc-4439-a3ab-ccd62e042295-tls-certs\") pod \"llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-sche7v8gn\" (UID: \"b0a0cbe7-40bc-4439-a3ab-ccd62e042295\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-sche7v8gn"
Sep 16 19:33:15.267852 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:33:15.267767    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4ht7g\" (UniqueName: \"kubernetes.io/projected/b0a0cbe7-40bc-4439-a3ab-ccd62e042295-kube-api-access-4ht7g\") pod \"llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-sche7v8gn\" (UID: \"b0a0cbe7-40bc-4439-a3ab-ccd62e042295\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-sche7v8gn"
Sep 16 19:33:15.283832 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:33:15.283806    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dt4qpr"
Sep 16 19:33:15.368536 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:33:15.368505    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b0a0cbe7-40bc-4439-a3ab-ccd62e042295-tls-certs\") pod \"llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-sche7v8gn\" (UID: \"b0a0cbe7-40bc-4439-a3ab-ccd62e042295\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-sche7v8gn"
Sep 16 19:33:15.368689 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:33:15.368569    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4ht7g\" (UniqueName: \"kubernetes.io/projected/b0a0cbe7-40bc-4439-a3ab-ccd62e042295-kube-api-access-4ht7g\") pod \"llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-sche7v8gn\" (UID: \"b0a0cbe7-40bc-4439-a3ab-ccd62e042295\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-sche7v8gn"
Sep 16 19:33:15.371382 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:33:15.371360    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b0a0cbe7-40bc-4439-a3ab-ccd62e042295-tls-certs\") pod \"llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-sche7v8gn\" (UID: \"b0a0cbe7-40bc-4439-a3ab-ccd62e042295\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-sche7v8gn"
Sep 16 19:33:15.382417 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:33:15.382381    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4ht7g\" (UniqueName: \"kubernetes.io/projected/b0a0cbe7-40bc-4439-a3ab-ccd62e042295-kube-api-access-4ht7g\") pod \"llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-sche7v8gn\" (UID: \"b0a0cbe7-40bc-4439-a3ab-ccd62e042295\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-sche7v8gn"
Sep 16 19:33:15.529237 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:33:15.529152    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-sche7v8gn"
Sep 16 19:33:15.561660 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:33:15.561635    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dt4qpr"]
Sep 16 19:33:15.628591 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:33:15.628573    2578 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 16 19:33:15.803849 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:33:15.803762    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-sche7v8gn"]
Sep 16 19:33:15.805866 ip-10-0-135-119 kubenswrapper[2578]: W0916 19:33:15.805833    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb0a0cbe7_40bc_4439_a3ab_ccd62e042295.slice/crio-41f4fb6e65418abbf579b5af29832af0621c26d0f6ed23c885805f79f929f108 WatchSource:0}: Error finding container 41f4fb6e65418abbf579b5af29832af0621c26d0f6ed23c885805f79f929f108: Status 404 returned error can't find the container with id 41f4fb6e65418abbf579b5af29832af0621c26d0f6ed23c885805f79f929f108
Sep 16 19:33:16.532543 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:33:16.532503    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dt4qpr" event={"ID":"46db97e1-daa2-4264-8a82-e6b465ff4905","Type":"ContainerStarted","Data":"79e9335e781609d96c264a808f53de4ba3440d4b69d8b7c480cea48dec618360"}
Sep 16 19:33:16.534514 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:33:16.534484    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-sche7v8gn" event={"ID":"b0a0cbe7-40bc-4439-a3ab-ccd62e042295","Type":"ContainerStarted","Data":"fb6369cdb42a6e53564fec5502a0b42f83a3f4616d290d3c7c9fea4b148e4a2e"}
Sep 16 19:33:16.534678 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:33:16.534540    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-sche7v8gn" event={"ID":"b0a0cbe7-40bc-4439-a3ab-ccd62e042295","Type":"ContainerStarted","Data":"41f4fb6e65418abbf579b5af29832af0621c26d0f6ed23c885805f79f929f108"}
Sep 16 19:33:16.534678 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:33:16.534660    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-sche7v8gn"
Sep 16 19:33:16.553316 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:33:16.553270    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-sche7v8gn" podStartSLOduration=1.5532562109999999 podStartE2EDuration="1.553256211s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:33:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:33:16.551290831 +0000 UTC m=+3177.617782330" watchObservedRunningTime="2026-09-16 19:33:16.553256211 +0000 UTC m=+3177.619747725"
Sep 16 19:33:17.001307 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:33:17.001265    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:33:17.104537 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:33:17.104499    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:33:22.570103 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:33:22.570008    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dt4qpr" event={"ID":"46db97e1-daa2-4264-8a82-e6b465ff4905","Type":"ContainerStarted","Data":"06ff6ec9474f90481923476c1ea5cbf35e667306263d54ff398829fd5c677540"}
Sep 16 19:33:24.580534 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:33:24.580496    2578 generic.go:356] "Generic (PLEG): container finished" podID="46db97e1-daa2-4264-8a82-e6b465ff4905" containerID="06ff6ec9474f90481923476c1ea5cbf35e667306263d54ff398829fd5c677540" exitCode=0
Sep 16 19:33:24.581048 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:33:24.580575    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dt4qpr" event={"ID":"46db97e1-daa2-4264-8a82-e6b465ff4905","Type":"ContainerDied","Data":"06ff6ec9474f90481923476c1ea5cbf35e667306263d54ff398829fd5c677540"}
Sep 16 19:33:28.304062 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:33:28.304023    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:33:30.189750 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:33:30.189687    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:33:47.543221 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:33:47.543184    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-sche7v8gn"
Sep 16 19:33:49.970712 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:33:49.970672    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:33:51.903182 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:33:51.903143    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:33:53.763037 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:33:53.763001    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dt4qpr" event={"ID":"46db97e1-daa2-4264-8a82-e6b465ff4905","Type":"ContainerStarted","Data":"72b8b5f99684ed4219f0165525610f48b608a5776b701fc72cbf2b3d48590b43"}
Sep 16 19:33:54.770495 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:33:54.770410    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dt4qpr" event={"ID":"46db97e1-daa2-4264-8a82-e6b465ff4905","Type":"ContainerStarted","Data":"978adf36e96d59d3044dc327e7129c5b9ed73f00bf6e18ad98dc5dd2f7f80dc7"}
Sep 16 19:33:54.794064 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:33:54.793949    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dt4qpr" podStartSLOduration=3.793924321 podStartE2EDuration="40.793933512s" totalImagesPullingTime="35.715257934s" totalInitContainerRuntime="1.284751257s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:33:14 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:33:54.793548723 +0000 UTC m=+3215.860040232" watchObservedRunningTime="2026-09-16 19:33:54.793933512 +0000 UTC m=+3215.860424994"
Sep 16 19:33:55.284283 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:33:55.284241    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dt4qpr"
Sep 16 19:33:55.284483 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:33:55.284333    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dt4qpr"
Sep 16 19:33:55.286127 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:33:55.286089    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dt4qpr" podUID="46db97e1-daa2-4264-8a82-e6b465ff4905" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/health\": dial tcp 10.132.0.66:8000: connect: connection refused"
Sep 16 19:34:05.285152 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:34:05.285105    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dt4qpr" podUID="46db97e1-daa2-4264-8a82-e6b465ff4905" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/health\": dial tcp 10.132.0.66:8000: connect: connection refused"
Sep 16 19:34:15.284366 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:34:15.284314    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dt4qpr" podUID="46db97e1-daa2-4264-8a82-e6b465ff4905" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/health\": dial tcp 10.132.0.66:8000: connect: connection refused"
Sep 16 19:34:25.284907 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:34:25.284861    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dt4qpr" podUID="46db97e1-daa2-4264-8a82-e6b465ff4905" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/health\": dial tcp 10.132.0.66:8000: connect: connection refused"
Sep 16 19:34:33.416501 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:34:33.416445    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:34:33.534388 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:34:33.534358    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:34:33.639297 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:34:33.639257    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:34:35.285363 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:34:35.285304    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dt4qpr" podUID="46db97e1-daa2-4264-8a82-e6b465ff4905" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/health\": dial tcp 10.132.0.66:8000: connect: connection refused"
Sep 16 19:34:45.284696 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:34:45.284638    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dt4qpr" podUID="46db97e1-daa2-4264-8a82-e6b465ff4905" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/health\": dial tcp 10.132.0.66:8000: connect: connection refused"
Sep 16 19:34:53.391505 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:34:53.391459    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:34:53.705325 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:34:53.705226    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:34:55.285332 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:34:55.285284    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dt4qpr" podUID="46db97e1-daa2-4264-8a82-e6b465ff4905" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/health\": dial tcp 10.132.0.66:8000: connect: connection refused"
Sep 16 19:35:05.284911 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:35:05.284852    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dt4qpr" podUID="46db97e1-daa2-4264-8a82-e6b465ff4905" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/health\": dial tcp 10.132.0.66:8000: connect: connection refused"
Sep 16 19:35:14.076168 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:35:14.076115    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:35:15.284861 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:35:15.284800    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dt4qpr" podUID="46db97e1-daa2-4264-8a82-e6b465ff4905" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/health\": dial tcp 10.132.0.66:8000: connect: connection refused"
Sep 16 19:35:19.632434 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:35:19.632406    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-tmrkx_4c2078bb-281a-4aa1-8e38-238897865434/console-operator/2.log"
Sep 16 19:35:19.649668 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:35:19.649616    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-k6gvs_6012728d-aee6-4f93-ab6e-fad616ef36e8/ovn-acl-logging/0.log"
Sep 16 19:35:19.653663 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:35:19.653636    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-tmrkx_4c2078bb-281a-4aa1-8e38-238897865434/console-operator/2.log"
Sep 16 19:35:19.666097 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:35:19.666075    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-k6gvs_6012728d-aee6-4f93-ab6e-fad616ef36e8/ovn-acl-logging/0.log"
Sep 16 19:35:25.285267 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:35:25.285219    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dt4qpr" podUID="46db97e1-daa2-4264-8a82-e6b465ff4905" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/health\": dial tcp 10.132.0.66:8000: connect: connection refused"
Sep 16 19:35:35.285264 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:35:35.285208    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dt4qpr" podUID="46db97e1-daa2-4264-8a82-e6b465ff4905" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/health\": dial tcp 10.132.0.66:8000: connect: connection refused"
Sep 16 19:35:45.284784 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:35:45.284666    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dt4qpr" podUID="46db97e1-daa2-4264-8a82-e6b465ff4905" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/health\": dial tcp 10.132.0.66:8000: connect: connection refused"
Sep 16 19:35:55.294730 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:35:55.294689    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dt4qpr"
Sep 16 19:35:55.303257 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:35:55.303227    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dt4qpr"
Sep 16 19:36:01.301502 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:36:01.301460    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-sche7v8gn"]
Sep 16 19:36:01.313823 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:36:01.313794    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dt4qpr"]
Sep 16 19:36:01.314171 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:36:01.314119    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dt4qpr" podUID="46db97e1-daa2-4264-8a82-e6b465ff4905" containerName="modelcar" containerID="cri-o://978adf36e96d59d3044dc327e7129c5b9ed73f00bf6e18ad98dc5dd2f7f80dc7" gracePeriod=60
Sep 16 19:36:01.332169 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:36:01.332142    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:36:01.432072 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:36:01.431919    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-7bc4bd10_llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dt4qpr_46db97e1-daa2-4264-8a82-e6b465ff4905/modelcar/0.log"
Sep 16 19:36:01.433203 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:36:01.433172    2578 generic.go:356] "Generic (PLEG): container finished" podID="46db97e1-daa2-4264-8a82-e6b465ff4905" containerID="978adf36e96d59d3044dc327e7129c5b9ed73f00bf6e18ad98dc5dd2f7f80dc7" exitCode=143
Sep 16 19:36:01.435276 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:36:01.435250    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dt4qpr" event={"ID":"46db97e1-daa2-4264-8a82-e6b465ff4905","Type":"ContainerDied","Data":"978adf36e96d59d3044dc327e7129c5b9ed73f00bf6e18ad98dc5dd2f7f80dc7"}
Sep 16 19:36:01.445743 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:36:01.445715    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:36:07.543202 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:36:07.543154    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-sche7v8gn" podUID="b0a0cbe7-40bc-4439-a3ab-ccd62e042295" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:36:16.325234 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:36:16.325183    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-sche7v8gn" podUID="b0a0cbe7-40bc-4439-a3ab-ccd62e042295" containerName="main" containerID="cri-o://fb6369cdb42a6e53564fec5502a0b42f83a3f4616d290d3c7c9fea4b148e4a2e" gracePeriod=45
Sep 16 19:36:16.338155 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:36:16.338113    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dt4qpr" podUID="46db97e1-daa2-4264-8a82-e6b465ff4905" containerName="main" containerID="cri-o://72b8b5f99684ed4219f0165525610f48b608a5776b701fc72cbf2b3d48590b43" gracePeriod=45
Sep 16 19:36:17.295857 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:36:17.295803    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dt4qpr" podUID="46db97e1-daa2-4264-8a82-e6b465ff4905" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/health\": dial tcp 10.132.0.66:8000: connect: connection refused"
Sep 16 19:36:17.511450 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:36:17.511419    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-7bc4bd10_llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dt4qpr_46db97e1-daa2-4264-8a82-e6b465ff4905/modelcar/0.log"
Sep 16 19:36:17.512255 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:36:17.512228    2578 generic.go:356] "Generic (PLEG): container finished" podID="46db97e1-daa2-4264-8a82-e6b465ff4905" containerID="72b8b5f99684ed4219f0165525610f48b608a5776b701fc72cbf2b3d48590b43" exitCode=0
Sep 16 19:36:17.512343 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:36:17.512290    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dt4qpr" event={"ID":"46db97e1-daa2-4264-8a82-e6b465ff4905","Type":"ContainerDied","Data":"72b8b5f99684ed4219f0165525610f48b608a5776b701fc72cbf2b3d48590b43"}
Sep 16 19:36:17.512386 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:36:17.512351    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dt4qpr" event={"ID":"46db97e1-daa2-4264-8a82-e6b465ff4905","Type":"ContainerDied","Data":"79e9335e781609d96c264a808f53de4ba3440d4b69d8b7c480cea48dec618360"}
Sep 16 19:36:17.512386 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:36:17.512371    2578 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="79e9335e781609d96c264a808f53de4ba3440d4b69d8b7c480cea48dec618360"
Sep 16 19:36:17.543073 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:36:17.543025    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-sche7v8gn" podUID="b0a0cbe7-40bc-4439-a3ab-ccd62e042295" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:36:17.602321 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:36:17.602295    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-7bc4bd10_llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dt4qpr_46db97e1-daa2-4264-8a82-e6b465ff4905/modelcar/0.log"
Sep 16 19:36:17.605305 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:36:17.605281    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dt4qpr"
Sep 16 19:36:17.611948 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:36:17.611928    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wm2kn\" (UniqueName: \"kubernetes.io/projected/46db97e1-daa2-4264-8a82-e6b465ff4905-kube-api-access-wm2kn\") pod \"46db97e1-daa2-4264-8a82-e6b465ff4905\" (UID: \"46db97e1-daa2-4264-8a82-e6b465ff4905\") "
Sep 16 19:36:17.612134 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:36:17.611967    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/46db97e1-daa2-4264-8a82-e6b465ff4905-kserve-provision-location\") pod \"46db97e1-daa2-4264-8a82-e6b465ff4905\" (UID: \"46db97e1-daa2-4264-8a82-e6b465ff4905\") "
Sep 16 19:36:17.612134 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:36:17.612023    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/46db97e1-daa2-4264-8a82-e6b465ff4905-home\") pod \"46db97e1-daa2-4264-8a82-e6b465ff4905\" (UID: \"46db97e1-daa2-4264-8a82-e6b465ff4905\") "
Sep 16 19:36:17.612134 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:36:17.612060    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/46db97e1-daa2-4264-8a82-e6b465ff4905-model-cache\") pod \"46db97e1-daa2-4264-8a82-e6b465ff4905\" (UID: \"46db97e1-daa2-4264-8a82-e6b465ff4905\") "
Sep 16 19:36:17.612134 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:36:17.612080    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/46db97e1-daa2-4264-8a82-e6b465ff4905-tmp-dir\") pod \"46db97e1-daa2-4264-8a82-e6b465ff4905\" (UID: \"46db97e1-daa2-4264-8a82-e6b465ff4905\") "
Sep 16 19:36:17.612134 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:36:17.612106    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/46db97e1-daa2-4264-8a82-e6b465ff4905-tls-certs\") pod \"46db97e1-daa2-4264-8a82-e6b465ff4905\" (UID: \"46db97e1-daa2-4264-8a82-e6b465ff4905\") "
Sep 16 19:36:17.612406 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:36:17.612323    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46db97e1-daa2-4264-8a82-e6b465ff4905-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "46db97e1-daa2-4264-8a82-e6b465ff4905" (UID: "46db97e1-daa2-4264-8a82-e6b465ff4905"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:36:17.612456 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:36:17.612402    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46db97e1-daa2-4264-8a82-e6b465ff4905-model-cache" (OuterVolumeSpecName: "model-cache") pod "46db97e1-daa2-4264-8a82-e6b465ff4905" (UID: "46db97e1-daa2-4264-8a82-e6b465ff4905"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:36:17.612764 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:36:17.612729    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46db97e1-daa2-4264-8a82-e6b465ff4905-home" (OuterVolumeSpecName: "home") pod "46db97e1-daa2-4264-8a82-e6b465ff4905" (UID: "46db97e1-daa2-4264-8a82-e6b465ff4905"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:36:17.614762 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:36:17.614738    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/46db97e1-daa2-4264-8a82-e6b465ff4905-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "46db97e1-daa2-4264-8a82-e6b465ff4905" (UID: "46db97e1-daa2-4264-8a82-e6b465ff4905"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:36:17.614762 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:36:17.614752    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/46db97e1-daa2-4264-8a82-e6b465ff4905-kube-api-access-wm2kn" (OuterVolumeSpecName: "kube-api-access-wm2kn") pod "46db97e1-daa2-4264-8a82-e6b465ff4905" (UID: "46db97e1-daa2-4264-8a82-e6b465ff4905"). InnerVolumeSpecName "kube-api-access-wm2kn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:36:17.628748 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:36:17.628711    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46db97e1-daa2-4264-8a82-e6b465ff4905-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "46db97e1-daa2-4264-8a82-e6b465ff4905" (UID: "46db97e1-daa2-4264-8a82-e6b465ff4905"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:36:17.712779 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:36:17.712741    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/46db97e1-daa2-4264-8a82-e6b465ff4905-dshm\") pod \"46db97e1-daa2-4264-8a82-e6b465ff4905\" (UID: \"46db97e1-daa2-4264-8a82-e6b465ff4905\") "
Sep 16 19:36:17.713089 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:36:17.713069    2578 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/46db97e1-daa2-4264-8a82-e6b465ff4905-kserve-provision-location\") on node \"ip-10-0-135-119.ec2.internal\" DevicePath \"\""
Sep 16 19:36:17.713089 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:36:17.713090    2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/46db97e1-daa2-4264-8a82-e6b465ff4905-home\") on node \"ip-10-0-135-119.ec2.internal\" DevicePath \"\""
Sep 16 19:36:17.713228 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:36:17.713105    2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/46db97e1-daa2-4264-8a82-e6b465ff4905-model-cache\") on node \"ip-10-0-135-119.ec2.internal\" DevicePath \"\""
Sep 16 19:36:17.713228 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:36:17.713138    2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/46db97e1-daa2-4264-8a82-e6b465ff4905-tmp-dir\") on node \"ip-10-0-135-119.ec2.internal\" DevicePath \"\""
Sep 16 19:36:17.713228 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:36:17.713146    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/46db97e1-daa2-4264-8a82-e6b465ff4905-tls-certs\") on node \"ip-10-0-135-119.ec2.internal\" DevicePath \"\""
Sep 16 19:36:17.713228 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:36:17.713154    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wm2kn\" (UniqueName: \"kubernetes.io/projected/46db97e1-daa2-4264-8a82-e6b465ff4905-kube-api-access-wm2kn\") on node \"ip-10-0-135-119.ec2.internal\" DevicePath \"\""
Sep 16 19:36:17.714915 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:36:17.714889    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46db97e1-daa2-4264-8a82-e6b465ff4905-dshm" (OuterVolumeSpecName: "dshm") pod "46db97e1-daa2-4264-8a82-e6b465ff4905" (UID: "46db97e1-daa2-4264-8a82-e6b465ff4905"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:36:17.813622 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:36:17.813583    2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/46db97e1-daa2-4264-8a82-e6b465ff4905-dshm\") on node \"ip-10-0-135-119.ec2.internal\" DevicePath \"\""
Sep 16 19:36:18.517723 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:36:18.517689    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dt4qpr"
Sep 16 19:36:18.542223 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:36:18.542188    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dt4qpr"]
Sep 16 19:36:18.549482 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:36:18.549437    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dt4qpr"]
Sep 16 19:36:19.440835 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:36:19.440796    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="46db97e1-daa2-4264-8a82-e6b465ff4905" path="/var/lib/kubelet/pods/46db97e1-daa2-4264-8a82-e6b465ff4905/volumes"
Sep 16 19:36:27.543030 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:36:27.542968    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-sche7v8gn" podUID="b0a0cbe7-40bc-4439-a3ab-ccd62e042295" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:36:27.543450 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:36:27.543105    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-sche7v8gn"
Sep 16 19:36:37.543120 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:36:37.543076    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-sche7v8gn" podUID="b0a0cbe7-40bc-4439-a3ab-ccd62e042295" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:36:46.583704 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:36:46.583676    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-sche7v8gn"
Sep 16 19:36:46.661082 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:36:46.661047    2578 generic.go:356] "Generic (PLEG): container finished" podID="b0a0cbe7-40bc-4439-a3ab-ccd62e042295" containerID="fb6369cdb42a6e53564fec5502a0b42f83a3f4616d290d3c7c9fea4b148e4a2e" exitCode=0
Sep 16 19:36:46.661273 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:36:46.661096    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-sche7v8gn"
Sep 16 19:36:46.661273 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:36:46.661129    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-sche7v8gn" event={"ID":"b0a0cbe7-40bc-4439-a3ab-ccd62e042295","Type":"ContainerDied","Data":"fb6369cdb42a6e53564fec5502a0b42f83a3f4616d290d3c7c9fea4b148e4a2e"}
Sep 16 19:36:46.661273 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:36:46.661180    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-sche7v8gn" event={"ID":"b0a0cbe7-40bc-4439-a3ab-ccd62e042295","Type":"ContainerDied","Data":"41f4fb6e65418abbf579b5af29832af0621c26d0f6ed23c885805f79f929f108"}
Sep 16 19:36:46.661273 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:36:46.661205    2578 scope.go:117] "RemoveContainer" containerID="fb6369cdb42a6e53564fec5502a0b42f83a3f4616d290d3c7c9fea4b148e4a2e"
Sep 16 19:36:46.664534 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:36:46.664506    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b0a0cbe7-40bc-4439-a3ab-ccd62e042295-tls-certs\") pod \"b0a0cbe7-40bc-4439-a3ab-ccd62e042295\" (UID: \"b0a0cbe7-40bc-4439-a3ab-ccd62e042295\") "
Sep 16 19:36:46.664657 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:36:46.664568    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4ht7g\" (UniqueName: \"kubernetes.io/projected/b0a0cbe7-40bc-4439-a3ab-ccd62e042295-kube-api-access-4ht7g\") pod \"b0a0cbe7-40bc-4439-a3ab-ccd62e042295\" (UID: \"b0a0cbe7-40bc-4439-a3ab-ccd62e042295\") "
Sep 16 19:36:46.666881 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:36:46.666856    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b0a0cbe7-40bc-4439-a3ab-ccd62e042295-kube-api-access-4ht7g" (OuterVolumeSpecName: "kube-api-access-4ht7g") pod "b0a0cbe7-40bc-4439-a3ab-ccd62e042295" (UID: "b0a0cbe7-40bc-4439-a3ab-ccd62e042295"). InnerVolumeSpecName "kube-api-access-4ht7g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:36:46.667026 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:36:46.666901    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b0a0cbe7-40bc-4439-a3ab-ccd62e042295-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "b0a0cbe7-40bc-4439-a3ab-ccd62e042295" (UID: "b0a0cbe7-40bc-4439-a3ab-ccd62e042295"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:36:46.670344 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:36:46.670324    2578 scope.go:117] "RemoveContainer" containerID="fb6369cdb42a6e53564fec5502a0b42f83a3f4616d290d3c7c9fea4b148e4a2e"
Sep 16 19:36:46.670667 ip-10-0-135-119 kubenswrapper[2578]: E0916 19:36:46.670622    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fb6369cdb42a6e53564fec5502a0b42f83a3f4616d290d3c7c9fea4b148e4a2e\": container with ID starting with fb6369cdb42a6e53564fec5502a0b42f83a3f4616d290d3c7c9fea4b148e4a2e not found: ID does not exist" containerID="fb6369cdb42a6e53564fec5502a0b42f83a3f4616d290d3c7c9fea4b148e4a2e"
Sep 16 19:36:46.670726 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:36:46.670676    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fb6369cdb42a6e53564fec5502a0b42f83a3f4616d290d3c7c9fea4b148e4a2e"} err="failed to get container status \"fb6369cdb42a6e53564fec5502a0b42f83a3f4616d290d3c7c9fea4b148e4a2e\": rpc error: code = NotFound desc = could not find container \"fb6369cdb42a6e53564fec5502a0b42f83a3f4616d290d3c7c9fea4b148e4a2e\": container with ID starting with fb6369cdb42a6e53564fec5502a0b42f83a3f4616d290d3c7c9fea4b148e4a2e not found: ID does not exist"
Sep 16 19:36:46.748912 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:36:46.748873    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:36:46.766145 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:36:46.766097    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4ht7g\" (UniqueName: \"kubernetes.io/projected/b0a0cbe7-40bc-4439-a3ab-ccd62e042295-kube-api-access-4ht7g\") on node \"ip-10-0-135-119.ec2.internal\" DevicePath \"\""
Sep 16 19:36:46.766145 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:36:46.766139    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b0a0cbe7-40bc-4439-a3ab-ccd62e042295-tls-certs\") on node \"ip-10-0-135-119.ec2.internal\" DevicePath \"\""
Sep 16 19:36:46.850009 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:36:46.849943    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:36:46.986551 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:36:46.986519    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-sche7v8gn"]
Sep 16 19:36:46.991223 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:36:46.991189    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-sche7v8gn"]
Sep 16 19:36:47.435304 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:36:47.435264    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b0a0cbe7-40bc-4439-a3ab-ccd62e042295" path="/var/lib/kubelet/pods/b0a0cbe7-40bc-4439-a3ab-ccd62e042295/volumes"
Sep 16 19:36:51.373525 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:36:51.373445    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:36:51.465179 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:36:51.465116    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:36:58.264304 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:36:58.264254    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:37:02.700156 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:37:02.700121    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:37:02.916427 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:37:02.916390    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc58j62ft"]
Sep 16 19:37:02.916836 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:37:02.916816    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="46db97e1-daa2-4264-8a82-e6b465ff4905" containerName="modelcar-init"
Sep 16 19:37:02.916926 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:37:02.916840    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="46db97e1-daa2-4264-8a82-e6b465ff4905" containerName="modelcar-init"
Sep 16 19:37:02.916926 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:37:02.916865    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b0a0cbe7-40bc-4439-a3ab-ccd62e042295" containerName="main"
Sep 16 19:37:02.916926 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:37:02.916874    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="b0a0cbe7-40bc-4439-a3ab-ccd62e042295" containerName="main"
Sep 16 19:37:02.916926 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:37:02.916887    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="46db97e1-daa2-4264-8a82-e6b465ff4905" containerName="modelcar"
Sep 16 19:37:02.916926 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:37:02.916896    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="46db97e1-daa2-4264-8a82-e6b465ff4905" containerName="modelcar"
Sep 16 19:37:02.916926 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:37:02.916911    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="46db97e1-daa2-4264-8a82-e6b465ff4905" containerName="main"
Sep 16 19:37:02.916926 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:37:02.916919    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="46db97e1-daa2-4264-8a82-e6b465ff4905" containerName="main"
Sep 16 19:37:02.917303 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:37:02.917041    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="46db97e1-daa2-4264-8a82-e6b465ff4905" containerName="main"
Sep 16 19:37:02.917303 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:37:02.917058    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="b0a0cbe7-40bc-4439-a3ab-ccd62e042295" containerName="main"
Sep 16 19:37:02.917303 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:37:02.917068    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="46db97e1-daa2-4264-8a82-e6b465ff4905" containerName="modelcar"
Sep 16 19:37:02.920115 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:37:02.920083    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc58j62ft"
Sep 16 19:37:02.922933 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:37:02.922904    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-caba0b4a\"/\"custom-route-timeout-test-epp-sa-dockercfg-b2jst\""
Sep 16 19:37:02.923078 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:37:02.922949    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-caba0b4a\"/\"openshift-service-ca.crt\""
Sep 16 19:37:02.923078 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:37:02.922908    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-caba0b4a\"/\"custom-route-timeout-test-kserve-self-signed-certs\""
Sep 16 19:37:02.923078 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:37:02.922907    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-caba0b4a\"/\"kube-root-ca.crt\""
Sep 16 19:37:02.923078 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:37:02.922915    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-caba0b4a\"/\"default-dockercfg-2vrc2\""
Sep 16 19:37:02.931035 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:37:02.931010    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc58j62ft"]
Sep 16 19:37:02.946584 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:37:02.946551    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:37:03.017615 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:37:03.017496    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vkx8j\" (UniqueName: \"kubernetes.io/projected/4fe1ae1d-4e91-4508-b36e-f7668ba6f121-kube-api-access-vkx8j\") pod \"custom-route-timeout-test-kserve-router-scheduler-54cbcc58j62ft\" (UID: \"4fe1ae1d-4e91-4508-b36e-f7668ba6f121\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc58j62ft"
Sep 16 19:37:03.017615 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:37:03.017575    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4fe1ae1d-4e91-4508-b36e-f7668ba6f121-tls-certs\") pod \"custom-route-timeout-test-kserve-router-scheduler-54cbcc58j62ft\" (UID: \"4fe1ae1d-4e91-4508-b36e-f7668ba6f121\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc58j62ft"
Sep 16 19:37:03.118244 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:37:03.118203    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vkx8j\" (UniqueName: \"kubernetes.io/projected/4fe1ae1d-4e91-4508-b36e-f7668ba6f121-kube-api-access-vkx8j\") pod \"custom-route-timeout-test-kserve-router-scheduler-54cbcc58j62ft\" (UID: \"4fe1ae1d-4e91-4508-b36e-f7668ba6f121\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc58j62ft"
Sep 16 19:37:03.118428 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:37:03.118271    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4fe1ae1d-4e91-4508-b36e-f7668ba6f121-tls-certs\") pod \"custom-route-timeout-test-kserve-router-scheduler-54cbcc58j62ft\" (UID: \"4fe1ae1d-4e91-4508-b36e-f7668ba6f121\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc58j62ft"
Sep 16 19:37:03.121005 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:37:03.120962    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4fe1ae1d-4e91-4508-b36e-f7668ba6f121-tls-certs\") pod \"custom-route-timeout-test-kserve-router-scheduler-54cbcc58j62ft\" (UID: \"4fe1ae1d-4e91-4508-b36e-f7668ba6f121\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc58j62ft"
Sep 16 19:37:03.127810 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:37:03.127784    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vkx8j\" (UniqueName: \"kubernetes.io/projected/4fe1ae1d-4e91-4508-b36e-f7668ba6f121-kube-api-access-vkx8j\") pod \"custom-route-timeout-test-kserve-router-scheduler-54cbcc58j62ft\" (UID: \"4fe1ae1d-4e91-4508-b36e-f7668ba6f121\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc58j62ft"
Sep 16 19:37:03.232142 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:37:03.232095    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc58j62ft"
Sep 16 19:37:03.510481 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:37:03.510442    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc58j62ft"]
Sep 16 19:37:03.518197 ip-10-0-135-119 kubenswrapper[2578]: W0916 19:37:03.518161    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4fe1ae1d_4e91_4508_b36e_f7668ba6f121.slice/crio-93bc6d343b8d53b26ffe6389b5a272f0a54928e580423c0d5cf4e0370acd616b WatchSource:0}: Error finding container 93bc6d343b8d53b26ffe6389b5a272f0a54928e580423c0d5cf4e0370acd616b: Status 404 returned error can't find the container with id 93bc6d343b8d53b26ffe6389b5a272f0a54928e580423c0d5cf4e0370acd616b
Sep 16 19:37:03.744069 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:37:03.744033    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc58j62ft" event={"ID":"4fe1ae1d-4e91-4508-b36e-f7668ba6f121","Type":"ContainerStarted","Data":"0ef7f469f7219d3848e19def502b925355f6cb943a3211976f508b72e402131c"}
Sep 16 19:37:03.744499 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:37:03.744096    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc58j62ft" event={"ID":"4fe1ae1d-4e91-4508-b36e-f7668ba6f121","Type":"ContainerStarted","Data":"93bc6d343b8d53b26ffe6389b5a272f0a54928e580423c0d5cf4e0370acd616b"}
Sep 16 19:37:03.744499 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:37:03.744151    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc58j62ft"
Sep 16 19:37:03.762480 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:37:03.762422    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc58j62ft" podStartSLOduration=1.762405821 podStartE2EDuration="1.762405821s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:37:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:37:03.760943671 +0000 UTC m=+3404.827435171" watchObservedRunningTime="2026-09-16 19:37:03.762405821 +0000 UTC m=+3404.828897308"
Sep 16 19:37:05.053098 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:37:05.053049    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:37:05.154588 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:37:05.154553    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:37:19.001154 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:37:19.001108    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:37:34.751192 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:37:34.751155    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc58j62ft"
Sep 16 19:37:35.231757 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:37:35.231720    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:37:35.332505 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:37:35.332467    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:37:35.431330 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:37:35.431291    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:37:42.073584 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:37:42.073542    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:37:48.088479 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:37:48.088435    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:37:48.153343 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:37:48.153309    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:37:49.016720 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:37:49.016677    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:38:12.772046 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:38:12.772010    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:38:12.834895 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:38:12.834852    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:38:58.400911 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:38:58.400812    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc58j62ft"]
Sep 16 19:38:58.401875 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:38:58.401817    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:38:59.820882 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:38:59.820831    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:39:00.425525 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:39:00.425480    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:39:04.750719 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:39:04.750661    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc58j62ft" podUID="4fe1ae1d-4e91-4508-b36e-f7668ba6f121" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:39:13.420956 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:39:13.420910    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc58j62ft" podUID="4fe1ae1d-4e91-4508-b36e-f7668ba6f121" containerName="main" containerID="cri-o://0ef7f469f7219d3848e19def502b925355f6cb943a3211976f508b72e402131c" gracePeriod=45
Sep 16 19:39:14.750511 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:39:14.750466    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc58j62ft" podUID="4fe1ae1d-4e91-4508-b36e-f7668ba6f121" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:39:24.750759 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:39:24.750716    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc58j62ft" podUID="4fe1ae1d-4e91-4508-b36e-f7668ba6f121" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:39:24.751195 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:39:24.750826    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc58j62ft"
Sep 16 19:39:34.750528 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:39:34.750487    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc58j62ft" podUID="4fe1ae1d-4e91-4508-b36e-f7668ba6f121" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:39:35.007706 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:39:35.007619    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:39:43.664374 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:39:43.664346    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc58j62ft"
Sep 16 19:39:43.759819 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:39:43.759724    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4fe1ae1d-4e91-4508-b36e-f7668ba6f121-tls-certs\") pod \"4fe1ae1d-4e91-4508-b36e-f7668ba6f121\" (UID: \"4fe1ae1d-4e91-4508-b36e-f7668ba6f121\") "
Sep 16 19:39:43.760006 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:39:43.759860    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vkx8j\" (UniqueName: \"kubernetes.io/projected/4fe1ae1d-4e91-4508-b36e-f7668ba6f121-kube-api-access-vkx8j\") pod \"4fe1ae1d-4e91-4508-b36e-f7668ba6f121\" (UID: \"4fe1ae1d-4e91-4508-b36e-f7668ba6f121\") "
Sep 16 19:39:43.762123 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:39:43.762090    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4fe1ae1d-4e91-4508-b36e-f7668ba6f121-kube-api-access-vkx8j" (OuterVolumeSpecName: "kube-api-access-vkx8j") pod "4fe1ae1d-4e91-4508-b36e-f7668ba6f121" (UID: "4fe1ae1d-4e91-4508-b36e-f7668ba6f121"). InnerVolumeSpecName "kube-api-access-vkx8j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:39:43.762123 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:39:43.762114    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4fe1ae1d-4e91-4508-b36e-f7668ba6f121-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "4fe1ae1d-4e91-4508-b36e-f7668ba6f121" (UID: "4fe1ae1d-4e91-4508-b36e-f7668ba6f121"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:39:43.860854 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:39:43.860822    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4fe1ae1d-4e91-4508-b36e-f7668ba6f121-tls-certs\") on node \"ip-10-0-135-119.ec2.internal\" DevicePath \"\""
Sep 16 19:39:43.860854 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:39:43.860850    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vkx8j\" (UniqueName: \"kubernetes.io/projected/4fe1ae1d-4e91-4508-b36e-f7668ba6f121-kube-api-access-vkx8j\") on node \"ip-10-0-135-119.ec2.internal\" DevicePath \"\""
Sep 16 19:39:44.514105 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:39:44.514060    2578 generic.go:356] "Generic (PLEG): container finished" podID="4fe1ae1d-4e91-4508-b36e-f7668ba6f121" containerID="0ef7f469f7219d3848e19def502b925355f6cb943a3211976f508b72e402131c" exitCode=0
Sep 16 19:39:44.514312 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:39:44.514141    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc58j62ft" event={"ID":"4fe1ae1d-4e91-4508-b36e-f7668ba6f121","Type":"ContainerDied","Data":"0ef7f469f7219d3848e19def502b925355f6cb943a3211976f508b72e402131c"}
Sep 16 19:39:44.514312 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:39:44.514224    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc58j62ft" event={"ID":"4fe1ae1d-4e91-4508-b36e-f7668ba6f121","Type":"ContainerDied","Data":"93bc6d343b8d53b26ffe6389b5a272f0a54928e580423c0d5cf4e0370acd616b"}
Sep 16 19:39:44.514312 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:39:44.514263    2578 scope.go:117] "RemoveContainer" containerID="0ef7f469f7219d3848e19def502b925355f6cb943a3211976f508b72e402131c"
Sep 16 19:39:44.514552 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:39:44.514520    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc58j62ft"
Sep 16 19:39:44.524430 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:39:44.524409    2578 scope.go:117] "RemoveContainer" containerID="0ef7f469f7219d3848e19def502b925355f6cb943a3211976f508b72e402131c"
Sep 16 19:39:44.524706 ip-10-0-135-119 kubenswrapper[2578]: E0916 19:39:44.524685    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0ef7f469f7219d3848e19def502b925355f6cb943a3211976f508b72e402131c\": container with ID starting with 0ef7f469f7219d3848e19def502b925355f6cb943a3211976f508b72e402131c not found: ID does not exist" containerID="0ef7f469f7219d3848e19def502b925355f6cb943a3211976f508b72e402131c"
Sep 16 19:39:44.524775 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:39:44.524717    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0ef7f469f7219d3848e19def502b925355f6cb943a3211976f508b72e402131c"} err="failed to get container status \"0ef7f469f7219d3848e19def502b925355f6cb943a3211976f508b72e402131c\": rpc error: code = NotFound desc = could not find container \"0ef7f469f7219d3848e19def502b925355f6cb943a3211976f508b72e402131c\": container with ID starting with 0ef7f469f7219d3848e19def502b925355f6cb943a3211976f508b72e402131c not found: ID does not exist"
Sep 16 19:39:44.538743 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:39:44.538721    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc58j62ft"]
Sep 16 19:39:44.543534 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:39:44.543514    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc58j62ft"]
Sep 16 19:39:45.434437 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:39:45.434406    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4fe1ae1d-4e91-4508-b36e-f7668ba6f121" path="/var/lib/kubelet/pods/4fe1ae1d-4e91-4508-b36e-f7668ba6f121/volumes"
Sep 16 19:39:48.350046 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:39:48.350006    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:39:48.448416 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:39:48.448381    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:39:48.563319 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:39:48.563288    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-785d98b7d8-mkwgn"]
Sep 16 19:39:48.563741 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:39:48.563721    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4fe1ae1d-4e91-4508-b36e-f7668ba6f121" containerName="main"
Sep 16 19:39:48.563800 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:39:48.563743    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="4fe1ae1d-4e91-4508-b36e-f7668ba6f121" containerName="main"
Sep 16 19:39:48.563836 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:39:48.563813    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="4fe1ae1d-4e91-4508-b36e-f7668ba6f121" containerName="main"
Sep 16 19:39:48.568036 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:39:48.568015    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-785d98b7d8-mkwgn"
Sep 16 19:39:48.570438 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:39:48.570414    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-028f7809\"/\"openshift-gw-ca-root-cert\""
Sep 16 19:39:48.570759 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:39:48.570744    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-028f7809\"/\"router-gateway-1-openshift-default-dockercfg-dh2kp\""
Sep 16 19:39:48.571084 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:39:48.571071    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-028f7809\"/\"kube-root-ca.crt\""
Sep 16 19:39:48.571136 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:39:48.571074    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-028f7809\"/\"openshift-service-ca.crt\""
Sep 16 19:39:48.580802 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:39:48.580771    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-785d98b7d8-mkwgn"]
Sep 16 19:39:48.595685 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:39:48.595658    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:39:48.603325 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:39:48.603248    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7618c478-85ac-40e1-b081-3aac9e28041d-credential-socket\") pod \"router-gateway-1-openshift-default-785d98b7d8-mkwgn\" (UID: \"7618c478-85ac-40e1-b081-3aac9e28041d\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-785d98b7d8-mkwgn"
Sep 16 19:39:48.603325 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:39:48.603302    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7618c478-85ac-40e1-b081-3aac9e28041d-istio-envoy\") pod \"router-gateway-1-openshift-default-785d98b7d8-mkwgn\" (UID: \"7618c478-85ac-40e1-b081-3aac9e28041d\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-785d98b7d8-mkwgn"
Sep 16 19:39:48.603521 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:39:48.603333    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7618c478-85ac-40e1-b081-3aac9e28041d-workload-certs\") pod \"router-gateway-1-openshift-default-785d98b7d8-mkwgn\" (UID: \"7618c478-85ac-40e1-b081-3aac9e28041d\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-785d98b7d8-mkwgn"
Sep 16 19:39:48.603521 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:39:48.603383    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7618c478-85ac-40e1-b081-3aac9e28041d-istio-data\") pod \"router-gateway-1-openshift-default-785d98b7d8-mkwgn\" (UID: \"7618c478-85ac-40e1-b081-3aac9e28041d\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-785d98b7d8-mkwgn"
Sep 16 19:39:48.603521 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:39:48.603476    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7618c478-85ac-40e1-b081-3aac9e28041d-workload-socket\") pod \"router-gateway-1-openshift-default-785d98b7d8-mkwgn\" (UID: \"7618c478-85ac-40e1-b081-3aac9e28041d\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-785d98b7d8-mkwgn"
Sep 16 19:39:48.603521 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:39:48.603513    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7618c478-85ac-40e1-b081-3aac9e28041d-istio-token\") pod \"router-gateway-1-openshift-default-785d98b7d8-mkwgn\" (UID: \"7618c478-85ac-40e1-b081-3aac9e28041d\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-785d98b7d8-mkwgn"
Sep 16 19:39:48.603758 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:39:48.603533    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7618c478-85ac-40e1-b081-3aac9e28041d-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-785d98b7d8-mkwgn\" (UID: \"7618c478-85ac-40e1-b081-3aac9e28041d\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-785d98b7d8-mkwgn"
Sep 16 19:39:48.603758 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:39:48.603558    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fpqj5\" (UniqueName: \"kubernetes.io/projected/7618c478-85ac-40e1-b081-3aac9e28041d-kube-api-access-fpqj5\") pod \"router-gateway-1-openshift-default-785d98b7d8-mkwgn\" (UID: \"7618c478-85ac-40e1-b081-3aac9e28041d\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-785d98b7d8-mkwgn"
Sep 16 19:39:48.603758 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:39:48.603614    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7618c478-85ac-40e1-b081-3aac9e28041d-istio-podinfo\") pod \"router-gateway-1-openshift-default-785d98b7d8-mkwgn\" (UID: \"7618c478-85ac-40e1-b081-3aac9e28041d\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-785d98b7d8-mkwgn"
Sep 16 19:39:48.692935 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:39:48.692885    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:39:48.705100 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:39:48.705065    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7618c478-85ac-40e1-b081-3aac9e28041d-istio-envoy\") pod \"router-gateway-1-openshift-default-785d98b7d8-mkwgn\" (UID: \"7618c478-85ac-40e1-b081-3aac9e28041d\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-785d98b7d8-mkwgn"
Sep 16 19:39:48.705243 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:39:48.705105    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7618c478-85ac-40e1-b081-3aac9e28041d-workload-certs\") pod \"router-gateway-1-openshift-default-785d98b7d8-mkwgn\" (UID: \"7618c478-85ac-40e1-b081-3aac9e28041d\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-785d98b7d8-mkwgn"
Sep 16 19:39:48.705243 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:39:48.705126    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7618c478-85ac-40e1-b081-3aac9e28041d-istio-data\") pod \"router-gateway-1-openshift-default-785d98b7d8-mkwgn\" (UID: \"7618c478-85ac-40e1-b081-3aac9e28041d\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-785d98b7d8-mkwgn"
Sep 16 19:39:48.705243 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:39:48.705173    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7618c478-85ac-40e1-b081-3aac9e28041d-workload-socket\") pod \"router-gateway-1-openshift-default-785d98b7d8-mkwgn\" (UID: \"7618c478-85ac-40e1-b081-3aac9e28041d\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-785d98b7d8-mkwgn"
Sep 16 19:39:48.705243 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:39:48.705191    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7618c478-85ac-40e1-b081-3aac9e28041d-istio-token\") pod \"router-gateway-1-openshift-default-785d98b7d8-mkwgn\" (UID: \"7618c478-85ac-40e1-b081-3aac9e28041d\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-785d98b7d8-mkwgn"
Sep 16 19:39:48.705243 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:39:48.705209    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7618c478-85ac-40e1-b081-3aac9e28041d-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-785d98b7d8-mkwgn\" (UID: \"7618c478-85ac-40e1-b081-3aac9e28041d\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-785d98b7d8-mkwgn"
Sep 16 19:39:48.705243 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:39:48.705229    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fpqj5\" (UniqueName: \"kubernetes.io/projected/7618c478-85ac-40e1-b081-3aac9e28041d-kube-api-access-fpqj5\") pod \"router-gateway-1-openshift-default-785d98b7d8-mkwgn\" (UID: \"7618c478-85ac-40e1-b081-3aac9e28041d\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-785d98b7d8-mkwgn"
Sep 16 19:39:48.705763 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:39:48.705260    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7618c478-85ac-40e1-b081-3aac9e28041d-istio-podinfo\") pod \"router-gateway-1-openshift-default-785d98b7d8-mkwgn\" (UID: \"7618c478-85ac-40e1-b081-3aac9e28041d\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-785d98b7d8-mkwgn"
Sep 16 19:39:48.705763 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:39:48.705327    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7618c478-85ac-40e1-b081-3aac9e28041d-credential-socket\") pod \"router-gateway-1-openshift-default-785d98b7d8-mkwgn\" (UID: \"7618c478-85ac-40e1-b081-3aac9e28041d\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-785d98b7d8-mkwgn"
Sep 16 19:39:48.705877 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:39:48.705782    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7618c478-85ac-40e1-b081-3aac9e28041d-workload-socket\") pod \"router-gateway-1-openshift-default-785d98b7d8-mkwgn\" (UID: \"7618c478-85ac-40e1-b081-3aac9e28041d\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-785d98b7d8-mkwgn"
Sep 16 19:39:48.705877 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:39:48.705806    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7618c478-85ac-40e1-b081-3aac9e28041d-istio-data\") pod \"router-gateway-1-openshift-default-785d98b7d8-mkwgn\" (UID: \"7618c478-85ac-40e1-b081-3aac9e28041d\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-785d98b7d8-mkwgn"
Sep 16 19:39:48.705877 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:39:48.705839    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7618c478-85ac-40e1-b081-3aac9e28041d-credential-socket\") pod \"router-gateway-1-openshift-default-785d98b7d8-mkwgn\" (UID: \"7618c478-85ac-40e1-b081-3aac9e28041d\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-785d98b7d8-mkwgn"
Sep 16 19:39:48.706231 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:39:48.706198    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7618c478-85ac-40e1-b081-3aac9e28041d-workload-certs\") pod \"router-gateway-1-openshift-default-785d98b7d8-mkwgn\" (UID: \"7618c478-85ac-40e1-b081-3aac9e28041d\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-785d98b7d8-mkwgn"
Sep 16 19:39:48.706339 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:39:48.706301    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7618c478-85ac-40e1-b081-3aac9e28041d-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-785d98b7d8-mkwgn\" (UID: \"7618c478-85ac-40e1-b081-3aac9e28041d\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-785d98b7d8-mkwgn"
Sep 16 19:39:48.707788 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:39:48.707764    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7618c478-85ac-40e1-b081-3aac9e28041d-istio-envoy\") pod \"router-gateway-1-openshift-default-785d98b7d8-mkwgn\" (UID: \"7618c478-85ac-40e1-b081-3aac9e28041d\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-785d98b7d8-mkwgn"
Sep 16 19:39:48.708036 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:39:48.708018    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7618c478-85ac-40e1-b081-3aac9e28041d-istio-podinfo\") pod \"router-gateway-1-openshift-default-785d98b7d8-mkwgn\" (UID: \"7618c478-85ac-40e1-b081-3aac9e28041d\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-785d98b7d8-mkwgn"
Sep 16 19:39:48.716401 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:39:48.716379    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7618c478-85ac-40e1-b081-3aac9e28041d-istio-token\") pod \"router-gateway-1-openshift-default-785d98b7d8-mkwgn\" (UID: \"7618c478-85ac-40e1-b081-3aac9e28041d\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-785d98b7d8-mkwgn"
Sep 16 19:39:48.716493 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:39:48.716457    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fpqj5\" (UniqueName: \"kubernetes.io/projected/7618c478-85ac-40e1-b081-3aac9e28041d-kube-api-access-fpqj5\") pod \"router-gateway-1-openshift-default-785d98b7d8-mkwgn\" (UID: \"7618c478-85ac-40e1-b081-3aac9e28041d\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-785d98b7d8-mkwgn"
Sep 16 19:39:48.878828 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:39:48.878799    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-785d98b7d8-mkwgn"
Sep 16 19:39:49.158208 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:39:49.158181    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-785d98b7d8-mkwgn"]
Sep 16 19:39:49.160999 ip-10-0-135-119 kubenswrapper[2578]: W0916 19:39:49.160955    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7618c478_85ac_40e1_b081_3aac9e28041d.slice/crio-b196bbaefd58a82436d08d8b6a0f8e9fee04a0ab57d5098e642a23ad3a2e542e WatchSource:0}: Error finding container b196bbaefd58a82436d08d8b6a0f8e9fee04a0ab57d5098e642a23ad3a2e542e: Status 404 returned error can't find the container with id b196bbaefd58a82436d08d8b6a0f8e9fee04a0ab57d5098e642a23ad3a2e542e
Sep 16 19:39:49.163066 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:39:49.163048    2578 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 16 19:39:49.543054 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:39:49.542945    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-785d98b7d8-mkwgn" event={"ID":"7618c478-85ac-40e1-b081-3aac9e28041d","Type":"ContainerStarted","Data":"b196bbaefd58a82436d08d8b6a0f8e9fee04a0ab57d5098e642a23ad3a2e542e"}
Sep 16 19:39:52.001527 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:39:52.001478    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"}
Sep 16 19:39:52.001804 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:39:52.001586    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"}
Sep 16 19:39:52.001804 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:39:52.001665    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"}
Sep 16 19:39:52.478279 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:39:52.478236    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:39:52.562454 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:39:52.562416    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-785d98b7d8-mkwgn" event={"ID":"7618c478-85ac-40e1-b081-3aac9e28041d","Type":"ContainerStarted","Data":"b938505211a42d5f04fdc06362d565163464382e5a2b13924c64d87b99b58e11"}
Sep 16 19:39:52.598200 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:39:52.597226    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-785d98b7d8-mkwgn" podStartSLOduration=1.759193272 podStartE2EDuration="4.597199358s" totalImagesPullingTime="2.838006086s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:39:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:39:52.581349103 +0000 UTC m=+3573.647840647" watchObservedRunningTime="2026-09-16 19:39:52.597199358 +0000 UTC m=+3573.663690847"
Sep 16 19:39:52.879100 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:39:52.879065    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-785d98b7d8-mkwgn"
Sep 16 19:39:53.881532 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:39:53.881483    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-785d98b7d8-mkwgn" podUID="7618c478-85ac-40e1-b081-3aac9e28041d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused"
Sep 16 19:39:54.880173 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:39:54.880124    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-785d98b7d8-mkwgn" podUID="7618c478-85ac-40e1-b081-3aac9e28041d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused"
Sep 16 19:39:54.979671 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:39:54.979634    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:39:55.392564 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:39:55.392521    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:39:55.499258 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:39:55.499214    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:39:55.593554 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:39:55.593509    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:39:55.879820 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:39:55.879774    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-785d98b7d8-mkwgn" podUID="7618c478-85ac-40e1-b081-3aac9e28041d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused"
Sep 16 19:39:56.879492 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:39:56.879438    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-785d98b7d8-mkwgn" podUID="7618c478-85ac-40e1-b081-3aac9e28041d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused"
Sep 16 19:39:57.879939 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:39:57.879888    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-785d98b7d8-mkwgn" podUID="7618c478-85ac-40e1-b081-3aac9e28041d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused"
Sep 16 19:39:58.884162 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:39:58.884124    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-785d98b7d8-mkwgn"
Sep 16 19:39:58.884572 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:39:58.884424    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-785d98b7d8-mkwgn"
Sep 16 19:39:58.885098 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:39:58.885078    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-785d98b7d8-mkwgn"
Sep 16 19:40:01.576971 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:40:01.576933    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:40:01.644517 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:40:01.644449    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:40:02.107875 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:40:02.107833    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:40:02.208718 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:40:02.208673    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:40:02.322700 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:40:02.322657    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:40:02.463519 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:40:02.463479    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:40:02.556705 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:40:02.556670    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:40:19.675312 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:40:19.675279    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-tmrkx_4c2078bb-281a-4aa1-8e38-238897865434/console-operator/2.log"
Sep 16 19:40:19.684646 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:40:19.684616    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-k6gvs_6012728d-aee6-4f93-ab6e-fad616ef36e8/ovn-acl-logging/0.log"
Sep 16 19:40:19.692911 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:40:19.692889    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-tmrkx_4c2078bb-281a-4aa1-8e38-238897865434/console-operator/2.log"
Sep 16 19:40:19.701208 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:40:19.701184    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-k6gvs_6012728d-aee6-4f93-ab6e-fad616ef36e8/ovn-acl-logging/0.log"
Sep 16 19:40:19.936102 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:40:19.936019    2578 scope.go:117] "RemoveContainer" containerID="06ff6ec9474f90481923476c1ea5cbf35e667306263d54ff398829fd5c677540"
Sep 16 19:40:19.945212 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:40:19.945186    2578 scope.go:117] "RemoveContainer" containerID="978adf36e96d59d3044dc327e7129c5b9ed73f00bf6e18ad98dc5dd2f7f80dc7"
Sep 16 19:40:19.953505 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:40:19.953482    2578 scope.go:117] "RemoveContainer" containerID="72b8b5f99684ed4219f0165525610f48b608a5776b701fc72cbf2b3d48590b43"
Sep 16 19:40:27.479383 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:40:27.479346    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:40:28.653101 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:40:28.653058    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:40:47.821462 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:40:47.821412    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:42:19.789467 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:42:19.789425    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:42:19.911236 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:42:19.911200    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:42:22.590129 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:42:22.590088    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:42:23.435639 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:42:23.435603    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:42:42.600707 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:42:42.600666    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:42:42.702332 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:42:42.702297    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:42:42.801654 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:42:42.801617    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:42:42.900953 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:42:42.900916    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:42:43.001371 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:42:43.001332    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:42:43.102338 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:42:43.102297    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:42:45.281065 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:42:45.281030    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:42:45.383791 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:42:45.383756    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:42:46.055600 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:42:46.055564    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:42:49.297625 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:42:49.297573    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:42:49.498616 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:42:49.498574    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:43:09.778280 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:43:09.778238    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:43:09.876080 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:43:09.876047    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:43:10.902120 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:43:10.902085    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:43:14.736761 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:43:14.736719    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-785d98b7d8-mkwgn"]
Sep 16 19:43:14.737257 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:43:14.736968    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-785d98b7d8-mkwgn" podUID="7618c478-85ac-40e1-b081-3aac9e28041d" containerName="istio-proxy" containerID="cri-o://b938505211a42d5f04fdc06362d565163464382e5a2b13924c64d87b99b58e11" gracePeriod=30
Sep 16 19:43:14.834589 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:43:14.834549    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:43:19.997522 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:43:19.997462    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-785d98b7d8-mkwgn"
Sep 16 19:43:20.089440 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:43:20.089408    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7618c478-85ac-40e1-b081-3aac9e28041d-credential-socket\") pod \"7618c478-85ac-40e1-b081-3aac9e28041d\" (UID: \"7618c478-85ac-40e1-b081-3aac9e28041d\") "
Sep 16 19:43:20.089606 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:43:20.089464    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7618c478-85ac-40e1-b081-3aac9e28041d-istio-podinfo\") pod \"7618c478-85ac-40e1-b081-3aac9e28041d\" (UID: \"7618c478-85ac-40e1-b081-3aac9e28041d\") "
Sep 16 19:43:20.089606 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:43:20.089497    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7618c478-85ac-40e1-b081-3aac9e28041d-istio-data\") pod \"7618c478-85ac-40e1-b081-3aac9e28041d\" (UID: \"7618c478-85ac-40e1-b081-3aac9e28041d\") "
Sep 16 19:43:20.089606 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:43:20.089519    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7618c478-85ac-40e1-b081-3aac9e28041d-istio-envoy\") pod \"7618c478-85ac-40e1-b081-3aac9e28041d\" (UID: \"7618c478-85ac-40e1-b081-3aac9e28041d\") "
Sep 16 19:43:20.089606 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:43:20.089545    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fpqj5\" (UniqueName: \"kubernetes.io/projected/7618c478-85ac-40e1-b081-3aac9e28041d-kube-api-access-fpqj5\") pod \"7618c478-85ac-40e1-b081-3aac9e28041d\" (UID: \"7618c478-85ac-40e1-b081-3aac9e28041d\") "
Sep 16 19:43:20.089606 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:43:20.089565    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7618c478-85ac-40e1-b081-3aac9e28041d-workload-socket\") pod \"7618c478-85ac-40e1-b081-3aac9e28041d\" (UID: \"7618c478-85ac-40e1-b081-3aac9e28041d\") "
Sep 16 19:43:20.089898 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:43:20.089632    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7618c478-85ac-40e1-b081-3aac9e28041d-workload-certs\") pod \"7618c478-85ac-40e1-b081-3aac9e28041d\" (UID: \"7618c478-85ac-40e1-b081-3aac9e28041d\") "
Sep 16 19:43:20.089898 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:43:20.089693    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7618c478-85ac-40e1-b081-3aac9e28041d-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "7618c478-85ac-40e1-b081-3aac9e28041d" (UID: "7618c478-85ac-40e1-b081-3aac9e28041d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:43:20.089898 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:43:20.089755    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7618c478-85ac-40e1-b081-3aac9e28041d-istio-token\") pod \"7618c478-85ac-40e1-b081-3aac9e28041d\" (UID: \"7618c478-85ac-40e1-b081-3aac9e28041d\") "
Sep 16 19:43:20.089898 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:43:20.089781    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7618c478-85ac-40e1-b081-3aac9e28041d-istio-data" (OuterVolumeSpecName: "istio-data") pod "7618c478-85ac-40e1-b081-3aac9e28041d" (UID: "7618c478-85ac-40e1-b081-3aac9e28041d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:43:20.089898 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:43:20.089822    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7618c478-85ac-40e1-b081-3aac9e28041d-istiod-ca-cert\") pod \"7618c478-85ac-40e1-b081-3aac9e28041d\" (UID: \"7618c478-85ac-40e1-b081-3aac9e28041d\") "
Sep 16 19:43:20.090194 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:43:20.090003    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7618c478-85ac-40e1-b081-3aac9e28041d-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "7618c478-85ac-40e1-b081-3aac9e28041d" (UID: "7618c478-85ac-40e1-b081-3aac9e28041d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:43:20.090194 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:43:20.090036    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7618c478-85ac-40e1-b081-3aac9e28041d-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "7618c478-85ac-40e1-b081-3aac9e28041d" (UID: "7618c478-85ac-40e1-b081-3aac9e28041d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:43:20.090305 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:43:20.090193    2578 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7618c478-85ac-40e1-b081-3aac9e28041d-workload-certs\") on node \"ip-10-0-135-119.ec2.internal\" DevicePath \"\""
Sep 16 19:43:20.090305 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:43:20.090213    2578 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7618c478-85ac-40e1-b081-3aac9e28041d-credential-socket\") on node \"ip-10-0-135-119.ec2.internal\" DevicePath \"\""
Sep 16 19:43:20.090305 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:43:20.090229    2578 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7618c478-85ac-40e1-b081-3aac9e28041d-istio-data\") on node \"ip-10-0-135-119.ec2.internal\" DevicePath \"\""
Sep 16 19:43:20.090305 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:43:20.090244    2578 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7618c478-85ac-40e1-b081-3aac9e28041d-workload-socket\") on node \"ip-10-0-135-119.ec2.internal\" DevicePath \"\""
Sep 16 19:43:20.090681 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:43:20.090340    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7618c478-85ac-40e1-b081-3aac9e28041d-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "7618c478-85ac-40e1-b081-3aac9e28041d" (UID: "7618c478-85ac-40e1-b081-3aac9e28041d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:43:20.092216 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:43:20.092186    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7618c478-85ac-40e1-b081-3aac9e28041d-kube-api-access-fpqj5" (OuterVolumeSpecName: "kube-api-access-fpqj5") pod "7618c478-85ac-40e1-b081-3aac9e28041d" (UID: "7618c478-85ac-40e1-b081-3aac9e28041d"). InnerVolumeSpecName "kube-api-access-fpqj5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:43:20.092334 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:43:20.092222    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7618c478-85ac-40e1-b081-3aac9e28041d-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "7618c478-85ac-40e1-b081-3aac9e28041d" (UID: "7618c478-85ac-40e1-b081-3aac9e28041d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:43:20.092334 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:43:20.092240    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7618c478-85ac-40e1-b081-3aac9e28041d-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "7618c478-85ac-40e1-b081-3aac9e28041d" (UID: "7618c478-85ac-40e1-b081-3aac9e28041d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 16 19:43:20.092488 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:43:20.092466    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7618c478-85ac-40e1-b081-3aac9e28041d-istio-token" (OuterVolumeSpecName: "istio-token") pod "7618c478-85ac-40e1-b081-3aac9e28041d" (UID: "7618c478-85ac-40e1-b081-3aac9e28041d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:43:20.191158 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:43:20.191123    2578 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7618c478-85ac-40e1-b081-3aac9e28041d-istio-token\") on node \"ip-10-0-135-119.ec2.internal\" DevicePath \"\""
Sep 16 19:43:20.191158 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:43:20.191151    2578 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7618c478-85ac-40e1-b081-3aac9e28041d-istiod-ca-cert\") on node \"ip-10-0-135-119.ec2.internal\" DevicePath \"\""
Sep 16 19:43:20.191158 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:43:20.191160    2578 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7618c478-85ac-40e1-b081-3aac9e28041d-istio-podinfo\") on node \"ip-10-0-135-119.ec2.internal\" DevicePath \"\""
Sep 16 19:43:20.191158 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:43:20.191168    2578 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7618c478-85ac-40e1-b081-3aac9e28041d-istio-envoy\") on node \"ip-10-0-135-119.ec2.internal\" DevicePath \"\""
Sep 16 19:43:20.191415 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:43:20.191177    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fpqj5\" (UniqueName: \"kubernetes.io/projected/7618c478-85ac-40e1-b081-3aac9e28041d-kube-api-access-fpqj5\") on node \"ip-10-0-135-119.ec2.internal\" DevicePath \"\""
Sep 16 19:43:20.560111 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:43:20.560072    2578 generic.go:356] "Generic (PLEG): container finished" podID="7618c478-85ac-40e1-b081-3aac9e28041d" containerID="b938505211a42d5f04fdc06362d565163464382e5a2b13924c64d87b99b58e11" exitCode=0
Sep 16 19:43:20.560308 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:43:20.560122    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-785d98b7d8-mkwgn"
Sep 16 19:43:20.560308 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:43:20.560156    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-785d98b7d8-mkwgn" event={"ID":"7618c478-85ac-40e1-b081-3aac9e28041d","Type":"ContainerDied","Data":"b938505211a42d5f04fdc06362d565163464382e5a2b13924c64d87b99b58e11"}
Sep 16 19:43:20.560308 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:43:20.560212    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-785d98b7d8-mkwgn" event={"ID":"7618c478-85ac-40e1-b081-3aac9e28041d","Type":"ContainerDied","Data":"b196bbaefd58a82436d08d8b6a0f8e9fee04a0ab57d5098e642a23ad3a2e542e"}
Sep 16 19:43:20.560308 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:43:20.560242    2578 scope.go:117] "RemoveContainer" containerID="b938505211a42d5f04fdc06362d565163464382e5a2b13924c64d87b99b58e11"
Sep 16 19:43:20.569234 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:43:20.569212    2578 scope.go:117] "RemoveContainer" containerID="b938505211a42d5f04fdc06362d565163464382e5a2b13924c64d87b99b58e11"
Sep 16 19:43:20.569484 ip-10-0-135-119 kubenswrapper[2578]: E0916 19:43:20.569463    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b938505211a42d5f04fdc06362d565163464382e5a2b13924c64d87b99b58e11\": container with ID starting with b938505211a42d5f04fdc06362d565163464382e5a2b13924c64d87b99b58e11 not found: ID does not exist" containerID="b938505211a42d5f04fdc06362d565163464382e5a2b13924c64d87b99b58e11"
Sep 16 19:43:20.569545 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:43:20.569496    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b938505211a42d5f04fdc06362d565163464382e5a2b13924c64d87b99b58e11"} err="failed to get container status \"b938505211a42d5f04fdc06362d565163464382e5a2b13924c64d87b99b58e11\": rpc error: code = NotFound desc = could not find container \"b938505211a42d5f04fdc06362d565163464382e5a2b13924c64d87b99b58e11\": container with ID starting with b938505211a42d5f04fdc06362d565163464382e5a2b13924c64d87b99b58e11 not found: ID does not exist"
Sep 16 19:43:20.583533 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:43:20.583507    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-785d98b7d8-mkwgn"]
Sep 16 19:43:20.590545 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:43:20.590523    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-785d98b7d8-mkwgn"]
Sep 16 19:43:21.434882 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:43:21.434846    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7618c478-85ac-40e1-b081-3aac9e28041d" path="/var/lib/kubelet/pods/7618c478-85ac-40e1-b081-3aac9e28041d/volumes"
Sep 16 19:43:35.929694 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:43:35.929660    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:43:37.995273 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:43:37.995243    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:43:43.800207 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:43:43.800171    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:43:51.594159 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:43:51.594117    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:43:51.799082 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:43:51.799043    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schej4gsm"]
Sep 16 19:43:51.799478 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:43:51.799460    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7618c478-85ac-40e1-b081-3aac9e28041d" containerName="istio-proxy"
Sep 16 19:43:51.799576 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:43:51.799482    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="7618c478-85ac-40e1-b081-3aac9e28041d" containerName="istio-proxy"
Sep 16 19:43:51.799631 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:43:51.799597    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="7618c478-85ac-40e1-b081-3aac9e28041d" containerName="istio-proxy"
Sep 16 19:43:51.802701 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:43:51.802669    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schej4gsm"
Sep 16 19:43:51.805543 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:43:51.805520    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"kube-root-ca.crt\""
Sep 16 19:43:51.805656 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:43:51.805521    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"openshift-service-ca.crt\""
Sep 16 19:43:51.805656 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:43:51.805606    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"llmisvc-model-fb-opt-125m-route-50bc673d-epp-sa-dockercfg-724ht\""
Sep 16 19:43:51.805767 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:43:51.805675    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"llmisv44d181485fad85e662eb092f3749502f-kserve-self-signed-certs\""
Sep 16 19:43:51.805767 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:43:51.805739    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"default-dockercfg-mknvd\""
Sep 16 19:43:51.813668 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:43:51.813646    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schej4gsm"]
Sep 16 19:43:51.849323 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:43:51.849216    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d4356bd9-3eff-440e-83b7-8f23dcc381bb-tls-certs\") pod \"llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schej4gsm\" (UID: \"d4356bd9-3eff-440e-83b7-8f23dcc381bb\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schej4gsm"
Sep 16 19:43:51.849323 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:43:51.849259    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tn7tc\" (UniqueName: \"kubernetes.io/projected/d4356bd9-3eff-440e-83b7-8f23dcc381bb-kube-api-access-tn7tc\") pod \"llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schej4gsm\" (UID: \"d4356bd9-3eff-440e-83b7-8f23dcc381bb\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schej4gsm"
Sep 16 19:43:51.853365 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:43:51.853334    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:43:51.950207 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:43:51.950178    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d4356bd9-3eff-440e-83b7-8f23dcc381bb-tls-certs\") pod \"llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schej4gsm\" (UID: \"d4356bd9-3eff-440e-83b7-8f23dcc381bb\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schej4gsm"
Sep 16 19:43:51.950346 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:43:51.950217    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tn7tc\" (UniqueName: \"kubernetes.io/projected/d4356bd9-3eff-440e-83b7-8f23dcc381bb-kube-api-access-tn7tc\") pod \"llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schej4gsm\" (UID: \"d4356bd9-3eff-440e-83b7-8f23dcc381bb\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schej4gsm"
Sep 16 19:43:51.953052 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:43:51.953025    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d4356bd9-3eff-440e-83b7-8f23dcc381bb-tls-certs\") pod \"llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schej4gsm\" (UID: \"d4356bd9-3eff-440e-83b7-8f23dcc381bb\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schej4gsm"
Sep 16 19:43:51.959726 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:43:51.959702    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tn7tc\" (UniqueName: \"kubernetes.io/projected/d4356bd9-3eff-440e-83b7-8f23dcc381bb-kube-api-access-tn7tc\") pod \"llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schej4gsm\" (UID: \"d4356bd9-3eff-440e-83b7-8f23dcc381bb\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schej4gsm"
Sep 16 19:43:52.113949 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:43:52.113867    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schej4gsm"
Sep 16 19:43:52.388461 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:43:52.388428    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schej4gsm"]
Sep 16 19:43:52.390112 ip-10-0-135-119 kubenswrapper[2578]: W0916 19:43:52.390022    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd4356bd9_3eff_440e_83b7_8f23dcc381bb.slice/crio-716a398efd7e25ccb45ab1a98cb81ef580e4ecd6952c05f6120a0e9341fdc2b0 WatchSource:0}: Error finding container 716a398efd7e25ccb45ab1a98cb81ef580e4ecd6952c05f6120a0e9341fdc2b0: Status 404 returned error can't find the container with id 716a398efd7e25ccb45ab1a98cb81ef580e4ecd6952c05f6120a0e9341fdc2b0
Sep 16 19:43:52.430739 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:43:52.430711    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:43:52.708305 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:43:52.708222    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schej4gsm" event={"ID":"d4356bd9-3eff-440e-83b7-8f23dcc381bb","Type":"ContainerStarted","Data":"16f73ef02ef52d7dca4abedaa2a8e51429c90b978579d4166642ebe2e243cf11"}
Sep 16 19:43:52.708305 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:43:52.708271    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schej4gsm" event={"ID":"d4356bd9-3eff-440e-83b7-8f23dcc381bb","Type":"ContainerStarted","Data":"716a398efd7e25ccb45ab1a98cb81ef580e4ecd6952c05f6120a0e9341fdc2b0"}
Sep 16 19:43:52.708305 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:43:52.708300    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schej4gsm"
Sep 16 19:43:52.730027 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:43:52.729965    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schej4gsm" podStartSLOduration=1.729947933 podStartE2EDuration="1.729947933s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:43:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:43:52.728528703 +0000 UTC m=+3813.795020209" watchObservedRunningTime="2026-09-16 19:43:52.729947933 +0000 UTC m=+3813.796439421"
Sep 16 19:43:58.621927 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:43:58.621891    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:43:58.686068 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:43:58.686037    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:44:23.716400 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:44:23.716357    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schej4gsm"
Sep 16 19:44:31.071879 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:44:31.071837    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:45:02.104297 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:45:02.104203    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:45:19.708891 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:45:19.708860    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-tmrkx_4c2078bb-281a-4aa1-8e38-238897865434/console-operator/2.log"
Sep 16 19:45:19.717686 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:45:19.717662    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-k6gvs_6012728d-aee6-4f93-ab6e-fad616ef36e8/ovn-acl-logging/0.log"
Sep 16 19:45:19.723714 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:45:19.723692    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-tmrkx_4c2078bb-281a-4aa1-8e38-238897865434/console-operator/2.log"
Sep 16 19:45:19.730988 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:45:19.730951    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-k6gvs_6012728d-aee6-4f93-ab6e-fad616ef36e8/ovn-acl-logging/0.log"
Sep 16 19:45:22.355715 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:45:22.355666    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:45:22.456438 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:45:22.456405    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:45:22.555206 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:45:22.555170    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:45:22.670060 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:45:22.670014    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:45:22.771268 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:45:22.771228    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:45:24.892924 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:45:24.892878    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:45:24.993891 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:45:24.993853    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:45:25.753373 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:45:25.753320    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:45:27.759500 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:45:27.759457    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:45:27.840865 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:45:27.840819    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:45:29.437258 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:45:29.437228    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:45:34.935347 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:45:34.935309    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:45:35.035221 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:45:35.035181    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:45:35.523355 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:45:35.523318    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:45:35.592124 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:45:35.592083    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:45:58.670351 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:45:58.670313    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-pq2jb"]
Sep 16 19:45:58.673766 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:45:58.673739    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-pq2jb"
Sep 16 19:45:58.676221 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:45:58.676197    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-with-lora-adapters-a290d4b0\"/\"lora-pvc-adapters-test-kserve-self-signed-certs\""
Sep 16 19:45:58.676360 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:45:58.676243    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-with-lora-adapters-a290d4b0\"/\"openshift-service-ca.crt\""
Sep 16 19:45:58.676360 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:45:58.676351    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-with-lora-adapters-a290d4b0\"/\"default-dockercfg-cp97w\""
Sep 16 19:45:58.677105 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:45:58.677088    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-with-lora-adapters-a290d4b0\"/\"kube-root-ca.crt\""
Sep 16 19:45:58.703026 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:45:58.702991    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-pq2jb"]
Sep 16 19:45:58.731826 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:45:58.731785    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:45:58.782365 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:45:58.782303    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6641132a-39f9-40cc-9056-550afe8a6dad-tmp-dir\") pod \"lora-pvc-adapters-test-kserve-8cfd5947f-pq2jb\" (UID: \"6641132a-39f9-40cc-9056-550afe8a6dad\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-pq2jb"
Sep 16 19:45:58.782365 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:45:58.782373    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6641132a-39f9-40cc-9056-550afe8a6dad-model-cache\") pod \"lora-pvc-adapters-test-kserve-8cfd5947f-pq2jb\" (UID: \"6641132a-39f9-40cc-9056-550afe8a6dad\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-pq2jb"
Sep 16 19:45:58.782605 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:45:58.782428    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6641132a-39f9-40cc-9056-550afe8a6dad-tls-certs\") pod \"lora-pvc-adapters-test-kserve-8cfd5947f-pq2jb\" (UID: \"6641132a-39f9-40cc-9056-550afe8a6dad\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-pq2jb"
Sep 16 19:45:58.782645 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:45:58.782600    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gpr5w\" (UniqueName: \"kubernetes.io/projected/6641132a-39f9-40cc-9056-550afe8a6dad-kube-api-access-gpr5w\") pod \"lora-pvc-adapters-test-kserve-8cfd5947f-pq2jb\" (UID: \"6641132a-39f9-40cc-9056-550afe8a6dad\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-pq2jb"
Sep 16 19:45:58.782722 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:45:58.782701    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6641132a-39f9-40cc-9056-550afe8a6dad-dshm\") pod \"lora-pvc-adapters-test-kserve-8cfd5947f-pq2jb\" (UID: \"6641132a-39f9-40cc-9056-550afe8a6dad\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-pq2jb"
Sep 16 19:45:58.782810 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:45:58.782796    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6641132a-39f9-40cc-9056-550afe8a6dad-home\") pod \"lora-pvc-adapters-test-kserve-8cfd5947f-pq2jb\" (UID: \"6641132a-39f9-40cc-9056-550afe8a6dad\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-pq2jb"
Sep 16 19:45:58.884251 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:45:58.884202    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gpr5w\" (UniqueName: \"kubernetes.io/projected/6641132a-39f9-40cc-9056-550afe8a6dad-kube-api-access-gpr5w\") pod \"lora-pvc-adapters-test-kserve-8cfd5947f-pq2jb\" (UID: \"6641132a-39f9-40cc-9056-550afe8a6dad\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-pq2jb"
Sep 16 19:45:58.884450 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:45:58.884298    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6641132a-39f9-40cc-9056-550afe8a6dad-dshm\") pod \"lora-pvc-adapters-test-kserve-8cfd5947f-pq2jb\" (UID: \"6641132a-39f9-40cc-9056-550afe8a6dad\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-pq2jb"
Sep 16 19:45:58.884450 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:45:58.884382    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6641132a-39f9-40cc-9056-550afe8a6dad-home\") pod \"lora-pvc-adapters-test-kserve-8cfd5947f-pq2jb\" (UID: \"6641132a-39f9-40cc-9056-550afe8a6dad\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-pq2jb"
Sep 16 19:45:58.884450 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:45:58.884412    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6641132a-39f9-40cc-9056-550afe8a6dad-tmp-dir\") pod \"lora-pvc-adapters-test-kserve-8cfd5947f-pq2jb\" (UID: \"6641132a-39f9-40cc-9056-550afe8a6dad\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-pq2jb"
Sep 16 19:45:58.884450 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:45:58.884438    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6641132a-39f9-40cc-9056-550afe8a6dad-model-cache\") pod \"lora-pvc-adapters-test-kserve-8cfd5947f-pq2jb\" (UID: \"6641132a-39f9-40cc-9056-550afe8a6dad\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-pq2jb"
Sep 16 19:45:58.884679 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:45:58.884598    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6641132a-39f9-40cc-9056-550afe8a6dad-tls-certs\") pod \"lora-pvc-adapters-test-kserve-8cfd5947f-pq2jb\" (UID: \"6641132a-39f9-40cc-9056-550afe8a6dad\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-pq2jb"
Sep 16 19:45:58.884865 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:45:58.884842    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6641132a-39f9-40cc-9056-550afe8a6dad-home\") pod \"lora-pvc-adapters-test-kserve-8cfd5947f-pq2jb\" (UID: \"6641132a-39f9-40cc-9056-550afe8a6dad\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-pq2jb"
Sep 16 19:45:58.884953 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:45:58.884889    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6641132a-39f9-40cc-9056-550afe8a6dad-model-cache\") pod \"lora-pvc-adapters-test-kserve-8cfd5947f-pq2jb\" (UID: \"6641132a-39f9-40cc-9056-550afe8a6dad\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-pq2jb"
Sep 16 19:45:58.885032 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:45:58.885013    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6641132a-39f9-40cc-9056-550afe8a6dad-tmp-dir\") pod \"lora-pvc-adapters-test-kserve-8cfd5947f-pq2jb\" (UID: \"6641132a-39f9-40cc-9056-550afe8a6dad\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-pq2jb"
Sep 16 19:45:58.886931 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:45:58.886900    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6641132a-39f9-40cc-9056-550afe8a6dad-dshm\") pod \"lora-pvc-adapters-test-kserve-8cfd5947f-pq2jb\" (UID: \"6641132a-39f9-40cc-9056-550afe8a6dad\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-pq2jb"
Sep 16 19:45:58.887424 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:45:58.887404    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6641132a-39f9-40cc-9056-550afe8a6dad-tls-certs\") pod \"lora-pvc-adapters-test-kserve-8cfd5947f-pq2jb\" (UID: \"6641132a-39f9-40cc-9056-550afe8a6dad\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-pq2jb"
Sep 16 19:45:58.897207 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:45:58.897184    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gpr5w\" (UniqueName: \"kubernetes.io/projected/6641132a-39f9-40cc-9056-550afe8a6dad-kube-api-access-gpr5w\") pod \"lora-pvc-adapters-test-kserve-8cfd5947f-pq2jb\" (UID: \"6641132a-39f9-40cc-9056-550afe8a6dad\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-pq2jb"
Sep 16 19:46:01.106221 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:46:01.106180    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-86b39b6d-ce90-4025-823c-c0244a45ed72\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-09b88cffeeadd7a3d\") pod \"lora-pvc-adapters-test-kserve-8cfd5947f-pq2jb\" (UID: \"6641132a-39f9-40cc-9056-550afe8a6dad\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-pq2jb"
Sep 16 19:46:01.113457 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:46:01.113413    2578 operation_generator.go:1470] "Controller attach succeeded for volume \"pvc-86b39b6d-ce90-4025-823c-c0244a45ed72\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-09b88cffeeadd7a3d\") pod \"lora-pvc-adapters-test-kserve-8cfd5947f-pq2jb\" (UID: \"6641132a-39f9-40cc-9056-550afe8a6dad\") device path: \"\"" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-pq2jb"
Sep 16 19:46:01.207329 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:46:01.207285    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"pvc-86b39b6d-ce90-4025-823c-c0244a45ed72\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-09b88cffeeadd7a3d\") pod \"lora-pvc-adapters-test-kserve-8cfd5947f-pq2jb\" (UID: \"6641132a-39f9-40cc-9056-550afe8a6dad\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-pq2jb"
Sep 16 19:46:01.207516 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:46:01.207438    2578 operation_generator.go:515] "MountVolume.WaitForAttach entering for volume \"pvc-86b39b6d-ce90-4025-823c-c0244a45ed72\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-09b88cffeeadd7a3d\") pod \"lora-pvc-adapters-test-kserve-8cfd5947f-pq2jb\" (UID: \"6641132a-39f9-40cc-9056-550afe8a6dad\") DevicePath \"\"" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-pq2jb"
Sep 16 19:46:01.210619 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:46:01.210593    2578 operation_generator.go:525] "MountVolume.WaitForAttach succeeded for volume \"pvc-86b39b6d-ce90-4025-823c-c0244a45ed72\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-09b88cffeeadd7a3d\") pod \"lora-pvc-adapters-test-kserve-8cfd5947f-pq2jb\" (UID: \"6641132a-39f9-40cc-9056-550afe8a6dad\") DevicePath \"csi-e91f30bbc78f1217c50687ec8c32318377eaf05aaf90686263f4c4fcfea9926f\"" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-pq2jb"
Sep 16 19:46:01.305463 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:46:01.305423    2578 operation_generator.go:557] "MountVolume.MountDevice succeeded for volume \"pvc-86b39b6d-ce90-4025-823c-c0244a45ed72\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-09b88cffeeadd7a3d\") pod \"lora-pvc-adapters-test-kserve-8cfd5947f-pq2jb\" (UID: \"6641132a-39f9-40cc-9056-550afe8a6dad\") device mount path \"/var/lib/kubelet/plugins/kubernetes.io/csi/ebs.csi.aws.com/6c63b96b19d1080af6156134d13ea2983e81b0ca8abb7c134acde6f58b7b04b2/globalmount\"" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-pq2jb"
Sep 16 19:46:01.325507 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:46:01.325475    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"pvc-86b39b6d-ce90-4025-823c-c0244a45ed72\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-09b88cffeeadd7a3d\") pod \"lora-pvc-adapters-test-kserve-8cfd5947f-pq2jb\" (UID: \"6641132a-39f9-40cc-9056-550afe8a6dad\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-pq2jb"
Sep 16 19:46:01.385060 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:46:01.385022    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-pq2jb"
Sep 16 19:46:01.673845 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:46:01.673808    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-pq2jb"]
Sep 16 19:46:01.676930 ip-10-0-135-119 kubenswrapper[2578]: W0916 19:46:01.676896    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6641132a_39f9_40cc_9056_550afe8a6dad.slice/crio-30e879e71fc9bf6fee44a3c19c12d8cf6654b30a18a0009d704ca5f0d6e44cc2 WatchSource:0}: Error finding container 30e879e71fc9bf6fee44a3c19c12d8cf6654b30a18a0009d704ca5f0d6e44cc2: Status 404 returned error can't find the container with id 30e879e71fc9bf6fee44a3c19c12d8cf6654b30a18a0009d704ca5f0d6e44cc2
Sep 16 19:46:02.322600 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:46:02.322558    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-pq2jb" event={"ID":"6641132a-39f9-40cc-9056-550afe8a6dad","Type":"ContainerStarted","Data":"dca39ce8c751d2702770e6ef7de3c2858550022890ddd4af062cbfd1da762625"}
Sep 16 19:46:02.323070 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:46:02.322630    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-pq2jb" event={"ID":"6641132a-39f9-40cc-9056-550afe8a6dad","Type":"ContainerStarted","Data":"30e879e71fc9bf6fee44a3c19c12d8cf6654b30a18a0009d704ca5f0d6e44cc2"}
Sep 16 19:46:02.346575 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:46:02.346497    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-pq2jb" podStartSLOduration=4.346471596 podStartE2EDuration="4.346471596s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 19:45:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:46:02.344063063 +0000 UTC m=+3943.410554571" watchObservedRunningTime="2026-09-16 19:46:02.346471596 +0000 UTC m=+3943.412963085"
Sep 16 19:46:02.386090 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:46:02.386050    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-pq2jb"
Sep 16 19:46:11.385528 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:46:11.385486    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-pq2jb"
Sep 16 19:46:11.387293 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:46:11.387243    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-pq2jb" podUID="6641132a-39f9-40cc-9056-550afe8a6dad" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/health\": dial tcp 10.132.0.71:8000: connect: connection refused"
Sep 16 19:46:11.449583 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:46:11.449542    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:46:11.481664 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:46:11.481626    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schej4gsm"]
Sep 16 19:46:11.531590 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:46:11.531556    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:46:13.715574 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:46:13.715520    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schej4gsm" podUID="d4356bd9-3eff-440e-83b7-8f23dcc381bb" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:46:21.385584 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:46:21.385534    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-pq2jb" podUID="6641132a-39f9-40cc-9056-550afe8a6dad" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/health\": dial tcp 10.132.0.71:8000: connect: connection refused"
Sep 16 19:46:23.716220 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:46:23.716177    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schej4gsm" podUID="d4356bd9-3eff-440e-83b7-8f23dcc381bb" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:46:26.509555 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:46:26.509501    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schej4gsm" podUID="d4356bd9-3eff-440e-83b7-8f23dcc381bb" containerName="main" containerID="cri-o://16f73ef02ef52d7dca4abedaa2a8e51429c90b978579d4166642ebe2e243cf11" gracePeriod=45
Sep 16 19:46:31.385666 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:46:31.385622    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-pq2jb" podUID="6641132a-39f9-40cc-9056-550afe8a6dad" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/health\": dial tcp 10.132.0.71:8000: connect: connection refused"
Sep 16 19:46:33.716469 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:46:33.716429    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schej4gsm" podUID="d4356bd9-3eff-440e-83b7-8f23dcc381bb" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:46:33.716895 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:46:33.716582    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schej4gsm"
Sep 16 19:46:41.385920 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:46:41.385870    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-pq2jb" podUID="6641132a-39f9-40cc-9056-550afe8a6dad" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/health\": dial tcp 10.132.0.71:8000: connect: connection refused"
Sep 16 19:46:43.716313 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:46:43.716272    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schej4gsm" podUID="d4356bd9-3eff-440e-83b7-8f23dcc381bb" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:46:51.385847 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:46:51.385797    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-pq2jb" podUID="6641132a-39f9-40cc-9056-550afe8a6dad" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/health\": dial tcp 10.132.0.71:8000: connect: connection refused"
Sep 16 19:46:53.716272 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:46:53.716231    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schej4gsm" podUID="d4356bd9-3eff-440e-83b7-8f23dcc381bb" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:46:56.776022 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:46:56.775933    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schej4gsm"
Sep 16 19:46:56.845716 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:46:56.845670    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d4356bd9-3eff-440e-83b7-8f23dcc381bb-tls-certs\") pod \"d4356bd9-3eff-440e-83b7-8f23dcc381bb\" (UID: \"d4356bd9-3eff-440e-83b7-8f23dcc381bb\") "
Sep 16 19:46:56.845938 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:46:56.845734    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tn7tc\" (UniqueName: \"kubernetes.io/projected/d4356bd9-3eff-440e-83b7-8f23dcc381bb-kube-api-access-tn7tc\") pod \"d4356bd9-3eff-440e-83b7-8f23dcc381bb\" (UID: \"d4356bd9-3eff-440e-83b7-8f23dcc381bb\") "
Sep 16 19:46:56.848284 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:46:56.848246    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d4356bd9-3eff-440e-83b7-8f23dcc381bb-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "d4356bd9-3eff-440e-83b7-8f23dcc381bb" (UID: "d4356bd9-3eff-440e-83b7-8f23dcc381bb"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:46:56.848401 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:46:56.848319    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d4356bd9-3eff-440e-83b7-8f23dcc381bb-kube-api-access-tn7tc" (OuterVolumeSpecName: "kube-api-access-tn7tc") pod "d4356bd9-3eff-440e-83b7-8f23dcc381bb" (UID: "d4356bd9-3eff-440e-83b7-8f23dcc381bb"). InnerVolumeSpecName "kube-api-access-tn7tc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:46:56.947197 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:46:56.947158    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tn7tc\" (UniqueName: \"kubernetes.io/projected/d4356bd9-3eff-440e-83b7-8f23dcc381bb-kube-api-access-tn7tc\") on node \"ip-10-0-135-119.ec2.internal\" DevicePath \"\""
Sep 16 19:46:56.947197 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:46:56.947190    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d4356bd9-3eff-440e-83b7-8f23dcc381bb-tls-certs\") on node \"ip-10-0-135-119.ec2.internal\" DevicePath \"\""
Sep 16 19:46:57.599060 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:46:57.599023    2578 generic.go:356] "Generic (PLEG): container finished" podID="d4356bd9-3eff-440e-83b7-8f23dcc381bb" containerID="16f73ef02ef52d7dca4abedaa2a8e51429c90b978579d4166642ebe2e243cf11" exitCode=0
Sep 16 19:46:57.599252 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:46:57.599079    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schej4gsm"
Sep 16 19:46:57.599252 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:46:57.599120    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schej4gsm" event={"ID":"d4356bd9-3eff-440e-83b7-8f23dcc381bb","Type":"ContainerDied","Data":"16f73ef02ef52d7dca4abedaa2a8e51429c90b978579d4166642ebe2e243cf11"}
Sep 16 19:46:57.599252 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:46:57.599197    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schej4gsm" event={"ID":"d4356bd9-3eff-440e-83b7-8f23dcc381bb","Type":"ContainerDied","Data":"716a398efd7e25ccb45ab1a98cb81ef580e4ecd6952c05f6120a0e9341fdc2b0"}
Sep 16 19:46:57.599252 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:46:57.599226    2578 scope.go:117] "RemoveContainer" containerID="16f73ef02ef52d7dca4abedaa2a8e51429c90b978579d4166642ebe2e243cf11"
Sep 16 19:46:57.609434 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:46:57.609409    2578 scope.go:117] "RemoveContainer" containerID="16f73ef02ef52d7dca4abedaa2a8e51429c90b978579d4166642ebe2e243cf11"
Sep 16 19:46:57.609739 ip-10-0-135-119 kubenswrapper[2578]: E0916 19:46:57.609716    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"16f73ef02ef52d7dca4abedaa2a8e51429c90b978579d4166642ebe2e243cf11\": container with ID starting with 16f73ef02ef52d7dca4abedaa2a8e51429c90b978579d4166642ebe2e243cf11 not found: ID does not exist" containerID="16f73ef02ef52d7dca4abedaa2a8e51429c90b978579d4166642ebe2e243cf11"
Sep 16 19:46:57.609825 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:46:57.609747    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"16f73ef02ef52d7dca4abedaa2a8e51429c90b978579d4166642ebe2e243cf11"} err="failed to get container status \"16f73ef02ef52d7dca4abedaa2a8e51429c90b978579d4166642ebe2e243cf11\": rpc error: code = NotFound desc = could not find container \"16f73ef02ef52d7dca4abedaa2a8e51429c90b978579d4166642ebe2e243cf11\": container with ID starting with 16f73ef02ef52d7dca4abedaa2a8e51429c90b978579d4166642ebe2e243cf11 not found: ID does not exist"
Sep 16 19:46:57.618079 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:46:57.618052    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schej4gsm"]
Sep 16 19:46:57.625215 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:46:57.625191    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schej4gsm"]
Sep 16 19:46:59.436346 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:46:59.436300    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d4356bd9-3eff-440e-83b7-8f23dcc381bb" path="/var/lib/kubelet/pods/d4356bd9-3eff-440e-83b7-8f23dcc381bb/volumes"
Sep 16 19:47:01.114485 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:47:01.114446    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:47:01.214260 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:47:01.214223    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:47:01.385838 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:47:01.385790    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-pq2jb" podUID="6641132a-39f9-40cc-9056-550afe8a6dad" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/health\": dial tcp 10.132.0.71:8000: connect: connection refused"
Sep 16 19:47:05.127194 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:47:05.127147    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:47:05.354783 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:47:05.354731    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:47:09.064630 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:47:09.064591    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:47:11.386418 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:47:11.386379    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-pq2jb" podUID="6641132a-39f9-40cc-9056-550afe8a6dad" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/health\": dial tcp 10.132.0.71:8000: connect: connection refused"
Sep 16 19:47:15.110274 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:47:15.110229    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:47:15.183373 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:47:15.183327    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:47:21.148570 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:47:21.148530    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:47:21.247592 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:47:21.247553    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:47:21.395321 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:47:21.395284    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-pq2jb"
Sep 16 19:47:21.403113 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:47:21.403028    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-pq2jb"
Sep 16 19:47:23.428280 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:47:23.428243    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:47:53.467865 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:47:53.467770    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-with-lora-adapters-a290d4b0_lora-pvc-adapters-test-kserve-8cfd5947f-pq2jb_6641132a-39f9-40cc-9056-550afe8a6dad/main/0.log"
Sep 16 19:47:54.823564 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:47:54.823529    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-pq2jb"]
Sep 16 19:47:54.851368 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:47:54.851336    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:48:09.846524 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:48:09.846460    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-pq2jb" podUID="6641132a-39f9-40cc-9056-550afe8a6dad" containerName="main" containerID="cri-o://dca39ce8c751d2702770e6ef7de3c2858550022890ddd4af062cbfd1da762625" gracePeriod=45
Sep 16 19:48:10.396691 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:48:10.396652    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-pq2jb" podUID="6641132a-39f9-40cc-9056-550afe8a6dad" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/health\": dial tcp 10.132.0.71:8000: connect: connection refused"
Sep 16 19:48:10.506417 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:48:10.506380    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-pq2jb"
Sep 16 19:48:10.542790 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:48:10.542755    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gpr5w\" (UniqueName: \"kubernetes.io/projected/6641132a-39f9-40cc-9056-550afe8a6dad-kube-api-access-gpr5w\") pod \"6641132a-39f9-40cc-9056-550afe8a6dad\" (UID: \"6641132a-39f9-40cc-9056-550afe8a6dad\") "
Sep 16 19:48:10.543038 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:48:10.542805    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6641132a-39f9-40cc-9056-550afe8a6dad-tls-certs\") pod \"6641132a-39f9-40cc-9056-550afe8a6dad\" (UID: \"6641132a-39f9-40cc-9056-550afe8a6dad\") "
Sep 16 19:48:10.543038 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:48:10.542845    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6641132a-39f9-40cc-9056-550afe8a6dad-tmp-dir\") pod \"6641132a-39f9-40cc-9056-550afe8a6dad\" (UID: \"6641132a-39f9-40cc-9056-550afe8a6dad\") "
Sep 16 19:48:10.543182 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:48:10.543037    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6641132a-39f9-40cc-9056-550afe8a6dad-home\") pod \"6641132a-39f9-40cc-9056-550afe8a6dad\" (UID: \"6641132a-39f9-40cc-9056-550afe8a6dad\") "
Sep 16 19:48:10.543182 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:48:10.543126    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6641132a-39f9-40cc-9056-550afe8a6dad-model-cache\") pod \"6641132a-39f9-40cc-9056-550afe8a6dad\" (UID: \"6641132a-39f9-40cc-9056-550afe8a6dad\") "
Sep 16 19:48:10.543318 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:48:10.543269    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-pvc-source\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-09b88cffeeadd7a3d\") pod \"6641132a-39f9-40cc-9056-550afe8a6dad\" (UID: \"6641132a-39f9-40cc-9056-550afe8a6dad\") "
Sep 16 19:48:10.543375 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:48:10.543318    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6641132a-39f9-40cc-9056-550afe8a6dad-dshm\") pod \"6641132a-39f9-40cc-9056-550afe8a6dad\" (UID: \"6641132a-39f9-40cc-9056-550afe8a6dad\") "
Sep 16 19:48:10.543434 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:48:10.543374    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6641132a-39f9-40cc-9056-550afe8a6dad-home" (OuterVolumeSpecName: "home") pod "6641132a-39f9-40cc-9056-550afe8a6dad" (UID: "6641132a-39f9-40cc-9056-550afe8a6dad"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:48:10.543643 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:48:10.543618    2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6641132a-39f9-40cc-9056-550afe8a6dad-home\") on node \"ip-10-0-135-119.ec2.internal\" DevicePath \"\""
Sep 16 19:48:10.543643 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:48:10.543616    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6641132a-39f9-40cc-9056-550afe8a6dad-model-cache" (OuterVolumeSpecName: "model-cache") pod "6641132a-39f9-40cc-9056-550afe8a6dad" (UID: "6641132a-39f9-40cc-9056-550afe8a6dad"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:48:10.545606 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:48:10.545565    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6641132a-39f9-40cc-9056-550afe8a6dad-kube-api-access-gpr5w" (OuterVolumeSpecName: "kube-api-access-gpr5w") pod "6641132a-39f9-40cc-9056-550afe8a6dad" (UID: "6641132a-39f9-40cc-9056-550afe8a6dad"). InnerVolumeSpecName "kube-api-access-gpr5w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:48:10.545766 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:48:10.545736    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6641132a-39f9-40cc-9056-550afe8a6dad-dshm" (OuterVolumeSpecName: "dshm") pod "6641132a-39f9-40cc-9056-550afe8a6dad" (UID: "6641132a-39f9-40cc-9056-550afe8a6dad"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:48:10.545871 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:48:10.545783    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6641132a-39f9-40cc-9056-550afe8a6dad-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "6641132a-39f9-40cc-9056-550afe8a6dad" (UID: "6641132a-39f9-40cc-9056-550afe8a6dad"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:48:10.553473 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:48:10.553444    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/ebs.csi.aws.com^vol-09b88cffeeadd7a3d" (OuterVolumeSpecName: "kserve-pvc-source") pod "6641132a-39f9-40cc-9056-550afe8a6dad" (UID: "6641132a-39f9-40cc-9056-550afe8a6dad"). InnerVolumeSpecName "pvc-86b39b6d-ce90-4025-823c-c0244a45ed72". PluginName "kubernetes.io/csi", VolumeGIDValue ""
Sep 16 19:48:10.554498 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:48:10.554466    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6641132a-39f9-40cc-9056-550afe8a6dad-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "6641132a-39f9-40cc-9056-550afe8a6dad" (UID: "6641132a-39f9-40cc-9056-550afe8a6dad"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:48:10.644887 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:48:10.644846    2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6641132a-39f9-40cc-9056-550afe8a6dad-tmp-dir\") on node \"ip-10-0-135-119.ec2.internal\" DevicePath \"\""
Sep 16 19:48:10.644887 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:48:10.644886    2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6641132a-39f9-40cc-9056-550afe8a6dad-model-cache\") on node \"ip-10-0-135-119.ec2.internal\" DevicePath \"\""
Sep 16 19:48:10.645142 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:48:10.644918    2578 reconciler_common.go:292] "operationExecutor.UnmountDevice started for volume \"pvc-86b39b6d-ce90-4025-823c-c0244a45ed72\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-09b88cffeeadd7a3d\") on node \"ip-10-0-135-119.ec2.internal\" "
Sep 16 19:48:10.645142 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:48:10.644928    2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6641132a-39f9-40cc-9056-550afe8a6dad-dshm\") on node \"ip-10-0-135-119.ec2.internal\" DevicePath \"\""
Sep 16 19:48:10.645142 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:48:10.644937    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gpr5w\" (UniqueName: \"kubernetes.io/projected/6641132a-39f9-40cc-9056-550afe8a6dad-kube-api-access-gpr5w\") on node \"ip-10-0-135-119.ec2.internal\" DevicePath \"\""
Sep 16 19:48:10.645142 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:48:10.644945    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6641132a-39f9-40cc-9056-550afe8a6dad-tls-certs\") on node \"ip-10-0-135-119.ec2.internal\" DevicePath \"\""
Sep 16 19:48:10.672497 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:48:10.672475    2578 operation_generator.go:896] UnmountDevice succeeded for volume "pvc-86b39b6d-ce90-4025-823c-c0244a45ed72" (UniqueName: "kubernetes.io/csi/ebs.csi.aws.com^vol-09b88cffeeadd7a3d") on node "ip-10-0-135-119.ec2.internal"
Sep 16 19:48:10.746277 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:48:10.746252    2578 reconciler_common.go:299] "Volume detached for volume \"pvc-86b39b6d-ce90-4025-823c-c0244a45ed72\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-09b88cffeeadd7a3d\") on node \"ip-10-0-135-119.ec2.internal\" DevicePath \"csi-e91f30bbc78f1217c50687ec8c32318377eaf05aaf90686263f4c4fcfea9926f\""
Sep 16 19:48:10.974404 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:48:10.974307    2578 generic.go:356] "Generic (PLEG): container finished" podID="6641132a-39f9-40cc-9056-550afe8a6dad" containerID="dca39ce8c751d2702770e6ef7de3c2858550022890ddd4af062cbfd1da762625" exitCode=0
Sep 16 19:48:10.974404 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:48:10.974352    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-pq2jb"
Sep 16 19:48:10.974404 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:48:10.974394    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-pq2jb" event={"ID":"6641132a-39f9-40cc-9056-550afe8a6dad","Type":"ContainerDied","Data":"dca39ce8c751d2702770e6ef7de3c2858550022890ddd4af062cbfd1da762625"}
Sep 16 19:48:10.974936 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:48:10.974451    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-pq2jb" event={"ID":"6641132a-39f9-40cc-9056-550afe8a6dad","Type":"ContainerDied","Data":"30e879e71fc9bf6fee44a3c19c12d8cf6654b30a18a0009d704ca5f0d6e44cc2"}
Sep 16 19:48:10.974936 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:48:10.974476    2578 scope.go:117] "RemoveContainer" containerID="dca39ce8c751d2702770e6ef7de3c2858550022890ddd4af062cbfd1da762625"
Sep 16 19:48:10.984263 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:48:10.984239    2578 scope.go:117] "RemoveContainer" containerID="dca39ce8c751d2702770e6ef7de3c2858550022890ddd4af062cbfd1da762625"
Sep 16 19:48:10.984598 ip-10-0-135-119 kubenswrapper[2578]: E0916 19:48:10.984575    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dca39ce8c751d2702770e6ef7de3c2858550022890ddd4af062cbfd1da762625\": container with ID starting with dca39ce8c751d2702770e6ef7de3c2858550022890ddd4af062cbfd1da762625 not found: ID does not exist" containerID="dca39ce8c751d2702770e6ef7de3c2858550022890ddd4af062cbfd1da762625"
Sep 16 19:48:10.984654 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:48:10.984606    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dca39ce8c751d2702770e6ef7de3c2858550022890ddd4af062cbfd1da762625"} err="failed to get container status \"dca39ce8c751d2702770e6ef7de3c2858550022890ddd4af062cbfd1da762625\": rpc error: code = NotFound desc = could not find container \"dca39ce8c751d2702770e6ef7de3c2858550022890ddd4af062cbfd1da762625\": container with ID starting with dca39ce8c751d2702770e6ef7de3c2858550022890ddd4af062cbfd1da762625 not found: ID does not exist"
Sep 16 19:48:11.000831 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:48:11.000796    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-pq2jb"]
Sep 16 19:48:11.006080 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:48:11.006050    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-pq2jb"]
Sep 16 19:48:11.434441 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:48:11.434397    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6641132a-39f9-40cc-9056-550afe8a6dad" path="/var/lib/kubelet/pods/6641132a-39f9-40cc-9056-550afe8a6dad/volumes"
Sep 16 19:48:14.878832 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:48:14.878796    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:48:14.979107 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:48:14.979057    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:48:20.203828 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:48:20.203773    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:48:20.304111 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:48:20.304018    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:48:20.404183 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:48:20.404152    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:48:20.506426 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:48:20.506389    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:48:21.621940 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:48:21.621904    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:48:28.046722 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:48:28.046690    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:48:28.146374 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:48:28.146337    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:48:48.550727 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:48:48.550688    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:48:48.799619 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:48:48.799584    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-pwqwv"]
Sep 16 19:48:48.799959 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:48:48.799946    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d4356bd9-3eff-440e-83b7-8f23dcc381bb" containerName="main"
Sep 16 19:48:48.800030 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:48:48.799961    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="d4356bd9-3eff-440e-83b7-8f23dcc381bb" containerName="main"
Sep 16 19:48:48.800030 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:48:48.799969    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6641132a-39f9-40cc-9056-550afe8a6dad" containerName="main"
Sep 16 19:48:48.800030 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:48:48.799993    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="6641132a-39f9-40cc-9056-550afe8a6dad" containerName="main"
Sep 16 19:48:48.800133 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:48:48.800054    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="d4356bd9-3eff-440e-83b7-8f23dcc381bb" containerName="main"
Sep 16 19:48:48.800133 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:48:48.800063    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="6641132a-39f9-40cc-9056-550afe8a6dad" containerName="main"
Sep 16 19:48:48.802762 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:48:48.802697    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-pwqwv"
Sep 16 19:48:48.806175 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:48:48.806135    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-tls-resources-1d9e6e79\"/\"openshift-service-ca.crt\""
Sep 16 19:48:48.807712 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:48:48.807681    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-tls-resources-1d9e6e79\"/\"default-dockercfg-d6svp\""
Sep 16 19:48:48.808010 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:48:48.807717    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-tls-resources-1d9e6e79\"/\"tls-verification-test-kserve-self-signed-certs\""
Sep 16 19:48:48.808130 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:48:48.807829    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-tls-resources-1d9e6e79\"/\"kube-root-ca.crt\""
Sep 16 19:48:48.808195 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:48:48.807835    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-tls-resources-1d9e6e79\"/\"tls-verification-test-epp-sa-dockercfg-bwqrb\""
Sep 16 19:48:48.817827 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:48:48.817790    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-pwqwv"]
Sep 16 19:48:48.850536 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:48:48.850504    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:48:48.897442 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:48:48.897382    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/43379ca3-f36b-4bf0-8a5e-45f514086807-tls-certs\") pod \"tls-verification-test-kserve-router-scheduler-77cd9c86c9-pwqwv\" (UID: \"43379ca3-f36b-4bf0-8a5e-45f514086807\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-pwqwv"
Sep 16 19:48:48.897669 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:48:48.897474    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jn4lj\" (UniqueName: \"kubernetes.io/projected/43379ca3-f36b-4bf0-8a5e-45f514086807-kube-api-access-jn4lj\") pod \"tls-verification-test-kserve-router-scheduler-77cd9c86c9-pwqwv\" (UID: \"43379ca3-f36b-4bf0-8a5e-45f514086807\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-pwqwv"
Sep 16 19:48:48.998036 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:48:48.997988    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/43379ca3-f36b-4bf0-8a5e-45f514086807-tls-certs\") pod \"tls-verification-test-kserve-router-scheduler-77cd9c86c9-pwqwv\" (UID: \"43379ca3-f36b-4bf0-8a5e-45f514086807\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-pwqwv"
Sep 16 19:48:48.998247 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:48:48.998045    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jn4lj\" (UniqueName: \"kubernetes.io/projected/43379ca3-f36b-4bf0-8a5e-45f514086807-kube-api-access-jn4lj\") pod \"tls-verification-test-kserve-router-scheduler-77cd9c86c9-pwqwv\" (UID: \"43379ca3-f36b-4bf0-8a5e-45f514086807\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-pwqwv"
Sep 16 19:48:49.000909 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:48:49.000883    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/43379ca3-f36b-4bf0-8a5e-45f514086807-tls-certs\") pod \"tls-verification-test-kserve-router-scheduler-77cd9c86c9-pwqwv\" (UID: \"43379ca3-f36b-4bf0-8a5e-45f514086807\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-pwqwv"
Sep 16 19:48:49.008207 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:48:49.008177    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jn4lj\" (UniqueName: \"kubernetes.io/projected/43379ca3-f36b-4bf0-8a5e-45f514086807-kube-api-access-jn4lj\") pod \"tls-verification-test-kserve-router-scheduler-77cd9c86c9-pwqwv\" (UID: \"43379ca3-f36b-4bf0-8a5e-45f514086807\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-pwqwv"
Sep 16 19:48:49.115599 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:48:49.115483    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-pwqwv"
Sep 16 19:48:49.390010 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:48:49.389953    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-pwqwv"]
Sep 16 19:48:49.393859 ip-10-0-135-119 kubenswrapper[2578]: W0916 19:48:49.393829    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod43379ca3_f36b_4bf0_8a5e_45f514086807.slice/crio-b17aa58c34045a2a5594431bbbda9234b300d96eff3218221ef7b967286c396c WatchSource:0}: Error finding container b17aa58c34045a2a5594431bbbda9234b300d96eff3218221ef7b967286c396c: Status 404 returned error can't find the container with id b17aa58c34045a2a5594431bbbda9234b300d96eff3218221ef7b967286c396c
Sep 16 19:48:50.175129 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:48:50.175081    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-pwqwv" event={"ID":"43379ca3-f36b-4bf0-8a5e-45f514086807","Type":"ContainerStarted","Data":"f048676ed805a7f7262bcf4349d9c7a870f89d1556a2528c9d1f99db49e2cbfc"}
Sep 16 19:48:50.175647 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:48:50.175147    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-pwqwv" event={"ID":"43379ca3-f36b-4bf0-8a5e-45f514086807","Type":"ContainerStarted","Data":"b17aa58c34045a2a5594431bbbda9234b300d96eff3218221ef7b967286c396c"}
Sep 16 19:48:50.175647 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:48:50.175222    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-pwqwv"
Sep 16 19:48:50.194687 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:48:50.194625    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-pwqwv" podStartSLOduration=2.194597103 podStartE2EDuration="2.194597103s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:48:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:48:50.192501114 +0000 UTC m=+4111.258992602" watchObservedRunningTime="2026-09-16 19:48:50.194597103 +0000 UTC m=+4111.261088592"
Sep 16 19:48:50.861834 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:48:50.861796    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:49:04.200477 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:49:04.200439    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:49:15.401038 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:49:15.400966    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:49:15.484644 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:49:15.484608    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:49:15.553043 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:49:15.552965    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:49:21.181473 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:49:21.181436    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-pwqwv"
Sep 16 19:49:43.714499 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:49:43.714432    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:50:05.304302 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:50:05.304262    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:50:05.407077 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:50:05.407034    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:50:05.653276 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:50:05.653231    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:50:05.734205 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:50:05.734166    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:50:09.126993 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:50:09.126944    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:50:12.398234 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:50:12.398197    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:50:14.674789 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:50:14.674745    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:50:14.795574 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:50:14.795529    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b45pf2wc"]
Sep 16 19:50:14.799419 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:50:14.799393    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b45pf2wc"
Sep 16 19:50:14.802673 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:50:14.802623    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"default-dockercfg-f7z8n\""
Sep 16 19:50:14.802840 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:50:14.802781    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"router-with-refs-pd-test-kserve-self-signed-certs\""
Sep 16 19:50:14.802840 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:50:14.802785    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"openshift-service-ca.crt\""
Sep 16 19:50:14.803002 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:50:14.802847    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"router-with-refs-pd-test-epp-sa-dockercfg-lnhm5\""
Sep 16 19:50:14.803002 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:50:14.802921    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"kube-root-ca.crt\""
Sep 16 19:50:14.810147 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:50:14.810115    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b45pf2wc"]
Sep 16 19:50:14.848501 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:50:14.848461    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:50:14.868718 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:50:14.868686    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7b1f2cd0-df16-41e4-9ee6-701f678537c1-tls-certs\") pod \"router-with-refs-pd-test-kserve-router-scheduler-99c668b45pf2wc\" (UID: \"7b1f2cd0-df16-41e4-9ee6-701f678537c1\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b45pf2wc"
Sep 16 19:50:14.868825 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:50:14.868721    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xj97j\" (UniqueName: \"kubernetes.io/projected/7b1f2cd0-df16-41e4-9ee6-701f678537c1-kube-api-access-xj97j\") pod \"router-with-refs-pd-test-kserve-router-scheduler-99c668b45pf2wc\" (UID: \"7b1f2cd0-df16-41e4-9ee6-701f678537c1\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b45pf2wc"
Sep 16 19:50:14.970251 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:50:14.970154    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7b1f2cd0-df16-41e4-9ee6-701f678537c1-tls-certs\") pod \"router-with-refs-pd-test-kserve-router-scheduler-99c668b45pf2wc\" (UID: \"7b1f2cd0-df16-41e4-9ee6-701f678537c1\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b45pf2wc"
Sep 16 19:50:14.970251 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:50:14.970197    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xj97j\" (UniqueName: \"kubernetes.io/projected/7b1f2cd0-df16-41e4-9ee6-701f678537c1-kube-api-access-xj97j\") pod \"router-with-refs-pd-test-kserve-router-scheduler-99c668b45pf2wc\" (UID: \"7b1f2cd0-df16-41e4-9ee6-701f678537c1\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b45pf2wc"
Sep 16 19:50:14.972943 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:50:14.972919    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7b1f2cd0-df16-41e4-9ee6-701f678537c1-tls-certs\") pod \"router-with-refs-pd-test-kserve-router-scheduler-99c668b45pf2wc\" (UID: \"7b1f2cd0-df16-41e4-9ee6-701f678537c1\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b45pf2wc"
Sep 16 19:50:14.980273 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:50:14.980249    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xj97j\" (UniqueName: \"kubernetes.io/projected/7b1f2cd0-df16-41e4-9ee6-701f678537c1-kube-api-access-xj97j\") pod \"router-with-refs-pd-test-kserve-router-scheduler-99c668b45pf2wc\" (UID: \"7b1f2cd0-df16-41e4-9ee6-701f678537c1\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b45pf2wc"
Sep 16 19:50:15.110784 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:50:15.110749    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b45pf2wc"
Sep 16 19:50:15.152767 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:50:15.152720    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:50:15.387450 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:50:15.387422    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b45pf2wc"]
Sep 16 19:50:15.389657 ip-10-0-135-119 kubenswrapper[2578]: W0916 19:50:15.389625    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7b1f2cd0_df16_41e4_9ee6_701f678537c1.slice/crio-d4f0ef149e69c0ecef5dd64ab3bf0e012635d8d2bfde9b7882c9dc99a0d8f9aa WatchSource:0}: Error finding container d4f0ef149e69c0ecef5dd64ab3bf0e012635d8d2bfde9b7882c9dc99a0d8f9aa: Status 404 returned error can't find the container with id d4f0ef149e69c0ecef5dd64ab3bf0e012635d8d2bfde9b7882c9dc99a0d8f9aa
Sep 16 19:50:15.586497 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:50:15.586459    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b45pf2wc" event={"ID":"7b1f2cd0-df16-41e4-9ee6-701f678537c1","Type":"ContainerStarted","Data":"8e1e18f3ed2ec31d900f3ff368662e8ec9802199a73c30facd1fe51dc7586cfa"}
Sep 16 19:50:15.586698 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:50:15.586518    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b45pf2wc" event={"ID":"7b1f2cd0-df16-41e4-9ee6-701f678537c1","Type":"ContainerStarted","Data":"d4f0ef149e69c0ecef5dd64ab3bf0e012635d8d2bfde9b7882c9dc99a0d8f9aa"}
Sep 16 19:50:15.586698 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:50:15.586547    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b45pf2wc"
Sep 16 19:50:15.610313 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:50:15.610254    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b45pf2wc" podStartSLOduration=1.610234419 podStartE2EDuration="1.610234419s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:50:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:50:15.60817531 +0000 UTC m=+4196.674666829" watchObservedRunningTime="2026-09-16 19:50:15.610234419 +0000 UTC m=+4196.676725963"
Sep 16 19:50:18.928701 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:50:18.928663    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:50:19.052531 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:50:19.052490    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:50:19.738683 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:50:19.738638    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-tmrkx_4c2078bb-281a-4aa1-8e38-238897865434/console-operator/2.log"
Sep 16 19:50:19.747991 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:50:19.747954    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-k6gvs_6012728d-aee6-4f93-ab6e-fad616ef36e8/ovn-acl-logging/0.log"
Sep 16 19:50:19.755093 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:50:19.755065    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-tmrkx_4c2078bb-281a-4aa1-8e38-238897865434/console-operator/2.log"
Sep 16 19:50:19.763291 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:50:19.763264    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-k6gvs_6012728d-aee6-4f93-ab6e-fad616ef36e8/ovn-acl-logging/0.log"
Sep 16 19:50:46.593421 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:50:46.593339    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b45pf2wc"
Sep 16 19:50:53.908997 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:50:53.908943    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:50:53.938754 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:50:53.938715    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-pwqwv"]
Sep 16 19:50:53.995489 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:50:53.995450    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:51:01.180901 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:51:01.180853    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-pwqwv" podUID="43379ca3-f36b-4bf0-8a5e-45f514086807" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:51:08.959532 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:51:08.959476    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-pwqwv" podUID="43379ca3-f36b-4bf0-8a5e-45f514086807" containerName="main" containerID="cri-o://f048676ed805a7f7262bcf4349d9c7a870f89d1556a2528c9d1f99db49e2cbfc" gracePeriod=45
Sep 16 19:51:11.180658 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:51:11.180609    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-pwqwv" podUID="43379ca3-f36b-4bf0-8a5e-45f514086807" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:51:21.181193 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:51:21.181153    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-pwqwv" podUID="43379ca3-f36b-4bf0-8a5e-45f514086807" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:51:21.181688 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:51:21.181259    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-pwqwv"
Sep 16 19:51:31.180941 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:51:31.180901    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-pwqwv" podUID="43379ca3-f36b-4bf0-8a5e-45f514086807" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:51:39.218640 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:51:39.218614    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-pwqwv"
Sep 16 19:51:39.337736 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:51:39.337632    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jn4lj\" (UniqueName: \"kubernetes.io/projected/43379ca3-f36b-4bf0-8a5e-45f514086807-kube-api-access-jn4lj\") pod \"43379ca3-f36b-4bf0-8a5e-45f514086807\" (UID: \"43379ca3-f36b-4bf0-8a5e-45f514086807\") "
Sep 16 19:51:39.337736 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:51:39.337691    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/43379ca3-f36b-4bf0-8a5e-45f514086807-tls-certs\") pod \"43379ca3-f36b-4bf0-8a5e-45f514086807\" (UID: \"43379ca3-f36b-4bf0-8a5e-45f514086807\") "
Sep 16 19:51:39.340183 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:51:39.340144    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/43379ca3-f36b-4bf0-8a5e-45f514086807-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "43379ca3-f36b-4bf0-8a5e-45f514086807" (UID: "43379ca3-f36b-4bf0-8a5e-45f514086807"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:51:39.340314 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:51:39.340195    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/43379ca3-f36b-4bf0-8a5e-45f514086807-kube-api-access-jn4lj" (OuterVolumeSpecName: "kube-api-access-jn4lj") pod "43379ca3-f36b-4bf0-8a5e-45f514086807" (UID: "43379ca3-f36b-4bf0-8a5e-45f514086807"). InnerVolumeSpecName "kube-api-access-jn4lj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:51:39.438338 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:51:39.438300    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jn4lj\" (UniqueName: \"kubernetes.io/projected/43379ca3-f36b-4bf0-8a5e-45f514086807-kube-api-access-jn4lj\") on node \"ip-10-0-135-119.ec2.internal\" DevicePath \"\""
Sep 16 19:51:39.438338 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:51:39.438329    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/43379ca3-f36b-4bf0-8a5e-45f514086807-tls-certs\") on node \"ip-10-0-135-119.ec2.internal\" DevicePath \"\""
Sep 16 19:51:39.994288 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:51:39.994249    2578 generic.go:356] "Generic (PLEG): container finished" podID="43379ca3-f36b-4bf0-8a5e-45f514086807" containerID="f048676ed805a7f7262bcf4349d9c7a870f89d1556a2528c9d1f99db49e2cbfc" exitCode=0
Sep 16 19:51:39.994501 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:51:39.994300    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-pwqwv"
Sep 16 19:51:39.994501 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:51:39.994335    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-pwqwv" event={"ID":"43379ca3-f36b-4bf0-8a5e-45f514086807","Type":"ContainerDied","Data":"f048676ed805a7f7262bcf4349d9c7a870f89d1556a2528c9d1f99db49e2cbfc"}
Sep 16 19:51:39.994501 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:51:39.994387    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-pwqwv" event={"ID":"43379ca3-f36b-4bf0-8a5e-45f514086807","Type":"ContainerDied","Data":"b17aa58c34045a2a5594431bbbda9234b300d96eff3218221ef7b967286c396c"}
Sep 16 19:51:39.994501 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:51:39.994411    2578 scope.go:117] "RemoveContainer" containerID="f048676ed805a7f7262bcf4349d9c7a870f89d1556a2528c9d1f99db49e2cbfc"
Sep 16 19:51:40.003990 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:51:40.003954    2578 scope.go:117] "RemoveContainer" containerID="f048676ed805a7f7262bcf4349d9c7a870f89d1556a2528c9d1f99db49e2cbfc"
Sep 16 19:51:40.004284 ip-10-0-135-119 kubenswrapper[2578]: E0916 19:51:40.004262    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f048676ed805a7f7262bcf4349d9c7a870f89d1556a2528c9d1f99db49e2cbfc\": container with ID starting with f048676ed805a7f7262bcf4349d9c7a870f89d1556a2528c9d1f99db49e2cbfc not found: ID does not exist" containerID="f048676ed805a7f7262bcf4349d9c7a870f89d1556a2528c9d1f99db49e2cbfc"
Sep 16 19:51:40.004333 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:51:40.004297    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f048676ed805a7f7262bcf4349d9c7a870f89d1556a2528c9d1f99db49e2cbfc"} err="failed to get container status \"f048676ed805a7f7262bcf4349d9c7a870f89d1556a2528c9d1f99db49e2cbfc\": rpc error: code = NotFound desc = could not find container \"f048676ed805a7f7262bcf4349d9c7a870f89d1556a2528c9d1f99db49e2cbfc\": container with ID starting with f048676ed805a7f7262bcf4349d9c7a870f89d1556a2528c9d1f99db49e2cbfc not found: ID does not exist"
Sep 16 19:51:40.011897 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:51:40.011870    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-pwqwv"]
Sep 16 19:51:40.016316 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:51:40.016284    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-pwqwv"]
Sep 16 19:51:41.437302 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:51:41.437263    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="43379ca3-f36b-4bf0-8a5e-45f514086807" path="/var/lib/kubelet/pods/43379ca3-f36b-4bf0-8a5e-45f514086807/volumes"
Sep 16 19:51:43.951218 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:51:43.951187    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:51:44.075308 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:51:44.075267    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:51:48.924098 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:51:48.924057    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:51:49.161746 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:51:49.161713    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:51:50.255488 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:51:50.255449    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:51:51.512107 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:51:51.512068    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:51:57.622214 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:51:57.622171    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:51:57.722420 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:51:57.722380    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:52:22.710798 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:52:22.710761    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:52:33.320491 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:52:33.320458    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:52:33.324614 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:52:33.324572    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b45pf2wc"]
Sep 16 19:52:33.391272 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:52:33.391234    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:52:36.593103 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:52:36.593055    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b45pf2wc" podUID="7b1f2cd0-df16-41e4-9ee6-701f678537c1" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:52:46.593657 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:52:46.593598    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b45pf2wc" podUID="7b1f2cd0-df16-41e4-9ee6-701f678537c1" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:52:48.344396 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:52:48.344343    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b45pf2wc" podUID="7b1f2cd0-df16-41e4-9ee6-701f678537c1" containerName="main" containerID="cri-o://8e1e18f3ed2ec31d900f3ff368662e8ec9802199a73c30facd1fe51dc7586cfa" gracePeriod=45
Sep 16 19:52:56.593514 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:52:56.593467    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b45pf2wc" podUID="7b1f2cd0-df16-41e4-9ee6-701f678537c1" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:52:56.593937 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:52:56.593595    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b45pf2wc"
Sep 16 19:53:00.251319 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:53:00.251283    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:53:06.593453 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:53:06.593409    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b45pf2wc" podUID="7b1f2cd0-df16-41e4-9ee6-701f678537c1" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:53:16.593358 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:53:16.593317    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b45pf2wc" podUID="7b1f2cd0-df16-41e4-9ee6-701f678537c1" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:53:18.466277 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:53:18.466244    2578 generic.go:356] "Generic (PLEG): container finished" podID="7b1f2cd0-df16-41e4-9ee6-701f678537c1" containerID="8e1e18f3ed2ec31d900f3ff368662e8ec9802199a73c30facd1fe51dc7586cfa" exitCode=0
Sep 16 19:53:18.466597 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:53:18.466310    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b45pf2wc" event={"ID":"7b1f2cd0-df16-41e4-9ee6-701f678537c1","Type":"ContainerDied","Data":"8e1e18f3ed2ec31d900f3ff368662e8ec9802199a73c30facd1fe51dc7586cfa"}
Sep 16 19:53:18.600795 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:53:18.600770    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b45pf2wc"
Sep 16 19:53:18.738022 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:53:18.737959    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7b1f2cd0-df16-41e4-9ee6-701f678537c1-tls-certs\") pod \"7b1f2cd0-df16-41e4-9ee6-701f678537c1\" (UID: \"7b1f2cd0-df16-41e4-9ee6-701f678537c1\") "
Sep 16 19:53:18.738196 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:53:18.738034    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xj97j\" (UniqueName: \"kubernetes.io/projected/7b1f2cd0-df16-41e4-9ee6-701f678537c1-kube-api-access-xj97j\") pod \"7b1f2cd0-df16-41e4-9ee6-701f678537c1\" (UID: \"7b1f2cd0-df16-41e4-9ee6-701f678537c1\") "
Sep 16 19:53:18.740395 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:53:18.740364    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7b1f2cd0-df16-41e4-9ee6-701f678537c1-kube-api-access-xj97j" (OuterVolumeSpecName: "kube-api-access-xj97j") pod "7b1f2cd0-df16-41e4-9ee6-701f678537c1" (UID: "7b1f2cd0-df16-41e4-9ee6-701f678537c1"). InnerVolumeSpecName "kube-api-access-xj97j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:53:18.740395 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:53:18.740383    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7b1f2cd0-df16-41e4-9ee6-701f678537c1-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "7b1f2cd0-df16-41e4-9ee6-701f678537c1" (UID: "7b1f2cd0-df16-41e4-9ee6-701f678537c1"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:53:18.839180 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:53:18.839138    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7b1f2cd0-df16-41e4-9ee6-701f678537c1-tls-certs\") on node \"ip-10-0-135-119.ec2.internal\" DevicePath \"\""
Sep 16 19:53:18.839180 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:53:18.839172    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xj97j\" (UniqueName: \"kubernetes.io/projected/7b1f2cd0-df16-41e4-9ee6-701f678537c1-kube-api-access-xj97j\") on node \"ip-10-0-135-119.ec2.internal\" DevicePath \"\""
Sep 16 19:53:19.472343 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:53:19.472319    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b45pf2wc"
Sep 16 19:53:19.472705 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:53:19.472359    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b45pf2wc" event={"ID":"7b1f2cd0-df16-41e4-9ee6-701f678537c1","Type":"ContainerDied","Data":"d4f0ef149e69c0ecef5dd64ab3bf0e012635d8d2bfde9b7882c9dc99a0d8f9aa"}
Sep 16 19:53:19.472705 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:53:19.472417    2578 scope.go:117] "RemoveContainer" containerID="8e1e18f3ed2ec31d900f3ff368662e8ec9802199a73c30facd1fe51dc7586cfa"
Sep 16 19:53:19.491393 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:53:19.491370    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b45pf2wc"]
Sep 16 19:53:19.500197 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:53:19.500174    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b45pf2wc"]
Sep 16 19:53:21.441239 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:53:21.441192    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7b1f2cd0-df16-41e4-9ee6-701f678537c1" path="/var/lib/kubelet/pods/7b1f2cd0-df16-41e4-9ee6-701f678537c1/volumes"
Sep 16 19:53:23.429384 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:53:23.429325    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:53:23.526673 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:53:23.526626    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:53:28.484861 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:53:28.484827    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:53:32.301877 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:53:32.301836    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:53:49.969215 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:53:49.969127    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:53:51.675388 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:53:51.675351    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:53:57.498039 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:53:57.498002    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:54:14.873649 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:54:14.873608    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:54:14.937215 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:54:14.937182    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:54:16.745760 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:54:16.745728    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:54:21.415326 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:54:21.415287    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:54:21.477538 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:54:21.477499    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:55:03.393044 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:55:03.393010    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:55:04.556366 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:55:04.556329    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:55:09.782965 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:55:09.782927    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:55:15.670745 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:55:15.670711    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:55:15.768519 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:55:15.768488    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:55:19.769708 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:55:19.769683    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-tmrkx_4c2078bb-281a-4aa1-8e38-238897865434/console-operator/2.log"
Sep 16 19:55:19.778826 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:55:19.778803    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-k6gvs_6012728d-aee6-4f93-ab6e-fad616ef36e8/ovn-acl-logging/0.log"
Sep 16 19:55:19.786047 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:55:19.786027    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-tmrkx_4c2078bb-281a-4aa1-8e38-238897865434/console-operator/2.log"
Sep 16 19:55:19.794421 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:55:19.794403    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-k6gvs_6012728d-aee6-4f93-ab6e-fad616ef36e8/ovn-acl-logging/0.log"
Sep 16 19:55:23.160093 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:55:23.160061    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-hsdh7"]
Sep 16 19:55:23.160456 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:55:23.160410    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="43379ca3-f36b-4bf0-8a5e-45f514086807" containerName="main"
Sep 16 19:55:23.160456 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:55:23.160421    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="43379ca3-f36b-4bf0-8a5e-45f514086807" containerName="main"
Sep 16 19:55:23.160456 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:55:23.160437    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7b1f2cd0-df16-41e4-9ee6-701f678537c1" containerName="main"
Sep 16 19:55:23.160456 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:55:23.160442    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="7b1f2cd0-df16-41e4-9ee6-701f678537c1" containerName="main"
Sep 16 19:55:23.160589 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:55:23.160490    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="7b1f2cd0-df16-41e4-9ee6-701f678537c1" containerName="main"
Sep 16 19:55:23.160589 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:55:23.160503    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="43379ca3-f36b-4bf0-8a5e-45f514086807" containerName="main"
Sep 16 19:55:23.163442 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:55:23.163423    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-hsdh7"
Sep 16 19:55:23.165889 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:55:23.165862    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-rolling-upgrade-coordination-a9229638\"/\"kube-root-ca.crt\""
Sep 16 19:55:23.166631 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:55:23.166605    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-rolling-upgrade-coordination-a9229638\"/\"openshift-service-ca.crt\""
Sep 16 19:55:23.166758 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:55:23.166674    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-rolling-upgrade-coordination-a9229638\"/\"rolling-upgrade-test-kserve-self-signed-certs\""
Sep 16 19:55:23.166876 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:55:23.166861    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-rolling-upgrade-coordination-a9229638\"/\"default-dockercfg-b6wch\""
Sep 16 19:55:23.173345 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:55:23.173324    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-hsdh7"]
Sep 16 19:55:23.189886 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:55:23.189861    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:55:23.206209 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:55:23.206186    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/381e5008-d794-4e30-87f7-da2f23e65f0c-model-cache\") pod \"rolling-upgrade-test-kserve-74b79d7f77-hsdh7\" (UID: \"381e5008-d794-4e30-87f7-da2f23e65f0c\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-hsdh7"
Sep 16 19:55:23.206333 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:55:23.206214    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dndhl\" (UniqueName: \"kubernetes.io/projected/381e5008-d794-4e30-87f7-da2f23e65f0c-kube-api-access-dndhl\") pod \"rolling-upgrade-test-kserve-74b79d7f77-hsdh7\" (UID: \"381e5008-d794-4e30-87f7-da2f23e65f0c\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-hsdh7"
Sep 16 19:55:23.206333 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:55:23.206237    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/381e5008-d794-4e30-87f7-da2f23e65f0c-tmp-dir\") pod \"rolling-upgrade-test-kserve-74b79d7f77-hsdh7\" (UID: \"381e5008-d794-4e30-87f7-da2f23e65f0c\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-hsdh7"
Sep 16 19:55:23.206333 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:55:23.206266    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/381e5008-d794-4e30-87f7-da2f23e65f0c-home\") pod \"rolling-upgrade-test-kserve-74b79d7f77-hsdh7\" (UID: \"381e5008-d794-4e30-87f7-da2f23e65f0c\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-hsdh7"
Sep 16 19:55:23.206333 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:55:23.206314    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/381e5008-d794-4e30-87f7-da2f23e65f0c-dshm\") pod \"rolling-upgrade-test-kserve-74b79d7f77-hsdh7\" (UID: \"381e5008-d794-4e30-87f7-da2f23e65f0c\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-hsdh7"
Sep 16 19:55:23.206481 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:55:23.206357    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/381e5008-d794-4e30-87f7-da2f23e65f0c-tls-certs\") pod \"rolling-upgrade-test-kserve-74b79d7f77-hsdh7\" (UID: \"381e5008-d794-4e30-87f7-da2f23e65f0c\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-hsdh7"
Sep 16 19:55:23.307646 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:55:23.307612    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/381e5008-d794-4e30-87f7-da2f23e65f0c-dshm\") pod \"rolling-upgrade-test-kserve-74b79d7f77-hsdh7\" (UID: \"381e5008-d794-4e30-87f7-da2f23e65f0c\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-hsdh7"
Sep 16 19:55:23.307824 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:55:23.307660    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/381e5008-d794-4e30-87f7-da2f23e65f0c-tls-certs\") pod \"rolling-upgrade-test-kserve-74b79d7f77-hsdh7\" (UID: \"381e5008-d794-4e30-87f7-da2f23e65f0c\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-hsdh7"
Sep 16 19:55:23.307824 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:55:23.307721    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/381e5008-d794-4e30-87f7-da2f23e65f0c-model-cache\") pod \"rolling-upgrade-test-kserve-74b79d7f77-hsdh7\" (UID: \"381e5008-d794-4e30-87f7-da2f23e65f0c\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-hsdh7"
Sep 16 19:55:23.307824 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:55:23.307738    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dndhl\" (UniqueName: \"kubernetes.io/projected/381e5008-d794-4e30-87f7-da2f23e65f0c-kube-api-access-dndhl\") pod \"rolling-upgrade-test-kserve-74b79d7f77-hsdh7\" (UID: \"381e5008-d794-4e30-87f7-da2f23e65f0c\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-hsdh7"
Sep 16 19:55:23.307824 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:55:23.307764    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/381e5008-d794-4e30-87f7-da2f23e65f0c-tmp-dir\") pod \"rolling-upgrade-test-kserve-74b79d7f77-hsdh7\" (UID: \"381e5008-d794-4e30-87f7-da2f23e65f0c\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-hsdh7"
Sep 16 19:55:23.307824 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:55:23.307803    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/381e5008-d794-4e30-87f7-da2f23e65f0c-home\") pod \"rolling-upgrade-test-kserve-74b79d7f77-hsdh7\" (UID: \"381e5008-d794-4e30-87f7-da2f23e65f0c\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-hsdh7"
Sep 16 19:55:23.308215 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:55:23.308190    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/381e5008-d794-4e30-87f7-da2f23e65f0c-model-cache\") pod \"rolling-upgrade-test-kserve-74b79d7f77-hsdh7\" (UID: \"381e5008-d794-4e30-87f7-da2f23e65f0c\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-hsdh7"
Sep 16 19:55:23.308274 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:55:23.308233    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/381e5008-d794-4e30-87f7-da2f23e65f0c-tmp-dir\") pod \"rolling-upgrade-test-kserve-74b79d7f77-hsdh7\" (UID: \"381e5008-d794-4e30-87f7-da2f23e65f0c\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-hsdh7"
Sep 16 19:55:23.308274 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:55:23.308247    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/381e5008-d794-4e30-87f7-da2f23e65f0c-home\") pod \"rolling-upgrade-test-kserve-74b79d7f77-hsdh7\" (UID: \"381e5008-d794-4e30-87f7-da2f23e65f0c\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-hsdh7"
Sep 16 19:55:23.310159 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:55:23.310140    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/381e5008-d794-4e30-87f7-da2f23e65f0c-dshm\") pod \"rolling-upgrade-test-kserve-74b79d7f77-hsdh7\" (UID: \"381e5008-d794-4e30-87f7-da2f23e65f0c\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-hsdh7"
Sep 16 19:55:23.310334 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:55:23.310318    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/381e5008-d794-4e30-87f7-da2f23e65f0c-tls-certs\") pod \"rolling-upgrade-test-kserve-74b79d7f77-hsdh7\" (UID: \"381e5008-d794-4e30-87f7-da2f23e65f0c\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-hsdh7"
Sep 16 19:55:23.316606 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:55:23.316588    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dndhl\" (UniqueName: \"kubernetes.io/projected/381e5008-d794-4e30-87f7-da2f23e65f0c-kube-api-access-dndhl\") pod \"rolling-upgrade-test-kserve-74b79d7f77-hsdh7\" (UID: \"381e5008-d794-4e30-87f7-da2f23e65f0c\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-hsdh7"
Sep 16 19:55:23.455785 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:55:23.455706    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:55:23.474831 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:55:23.474796    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-hsdh7"
Sep 16 19:55:23.749597 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:55:23.748493    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-hsdh7"]
Sep 16 19:55:24.052720 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:55:24.052631    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-hsdh7" event={"ID":"381e5008-d794-4e30-87f7-da2f23e65f0c","Type":"ContainerStarted","Data":"e80b566b19004efb1e019c5ec2fd16f65e97620e986a2f62091d958359b9785a"}
Sep 16 19:55:24.052720 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:55:24.052688    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-hsdh7" event={"ID":"381e5008-d794-4e30-87f7-da2f23e65f0c","Type":"ContainerStarted","Data":"622b374a201a30ecaa2360379f6a842309f6a9521201f3cf1311f386f7f66b41"}
Sep 16 19:55:24.071998 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:55:24.071926    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-hsdh7" podStartSLOduration=1.071909707 podStartE2EDuration="1.071909707s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:55:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:55:24.070640153 +0000 UTC m=+4505.137131666" watchObservedRunningTime="2026-09-16 19:55:24.071909707 +0000 UTC m=+4505.138401194"
Sep 16 19:55:24.475775 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:55:24.475740    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-hsdh7"
Sep 16 19:55:33.475891 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:55:33.475856    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-hsdh7"
Sep 16 19:55:33.488250 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:55:33.488224    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-hsdh7"
Sep 16 19:55:34.111554 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:55:34.111524    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-hsdh7"
Sep 16 19:55:50.834091 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:55:50.834029    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:55:56.089951 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:55:56.089911    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:56:03.660728 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:56:03.660693    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-hsdh7"]
Sep 16 19:56:03.670939 ip-10-0-135-119 kubenswrapper[2578]: E0916 19:56:03.670905    2578 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 16 19:56:03.670939 ip-10-0-135-119 kubenswrapper[2578]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 19:56:03.670939 ip-10-0-135-119 kubenswrapper[2578]:         , stdout: , stderr: , exit code -1
Sep 16 19:56:03.670939 ip-10-0-135-119 kubenswrapper[2578]:  > containerID="e80b566b19004efb1e019c5ec2fd16f65e97620e986a2f62091d958359b9785a" cmd=["/bin/sleep","15"]
Sep 16 19:56:03.671187 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:56:03.670957    2578 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-hsdh7" message="" err=<
Sep 16 19:56:03.671187 ip-10-0-135-119 kubenswrapper[2578]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 19:56:03.671187 ip-10-0-135-119 kubenswrapper[2578]:         , stdout: , stderr: , exit code -1
Sep 16 19:56:03.671187 ip-10-0-135-119 kubenswrapper[2578]:  >
Sep 16 19:56:03.671187 ip-10-0-135-119 kubenswrapper[2578]: E0916 19:56:03.670971    2578 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 16 19:56:03.671187 ip-10-0-135-119 kubenswrapper[2578]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 19:56:03.671187 ip-10-0-135-119 kubenswrapper[2578]:         , stdout: , stderr: , exit code -1
Sep 16 19:56:03.671187 ip-10-0-135-119 kubenswrapper[2578]:  > pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-hsdh7" podUID="381e5008-d794-4e30-87f7-da2f23e65f0c" containerName="main" containerID="cri-o://e80b566b19004efb1e019c5ec2fd16f65e97620e986a2f62091d958359b9785a"
Sep 16 19:56:03.671187 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:56:03.671028    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-hsdh7" podUID="381e5008-d794-4e30-87f7-da2f23e65f0c" containerName="main" containerID="cri-o://e80b566b19004efb1e019c5ec2fd16f65e97620e986a2f62091d958359b9785a" gracePeriod=60
Sep 16 19:56:03.674190 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:56:03.674163    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:56:03.744277 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:56:03.744241    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:56:04.019748 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:56:04.019720    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-hsdh7"
Sep 16 19:56:04.134127 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:56:04.134097    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/381e5008-d794-4e30-87f7-da2f23e65f0c-home\") pod \"381e5008-d794-4e30-87f7-da2f23e65f0c\" (UID: \"381e5008-d794-4e30-87f7-da2f23e65f0c\") "
Sep 16 19:56:04.134309 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:56:04.134141    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dndhl\" (UniqueName: \"kubernetes.io/projected/381e5008-d794-4e30-87f7-da2f23e65f0c-kube-api-access-dndhl\") pod \"381e5008-d794-4e30-87f7-da2f23e65f0c\" (UID: \"381e5008-d794-4e30-87f7-da2f23e65f0c\") "
Sep 16 19:56:04.134309 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:56:04.134198    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/381e5008-d794-4e30-87f7-da2f23e65f0c-model-cache\") pod \"381e5008-d794-4e30-87f7-da2f23e65f0c\" (UID: \"381e5008-d794-4e30-87f7-da2f23e65f0c\") "
Sep 16 19:56:04.134309 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:56:04.134216    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/381e5008-d794-4e30-87f7-da2f23e65f0c-dshm\") pod \"381e5008-d794-4e30-87f7-da2f23e65f0c\" (UID: \"381e5008-d794-4e30-87f7-da2f23e65f0c\") "
Sep 16 19:56:04.134309 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:56:04.134234    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/381e5008-d794-4e30-87f7-da2f23e65f0c-tmp-dir\") pod \"381e5008-d794-4e30-87f7-da2f23e65f0c\" (UID: \"381e5008-d794-4e30-87f7-da2f23e65f0c\") "
Sep 16 19:56:04.134309 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:56:04.134258    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/381e5008-d794-4e30-87f7-da2f23e65f0c-tls-certs\") pod \"381e5008-d794-4e30-87f7-da2f23e65f0c\" (UID: \"381e5008-d794-4e30-87f7-da2f23e65f0c\") "
Sep 16 19:56:04.134547 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:56:04.134380    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/381e5008-d794-4e30-87f7-da2f23e65f0c-home" (OuterVolumeSpecName: "home") pod "381e5008-d794-4e30-87f7-da2f23e65f0c" (UID: "381e5008-d794-4e30-87f7-da2f23e65f0c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:56:04.134547 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:56:04.134439    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/381e5008-d794-4e30-87f7-da2f23e65f0c-model-cache" (OuterVolumeSpecName: "model-cache") pod "381e5008-d794-4e30-87f7-da2f23e65f0c" (UID: "381e5008-d794-4e30-87f7-da2f23e65f0c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:56:04.134547 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:56:04.134542    2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/381e5008-d794-4e30-87f7-da2f23e65f0c-home\") on node \"ip-10-0-135-119.ec2.internal\" DevicePath \"\""
Sep 16 19:56:04.134658 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:56:04.134555    2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/381e5008-d794-4e30-87f7-da2f23e65f0c-model-cache\") on node \"ip-10-0-135-119.ec2.internal\" DevicePath \"\""
Sep 16 19:56:04.134658 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:56:04.134560    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/381e5008-d794-4e30-87f7-da2f23e65f0c-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "381e5008-d794-4e30-87f7-da2f23e65f0c" (UID: "381e5008-d794-4e30-87f7-da2f23e65f0c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:56:04.136701 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:56:04.136670    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/381e5008-d794-4e30-87f7-da2f23e65f0c-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "381e5008-d794-4e30-87f7-da2f23e65f0c" (UID: "381e5008-d794-4e30-87f7-da2f23e65f0c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:56:04.136812 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:56:04.136717    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/381e5008-d794-4e30-87f7-da2f23e65f0c-dshm" (OuterVolumeSpecName: "dshm") pod "381e5008-d794-4e30-87f7-da2f23e65f0c" (UID: "381e5008-d794-4e30-87f7-da2f23e65f0c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:56:04.136812 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:56:04.136752    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/381e5008-d794-4e30-87f7-da2f23e65f0c-kube-api-access-dndhl" (OuterVolumeSpecName: "kube-api-access-dndhl") pod "381e5008-d794-4e30-87f7-da2f23e65f0c" (UID: "381e5008-d794-4e30-87f7-da2f23e65f0c"). InnerVolumeSpecName "kube-api-access-dndhl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:56:04.235353 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:56:04.235288    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dndhl\" (UniqueName: \"kubernetes.io/projected/381e5008-d794-4e30-87f7-da2f23e65f0c-kube-api-access-dndhl\") on node \"ip-10-0-135-119.ec2.internal\" DevicePath \"\""
Sep 16 19:56:04.235353 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:56:04.235311    2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/381e5008-d794-4e30-87f7-da2f23e65f0c-dshm\") on node \"ip-10-0-135-119.ec2.internal\" DevicePath \"\""
Sep 16 19:56:04.235353 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:56:04.235320    2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/381e5008-d794-4e30-87f7-da2f23e65f0c-tmp-dir\") on node \"ip-10-0-135-119.ec2.internal\" DevicePath \"\""
Sep 16 19:56:04.235353 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:56:04.235328    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/381e5008-d794-4e30-87f7-da2f23e65f0c-tls-certs\") on node \"ip-10-0-135-119.ec2.internal\" DevicePath \"\""
Sep 16 19:56:04.237478 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:56:04.237454    2578 generic.go:356] "Generic (PLEG): container finished" podID="381e5008-d794-4e30-87f7-da2f23e65f0c" containerID="e80b566b19004efb1e019c5ec2fd16f65e97620e986a2f62091d958359b9785a" exitCode=0
Sep 16 19:56:04.237595 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:56:04.237509    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-hsdh7" event={"ID":"381e5008-d794-4e30-87f7-da2f23e65f0c","Type":"ContainerDied","Data":"e80b566b19004efb1e019c5ec2fd16f65e97620e986a2f62091d958359b9785a"}
Sep 16 19:56:04.237595 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:56:04.237545    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-hsdh7" event={"ID":"381e5008-d794-4e30-87f7-da2f23e65f0c","Type":"ContainerDied","Data":"622b374a201a30ecaa2360379f6a842309f6a9521201f3cf1311f386f7f66b41"}
Sep 16 19:56:04.237595 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:56:04.237566    2578 scope.go:117] "RemoveContainer" containerID="e80b566b19004efb1e019c5ec2fd16f65e97620e986a2f62091d958359b9785a"
Sep 16 19:56:04.237595 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:56:04.237509    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-hsdh7"
Sep 16 19:56:04.249612 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:56:04.249594    2578 scope.go:117] "RemoveContainer" containerID="e80b566b19004efb1e019c5ec2fd16f65e97620e986a2f62091d958359b9785a"
Sep 16 19:56:04.249903 ip-10-0-135-119 kubenswrapper[2578]: E0916 19:56:04.249884    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e80b566b19004efb1e019c5ec2fd16f65e97620e986a2f62091d958359b9785a\": container with ID starting with e80b566b19004efb1e019c5ec2fd16f65e97620e986a2f62091d958359b9785a not found: ID does not exist" containerID="e80b566b19004efb1e019c5ec2fd16f65e97620e986a2f62091d958359b9785a"
Sep 16 19:56:04.249958 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:56:04.249912    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e80b566b19004efb1e019c5ec2fd16f65e97620e986a2f62091d958359b9785a"} err="failed to get container status \"e80b566b19004efb1e019c5ec2fd16f65e97620e986a2f62091d958359b9785a\": rpc error: code = NotFound desc = could not find container \"e80b566b19004efb1e019c5ec2fd16f65e97620e986a2f62091d958359b9785a\": container with ID starting with e80b566b19004efb1e019c5ec2fd16f65e97620e986a2f62091d958359b9785a not found: ID does not exist"
Sep 16 19:56:04.260468 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:56:04.260444    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-hsdh7"]
Sep 16 19:56:04.265066 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:56:04.265047    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-hsdh7"]
Sep 16 19:56:05.433892 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:56:05.433856    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="381e5008-d794-4e30-87f7-da2f23e65f0c" path="/var/lib/kubelet/pods/381e5008-d794-4e30-87f7-da2f23e65f0c/volumes"
Sep 16 19:56:15.989485 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:56:15.989447    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:56:20.658256 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:56:20.658221    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:56:22.369147 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:56:22.369111    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:56:23.711197 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:56:23.711158    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:56:28.361639 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:56:28.361594    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:56:28.431062 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:56:28.431032    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:56:48.892891 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:56:48.892780    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:56:48.965032 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:56:48.965004    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:56:54.557292 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:56:54.557247    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:56:58.548158 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:56:58.548121    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:56:58.688166 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:56:58.688129    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:57:00.825341 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:57:00.825289    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:57:07.097299 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:57:07.097263    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:57:07.175523 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:57:07.175492    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:57:23.898417 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:57:23.898384    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-65dc8f76f5-4f2xb_3f409571-9229-4506-898f-145d06ea77b2/manager/0.log"
Sep 16 19:57:23.928218 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:57:23.928195    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-fbdd4f445-cvtk5_2a534cde-c0df-4cbd-bd26-e3f08bb0468a/kuadrant-console-plugin/0.log"
Sep 16 19:57:24.159052 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:57:24.158941    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5ffb4b4ccb-4djgv_734cbe34-515a-4d3a-bdb5-b6914ab46948/manager/0.log"
Sep 16 19:57:30.013197 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:57:30.013156    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:57:30.158315 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:57:30.158269    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-mkvvc/must-gather-tgc8m"]
Sep 16 19:57:30.158790 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:57:30.158773    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="381e5008-d794-4e30-87f7-da2f23e65f0c" containerName="main"
Sep 16 19:57:30.158843 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:57:30.158792    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="381e5008-d794-4e30-87f7-da2f23e65f0c" containerName="main"
Sep 16 19:57:30.158878 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:57:30.158861    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="381e5008-d794-4e30-87f7-da2f23e65f0c" containerName="main"
Sep 16 19:57:30.162186 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:57:30.162164    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-mkvvc/must-gather-tgc8m"
Sep 16 19:57:30.164352 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:57:30.164333    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-mkvvc\"/\"default-dockercfg-h7jhd\""
Sep 16 19:57:30.165078 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:57:30.165054    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-mkvvc\"/\"kube-root-ca.crt\""
Sep 16 19:57:30.165208 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:57:30.165067    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-mkvvc\"/\"openshift-service-ca.crt\""
Sep 16 19:57:30.167831 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:57:30.167811    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:57:30.171767 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:57:30.171745    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-mkvvc/must-gather-tgc8m"]
Sep 16 19:57:30.259280 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:57:30.259241    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wsqc6\" (UniqueName: \"kubernetes.io/projected/24643bc5-90cb-4210-a169-fa655b2cb11b-kube-api-access-wsqc6\") pod \"must-gather-tgc8m\" (UID: \"24643bc5-90cb-4210-a169-fa655b2cb11b\") " pod="openshift-must-gather-mkvvc/must-gather-tgc8m"
Sep 16 19:57:30.259465 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:57:30.259298    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/24643bc5-90cb-4210-a169-fa655b2cb11b-must-gather-output\") pod \"must-gather-tgc8m\" (UID: \"24643bc5-90cb-4210-a169-fa655b2cb11b\") " pod="openshift-must-gather-mkvvc/must-gather-tgc8m"
Sep 16 19:57:30.270136 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:57:30.270065    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:57:30.360192 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:57:30.360154    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wsqc6\" (UniqueName: \"kubernetes.io/projected/24643bc5-90cb-4210-a169-fa655b2cb11b-kube-api-access-wsqc6\") pod \"must-gather-tgc8m\" (UID: \"24643bc5-90cb-4210-a169-fa655b2cb11b\") " pod="openshift-must-gather-mkvvc/must-gather-tgc8m"
Sep 16 19:57:30.360375 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:57:30.360210    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/24643bc5-90cb-4210-a169-fa655b2cb11b-must-gather-output\") pod \"must-gather-tgc8m\" (UID: \"24643bc5-90cb-4210-a169-fa655b2cb11b\") " pod="openshift-must-gather-mkvvc/must-gather-tgc8m"
Sep 16 19:57:30.360519 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:57:30.360502    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/24643bc5-90cb-4210-a169-fa655b2cb11b-must-gather-output\") pod \"must-gather-tgc8m\" (UID: \"24643bc5-90cb-4210-a169-fa655b2cb11b\") " pod="openshift-must-gather-mkvvc/must-gather-tgc8m"
Sep 16 19:57:30.369771 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:57:30.369743    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wsqc6\" (UniqueName: \"kubernetes.io/projected/24643bc5-90cb-4210-a169-fa655b2cb11b-kube-api-access-wsqc6\") pod \"must-gather-tgc8m\" (UID: \"24643bc5-90cb-4210-a169-fa655b2cb11b\") " pod="openshift-must-gather-mkvvc/must-gather-tgc8m"
Sep 16 19:57:30.472241 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:57:30.472202    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-mkvvc/must-gather-tgc8m"
Sep 16 19:57:30.743037 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:57:30.743006    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-mkvvc/must-gather-tgc8m"]
Sep 16 19:57:30.745507 ip-10-0-135-119 kubenswrapper[2578]: W0916 19:57:30.745476    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod24643bc5_90cb_4210_a169_fa655b2cb11b.slice/crio-1b4c129ca1e34fec649658ae159f9f2308e8d6274fd993cdab1d990410732c40 WatchSource:0}: Error finding container 1b4c129ca1e34fec649658ae159f9f2308e8d6274fd993cdab1d990410732c40: Status 404 returned error can't find the container with id 1b4c129ca1e34fec649658ae159f9f2308e8d6274fd993cdab1d990410732c40
Sep 16 19:57:30.747452 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:57:30.747431    2578 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 16 19:57:31.650395 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:57:31.650361    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-mkvvc/must-gather-tgc8m" event={"ID":"24643bc5-90cb-4210-a169-fa655b2cb11b","Type":"ContainerStarted","Data":"b67477f6edef85c583f255b36061b70446d4bd163a805153b0ea50c78c62f43a"}
Sep 16 19:57:31.650797 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:57:31.650406    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-mkvvc/must-gather-tgc8m" event={"ID":"24643bc5-90cb-4210-a169-fa655b2cb11b","Type":"ContainerStarted","Data":"2b173e3807e8d396b080426edc3f5fff559b2bd76d23480813bc5bccad6896c9"}
Sep 16 19:57:31.650797 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:57:31.650420    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-mkvvc/must-gather-tgc8m" event={"ID":"24643bc5-90cb-4210-a169-fa655b2cb11b","Type":"ContainerStarted","Data":"1b4c129ca1e34fec649658ae159f9f2308e8d6274fd993cdab1d990410732c40"}
Sep 16 19:57:31.667575 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:57:31.667507    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-mkvvc/must-gather-tgc8m" podStartSLOduration=0.986867908 podStartE2EDuration="1.667485438s" totalImagesPullingTime="680.61753ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:57:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:57:31.666414541 +0000 UTC m=+4632.732906064" watchObservedRunningTime="2026-09-16 19:57:31.667485438 +0000 UTC m=+4632.733976929"
Sep 16 19:57:33.123826 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:57:33.123796    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-8ndsh_3377e453-9418-49a8-870b-64424f13d694/global-pull-secret-syncer/0.log"
Sep 16 19:57:33.221600 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:57:33.221567    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log"
Sep 16 19:57:33.383647 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:57:33.383608    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-135-119.ec2.internal_ce93b5afb1de6b9a80d64df7da81462d/haproxy/0.log"
Sep 16 19:57:36.355604 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:57:36.355529    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:57:36.501246 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:57:36.501206    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:57:38.359276 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:57:38.359240    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-65dc8f76f5-4f2xb_3f409571-9229-4506-898f-145d06ea77b2/manager/0.log"
Sep 16 19:57:38.399387 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:57:38.399354    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-fbdd4f445-cvtk5_2a534cde-c0df-4cbd-bd26-e3f08bb0468a/kuadrant-console-plugin/0.log"
Sep 16 19:57:38.699388 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:57:38.699352    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5ffb4b4ccb-4djgv_734cbe34-515a-4d3a-bdb5-b6914ab46948/manager/0.log"
Sep 16 19:57:40.283556 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:57:40.283526    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-67669cdfc8-hcgrd_ab57f3d7-1613-44da-b990-5e4d91689278/metrics-server/0.log"
Sep 16 19:57:40.541671 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:57:40.541643    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-wsn6q_b38e139a-817f-4138-8f91-70823ef862aa/node-exporter/0.log"
Sep 16 19:57:40.566290 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:57:40.566242    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-wsn6q_b38e139a-817f-4138-8f91-70823ef862aa/kube-rbac-proxy/0.log"
Sep 16 19:57:40.605694 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:57:40.603840    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-wsn6q_b38e139a-817f-4138-8f91-70823ef862aa/init-textfile/0.log"
Sep 16 19:57:40.650222 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:57:40.650139    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-67dcf8c979-8vxlk_88c01bb5-1e07-406b-bc3a-b7e54f3b8c23/kube-rbac-proxy-main/0.log"
Sep 16 19:57:40.685470 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:57:40.685437    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-67dcf8c979-8vxlk_88c01bb5-1e07-406b-bc3a-b7e54f3b8c23/kube-rbac-proxy-self/0.log"
Sep 16 19:57:40.713053 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:57:40.713002    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-67dcf8c979-8vxlk_88c01bb5-1e07-406b-bc3a-b7e54f3b8c23/openshift-state-metrics/0.log"
Sep 16 19:57:40.775399 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:57:40.775364    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_f501e149-38f4-4f16-aaed-225fcd0faf34/prometheus/0.log"
Sep 16 19:57:40.806225 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:57:40.806128    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_f501e149-38f4-4f16-aaed-225fcd0faf34/config-reloader/0.log"
Sep 16 19:57:40.836159 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:57:40.836100    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_f501e149-38f4-4f16-aaed-225fcd0faf34/thanos-sidecar/0.log"
Sep 16 19:57:40.874219 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:57:40.874180    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_f501e149-38f4-4f16-aaed-225fcd0faf34/kube-rbac-proxy-web/0.log"
Sep 16 19:57:40.904408 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:57:40.904359    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_f501e149-38f4-4f16-aaed-225fcd0faf34/kube-rbac-proxy/0.log"
Sep 16 19:57:40.933387 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:57:40.933358    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_f501e149-38f4-4f16-aaed-225fcd0faf34/kube-rbac-proxy-thanos/0.log"
Sep 16 19:57:40.967300 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:57:40.967266    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_f501e149-38f4-4f16-aaed-225fcd0faf34/init-config-reloader/0.log"
Sep 16 19:57:41.008441 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:57:41.008409    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-68bcd7f987-4ncz7_7254fad8-d179-4cfb-a7eb-2cd5a87578fa/prometheus-operator/0.log"
Sep 16 19:57:41.035513 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:57:41.035476    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-68bcd7f987-4ncz7_7254fad8-d179-4cfb-a7eb-2cd5a87578fa/kube-rbac-proxy/0.log"
Sep 16 19:57:41.067195 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:57:41.067110    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-5864cd99d7-rcq4t_ff22f15f-83ae-4530-81ff-6d81caaf1ddb/prometheus-operator-admission-webhook/0.log"
Sep 16 19:57:41.101659 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:57:41.101614    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-655cdcc48d-764cf_f1465039-5e01-4402-84a7-56d706f786a6/telemeter-client/0.log"
Sep 16 19:57:41.131635 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:57:41.131600    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-655cdcc48d-764cf_f1465039-5e01-4402-84a7-56d706f786a6/reload/0.log"
Sep 16 19:57:41.159313 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:57:41.159256    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-655cdcc48d-764cf_f1465039-5e01-4402-84a7-56d706f786a6/kube-rbac-proxy/0.log"
Sep 16 19:57:43.003438 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:57:43.003410    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-tmrkx_4c2078bb-281a-4aa1-8e38-238897865434/console-operator/2.log"
Sep 16 19:57:43.009776 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:57:43.009748    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-tmrkx_4c2078bb-281a-4aa1-8e38-238897865434/console-operator/3.log"
Sep 16 19:57:43.171745 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:57:43.171709    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-mkvvc/perf-node-gather-daemonset-9556d"]
Sep 16 19:57:43.177437 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:57:43.177402    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-mkvvc/perf-node-gather-daemonset-9556d"
Sep 16 19:57:43.185331 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:57:43.185304    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-mkvvc/perf-node-gather-daemonset-9556d"]
Sep 16 19:57:43.298660 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:57:43.298574    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/5af052a0-c4ad-4211-a1e9-6f6b35c26461-podres\") pod \"perf-node-gather-daemonset-9556d\" (UID: \"5af052a0-c4ad-4211-a1e9-6f6b35c26461\") " pod="openshift-must-gather-mkvvc/perf-node-gather-daemonset-9556d"
Sep 16 19:57:43.298660 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:57:43.298612    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-795m8\" (UniqueName: \"kubernetes.io/projected/5af052a0-c4ad-4211-a1e9-6f6b35c26461-kube-api-access-795m8\") pod \"perf-node-gather-daemonset-9556d\" (UID: \"5af052a0-c4ad-4211-a1e9-6f6b35c26461\") " pod="openshift-must-gather-mkvvc/perf-node-gather-daemonset-9556d"
Sep 16 19:57:43.298868 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:57:43.298711    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/5af052a0-c4ad-4211-a1e9-6f6b35c26461-lib-modules\") pod \"perf-node-gather-daemonset-9556d\" (UID: \"5af052a0-c4ad-4211-a1e9-6f6b35c26461\") " pod="openshift-must-gather-mkvvc/perf-node-gather-daemonset-9556d"
Sep 16 19:57:43.298868 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:57:43.298768    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5af052a0-c4ad-4211-a1e9-6f6b35c26461-sys\") pod \"perf-node-gather-daemonset-9556d\" (UID: \"5af052a0-c4ad-4211-a1e9-6f6b35c26461\") " pod="openshift-must-gather-mkvvc/perf-node-gather-daemonset-9556d"
Sep 16 19:57:43.298868 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:57:43.298794    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/5af052a0-c4ad-4211-a1e9-6f6b35c26461-proc\") pod \"perf-node-gather-daemonset-9556d\" (UID: \"5af052a0-c4ad-4211-a1e9-6f6b35c26461\") " pod="openshift-must-gather-mkvvc/perf-node-gather-daemonset-9556d"
Sep 16 19:57:43.400180 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:57:43.399846    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/5af052a0-c4ad-4211-a1e9-6f6b35c26461-podres\") pod \"perf-node-gather-daemonset-9556d\" (UID: \"5af052a0-c4ad-4211-a1e9-6f6b35c26461\") " pod="openshift-must-gather-mkvvc/perf-node-gather-daemonset-9556d"
Sep 16 19:57:43.400180 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:57:43.399905    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-795m8\" (UniqueName: \"kubernetes.io/projected/5af052a0-c4ad-4211-a1e9-6f6b35c26461-kube-api-access-795m8\") pod \"perf-node-gather-daemonset-9556d\" (UID: \"5af052a0-c4ad-4211-a1e9-6f6b35c26461\") " pod="openshift-must-gather-mkvvc/perf-node-gather-daemonset-9556d"
Sep 16 19:57:43.400180 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:57:43.399956    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/5af052a0-c4ad-4211-a1e9-6f6b35c26461-lib-modules\") pod \"perf-node-gather-daemonset-9556d\" (UID: \"5af052a0-c4ad-4211-a1e9-6f6b35c26461\") " pod="openshift-must-gather-mkvvc/perf-node-gather-daemonset-9556d"
Sep 16 19:57:43.400180 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:57:43.400018    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5af052a0-c4ad-4211-a1e9-6f6b35c26461-sys\") pod \"perf-node-gather-daemonset-9556d\" (UID: \"5af052a0-c4ad-4211-a1e9-6f6b35c26461\") " pod="openshift-must-gather-mkvvc/perf-node-gather-daemonset-9556d"
Sep 16 19:57:43.400180 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:57:43.400049    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/5af052a0-c4ad-4211-a1e9-6f6b35c26461-proc\") pod \"perf-node-gather-daemonset-9556d\" (UID: \"5af052a0-c4ad-4211-a1e9-6f6b35c26461\") " pod="openshift-must-gather-mkvvc/perf-node-gather-daemonset-9556d"
Sep 16 19:57:43.400180 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:57:43.400076    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/5af052a0-c4ad-4211-a1e9-6f6b35c26461-podres\") pod \"perf-node-gather-daemonset-9556d\" (UID: \"5af052a0-c4ad-4211-a1e9-6f6b35c26461\") " pod="openshift-must-gather-mkvvc/perf-node-gather-daemonset-9556d"
Sep 16 19:57:43.400180 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:57:43.400137    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5af052a0-c4ad-4211-a1e9-6f6b35c26461-sys\") pod \"perf-node-gather-daemonset-9556d\" (UID: \"5af052a0-c4ad-4211-a1e9-6f6b35c26461\") " pod="openshift-must-gather-mkvvc/perf-node-gather-daemonset-9556d"
Sep 16 19:57:43.400180 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:57:43.400142    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/5af052a0-c4ad-4211-a1e9-6f6b35c26461-proc\") pod \"perf-node-gather-daemonset-9556d\" (UID: \"5af052a0-c4ad-4211-a1e9-6f6b35c26461\") " pod="openshift-must-gather-mkvvc/perf-node-gather-daemonset-9556d"
Sep 16 19:57:43.400743 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:57:43.400294    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/5af052a0-c4ad-4211-a1e9-6f6b35c26461-lib-modules\") pod \"perf-node-gather-daemonset-9556d\" (UID: \"5af052a0-c4ad-4211-a1e9-6f6b35c26461\") " pod="openshift-must-gather-mkvvc/perf-node-gather-daemonset-9556d"
Sep 16 19:57:43.409121 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:57:43.409091    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-795m8\" (UniqueName: \"kubernetes.io/projected/5af052a0-c4ad-4211-a1e9-6f6b35c26461-kube-api-access-795m8\") pod \"perf-node-gather-daemonset-9556d\" (UID: \"5af052a0-c4ad-4211-a1e9-6f6b35c26461\") " pod="openshift-must-gather-mkvvc/perf-node-gather-daemonset-9556d"
Sep 16 19:57:43.489505 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:57:43.489470    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-mkvvc/perf-node-gather-daemonset-9556d"
Sep 16 19:57:43.785221 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:57:43.785192    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-mkvvc/perf-node-gather-daemonset-9556d"]
Sep 16 19:57:43.787571 ip-10-0-135-119 kubenswrapper[2578]: W0916 19:57:43.787534    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod5af052a0_c4ad_4211_a1e9_6f6b35c26461.slice/crio-b4c60cbf244282f6b889611756491a42b6a4f30052a62af01ce19283ece4c70e WatchSource:0}: Error finding container b4c60cbf244282f6b889611756491a42b6a4f30052a62af01ce19283ece4c70e: Status 404 returned error can't find the container with id b4c60cbf244282f6b889611756491a42b6a4f30052a62af01ce19283ece4c70e
Sep 16 19:57:44.737545 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:57:44.737511    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-mkvvc/perf-node-gather-daemonset-9556d" event={"ID":"5af052a0-c4ad-4211-a1e9-6f6b35c26461","Type":"ContainerStarted","Data":"a8753f7970a038637de2b33908f119469bf3fbaaea1290bd6a9b5e54e3d80773"}
Sep 16 19:57:44.738021 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:57:44.737566    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-mkvvc/perf-node-gather-daemonset-9556d" event={"ID":"5af052a0-c4ad-4211-a1e9-6f6b35c26461","Type":"ContainerStarted","Data":"b4c60cbf244282f6b889611756491a42b6a4f30052a62af01ce19283ece4c70e"}
Sep 16 19:57:44.738021 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:57:44.737635    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-mkvvc/perf-node-gather-daemonset-9556d"
Sep 16 19:57:44.756206 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:57:44.756151    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-mkvvc/perf-node-gather-daemonset-9556d" podStartSLOduration=1.756130106 podStartE2EDuration="1.756130106s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 19:57:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:57:44.753399585 +0000 UTC m=+4645.819891086" watchObservedRunningTime="2026-09-16 19:57:44.756130106 +0000 UTC m=+4645.822621594"
Sep 16 19:57:45.367969 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:57:45.367941    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-vpkvv_0ef1438e-2929-4fc5-9437-6e5b97e79006/dns/0.log"
Sep 16 19:57:45.392261 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:57:45.392226    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-vpkvv_0ef1438e-2929-4fc5-9437-6e5b97e79006/kube-rbac-proxy/0.log"
Sep 16 19:57:45.421629 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:57:45.421604    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-5rrj4_fc3365ef-dfd3-45fe-b80f-937949d65751/dns-node-resolver/0.log"
Sep 16 19:57:46.128396 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:57:46.128358    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-bl65n_454c7c51-a475-44f6-be51-1a8f1d29beaf/node-ca/0.log"
Sep 16 19:57:46.914607 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:57:46.914566    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f69wc_f4653546-2f57-4858-a038-9fd7c2139d56/konnectivity-agent/0.log" limit=1024
Sep 16 19:57:48.057746 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:57:48.057706    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-c42vj_ccaf4d24-a69d-4925-9f2a-8f77d2d3a16d/serve-healthcheck-canary/0.log"
Sep 16 19:57:48.624815 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:57:48.624781    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-5844b856db-mp7tt_cd2b7515-5e1e-429d-9c17-32a37b492b26/insights-operator/0.log"
Sep 16 19:57:48.625414 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:57:48.625369    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-5844b856db-mp7tt_cd2b7515-5e1e-429d-9c17-32a37b492b26/insights-operator/1.log"
Sep 16 19:57:48.649712 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:57:48.649681    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-9kn7r_2e9f2573-3279-4102-9777-5ff8f209ed6a/kube-rbac-proxy/0.log"
Sep 16 19:57:48.674449 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:57:48.674422    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-9kn7r_2e9f2573-3279-4102-9777-5ff8f209ed6a/exporter/0.log"
Sep 16 19:57:48.700902 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:57:48.700869    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-9kn7r_2e9f2573-3279-4102-9777-5ff8f209ed6a/extractor/0.log"
Sep 16 19:57:50.759958 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:57:50.759926    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-mkvvc/perf-node-gather-daemonset-9556d"
Sep 16 19:57:51.945387 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:57:51.945355    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-846d5d4569-4tqdv_339be3f7-ae11-4ab9-8a58-7aff3f24da03/manager/0.log"
Sep 16 19:57:52.001683 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:57:52.001650    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_openshift-lws-operator-fd8ccff4c-r9d6z_7f7087fe-a489-4aee-8e5e-c6b21fb64f8b/openshift-lws-operator/0.log"
Sep 16 19:57:53.337962 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:57:53.337870    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_llmisvc-controller-manager-6878c65f99-xl6qv_28083719-d2d0-41dd-93e2-08a91886fd0c/manager/0.log"
Sep 16 19:57:54.564785 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:57:54.564730    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_odh-model-controller-7b966d7984-svwvp_0e4010eb-b58d-43e3-a2ce-c2717edaf822/manager/0.log"
Sep 16 19:57:54.656339 ip-10-0-135-119 kubenswrapper[2578]: I0916 19:57:54.656301    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_seaweedfs-64568bcd49-sm5gv_b930538f-7cf6-4a9d-a43f-d96aa93595dc/seaweedfs/0.log"