Oct 30 02:52:46.892466 ip-10-0-143-156 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Oct 30 02:52:46.892475 ip-10-0-143-156 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Oct 30 02:52:46.892482 ip-10-0-143-156 systemd[1]: kubelet.service: Failed with result 'resources'.
Oct 30 02:52:46.892702 ip-10-0-143-156 systemd[1]: Failed to start Kubernetes Kubelet.
Oct 30 02:52:58.204304 ip-10-0-143-156 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found.
Oct 30 02:52:58.204323 ip-10-0-143-156 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot 05b7e5fcb59f440eac9bb26ec188cc4d --
Oct 30 02:54:51.459406 ip-10-0-143-156 systemd[1]: Starting Kubernetes Kubelet...
Oct 30 02:54:51.866762 ip-10-0-143-156 kubenswrapper[2574]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 02:54:51.866762 ip-10-0-143-156 kubenswrapper[2574]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Oct 30 02:54:51.866762 ip-10-0-143-156 kubenswrapper[2574]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 02:54:51.866762 ip-10-0-143-156 kubenswrapper[2574]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 02:54:51.871022 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.870936    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 02:54:51.871022 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.871023    2574 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 02:54:51.871105 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.871029    2574 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 02:54:51.871105 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.871033    2574 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 02:54:51.871105 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.871036    2574 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 02:54:51.871105 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.871040    2574 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 02:54:51.871105 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.871043    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 02:54:51.871105 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.871045    2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 02:54:51.871105 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.871048    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 02:54:51.871105 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.871051    2574 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 02:54:51.871105 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.871054    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 02:54:51.871105 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.871056    2574 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 02:54:51.871105 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.871059    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 02:54:51.871105 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.871062    2574 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 02:54:51.871105 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.871065    2574 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 02:54:51.871105 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.871067    2574 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 02:54:51.871105 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.871070    2574 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 02:54:51.871105 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.871072    2574 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 02:54:51.871105 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.871075    2574 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 02:54:51.871105 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.871078    2574 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 02:54:51.871105 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.871080    2574 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 02:54:51.871541 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.871083    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 02:54:51.871541 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.871086    2574 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 02:54:51.871541 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.871088    2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 02:54:51.871541 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.871091    2574 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 02:54:51.871541 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.871105    2574 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 02:54:51.871541 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.871109    2574 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 02:54:51.871541 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.871117    2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 02:54:51.871541 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.871123    2574 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 02:54:51.871541 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.871129    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 02:54:51.871541 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.871132    2574 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 02:54:51.871541 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.871135    2574 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 02:54:51.871541 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.871138    2574 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 02:54:51.871541 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.871140    2574 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 02:54:51.871541 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.871142    2574 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 02:54:51.871541 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.871145    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 02:54:51.871541 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.871148    2574 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 02:54:51.871541 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.871150    2574 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 02:54:51.871541 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.871153    2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 02:54:51.871541 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.871155    2574 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 02:54:51.871541 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.871158    2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 02:54:51.872013 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.871160    2574 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 02:54:51.872013 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.871163    2574 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 02:54:51.872013 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.871165    2574 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 02:54:51.872013 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.871168    2574 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 02:54:51.872013 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.871170    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 02:54:51.872013 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.871173    2574 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 02:54:51.872013 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.871175    2574 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 02:54:51.872013 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.871177    2574 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 02:54:51.872013 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.871180    2574 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 02:54:51.872013 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.871182    2574 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 02:54:51.872013 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.871186    2574 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 02:54:51.872013 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.871189    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 02:54:51.872013 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.871191    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 02:54:51.872013 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.871195    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 02:54:51.872013 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.871197    2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 02:54:51.872013 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.871200    2574 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 02:54:51.872013 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.871202    2574 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 02:54:51.872013 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.871205    2574 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 02:54:51.872013 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.871207    2574 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 02:54:51.872013 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.871210    2574 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 02:54:51.872497 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.871212    2574 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 02:54:51.872497 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.871215    2574 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 02:54:51.872497 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.871217    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 02:54:51.872497 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.871220    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 02:54:51.872497 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.871223    2574 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 02:54:51.872497 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.871225    2574 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 02:54:51.872497 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.871228    2574 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 02:54:51.872497 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.871231    2574 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 02:54:51.872497 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.871233    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 02:54:51.872497 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.871236    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 02:54:51.872497 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.871238    2574 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 02:54:51.872497 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.871241    2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 02:54:51.872497 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.871243    2574 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 02:54:51.872497 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.871246    2574 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 02:54:51.872497 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.871248    2574 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 02:54:51.872497 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.871250    2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 02:54:51.872497 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.871253    2574 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 02:54:51.872497 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.871255    2574 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 02:54:51.872497 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.871258    2574 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 02:54:51.872497 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.871260    2574 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 02:54:51.872972 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.871263    2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 02:54:51.872972 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.871265    2574 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 02:54:51.872972 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.871269    2574 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 02:54:51.872972 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.871272    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 02:54:51.872972 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.871274    2574 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 02:54:51.872972 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.871277    2574 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 02:54:51.872972 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.871280    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 02:54:51.872972 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.871283    2574 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 02:54:51.872972 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.871285    2574 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 02:54:51.872972 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.871288    2574 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 02:54:51.872972 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.871290    2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 02:54:51.872972 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.871293    2574 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 02:54:51.872972 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.871296    2574 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 02:54:51.872972 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.871298    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 02:54:51.872972 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.871301    2574 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 02:54:51.872972 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.871303    2574 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 02:54:51.872972 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.871306    2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 02:54:51.872972 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.871308    2574 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 02:54:51.872972 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.871311    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 02:54:51.873445 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.871313    2574 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 02:54:51.873445 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.871316    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 02:54:51.873445 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.871318    2574 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 02:54:51.873445 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.871321    2574 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 02:54:51.873445 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.871324    2574 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 02:54:51.873445 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.871327    2574 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 02:54:51.874199 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.874187    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 02:54:51.874199 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.874199    2574 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 02:54:51.874258 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.874203    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 02:54:51.874258 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.874207    2574 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 02:54:51.874258 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.874209    2574 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 02:54:51.874258 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.874212    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 02:54:51.874258 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.874215    2574 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 02:54:51.874258 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.874218    2574 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 02:54:51.874258 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.874221    2574 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 02:54:51.874258 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.874223    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 02:54:51.874258 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.874226    2574 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 02:54:51.874258 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.874230    2574 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 02:54:51.874258 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.874232    2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 02:54:51.874258 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.874235    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 02:54:51.874258 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.874237    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 02:54:51.874258 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.874240    2574 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 02:54:51.874258 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.874243    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 02:54:51.874258 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.874246    2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 02:54:51.874258 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.874248    2574 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 02:54:51.874258 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.874251    2574 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 02:54:51.874258 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.874253    2574 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 02:54:51.874726 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.874256    2574 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 02:54:51.874726 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.874259    2574 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 02:54:51.874726 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.874262    2574 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 02:54:51.874726 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.874265    2574 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 02:54:51.874726 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.874268    2574 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 02:54:51.874726 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.874271    2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 02:54:51.874726 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.874273    2574 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 02:54:51.874726 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.874276    2574 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 02:54:51.874726 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.874279    2574 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 02:54:51.874726 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.874281    2574 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 02:54:51.874726 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.874284    2574 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 02:54:51.874726 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.874286    2574 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 02:54:51.874726 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.874289    2574 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 02:54:51.874726 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.874291    2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 02:54:51.874726 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.874294    2574 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 02:54:51.874726 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.874296    2574 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 02:54:51.874726 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.874299    2574 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 02:54:51.874726 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.874302    2574 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 02:54:51.874726 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.874304    2574 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 02:54:51.874726 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.874307    2574 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 02:54:51.875212 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.874309    2574 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 02:54:51.875212 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.874311    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 02:54:51.875212 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.874314    2574 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 02:54:51.875212 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.874317    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 02:54:51.875212 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.874320    2574 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 02:54:51.875212 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.874322    2574 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 02:54:51.875212 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.874324    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 02:54:51.875212 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.874327    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 02:54:51.875212 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.874330    2574 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 02:54:51.875212 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.874333    2574 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 02:54:51.875212 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.874336    2574 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 02:54:51.875212 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.874339    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 02:54:51.875212 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.874343    2574 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 02:54:51.875212 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.874347    2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 02:54:51.875212 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.874351    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 02:54:51.875212 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.874354    2574 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 02:54:51.875212 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.874357    2574 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 02:54:51.875212 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.874359    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 02:54:51.875212 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.874362    2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 02:54:51.875699 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.874364    2574 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 02:54:51.875699 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.874367    2574 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 02:54:51.875699 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.874370    2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 02:54:51.875699 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.874372    2574 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 02:54:51.875699 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.874375    2574 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 02:54:51.875699 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.874378    2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 02:54:51.875699 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.874380    2574 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 02:54:51.875699 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.874383    2574 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 02:54:51.875699 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.874385    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 02:54:51.875699 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.874388    2574 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 02:54:51.875699 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.874390    2574 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 02:54:51.875699 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.874393    2574 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 02:54:51.875699 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.874396    2574 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 02:54:51.875699 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.874398    2574 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 02:54:51.875699 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.874401    2574 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 02:54:51.875699 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.874403    2574 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 02:54:51.875699 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.874406    2574 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 02:54:51.875699 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.874409    2574 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 02:54:51.875699 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.874412    2574 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 02:54:51.875699 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.874415    2574 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 02:54:51.876180 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.874417    2574 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 02:54:51.876180 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.874420    2574 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 02:54:51.876180 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.874423    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 02:54:51.876180 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.874425    2574 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 02:54:51.876180 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.874428    2574 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 02:54:51.876180 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.874431    2574 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 02:54:51.876180 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.874434    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 02:54:51.876180 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.874436    2574 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 02:54:51.876180 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.874439    2574 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 02:54:51.876180 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.874442    2574 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 02:54:51.876180 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.874444    2574 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 02:54:51.876180 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.874446    2574 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 02:54:51.876180 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.874449    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 02:54:51.876180 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.874451    2574 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 02:54:51.876180 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.874454    2574 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 02:54:51.876180 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.874456    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 02:54:51.876180 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.874459    2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 02:54:51.876180 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.874464    2574 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 02:54:51.876180 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.874468    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 02:54:51.876668 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.874471    2574 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 02:54:51.876668 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.874473    2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 02:54:51.876668 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.874476    2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 02:54:51.876668 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.874478    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 02:54:51.876668 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.874481    2574 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 02:54:51.876668 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.874483    2574 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 02:54:51.876668 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.874486    2574 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 02:54:51.876668 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.874971    2574 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 02:54:51.876668 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.874976    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 02:54:51.876668 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.874979    2574 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 02:54:51.876668 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.874982    2574 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 02:54:51.876668 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.874985    2574 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 02:54:51.876668 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.874988    2574 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 02:54:51.876668 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.874991    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 02:54:51.876668 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.874995    2574 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 02:54:51.876668 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.874998    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 02:54:51.876668 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.875001    2574 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 02:54:51.876668 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.875003    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 02:54:51.876668 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.875006    2574 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 02:54:51.877138 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.875009    2574 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 02:54:51.877138 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.875012    2574 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 02:54:51.877138 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.875014    2574 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 02:54:51.877138 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.875017    2574 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 02:54:51.877138 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.875020    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 02:54:51.877138 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.875023    2574 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 02:54:51.877138 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.875025    2574 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 02:54:51.877138 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.875028    2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 02:54:51.877138 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.875030    2574 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 02:54:51.877138 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.875033    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 02:54:51.877138 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.875035    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 02:54:51.877138 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.875038    2574 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 02:54:51.877138 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.875040    2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 02:54:51.877138 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.875043    2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 02:54:51.877138 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.875045    2574 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 02:54:51.877138 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.875048    2574 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 02:54:51.877138 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.875050    2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 02:54:51.877138 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.875053    2574 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 02:54:51.877138 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.875055    2574 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 02:54:51.877138 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.875058    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 02:54:51.877622 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.875060    2574 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 02:54:51.877622 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.875063    2574 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 02:54:51.877622 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.875065    2574 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 02:54:51.877622 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.875067    2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 02:54:51.877622 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.875070    2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 02:54:51.877622 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.875074    2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 02:54:51.877622 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.875076    2574 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 02:54:51.877622 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.875079    2574 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 02:54:51.877622 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.875082    2574 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 02:54:51.877622 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.875085    2574 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 02:54:51.877622 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.875088    2574 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 02:54:51.877622 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.875090    2574 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 02:54:51.877622 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.875106    2574 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 02:54:51.877622 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.875108    2574 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 02:54:51.877622 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.875111    2574 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 02:54:51.877622 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.875114    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 02:54:51.877622 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.875116    2574 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 02:54:51.877622 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.875119    2574 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 02:54:51.877622 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.875121    2574 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 02:54:51.877622 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.875124    2574 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 02:54:51.878135 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.875127    2574 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 02:54:51.878135 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.875131    2574 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 02:54:51.878135 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.875134    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 02:54:51.878135 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.875137    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 02:54:51.878135 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.875140    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 02:54:51.878135 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.875142    2574 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 02:54:51.878135 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.875145    2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 02:54:51.878135 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.875147    2574 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 02:54:51.878135 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.875150    2574 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 02:54:51.878135 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.875152    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 02:54:51.878135 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.875155    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 02:54:51.878135 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.875157    2574 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 02:54:51.878135 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.875160    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 02:54:51.878135 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.875162    2574 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 02:54:51.878135 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.875165    2574 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 02:54:51.878135 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.875167    2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 02:54:51.878135 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.875172    2574 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 02:54:51.878135 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.875176    2574 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 02:54:51.878135 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.875179    2574 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 02:54:51.878582 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.875183    2574 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 02:54:51.878582 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.875185    2574 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 02:54:51.878582 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.875188    2574 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 02:54:51.878582 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.875190    2574 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 02:54:51.878582 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.875193    2574 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 02:54:51.878582 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.875196    2574 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 02:54:51.878582 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.875198    2574 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 02:54:51.878582 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.875201    2574 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 02:54:51.878582 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.875203    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 02:54:51.878582 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.875206    2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 02:54:51.878582 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.875208    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 02:54:51.878582 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.875211    2574 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 02:54:51.878582 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.875213    2574 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 02:54:51.878582 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.875216    2574 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 02:54:51.878582 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.875219    2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 02:54:51.878582 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.875221    2574 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 02:54:51.878582 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.875224    2574 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 02:54:51.878582 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.875227    2574 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 02:54:51.878582 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.875230    2574 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 02:54:51.878582 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.875232    2574 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 02:54:51.879042 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.875235    2574 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 02:54:51.879042 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.875237    2574 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 02:54:51.879042 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.875242    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 02:54:51.879042 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.875244    2574 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 02:54:51.879042 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.875247    2574 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 02:54:51.879042 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.875250    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 02:54:51.879042 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.875252    2574 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 02:54:51.879042 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.875255    2574 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 02:54:51.879042 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.875257    2574 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 02:54:51.879042 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.875260    2574 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 02:54:51.879042 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.875263    2574 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 02:54:51.879042 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.875267    2574 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 02:54:51.879042 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.875271    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 02:54:51.879042 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.875274    2574 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 02:54:51.879042 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.875277    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 02:54:51.879042 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:51.876242    2574 flags.go:64] FLAG: --address="0.0.0.0"
Oct 30 02:54:51.879042 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:51.876251    2574 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Oct 30 02:54:51.879042 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:51.876257    2574 flags.go:64] FLAG: --anonymous-auth="true"
Oct 30 02:54:51.879042 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:51.876262    2574 flags.go:64] FLAG: --application-metrics-count-limit="100"
Oct 30 02:54:51.879042 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:51.876267    2574 flags.go:64] FLAG: --authentication-token-webhook="false"
Oct 30 02:54:51.879551 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:51.876270    2574 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Oct 30 02:54:51.879551 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:51.876275    2574 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Oct 30 02:54:51.879551 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:51.876279    2574 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Oct 30 02:54:51.879551 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:51.876282    2574 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Oct 30 02:54:51.879551 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:51.876285    2574 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Oct 30 02:54:51.879551 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:51.876289    2574 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Oct 30 02:54:51.879551 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:51.876292    2574 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Oct 30 02:54:51.879551 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:51.876295    2574 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Oct 30 02:54:51.879551 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:51.876298    2574 flags.go:64] FLAG: --cgroup-root=""
Oct 30 02:54:51.879551 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:51.876301    2574 flags.go:64] FLAG: --cgroups-per-qos="true"
Oct 30 02:54:51.879551 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:51.876304    2574 flags.go:64] FLAG: --client-ca-file=""
Oct 30 02:54:51.879551 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:51.876307    2574 flags.go:64] FLAG: --cloud-provider="external"
Oct 30 02:54:51.879551 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:51.876309    2574 flags.go:64] FLAG: --cluster-dns="[]"
Oct 30 02:54:51.879551 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:51.876314    2574 flags.go:64] FLAG: --cluster-domain=""
Oct 30 02:54:51.879551 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:51.876317    2574 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Oct 30 02:54:51.879551 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:51.876321    2574 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Oct 30 02:54:51.879551 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:51.876324    2574 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Oct 30 02:54:51.879551 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:51.876327    2574 flags.go:64] FLAG: --container-log-max-files="5"
Oct 30 02:54:51.879551 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:51.876332    2574 flags.go:64] FLAG: --container-log-max-size="10Mi"
Oct 30 02:54:51.879551 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:51.876335    2574 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Oct 30 02:54:51.879551 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:51.876339    2574 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Oct 30 02:54:51.879551 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:51.876342    2574 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Oct 30 02:54:51.879551 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:51.876345    2574 flags.go:64] FLAG: --contention-profiling="false"
Oct 30 02:54:51.879551 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:51.876348    2574 flags.go:64] FLAG: --cpu-cfs-quota="true"
Oct 30 02:54:51.880140 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:51.876352    2574 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Oct 30 02:54:51.880140 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:51.876355    2574 flags.go:64] FLAG: --cpu-manager-policy="none"
Oct 30 02:54:51.880140 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:51.876358    2574 flags.go:64] FLAG: --cpu-manager-policy-options=""
Oct 30 02:54:51.880140 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:51.876363    2574 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Oct 30 02:54:51.880140 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:51.876366    2574 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Oct 30 02:54:51.880140 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:51.876369    2574 flags.go:64] FLAG: --enable-debugging-handlers="true"
Oct 30 02:54:51.880140 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:51.876372    2574 flags.go:64] FLAG: --enable-load-reader="false"
Oct 30 02:54:51.880140 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:51.876375    2574 flags.go:64] FLAG: --enable-server="true"
Oct 30 02:54:51.880140 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:51.876378    2574 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Oct 30 02:54:51.880140 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:51.876382    2574 flags.go:64] FLAG: --event-burst="100"
Oct 30 02:54:51.880140 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:51.876385    2574 flags.go:64] FLAG: --event-qps="50"
Oct 30 02:54:51.880140 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:51.876388    2574 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Oct 30 02:54:51.880140 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:51.876391    2574 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Oct 30 02:54:51.880140 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:51.876394    2574 flags.go:64] FLAG: --eviction-hard=""
Oct 30 02:54:51.880140 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:51.876398    2574 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Oct 30 02:54:51.880140 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:51.876401    2574 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Oct 30 02:54:51.880140 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:51.876404    2574 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Oct 30 02:54:51.880140 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:51.876407    2574 flags.go:64] FLAG: --eviction-soft=""
Oct 30 02:54:51.880140 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:51.876411    2574 flags.go:64] FLAG: --eviction-soft-grace-period=""
Oct 30 02:54:51.880140 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:51.876414    2574 flags.go:64] FLAG: --exit-on-lock-contention="false"
Oct 30 02:54:51.880140 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:51.876417    2574 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Oct 30 02:54:51.880140 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:51.876420    2574 flags.go:64] FLAG: --experimental-mounter-path=""
Oct 30 02:54:51.880140 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:51.876424    2574 flags.go:64] FLAG: --fail-cgroupv1="true"
Oct 30 02:54:51.880140 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:51.876427    2574 flags.go:64] FLAG: --fail-swap-on="true"
Oct 30 02:54:51.880140 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:51.876430    2574 flags.go:64] FLAG: --feature-gates=""
Oct 30 02:54:51.880754 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:51.876434    2574 flags.go:64] FLAG: --file-check-frequency="20s"
Oct 30 02:54:51.880754 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:51.876437    2574 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Oct 30 02:54:51.880754 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:51.876441    2574 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Oct 30 02:54:51.880754 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:51.876444    2574 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Oct 30 02:54:51.880754 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:51.876447    2574 flags.go:64] FLAG: --healthz-port="10248"
Oct 30 02:54:51.880754 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:51.876450    2574 flags.go:64] FLAG: --help="false"
Oct 30 02:54:51.880754 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:51.876453    2574 flags.go:64] FLAG: --hostname-override="ip-10-0-143-156.ec2.internal"
Oct 30 02:54:51.880754 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:51.876457    2574 flags.go:64] FLAG: --housekeeping-interval="10s"
Oct 30 02:54:51.880754 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:51.876460    2574 flags.go:64] FLAG: --http-check-frequency="20s"
Oct 30 02:54:51.880754 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:51.876463    2574 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Oct 30 02:54:51.880754 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:51.876467    2574 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Oct 30 02:54:51.880754 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:51.876470    2574 flags.go:64] FLAG: --image-gc-high-threshold="85"
Oct 30 02:54:51.880754 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:51.876473    2574 flags.go:64] FLAG: --image-gc-low-threshold="80"
Oct 30 02:54:51.880754 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:51.876476    2574 flags.go:64] FLAG: --image-service-endpoint=""
Oct 30 02:54:51.880754 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:51.876479    2574 flags.go:64] FLAG: --kernel-memcg-notification="false"
Oct 30 02:54:51.880754 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:51.876482    2574 flags.go:64] FLAG: --kube-api-burst="100"
Oct 30 02:54:51.880754 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:51.876485    2574 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Oct 30 02:54:51.880754 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:51.876488    2574 flags.go:64] FLAG: --kube-api-qps="50"
Oct 30 02:54:51.880754 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:51.876491    2574 flags.go:64] FLAG: --kube-reserved=""
Oct 30 02:54:51.880754 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:51.876494    2574 flags.go:64] FLAG: --kube-reserved-cgroup=""
Oct 30 02:54:51.880754 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:51.876496    2574 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Oct 30 02:54:51.880754 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:51.876500    2574 flags.go:64] FLAG: --kubelet-cgroups=""
Oct 30 02:54:51.880754 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:51.876502    2574 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Oct 30 02:54:51.880754 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:51.876505    2574 flags.go:64] FLAG: --lock-file=""
Oct 30 02:54:51.881350 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:51.876508    2574 flags.go:64] FLAG: --log-cadvisor-usage="false"
Oct 30 02:54:51.881350 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:51.876511    2574 flags.go:64] FLAG: --log-flush-frequency="5s"
Oct 30 02:54:51.881350 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:51.876514    2574 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Oct 30 02:54:51.881350 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:51.876520    2574 flags.go:64] FLAG: --log-json-split-stream="false"
Oct 30 02:54:51.881350 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:51.876523    2574 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Oct 30 02:54:51.881350 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:51.876526    2574 flags.go:64] FLAG: --log-text-split-stream="false"
Oct 30 02:54:51.881350 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:51.876530    2574 flags.go:64] FLAG: --logging-format="text"
Oct 30 02:54:51.881350 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:51.876533    2574 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Oct 30 02:54:51.881350 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:51.876537    2574 flags.go:64] FLAG: --make-iptables-util-chains="true"
Oct 30 02:54:51.881350 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:51.876540    2574 flags.go:64] FLAG: --manifest-url=""
Oct 30 02:54:51.881350 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:51.876542    2574 flags.go:64] FLAG: --manifest-url-header=""
Oct 30 02:54:51.881350 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:51.876547    2574 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Oct 30 02:54:51.881350 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:51.876550    2574 flags.go:64] FLAG: --max-open-files="1000000"
Oct 30 02:54:51.881350 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:51.876554    2574 flags.go:64] FLAG: --max-pods="110"
Oct 30 02:54:51.881350 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:51.876557    2574 flags.go:64] FLAG: --maximum-dead-containers="-1"
Oct 30 02:54:51.881350 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:51.876560    2574 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Oct 30 02:54:51.881350 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:51.876563    2574 flags.go:64] FLAG: --memory-manager-policy="None"
Oct 30 02:54:51.881350 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:51.876566    2574 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Oct 30 02:54:51.881350 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:51.876569    2574 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Oct 30 02:54:51.881350 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:51.876573    2574 flags.go:64] FLAG: --node-ip="0.0.0.0"
Oct 30 02:54:51.881350 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:51.876575    2574 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Oct 30 02:54:51.881350 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:51.876583    2574 flags.go:64] FLAG: --node-status-max-images="50"
Oct 30 02:54:51.881350 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:51.876586    2574 flags.go:64] FLAG: --node-status-update-frequency="10s"
Oct 30 02:54:51.881350 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:51.876589    2574 flags.go:64] FLAG: --oom-score-adj="-999"
Oct 30 02:54:51.881350 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:51.876593    2574 flags.go:64] FLAG: --pod-cidr=""
Oct 30 02:54:51.882044 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:51.876596    2574 flags.go:64] FLAG: --pod-manifest-path=""
Oct 30 02:54:51.882044 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:51.876599    2574 flags.go:64] FLAG: --pod-max-pids="-1"
Oct 30 02:54:51.882044 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:51.876602    2574 flags.go:64] FLAG: --pods-per-core="0"
Oct 30 02:54:51.882044 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:51.876605    2574 flags.go:64] FLAG: --port="10250"
Oct 30 02:54:51.882044 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:51.876607    2574 flags.go:64] FLAG: --protect-kernel-defaults="false"
Oct 30 02:54:51.882044 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:51.876610    2574 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0ab09b2952fca25ca"
Oct 30 02:54:51.882044 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:51.876613    2574 flags.go:64] FLAG: --qos-reserved=""
Oct 30 02:54:51.882044 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:51.876616    2574 flags.go:64] FLAG: --read-only-port="10255"
Oct 30 02:54:51.882044 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:51.876619    2574 flags.go:64] FLAG: --register-node="true"
Oct 30 02:54:51.882044 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:51.876622    2574 flags.go:64] FLAG: --register-with-taints=""
Oct 30 02:54:51.882044 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:51.876626    2574 flags.go:64] FLAG: --registry-burst="10"
Oct 30 02:54:51.882044 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:51.876629    2574 flags.go:64] FLAG: --registry-qps="5"
Oct 30 02:54:51.882044 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:51.876632    2574 flags.go:64] FLAG: --reserved-cpus=""
Oct 30 02:54:51.882044 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:51.876637    2574 flags.go:64] FLAG: --reserved-memory=""
Oct 30 02:54:51.882044 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:51.876641    2574 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Oct 30 02:54:51.882044 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:51.876644    2574 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Oct 30 02:54:51.882044 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:51.876647    2574 flags.go:64] FLAG: --rotate-certificates="false"
Oct 30 02:54:51.882044 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:51.876650    2574 flags.go:64] FLAG: --rotate-server-certificates="false"
Oct 30 02:54:51.882044 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:51.876653    2574 flags.go:64] FLAG: --runonce="false"
Oct 30 02:54:51.882044 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:51.876656    2574 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Oct 30 02:54:51.882044 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:51.876659    2574 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Oct 30 02:54:51.882044 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:51.876662    2574 flags.go:64] FLAG: --seccomp-default="false"
Oct 30 02:54:51.882044 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:51.876665    2574 flags.go:64] FLAG: --serialize-image-pulls="true"
Oct 30 02:54:51.882044 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:51.876668    2574 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Oct 30 02:54:51.882044 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:51.876672    2574 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Oct 30 02:54:51.882044 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:51.876676    2574 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Oct 30 02:54:51.882709 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:51.876679    2574 flags.go:64] FLAG: --storage-driver-password="root"
Oct 30 02:54:51.882709 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:51.876682    2574 flags.go:64] FLAG: --storage-driver-secure="false"
Oct 30 02:54:51.882709 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:51.876685    2574 flags.go:64] FLAG: --storage-driver-table="stats"
Oct 30 02:54:51.882709 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:51.876689    2574 flags.go:64] FLAG: --storage-driver-user="root"
Oct 30 02:54:51.882709 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:51.876691    2574 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Oct 30 02:54:51.882709 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:51.876694    2574 flags.go:64] FLAG: --sync-frequency="1m0s"
Oct 30 02:54:51.882709 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:51.876697    2574 flags.go:64] FLAG: --system-cgroups=""
Oct 30 02:54:51.882709 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:51.876700    2574 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Oct 30 02:54:51.882709 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:51.876706    2574 flags.go:64] FLAG: --system-reserved-cgroup=""
Oct 30 02:54:51.882709 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:51.876709    2574 flags.go:64] FLAG: --tls-cert-file=""
Oct 30 02:54:51.882709 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:51.876712    2574 flags.go:64] FLAG: --tls-cipher-suites="[]"
Oct 30 02:54:51.882709 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:51.876716    2574 flags.go:64] FLAG: --tls-min-version=""
Oct 30 02:54:51.882709 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:51.876719    2574 flags.go:64] FLAG: --tls-private-key-file=""
Oct 30 02:54:51.882709 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:51.876721    2574 flags.go:64] FLAG: --topology-manager-policy="none"
Oct 30 02:54:51.882709 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:51.876724    2574 flags.go:64] FLAG: --topology-manager-policy-options=""
Oct 30 02:54:51.882709 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:51.876727    2574 flags.go:64] FLAG: --topology-manager-scope="container"
Oct 30 02:54:51.882709 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:51.876730    2574 flags.go:64] FLAG: --v="2"
Oct 30 02:54:51.882709 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:51.876736    2574 flags.go:64] FLAG: --version="false"
Oct 30 02:54:51.882709 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:51.876741    2574 flags.go:64] FLAG: --vmodule=""
Oct 30 02:54:51.882709 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:51.876746    2574 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Oct 30 02:54:51.882709 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:51.876749    2574 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Oct 30 02:54:51.882709 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.876886    2574 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 02:54:51.882709 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.876890    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 02:54:51.882709 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.876895    2574 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 02:54:51.883319 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.876899    2574 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 02:54:51.883319 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.876902    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 02:54:51.883319 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.876904    2574 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 02:54:51.883319 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.876907    2574 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 02:54:51.883319 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.876910    2574 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 02:54:51.883319 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.876912    2574 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 02:54:51.883319 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.876915    2574 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 02:54:51.883319 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.876918    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 02:54:51.883319 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.876921    2574 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 02:54:51.883319 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.876924    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 02:54:51.883319 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.876926    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 02:54:51.883319 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.876929    2574 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 02:54:51.883319 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.876931    2574 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 02:54:51.883319 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.876934    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 02:54:51.883319 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.876936    2574 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 02:54:51.883319 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.876939    2574 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 02:54:51.883319 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.876941    2574 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 02:54:51.883319 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.876944    2574 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 02:54:51.883319 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.876946    2574 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 02:54:51.883319 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.876949    2574 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 02:54:51.883803 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.876952    2574 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 02:54:51.883803 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.876954    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 02:54:51.883803 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.876957    2574 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 02:54:51.883803 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.876961    2574 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 02:54:51.883803 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.876964    2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 02:54:51.883803 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.876969    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 02:54:51.883803 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.876972    2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 02:54:51.883803 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.876976    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 02:54:51.883803 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.876979    2574 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 02:54:51.883803 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.876982    2574 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 02:54:51.883803 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.876985    2574 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 02:54:51.883803 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.876988    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 02:54:51.883803 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.876991    2574 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 02:54:51.883803 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.876993    2574 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 02:54:51.883803 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.876996    2574 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 02:54:51.883803 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.876999    2574 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 02:54:51.883803 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.877002    2574 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 02:54:51.883803 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.877004    2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 02:54:51.883803 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.877007    2574 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 02:54:51.884268 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.877009    2574 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 02:54:51.884268 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.877012    2574 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 02:54:51.884268 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.877015    2574 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 02:54:51.884268 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.877017    2574 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 02:54:51.884268 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.877020    2574 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 02:54:51.884268 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.877023    2574 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 02:54:51.884268 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.877025    2574 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 02:54:51.884268 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.877028    2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 02:54:51.884268 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.877030    2574 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 02:54:51.884268 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.877033    2574 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 02:54:51.884268 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.877035    2574 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 02:54:51.884268 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.877038    2574 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 02:54:51.884268 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.877040    2574 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 02:54:51.884268 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.877043    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 02:54:51.884268 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.877046    2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 02:54:51.884268 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.877049    2574 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 02:54:51.884268 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.877051    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 02:54:51.884268 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.877054    2574 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 02:54:51.884268 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.877058    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 02:54:51.884268 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.877061    2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 02:54:51.884743 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.877065    2574 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 02:54:51.884743 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.877067    2574 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 02:54:51.884743 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.877070    2574 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 02:54:51.884743 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.877072    2574 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 02:54:51.884743 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.877075    2574 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 02:54:51.884743 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.877078    2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 02:54:51.884743 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.877080    2574 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 02:54:51.884743 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.877083    2574 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 02:54:51.884743 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.877085    2574 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 02:54:51.884743 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.877088    2574 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 02:54:51.884743 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.877091    2574 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 02:54:51.884743 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.877104    2574 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 02:54:51.884743 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.877107    2574 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 02:54:51.884743 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.877110    2574 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 02:54:51.884743 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.877113    2574 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 02:54:51.884743 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.877116    2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 02:54:51.884743 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.877118    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 02:54:51.884743 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.877121    2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 02:54:51.884743 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.877124    2574 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 02:54:51.885470 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.877126    2574 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 02:54:51.885470 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.877129    2574 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 02:54:51.885470 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.877132    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 02:54:51.885470 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.877134    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 02:54:51.885470 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.877137    2574 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 02:54:51.885470 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.877140    2574 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 02:54:51.885470 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.877142    2574 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 02:54:51.885470 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.877145    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 02:54:51.885470 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.877148    2574 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 02:54:51.885470 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.877151    2574 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 02:54:51.885470 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.877154    2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 02:54:51.885470 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.877158    2574 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 02:54:51.885470 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.877161    2574 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 02:54:51.885470 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.877166    2574 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 02:54:51.885470 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.877168    2574 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 02:54:51.885470 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.877171    2574 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 02:54:51.885470 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.877174    2574 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 02:54:51.885470 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.877176    2574 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 02:54:51.885470 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.877179    2574 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 02:54:51.885470 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.877181    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 02:54:51.886235 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.877184    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 02:54:51.886235 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.877187    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 02:54:51.886235 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.877189    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 02:54:51.886235 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.877192    2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 02:54:51.886235 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.877194    2574 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 02:54:51.886235 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:51.877221    2574 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 02:54:51.886235 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.877262    2574 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 02:54:51.886235 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.877267    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 02:54:51.886235 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.877270    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 02:54:51.886235 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.877274    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 02:54:51.886235 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.877277    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 02:54:51.886235 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.877279    2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 02:54:51.886235 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.877282    2574 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 02:54:51.886235 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.877285    2574 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 02:54:51.886235 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.877288    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 02:54:51.886737 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.877292    2574 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 02:54:51.886737 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.877295    2574 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 02:54:51.886737 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.877298    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 02:54:51.886737 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.877301    2574 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 02:54:51.886737 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.877304    2574 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 02:54:51.886737 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.877306    2574 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 02:54:51.886737 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.877309    2574 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 02:54:51.886737 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.877311    2574 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 02:54:51.886737 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.877315    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 02:54:51.886737 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.877318    2574 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 02:54:51.886737 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.877321    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 02:54:51.886737 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.877326    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 02:54:51.886737 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.877329    2574 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 02:54:51.886737 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.877332    2574 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 02:54:51.886737 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.877334    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 02:54:51.886737 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.877337    2574 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 02:54:51.886737 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.877339    2574 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 02:54:51.886737 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.877342    2574 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 02:54:51.886737 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.877345    2574 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 02:54:51.886737 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.877347    2574 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 02:54:51.887296 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.877349    2574 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 02:54:51.887296 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.877352    2574 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 02:54:51.887296 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.877355    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 02:54:51.887296 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.877357    2574 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 02:54:51.887296 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.877361    2574 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 02:54:51.887296 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.877364    2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 02:54:51.887296 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.877368    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 02:54:51.887296 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.877371    2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 02:54:51.887296 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.877374    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 02:54:51.887296 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.877377    2574 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 02:54:51.887296 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.877379    2574 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 02:54:51.887296 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.877382    2574 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 02:54:51.887296 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.877385    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 02:54:51.887296 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.877387    2574 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 02:54:51.887296 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.877390    2574 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 02:54:51.887296 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.877401    2574 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 02:54:51.887296 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.877404    2574 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 02:54:51.887296 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.877407    2574 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 02:54:51.887296 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.877409    2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 02:54:51.887739 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.877412    2574 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 02:54:51.887739 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.877416    2574 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 02:54:51.887739 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.877418    2574 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 02:54:51.887739 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.877421    2574 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 02:54:51.887739 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.877425    2574 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 02:54:51.887739 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.877427    2574 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 02:54:51.887739 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.877430    2574 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 02:54:51.887739 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.877433    2574 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 02:54:51.887739 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.877435    2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 02:54:51.887739 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.877438    2574 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 02:54:51.887739 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.877440    2574 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 02:54:51.887739 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.877443    2574 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 02:54:51.887739 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.877446    2574 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 02:54:51.887739 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.877448    2574 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 02:54:51.887739 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.877452    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 02:54:51.887739 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.877454    2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 02:54:51.887739 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.877457    2574 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 02:54:51.887739 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.877459    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 02:54:51.887739 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.877462    2574 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 02:54:51.887739 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.877465    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 02:54:51.888225 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.877467    2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 02:54:51.888225 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.877471    2574 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 02:54:51.888225 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.877473    2574 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 02:54:51.888225 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.877476    2574 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 02:54:51.888225 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.877479    2574 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 02:54:51.888225 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.877481    2574 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 02:54:51.888225 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.877484    2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 02:54:51.888225 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.877487    2574 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 02:54:51.888225 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.877489    2574 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 02:54:51.888225 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.877492    2574 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 02:54:51.888225 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.877494    2574 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 02:54:51.888225 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.877497    2574 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 02:54:51.888225 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.877500    2574 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 02:54:51.888225 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.877504    2574 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 02:54:51.888225 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.877506    2574 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 02:54:51.888225 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.877509    2574 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 02:54:51.888225 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.877513    2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 02:54:51.888225 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.877516    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 02:54:51.888225 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.877519    2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 02:54:51.888671 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.877521    2574 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 02:54:51.888671 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.877524    2574 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 02:54:51.888671 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.877526    2574 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 02:54:51.888671 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.877529    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 02:54:51.888671 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.877531    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 02:54:51.888671 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.877534    2574 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 02:54:51.888671 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.877537    2574 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 02:54:51.888671 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.877539    2574 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 02:54:51.888671 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.877542    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 02:54:51.888671 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.877544    2574 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 02:54:51.888671 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.877548    2574 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 02:54:51.888671 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.877551    2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 02:54:51.888671 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.877553    2574 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 02:54:51.888671 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.877555    2574 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 02:54:51.888671 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.877558    2574 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 02:54:51.888671 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.877561    2574 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 02:54:51.888671 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.877564    2574 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 02:54:51.888671 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.877566    2574 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 02:54:51.888671 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.877569    2574 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 02:54:51.888671 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:51.886477    2574 server.go:525] "Kubelet version" kubeletVersion="v1.35.9"
Oct 30 02:54:51.889195 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:51.886494    2574 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Oct 30 02:54:51.889195 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:51.887207    2574 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Oct 30 02:54:51.889195 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:51.887217    2574 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Oct 30 02:54:51.889195 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.887274    2574 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 02:54:51.889195 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.887280    2574 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 02:54:51.889195 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.887283    2574 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 02:54:51.889195 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.887287    2574 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 02:54:51.889195 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.887290    2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 02:54:51.889195 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.887293    2574 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 02:54:51.889195 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.887295    2574 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 02:54:51.889195 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.887298    2574 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 02:54:51.889195 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.887301    2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 02:54:51.889195 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.887305    2574 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 02:54:51.889195 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.887308    2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 02:54:51.889195 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.887311    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 02:54:51.889195 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.887314    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 02:54:51.889195 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.887316    2574 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 02:54:51.889195 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.887319    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 02:54:51.889195 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.887322    2574 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 02:54:51.889664 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.887324    2574 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 02:54:51.889664 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.887327    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 02:54:51.889664 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.887330    2574 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 02:54:51.889664 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.887333    2574 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 02:54:51.889664 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.887335    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 02:54:51.889664 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.887338    2574 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 02:54:51.889664 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.887340    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 02:54:51.889664 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.887344    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 02:54:51.889664 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.887346    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 02:54:51.889664 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.887349    2574 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 02:54:51.889664 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.887352    2574 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 02:54:51.889664 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.887354    2574 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 02:54:51.889664 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.887357    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 02:54:51.889664 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.887360    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 02:54:51.889664 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.887362    2574 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 02:54:51.889664 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.887365    2574 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 02:54:51.889664 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.887368    2574 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 02:54:51.889664 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.887370    2574 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 02:54:51.889664 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.887373    2574 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 02:54:51.890116 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.887375    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 02:54:51.890116 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.887378    2574 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 02:54:51.890116 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.887381    2574 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 02:54:51.890116 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.887383    2574 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 02:54:51.890116 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.887386    2574 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 02:54:51.890116 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.887388    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 02:54:51.890116 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.887391    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 02:54:51.890116 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.887393    2574 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 02:54:51.890116 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.887396    2574 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 02:54:51.890116 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.887398    2574 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 02:54:51.890116 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.887401    2574 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 02:54:51.890116 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.887403    2574 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 02:54:51.890116 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.887406    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 02:54:51.890116 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.887408    2574 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 02:54:51.890116 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.887411    2574 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 02:54:51.890116 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.887413    2574 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 02:54:51.890116 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.887416    2574 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 02:54:51.890116 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.887419    2574 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 02:54:51.890116 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.887422    2574 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 02:54:51.890116 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.887424    2574 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 02:54:51.890584 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.887427    2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 02:54:51.890584 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.887431    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 02:54:51.890584 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.887433    2574 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 02:54:51.890584 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.887436    2574 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 02:54:51.890584 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.887438    2574 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 02:54:51.890584 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.887441    2574 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 02:54:51.890584 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.887444    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 02:54:51.890584 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.887447    2574 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 02:54:51.890584 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.887450    2574 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 02:54:51.890584 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.887452    2574 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 02:54:51.890584 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.887455    2574 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 02:54:51.890584 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.887457    2574 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 02:54:51.890584 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.887460    2574 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 02:54:51.890584 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.887463    2574 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 02:54:51.890584 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.887465    2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 02:54:51.890584 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.887468    2574 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 02:54:51.890584 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.887470    2574 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 02:54:51.890584 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.887473    2574 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 02:54:51.890584 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.887475    2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 02:54:51.890584 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.887478    2574 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 02:54:51.891052 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.887481    2574 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 02:54:51.891052 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.887483    2574 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 02:54:51.891052 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.887485    2574 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 02:54:51.891052 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.887488    2574 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 02:54:51.891052 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.887498    2574 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 02:54:51.891052 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.887501    2574 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 02:54:51.891052 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.887505    2574 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 02:54:51.891052 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.887509    2574 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 02:54:51.891052 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.887513    2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 02:54:51.891052 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.887516    2574 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 02:54:51.891052 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.887518    2574 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 02:54:51.891052 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.887521    2574 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 02:54:51.891052 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.887523    2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 02:54:51.891052 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.887526    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 02:54:51.891052 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.887529    2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 02:54:51.891052 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.887531    2574 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 02:54:51.891052 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.887534    2574 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 02:54:51.891052 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.887537    2574 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 02:54:51.891052 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.887540    2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 02:54:51.891608 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.887544    2574 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 02:54:51.891608 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.887547    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 02:54:51.891608 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.887550    2574 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 02:54:51.891608 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.887553    2574 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 02:54:51.891608 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.887555    2574 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 02:54:51.891608 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.887558    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 02:54:51.891608 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.887561    2574 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 02:54:51.891608 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.887563    2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 02:54:51.891608 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.887566    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 02:54:51.891608 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.887568    2574 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 02:54:51.891608 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.887571    2574 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 02:54:51.891608 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.887573    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 02:54:51.891608 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:51.887601    2574 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 02:54:51.891608 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.887745    2574 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 02:54:51.891608 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.887750    2574 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 02:54:51.891608 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.887753    2574 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 02:54:51.892025 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.887756    2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 02:54:51.892025 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.887759    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 02:54:51.892025 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.887768    2574 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 02:54:51.892025 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.887771    2574 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 02:54:51.892025 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.887773    2574 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 02:54:51.892025 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.887776    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 02:54:51.892025 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.887778    2574 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 02:54:51.892025 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.887781    2574 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 02:54:51.892025 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.887783    2574 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 02:54:51.892025 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.887786    2574 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 02:54:51.892025 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.887789    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 02:54:51.892025 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.887791    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 02:54:51.892025 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.887794    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 02:54:51.892025 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.887796    2574 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 02:54:51.892025 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.887799    2574 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 02:54:51.892025 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.887801    2574 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 02:54:51.892025 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.887804    2574 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 02:54:51.892025 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.887806    2574 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 02:54:51.892025 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.887809    2574 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 02:54:51.892538 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.887811    2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 02:54:51.892538 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.887814    2574 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 02:54:51.892538 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.887816    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 02:54:51.892538 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.887819    2574 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 02:54:51.892538 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.887821    2574 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 02:54:51.892538 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.887824    2574 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 02:54:51.892538 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.887826    2574 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 02:54:51.892538 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.887829    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 02:54:51.892538 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.887831    2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 02:54:51.892538 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.887834    2574 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 02:54:51.892538 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.887836    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 02:54:51.892538 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.887839    2574 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 02:54:51.892538 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.887842    2574 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 02:54:51.892538 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.887844    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 02:54:51.892538 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.887847    2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 02:54:51.892538 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.887849    2574 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 02:54:51.892538 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.887857    2574 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 02:54:51.892538 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.887860    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 02:54:51.892538 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.887863    2574 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 02:54:51.893108 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.887866    2574 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 02:54:51.893108 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.887868    2574 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 02:54:51.893108 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.887871    2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 02:54:51.893108 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.887873    2574 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 02:54:51.893108 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.887876    2574 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 02:54:51.893108 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.887878    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 02:54:51.893108 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.887881    2574 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 02:54:51.893108 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.887883    2574 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 02:54:51.893108 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.887886    2574 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 02:54:51.893108 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.887889    2574 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 02:54:51.893108 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.887891    2574 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 02:54:51.893108 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.887894    2574 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 02:54:51.893108 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.887896    2574 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 02:54:51.893108 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.887899    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 02:54:51.893108 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.887901    2574 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 02:54:51.893108 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.887904    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 02:54:51.893108 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.887906    2574 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 02:54:51.893108 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.887909    2574 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 02:54:51.893108 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.887912    2574 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 02:54:51.893108 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.887914    2574 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 02:54:51.893588 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.887916    2574 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 02:54:51.893588 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.887920    2574 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 02:54:51.893588 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.887922    2574 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 02:54:51.893588 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.887925    2574 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 02:54:51.893588 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.887928    2574 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 02:54:51.893588 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.887930    2574 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 02:54:51.893588 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.887932    2574 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 02:54:51.893588 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.887935    2574 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 02:54:51.893588 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.887938    2574 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 02:54:51.893588 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.887940    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 02:54:51.893588 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.887950    2574 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 02:54:51.893588 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.887953    2574 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 02:54:51.893588 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.887955    2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 02:54:51.893588 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.887957    2574 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 02:54:51.893588 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.887960    2574 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 02:54:51.893588 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.887962    2574 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 02:54:51.893588 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.887965    2574 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 02:54:51.893588 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.887968    2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 02:54:51.893588 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.887970    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 02:54:51.893588 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.887973    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 02:54:51.894116 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.887975    2574 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 02:54:51.894116 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.887978    2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 02:54:51.894116 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.887980    2574 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 02:54:51.894116 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.887983    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 02:54:51.894116 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.887985    2574 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 02:54:51.894116 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.887988    2574 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 02:54:51.894116 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.887990    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 02:54:51.894116 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.887993    2574 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 02:54:51.894116 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.887996    2574 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 02:54:51.894116 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.888000    2574 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 02:54:51.894116 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.888003    2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 02:54:51.894116 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.888006    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 02:54:51.894116 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.888008    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 02:54:51.894116 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.888011    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 02:54:51.894116 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.888013    2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 02:54:51.894116 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.888017    2574 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 02:54:51.894116 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.888020    2574 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 02:54:51.894116 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.888023    2574 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 02:54:51.894606 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.888025    2574 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 02:54:51.894606 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.888028    2574 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 02:54:51.894606 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.888031    2574 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 02:54:51.894606 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.888033    2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 02:54:51.894606 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.888036    2574 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 02:54:51.894606 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.888044    2574 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 02:54:51.894606 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.888047    2574 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 02:54:51.894606 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:51.888070    2574 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 02:54:51.894606 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.888119    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 02:54:51.894606 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.888124    2574 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 02:54:51.894606 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.888126    2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 02:54:51.894606 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.888129    2574 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 02:54:51.894606 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.888132    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 02:54:51.894606 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.888134    2574 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 02:54:51.894606 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.888137    2574 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 02:54:51.894606 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.888139    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 02:54:51.895257 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.888142    2574 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 02:54:51.895257 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.888146    2574 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 02:54:51.895257 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.888149    2574 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 02:54:51.895257 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.888152    2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 02:54:51.895257 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.888155    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 02:54:51.895257 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.888158    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 02:54:51.895257 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.888161    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 02:54:51.895257 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.888164    2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 02:54:51.895257 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.888167    2574 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 02:54:51.895257 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.888171    2574 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 02:54:51.895257 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.888174    2574 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 02:54:51.895257 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.888177    2574 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 02:54:51.895257 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.888179    2574 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 02:54:51.895257 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.888182    2574 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 02:54:51.895257 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.888184    2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 02:54:51.895257 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.888187    2574 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 02:54:51.895257 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.888189    2574 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 02:54:51.895257 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.888192    2574 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 02:54:51.895852 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.888194    2574 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 02:54:51.895852 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.888197    2574 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 02:54:51.895852 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.888199    2574 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 02:54:51.895852 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.888202    2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 02:54:51.895852 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.888211    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 02:54:51.895852 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.888214    2574 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 02:54:51.895852 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.888216    2574 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 02:54:51.895852 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.888219    2574 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 02:54:51.895852 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.888221    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 02:54:51.895852 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.888224    2574 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 02:54:51.895852 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.888227    2574 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 02:54:51.895852 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.888230    2574 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 02:54:51.895852 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.888232    2574 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 02:54:51.895852 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.888235    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 02:54:51.895852 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.888238    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 02:54:51.895852 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.888240    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 02:54:51.895852 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.888243    2574 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 02:54:51.895852 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.888245    2574 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 02:54:51.895852 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.888248    2574 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 02:54:51.895852 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.888250    2574 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 02:54:51.896463 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.888253    2574 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 02:54:51.896463 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.888255    2574 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 02:54:51.896463 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.888257    2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 02:54:51.896463 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.888260    2574 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 02:54:51.896463 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.888262    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 02:54:51.896463 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.888265    2574 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 02:54:51.896463 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.888267    2574 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 02:54:51.896463 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.888269    2574 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 02:54:51.896463 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.888272    2574 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 02:54:51.896463 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.888274    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 02:54:51.896463 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.888280    2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 02:54:51.896463 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.888282    2574 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 02:54:51.896463 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.888285    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 02:54:51.896463 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.888287    2574 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 02:54:51.896463 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.888290    2574 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 02:54:51.896463 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.888294    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 02:54:51.896463 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.888298    2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 02:54:51.896463 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.888306    2574 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 02:54:51.896463 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.888309    2574 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 02:54:51.896985 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.888311    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 02:54:51.896985 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.888314    2574 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 02:54:51.896985 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.888317    2574 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 02:54:51.896985 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.888319    2574 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 02:54:51.896985 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.888322    2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 02:54:51.896985 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.888324    2574 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 02:54:51.896985 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.888327    2574 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 02:54:51.896985 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.888330    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 02:54:51.896985 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.888332    2574 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 02:54:51.896985 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.888335    2574 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 02:54:51.896985 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.888337    2574 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 02:54:51.896985 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.888340    2574 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 02:54:51.896985 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.888342    2574 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 02:54:51.896985 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.888345    2574 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 02:54:51.896985 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.888347    2574 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 02:54:51.896985 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.888350    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 02:54:51.896985 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.888353    2574 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 02:54:51.896985 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.888355    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 02:54:51.896985 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.888358    2574 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 02:54:51.896985 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.888361    2574 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 02:54:51.897674 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.888363    2574 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 02:54:51.897674 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.888366    2574 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 02:54:51.897674 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.888368    2574 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 02:54:51.897674 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.888375    2574 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 02:54:51.897674 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.888378    2574 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 02:54:51.897674 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.888380    2574 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 02:54:51.897674 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.888383    2574 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 02:54:51.897674 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.888385    2574 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 02:54:51.897674 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.888390    2574 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 02:54:51.897674 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.888393    2574 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 02:54:51.897674 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.888395    2574 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 02:54:51.897674 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.888403    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 02:54:51.897674 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.888406    2574 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 02:54:51.897674 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.888408    2574 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 02:54:51.897674 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.888411    2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 02:54:51.897674 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.888414    2574 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 02:54:51.897674 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.888416    2574 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 02:54:51.897674 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.888419    2574 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 02:54:51.897674 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.888422    2574 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 02:54:51.897674 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.888425    2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 02:54:51.898235 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:51.888427    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 02:54:51.898235 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:51.888486    2574 server.go:957] "Client rotation is on, will bootstrap in background"
Oct 30 02:54:51.898235 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:51.891758    2574 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Oct 30 02:54:51.898235 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:51.892718    2574 server.go:1014] "Starting client certificate rotation"
Oct 30 02:54:51.898235 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:51.892832    2574 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Oct 30 02:54:51.898235 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:51.892882    2574 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Oct 30 02:54:51.898235 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:51.894627    2574 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 30 02:54:51.898235 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:51.897841    2574 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 30 02:54:51.903848 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:51.903831    2574 log.go:25] "Validated CRI v1 runtime API"
Oct 30 02:54:51.906402 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:51.906388    2574 log.go:25] "Validated CRI v1 image API"
Oct 30 02:54:51.907278 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:51.907257    2574 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Oct 30 02:54:51.909956 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:51.909933    2574 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 8bf05c95-fcd3-46e0-b571-1a0371d90777:/dev/nvme0n1p3 eab5acd2-acc1-4e27-a6b9-91c13694e430:/dev/nvme0n1p4]
Oct 30 02:54:51.910028 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:51.909954    2574 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}]
Oct 30 02:54:51.916883 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:51.916539    2574 manager.go:217] Machine: {Timestamp:2026-10-30 02:54:51.914783138 +0000 UTC m=+0.333534703 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3100043 MemoryCapacity:33164337152 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec24b99a3425169b54576a8e645a4a9b SystemUUID:ec24b99a-3425-169b-5457-6a8e645a4a9b BootID:05b7e5fc-b59f-440e-ac9b-b26ec188cc4d Filesystems:[{Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582168576 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582168576 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:e5:84:0a:c9 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:e5:84:0a:c9 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:ae:13:91:b2:51:44 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164337152 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Oct 30 02:54:51.916883 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:51.916879    2574 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Oct 30 02:54:51.917034 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:51.917020    2574 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Oct 30 02:54:51.920083 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:51.920055    2574 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Oct 30 02:54:51.920241 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:51.920085    2574 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-143-156.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Oct 30 02:54:51.920283 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:51.920244    2574 topology_manager.go:143] "Creating topology manager with none policy"
Oct 30 02:54:51.920283 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:51.920253    2574 container_manager_linux.go:311] "Creating device plugin manager"
Oct 30 02:54:51.920348 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:51.920284    2574 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 30 02:54:51.920348 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:51.920301    2574 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 30 02:54:51.921025 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:51.921014    2574 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager"
Oct 30 02:54:51.924805 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:51.924793    2574 state_mem.go:41] "Initialized" logger="CPUManager state memory"
Oct 30 02:54:51.924920 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:51.924911    2574 server.go:1235] "Using root directory" path="/var/lib/kubelet"
Oct 30 02:54:51.925540 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:51.925531    2574 kubelet.go:493] "Attempting to sync node with API server"
Oct 30 02:54:51.925578 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:51.925543    2574 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests"
Oct 30 02:54:51.925578 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:51.925553    2574 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Oct 30 02:54:51.925578 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:51.925561    2574 kubelet.go:405] "Adding apiserver pod source"
Oct 30 02:54:51.925578 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:51.925568    2574 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Oct 30 02:54:51.925990 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:51.925975    2574 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 30 02:54:51.928415 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:51.928400    2574 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1"
Oct 30 02:54:51.932624 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:51.932611    2574 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 30 02:54:51.933462 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:51.933447    2574 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Oct 30 02:54:51.933508 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:51.933497    2574 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Oct 30 02:54:51.933708 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:51.933694    2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Oct 30 02:54:51.933752 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:51.933723    2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Oct 30 02:54:51.933752 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:51.933735    2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Oct 30 02:54:51.933752 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:51.933743    2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Oct 30 02:54:51.933752 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:51.933752    2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Oct 30 02:54:51.933861 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:51.933761    2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Oct 30 02:54:51.934229 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:51.934217    2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Oct 30 02:54:51.934264 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:51.934233    2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Oct 30 02:54:51.934264 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:51.934244    2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Oct 30 02:54:51.934313 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:51.934270    2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Oct 30 02:54:51.934313 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:51.934291    2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Oct 30 02:54:51.934313 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:51.934304    2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Oct 30 02:54:51.934934 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:51.934923    2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Oct 30 02:54:51.934985 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:51.934935    2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Oct 30 02:54:51.938717 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:51.938702    2574 server.go:1263] "Started kubelet"
Oct 30 02:54:51.938833 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:51.938776    2574 server.go:182] "Starting to listen" address="0.0.0.0" port=10250
Oct 30 02:54:51.938885 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:51.938793    2574 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Oct 30 02:54:51.938943 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:51.938924    2574 server_v1.go:49] "podresources" method="list" useActivePods=true
Oct 30 02:54:51.939458 ip-10-0-143-156 systemd[1]: Started Kubernetes Kubelet.
Oct 30 02:54:51.940071 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:51.939966    2574 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Oct 30 02:54:51.941924 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:51.941358    2574 server.go:317] "Adding debug handlers to kubelet server"
Oct 30 02:54:51.946517 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:51.946499    2574 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Oct 30 02:54:51.946608 ip-10-0-143-156 kubenswrapper[2574]: E1030 02:54:51.946544    2574 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Oct 30 02:54:51.946987 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:51.946969    2574 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer"
Oct 30 02:54:51.947127 ip-10-0-143-156 kubenswrapper[2574]: E1030 02:54:51.947087    2574 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-143-156.ec2.internal\" not found"
Oct 30 02:54:51.947798 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:51.947776    2574 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Oct 30 02:54:51.947890 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:51.947781    2574 volume_manager.go:309] "The desired_state_of_world populator starts"
Oct 30 02:54:51.947890 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:51.947825    2574 volume_manager.go:311] "Starting Kubelet Volume Manager"
Oct 30 02:54:51.947985 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:51.947951    2574 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Oct 30 02:54:51.947985 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:51.947966    2574 factory.go:55] Registering systemd factory
Oct 30 02:54:51.947985 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:51.947975    2574 factory.go:223] Registration of the systemd container factory successfully
Oct 30 02:54:51.948133 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:51.947958    2574 reconstruct.go:97] "Volume reconstruction finished"
Oct 30 02:54:51.948133 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:51.948002    2574 reconciler.go:29] "Reconciler: start to sync state"
Oct 30 02:54:51.948224 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:51.948213    2574 factory.go:153] Registering CRI-O factory
Oct 30 02:54:51.948224 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:51.948223    2574 factory.go:223] Registration of the crio container factory successfully
Oct 30 02:54:51.948317 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:51.948236    2574 factory.go:103] Registering Raw factory
Oct 30 02:54:51.948317 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:51.948262    2574 manager.go:1199] Started watching for new ooms in manager
Oct 30 02:54:51.951267 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:51.951213    2574 manager.go:319] Starting recovery of all containers
Oct 30 02:54:51.958944 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:51.958912    2574 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Oct 30 02:54:51.963835 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:51.963821    2574 manager.go:324] Recovery completed
Oct 30 02:54:51.968124 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:51.968079    2574 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 02:54:51.973496 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:51.973478    2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-156.ec2.internal" event="NodeHasSufficientMemory"
Oct 30 02:54:51.973576 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:51.973503    2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-156.ec2.internal" event="NodeHasNoDiskPressure"
Oct 30 02:54:51.973576 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:51.973531    2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-156.ec2.internal" event="NodeHasSufficientPID"
Oct 30 02:54:51.973713 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:51.973672    2574 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-156.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Oct 30 02:54:51.973784 ip-10-0-143-156 kubenswrapper[2574]: E1030 02:54:51.973734    2574 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-143-156.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Oct 30 02:54:51.974578 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:51.974561    2574 cpu_manager.go:226] "Starting" policy="none"
Oct 30 02:54:51.974578 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:51.974573    2574 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 02:54:51.974720 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:51.974575    2574 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s"
Oct 30 02:54:51.974720 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:51.974612    2574 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory"
Oct 30 02:54:51.974720 ip-10-0-143-156 kubenswrapper[2574]: E1030 02:54:51.973668    2574 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-143-156.ec2.internal.18e32f203172efee  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-143-156.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-143-156.ec2.internal,},FirstTimestamp:2026-10-30 02:54:51.93866443 +0000 UTC m=+0.357415993,LastTimestamp:2026-10-30 02:54:51.93866443 +0000 UTC m=+0.357415993,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-143-156.ec2.internal,}"
Oct 30 02:54:51.975670 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:51.975652    2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-156.ec2.internal" event="NodeHasSufficientMemory"
Oct 30 02:54:51.975739 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:51.975681    2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-156.ec2.internal" event="NodeHasNoDiskPressure"
Oct 30 02:54:51.975739 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:51.975723    2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-156.ec2.internal" event="NodeHasSufficientPID"
Oct 30 02:54:51.976886 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:51.976872    2574 policy_none.go:50] "Start"
Oct 30 02:54:51.976933 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:51.976888    2574 memory_manager.go:187] "Starting memorymanager" policy="None"
Oct 30 02:54:51.976933 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:51.976898    2574 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Oct 30 02:54:51.978884 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:51.978872    2574 policy_none.go:44] "Start"
Oct 30 02:54:51.980126 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:51.980087    2574 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Oct 30 02:54:51.980210 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:51.980141    2574 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-j4bfc"
Oct 30 02:54:51.987341 ip-10-0-143-156 kubenswrapper[2574]: E1030 02:54:51.987321    2574 reflector.go:204] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver"
Oct 30 02:54:51.987473 ip-10-0-143-156 kubenswrapper[2574]: E1030 02:54:51.987376    2574 reflector.go:204] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service"
Oct 30 02:54:51.987473 ip-10-0-143-156 kubenswrapper[2574]: E1030 02:54:51.987394    2574 reflector.go:204] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-143-156.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Node"
Oct 30 02:54:51.987581 ip-10-0-143-156 kubenswrapper[2574]: E1030 02:54:51.987435    2574 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-143-156.ec2.internal.18e32f20338665d4  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-143-156.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-143-156.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-143-156.ec2.internal,},FirstTimestamp:2026-10-30 02:54:51.973494228 +0000 UTC m=+0.392245794,LastTimestamp:2026-10-30 02:54:51.973494228 +0000 UTC m=+0.392245794,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-143-156.ec2.internal,}"
Oct 30 02:54:51.988862 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:51.988846    2574 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-j4bfc"
Oct 30 02:54:52.035915 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:52.015722    2574 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"}
Oct 30 02:54:52.035915 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:52.022024    2574 manager.go:341] "Starting Device Plugin manager"
Oct 30 02:54:52.035915 ip-10-0-143-156 kubenswrapper[2574]: E1030 02:54:52.022070    2574 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Oct 30 02:54:52.035915 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:52.022081    2574 server.go:87] "Starting device plugin registration server"
Oct 30 02:54:52.035915 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:52.022359    2574 eviction_manager.go:194] "Eviction manager: starting control loop"
Oct 30 02:54:52.035915 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:52.022371    2574 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Oct 30 02:54:52.035915 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:52.022487    2574 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Oct 30 02:54:52.035915 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:52.022575    2574 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts"
Oct 30 02:54:52.035915 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:52.022584    2574 plugin_manager.go:121] "Starting Kubelet Plugin Manager"
Oct 30 02:54:52.035915 ip-10-0-143-156 kubenswrapper[2574]: E1030 02:54:52.023236    2574 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Oct 30 02:54:52.035915 ip-10-0-143-156 kubenswrapper[2574]: E1030 02:54:52.023297    2574 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-143-156.ec2.internal\" not found"
Oct 30 02:54:52.035915 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:52.023558    2574 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Oct 30 02:54:52.035915 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:52.023581    2574 status_manager.go:249] "Starting to sync pod status with apiserver"
Oct 30 02:54:52.035915 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:52.023600    2574 kubelet.go:2521] "Starting kubelet main sync loop"
Oct 30 02:54:52.035915 ip-10-0-143-156 kubenswrapper[2574]: E1030 02:54:52.023640    2574 kubelet.go:2545] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Oct 30 02:54:52.039337 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:52.039322    2574 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 02:54:52.123182 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:52.123126    2574 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 02:54:52.123764 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:52.123720    2574 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-156.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-143-156.ec2.internal"]
Oct 30 02:54:52.123877 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:52.123784    2574 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 02:54:52.124012 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:52.123994    2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-156.ec2.internal" event="NodeHasSufficientMemory"
Oct 30 02:54:52.124069 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:52.124027    2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-156.ec2.internal" event="NodeHasNoDiskPressure"
Oct 30 02:54:52.124069 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:52.124045    2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-156.ec2.internal" event="NodeHasSufficientPID"
Oct 30 02:54:52.124179 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:52.124078    2574 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-143-156.ec2.internal"
Oct 30 02:54:52.124543 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:52.124526    2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-156.ec2.internal" event="NodeHasSufficientMemory"
Oct 30 02:54:52.124593 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:52.124553    2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-156.ec2.internal" event="NodeHasNoDiskPressure"
Oct 30 02:54:52.124593 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:52.124563    2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-156.ec2.internal" event="NodeHasSufficientPID"
Oct 30 02:54:52.126645 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:52.126632    2574 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 02:54:52.126799 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:52.126785    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-156.ec2.internal"
Oct 30 02:54:52.126875 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:52.126814    2574 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 02:54:52.127390 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:52.127376    2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-156.ec2.internal" event="NodeHasSufficientMemory"
Oct 30 02:54:52.127448 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:52.127407    2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-156.ec2.internal" event="NodeHasNoDiskPressure"
Oct 30 02:54:52.127448 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:52.127422    2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-156.ec2.internal" event="NodeHasSufficientPID"
Oct 30 02:54:52.127554 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:52.127421    2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-156.ec2.internal" event="NodeHasSufficientMemory"
Oct 30 02:54:52.127597 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:52.127574    2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-156.ec2.internal" event="NodeHasNoDiskPressure"
Oct 30 02:54:52.127597 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:52.127591    2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-156.ec2.internal" event="NodeHasSufficientPID"
Oct 30 02:54:52.129492 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:52.129477    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-156.ec2.internal"
Oct 30 02:54:52.129548 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:52.129505    2574 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 02:54:52.130186 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:52.130170    2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-156.ec2.internal" event="NodeHasSufficientMemory"
Oct 30 02:54:52.130250 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:52.130199    2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-156.ec2.internal" event="NodeHasNoDiskPressure"
Oct 30 02:54:52.130250 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:52.130208    2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-156.ec2.internal" event="NodeHasSufficientPID"
Oct 30 02:54:52.142660 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:52.142634    2574 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-143-156.ec2.internal"
Oct 30 02:54:52.142660 ip-10-0-143-156 kubenswrapper[2574]: E1030 02:54:52.142663    2574 kubelet_node_status.go:524] "Error updating node status, will retry" err="error getting node \"ip-10-0-143-156.ec2.internal\": node \"ip-10-0-143-156.ec2.internal\" not found"
Oct 30 02:54:52.148430 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:52.148414    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/a7672f7a60a28c85d738430783c2cc68-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-143-156.ec2.internal\" (UID: \"a7672f7a60a28c85d738430783c2cc68\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-156.ec2.internal"
Oct 30 02:54:52.148491 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:52.148438    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a7672f7a60a28c85d738430783c2cc68-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-143-156.ec2.internal\" (UID: \"a7672f7a60a28c85d738430783c2cc68\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-156.ec2.internal"
Oct 30 02:54:52.148491 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:52.148455    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/fee27b4a19dca64d5627dff27844f71f-config\") pod \"kube-apiserver-proxy-ip-10-0-143-156.ec2.internal\" (UID: \"fee27b4a19dca64d5627dff27844f71f\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-143-156.ec2.internal"
Oct 30 02:54:52.161460 ip-10-0-143-156 kubenswrapper[2574]: E1030 02:54:52.161441    2574 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-143-156.ec2.internal\" not found" node="ip-10-0-143-156.ec2.internal"
Oct 30 02:54:52.164325 ip-10-0-143-156 kubenswrapper[2574]: E1030 02:54:52.164310    2574 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-143-156.ec2.internal\" not found" node="ip-10-0-143-156.ec2.internal"
Oct 30 02:54:52.186680 ip-10-0-143-156 kubenswrapper[2574]: E1030 02:54:52.186663    2574 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-143-156.ec2.internal\" not found"
Oct 30 02:54:52.248880 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:52.248844    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/a7672f7a60a28c85d738430783c2cc68-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-143-156.ec2.internal\" (UID: \"a7672f7a60a28c85d738430783c2cc68\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-156.ec2.internal"
Oct 30 02:54:52.248880 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:52.248883    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a7672f7a60a28c85d738430783c2cc68-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-143-156.ec2.internal\" (UID: \"a7672f7a60a28c85d738430783c2cc68\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-156.ec2.internal"
Oct 30 02:54:52.249036 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:52.248901    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/fee27b4a19dca64d5627dff27844f71f-config\") pod \"kube-apiserver-proxy-ip-10-0-143-156.ec2.internal\" (UID: \"fee27b4a19dca64d5627dff27844f71f\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-143-156.ec2.internal"
Oct 30 02:54:52.249036 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:52.248912    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/a7672f7a60a28c85d738430783c2cc68-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-143-156.ec2.internal\" (UID: \"a7672f7a60a28c85d738430783c2cc68\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-156.ec2.internal"
Oct 30 02:54:52.249036 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:52.248940    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/fee27b4a19dca64d5627dff27844f71f-config\") pod \"kube-apiserver-proxy-ip-10-0-143-156.ec2.internal\" (UID: \"fee27b4a19dca64d5627dff27844f71f\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-143-156.ec2.internal"
Oct 30 02:54:52.249036 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:52.248948    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a7672f7a60a28c85d738430783c2cc68-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-143-156.ec2.internal\" (UID: \"a7672f7a60a28c85d738430783c2cc68\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-156.ec2.internal"
Oct 30 02:54:52.286901 ip-10-0-143-156 kubenswrapper[2574]: E1030 02:54:52.286871    2574 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-143-156.ec2.internal\" not found"
Oct 30 02:54:52.387453 ip-10-0-143-156 kubenswrapper[2574]: E1030 02:54:52.387367    2574 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-143-156.ec2.internal\" not found"
Oct 30 02:54:52.465083 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:52.465058    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-156.ec2.internal"
Oct 30 02:54:52.467221 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:52.467205    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-156.ec2.internal"
Oct 30 02:54:52.488338 ip-10-0-143-156 kubenswrapper[2574]: E1030 02:54:52.488206    2574 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-143-156.ec2.internal\" not found"
Oct 30 02:54:52.588864 ip-10-0-143-156 kubenswrapper[2574]: E1030 02:54:52.588806    2574 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-143-156.ec2.internal\" not found"
Oct 30 02:54:52.689376 ip-10-0-143-156 kubenswrapper[2574]: E1030 02:54:52.689282    2574 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-143-156.ec2.internal\" not found"
Oct 30 02:54:52.790015 ip-10-0-143-156 kubenswrapper[2574]: E1030 02:54:52.789992    2574 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-143-156.ec2.internal\" not found"
Oct 30 02:54:52.890923 ip-10-0-143-156 kubenswrapper[2574]: E1030 02:54:52.890887    2574 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-143-156.ec2.internal\" not found"
Oct 30 02:54:52.892044 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:52.892024    2574 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Oct 30 02:54:52.892285 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:52.892259    2574 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received"
Oct 30 02:54:52.978637 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:52.978604    2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda7672f7a60a28c85d738430783c2cc68.slice/crio-c45974cae891b83781f22b2ed8a3e9c6dd117bb07816519904afa6a549a61b93 WatchSource:0}: Error finding container c45974cae891b83781f22b2ed8a3e9c6dd117bb07816519904afa6a549a61b93: Status 404 returned error can't find the container with id c45974cae891b83781f22b2ed8a3e9c6dd117bb07816519904afa6a549a61b93
Oct 30 02:54:52.979116 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:52.979083    2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfee27b4a19dca64d5627dff27844f71f.slice/crio-8e25068db3f97685e5f4f9df47b334f09abff36314e4419220b21069b574efb3 WatchSource:0}: Error finding container 8e25068db3f97685e5f4f9df47b334f09abff36314e4419220b21069b574efb3: Status 404 returned error can't find the container with id 8e25068db3f97685e5f4f9df47b334f09abff36314e4419220b21069b574efb3
Oct 30 02:54:52.983753 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:52.983733    2574 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 02:54:52.991965 ip-10-0-143-156 kubenswrapper[2574]: E1030 02:54:52.991944    2574 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-143-156.ec2.internal\" not found"
Oct 30 02:54:52.994164 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:52.994091    2574 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-29 02:49:52 +0000 UTC" deadline="2028-05-07 11:26:43.690290992 +0000 UTC"
Oct 30 02:54:52.994164 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:52.994157    2574 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13328h31m50.696136567s"
Oct 30 02:54:53.005202 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:53.005184    2574 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 02:54:53.026836 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:53.026774    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-156.ec2.internal" event={"ID":"fee27b4a19dca64d5627dff27844f71f","Type":"ContainerStarted","Data":"8e25068db3f97685e5f4f9df47b334f09abff36314e4419220b21069b574efb3"}
Oct 30 02:54:53.027868 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:53.027846    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-156.ec2.internal" event={"ID":"a7672f7a60a28c85d738430783c2cc68","Type":"ContainerStarted","Data":"c45974cae891b83781f22b2ed8a3e9c6dd117bb07816519904afa6a549a61b93"}
Oct 30 02:54:53.047700 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:53.047676    2574 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-156.ec2.internal"
Oct 30 02:54:53.074294 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:53.074275    2574 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 02:54:53.076379 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:53.076366    2574 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-156.ec2.internal"
Oct 30 02:54:53.102703 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:53.102678    2574 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 02:54:53.432778 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:53.432687    2574 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 02:54:53.526302 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:53.526264    2574 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 02:54:53.927398 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:53.927325    2574 apiserver.go:52] "Watching apiserver"
Oct 30 02:54:53.947986 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:53.947956    2574 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Oct 30 02:54:53.964630 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:53.964596    2574 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Oct 30 02:54:53.965017 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:53.964987    2574 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/node-ca-9mddj","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-156.ec2.internal","openshift-network-diagnostics/network-check-target-7rmdj","openshift-network-operator/iptables-alerter-jhq2g","openshift-ovn-kubernetes/ovnkube-node-g6nqm","kube-system/kube-apiserver-proxy-ip-10-0-143-156.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gqpp2","openshift-multus/multus-additional-cni-plugins-9l7k8","openshift-multus/network-metrics-daemon-mzk2s","kube-system/global-pull-secret-syncer-vbnwt","openshift-cluster-node-tuning-operator/tuned-h7cm4","kube-system/konnectivity-agent-djpqs","openshift-multus/multus-9qcgk","openshift-dns/node-resolver-zhb7b"]
Oct 30 02:54:53.967904 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:53.967865    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gqpp2"
Oct 30 02:54:53.971071 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:53.970626    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-9qcgk"
Oct 30 02:54:53.973746 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:53.973463    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7rmdj"
Oct 30 02:54:53.973746 ip-10-0-143-156 kubenswrapper[2574]: E1030 02:54:53.973547    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7rmdj" podUID="75d9e1ec-cd1b-4aba-b911-5a7465c653cf"
Oct 30 02:54:53.976203 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:53.976184    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-jhq2g"
Oct 30 02:54:53.978641 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:53.978619    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-g6nqm"
Oct 30 02:54:53.981850 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:53.981358    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-zhb7b"
Oct 30 02:54:53.984470 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:53.984450    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mzk2s"
Oct 30 02:54:53.984576 ip-10-0-143-156 kubenswrapper[2574]: E1030 02:54:53.984538    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mzk2s" podUID="3bd2c646-cebb-4e8a-b6c3-8b5b74049c11"
Oct 30 02:54:53.986398 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:53.986374    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Oct 30 02:54:53.986515 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:53.986485    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-8ljf8\""
Oct 30 02:54:53.986582 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:53.986521    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Oct 30 02:54:53.986642 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:53.986590    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Oct 30 02:54:53.987698 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:53.987680    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Oct 30 02:54:53.987875 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:53.987850    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-ft4zm\""
Oct 30 02:54:53.987965 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:53.987882    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Oct 30 02:54:53.987965 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:53.987916    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Oct 30 02:54:53.987965 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:53.987953    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Oct 30 02:54:53.988084 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:53.988027    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-9mddj"
Oct 30 02:54:53.992159 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:53.991812    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-9l7k8"
Oct 30 02:54:53.993692 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:53.993671    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-9qpgh\""
Oct 30 02:54:53.993789 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:53.993759    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Oct 30 02:54:53.993959 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:53.993877    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Oct 30 02:54:53.994775 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:53.994738    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Oct 30 02:54:53.994989 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:53.994972    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vbnwt"
Oct 30 02:54:53.995072 ip-10-0-143-156 kubenswrapper[2574]: E1030 02:54:53.995031    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vbnwt" podUID="98f9efe3-a732-4ba4-8543-941633d22170"
Oct 30 02:54:53.998240 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:53.997632    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-h7cm4"
Oct 30 02:54:54.000690 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:54.000329    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-djpqs"
Oct 30 02:54:54.017722 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:54.017697    2574 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Oct 30 02:54:54.018230 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:54.018086    2574 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-cw87c"
Oct 30 02:54:54.027354 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:54.027333    2574 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-cw87c"
Oct 30 02:54:54.049383 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:54.049355    2574 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Oct 30 02:54:54.062241 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:54.062211    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f425530a-8431-4c5b-8335-8cc1119745c1-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-g6nqm\" (UID: \"f425530a-8431-4c5b-8335-8cc1119745c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-g6nqm"
Oct 30 02:54:54.062346 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:54.062255    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/3e4730f0-d77e-4bff-b8af-b7fafd4525d8-etc-sysctl-d\") pod \"tuned-h7cm4\" (UID: \"3e4730f0-d77e-4bff-b8af-b7fafd4525d8\") " pod="openshift-cluster-node-tuning-operator/tuned-h7cm4"
Oct 30 02:54:54.062406 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:54.062358    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/487caf0f-5f2a-4e93-9c06-de9af02c470b-host\") pod \"node-ca-9mddj\" (UID: \"487caf0f-5f2a-4e93-9c06-de9af02c470b\") " pod="openshift-image-registry/node-ca-9mddj"
Oct 30 02:54:54.062460 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:54.062399    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/98f9efe3-a732-4ba4-8543-941633d22170-original-pull-secret\") pod \"global-pull-secret-syncer-vbnwt\" (UID: \"98f9efe3-a732-4ba4-8543-941633d22170\") " pod="kube-system/global-pull-secret-syncer-vbnwt"
Oct 30 02:54:54.062460 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:54.062428    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/f593f4c2-de6c-42db-b451-d0ade01317f7-konnectivity-ca\") pod \"konnectivity-agent-djpqs\" (UID: \"f593f4c2-de6c-42db-b451-d0ade01317f7\") " pod="kube-system/konnectivity-agent-djpqs"
Oct 30 02:54:54.062527 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:54.062456    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b4ce92fa-1612-4e46-ad6e-23049b2e279b-cnibin\") pod \"multus-9qcgk\" (UID: \"b4ce92fa-1612-4e46-ad6e-23049b2e279b\") " pod="openshift-multus/multus-9qcgk"
Oct 30 02:54:54.062527 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:54.062482    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/042af239-396c-4dae-8cf2-0c562af07c62-iptables-alerter-script\") pod \"iptables-alerter-jhq2g\" (UID: \"042af239-396c-4dae-8cf2-0c562af07c62\") " pod="openshift-network-operator/iptables-alerter-jhq2g"
Oct 30 02:54:54.062527 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:54.062507    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/f425530a-8431-4c5b-8335-8cc1119745c1-ovnkube-script-lib\") pod \"ovnkube-node-g6nqm\" (UID: \"f425530a-8431-4c5b-8335-8cc1119745c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-g6nqm"
Oct 30 02:54:54.062640 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:54.062529    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/3e4730f0-d77e-4bff-b8af-b7fafd4525d8-run\") pod \"tuned-h7cm4\" (UID: \"3e4730f0-d77e-4bff-b8af-b7fafd4525d8\") " pod="openshift-cluster-node-tuning-operator/tuned-h7cm4"
Oct 30 02:54:54.062640 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:54.062562    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4d8bf700-9e09-4741-954b-aba9ccd331c9-system-cni-dir\") pod \"multus-additional-cni-plugins-9l7k8\" (UID: \"4d8bf700-9e09-4741-954b-aba9ccd331c9\") " pod="openshift-multus/multus-additional-cni-plugins-9l7k8"
Oct 30 02:54:54.062640 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:54.062593    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/91a29ddf-25ca-40a8-b920-6d894360a050-registration-dir\") pod \"aws-ebs-csi-driver-node-gqpp2\" (UID: \"91a29ddf-25ca-40a8-b920-6d894360a050\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gqpp2"
Oct 30 02:54:54.062640 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:54.062618    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/b4ce92fa-1612-4e46-ad6e-23049b2e279b-host-run-multus-certs\") pod \"multus-9qcgk\" (UID: \"b4ce92fa-1612-4e46-ad6e-23049b2e279b\") " pod="openshift-multus/multus-9qcgk"
Oct 30 02:54:54.062784 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:54.062642    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/f425530a-8431-4c5b-8335-8cc1119745c1-host-kubelet\") pod \"ovnkube-node-g6nqm\" (UID: \"f425530a-8431-4c5b-8335-8cc1119745c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-g6nqm"
Oct 30 02:54:54.062784 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:54.062667    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lrvbf\" (UniqueName: \"kubernetes.io/projected/f425530a-8431-4c5b-8335-8cc1119745c1-kube-api-access-lrvbf\") pod \"ovnkube-node-g6nqm\" (UID: \"f425530a-8431-4c5b-8335-8cc1119745c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-g6nqm"
Oct 30 02:54:54.062784 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:54.062693    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/487caf0f-5f2a-4e93-9c06-de9af02c470b-serviceca\") pod \"node-ca-9mddj\" (UID: \"487caf0f-5f2a-4e93-9c06-de9af02c470b\") " pod="openshift-image-registry/node-ca-9mddj"
Oct 30 02:54:54.062784 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:54.062716    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b4ce92fa-1612-4e46-ad6e-23049b2e279b-cni-binary-copy\") pod \"multus-9qcgk\" (UID: \"b4ce92fa-1612-4e46-ad6e-23049b2e279b\") " pod="openshift-multus/multus-9qcgk"
Oct 30 02:54:54.062784 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:54.062740    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b4ce92fa-1612-4e46-ad6e-23049b2e279b-host-var-lib-cni-bin\") pod \"multus-9qcgk\" (UID: \"b4ce92fa-1612-4e46-ad6e-23049b2e279b\") " pod="openshift-multus/multus-9qcgk"
Oct 30 02:54:54.062784 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:54.062766    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p764l\" (UniqueName: \"kubernetes.io/projected/0e6f015d-0a2d-4579-ad4d-a88b8c11c3b9-kube-api-access-p764l\") pod \"node-resolver-zhb7b\" (UID: \"0e6f015d-0a2d-4579-ad4d-a88b8c11c3b9\") " pod="openshift-dns/node-resolver-zhb7b"
Oct 30 02:54:54.063058 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:54.062797    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/4d8bf700-9e09-4741-954b-aba9ccd331c9-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-9l7k8\" (UID: \"4d8bf700-9e09-4741-954b-aba9ccd331c9\") " pod="openshift-multus/multus-additional-cni-plugins-9l7k8"
Oct 30 02:54:54.063058 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:54.062826    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pfgdh\" (UniqueName: \"kubernetes.io/projected/4d8bf700-9e09-4741-954b-aba9ccd331c9-kube-api-access-pfgdh\") pod \"multus-additional-cni-plugins-9l7k8\" (UID: \"4d8bf700-9e09-4741-954b-aba9ccd331c9\") " pod="openshift-multus/multus-additional-cni-plugins-9l7k8"
Oct 30 02:54:54.063058 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:54.062851    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/91a29ddf-25ca-40a8-b920-6d894360a050-socket-dir\") pod \"aws-ebs-csi-driver-node-gqpp2\" (UID: \"91a29ddf-25ca-40a8-b920-6d894360a050\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gqpp2"
Oct 30 02:54:54.063058 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:54.062877    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/3e4730f0-d77e-4bff-b8af-b7fafd4525d8-etc-sysctl-conf\") pod \"tuned-h7cm4\" (UID: \"3e4730f0-d77e-4bff-b8af-b7fafd4525d8\") " pod="openshift-cluster-node-tuning-operator/tuned-h7cm4"
Oct 30 02:54:54.063058 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:54.062902    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3e4730f0-d77e-4bff-b8af-b7fafd4525d8-var-lib-kubelet\") pod \"tuned-h7cm4\" (UID: \"3e4730f0-d77e-4bff-b8af-b7fafd4525d8\") " pod="openshift-cluster-node-tuning-operator/tuned-h7cm4"
Oct 30 02:54:54.063058 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:54.062931    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/3e4730f0-d77e-4bff-b8af-b7fafd4525d8-host\") pod \"tuned-h7cm4\" (UID: \"3e4730f0-d77e-4bff-b8af-b7fafd4525d8\") " pod="openshift-cluster-node-tuning-operator/tuned-h7cm4"
Oct 30 02:54:54.063058 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:54.062955    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3bd2c646-cebb-4e8a-b6c3-8b5b74049c11-metrics-certs\") pod \"network-metrics-daemon-mzk2s\" (UID: \"3bd2c646-cebb-4e8a-b6c3-8b5b74049c11\") " pod="openshift-multus/network-metrics-daemon-mzk2s"
Oct 30 02:54:54.063058 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:54.062981    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g55cz\" (UniqueName: \"kubernetes.io/projected/91a29ddf-25ca-40a8-b920-6d894360a050-kube-api-access-g55cz\") pod \"aws-ebs-csi-driver-node-gqpp2\" (UID: \"91a29ddf-25ca-40a8-b920-6d894360a050\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gqpp2"
Oct 30 02:54:54.063058 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:54.063007    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/b4ce92fa-1612-4e46-ad6e-23049b2e279b-host-var-lib-cni-multus\") pod \"multus-9qcgk\" (UID: \"b4ce92fa-1612-4e46-ad6e-23049b2e279b\") " pod="openshift-multus/multus-9qcgk"
Oct 30 02:54:54.063058 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:54.063032    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/f425530a-8431-4c5b-8335-8cc1119745c1-log-socket\") pod \"ovnkube-node-g6nqm\" (UID: \"f425530a-8431-4c5b-8335-8cc1119745c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-g6nqm"
Oct 30 02:54:54.063058 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:54.063055    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/f425530a-8431-4c5b-8335-8cc1119745c1-env-overrides\") pod \"ovnkube-node-g6nqm\" (UID: \"f425530a-8431-4c5b-8335-8cc1119745c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-g6nqm"
Oct 30 02:54:54.063530 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:54.063079    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3e4730f0-d77e-4bff-b8af-b7fafd4525d8-tmp\") pod \"tuned-h7cm4\" (UID: \"3e4730f0-d77e-4bff-b8af-b7fafd4525d8\") " pod="openshift-cluster-node-tuning-operator/tuned-h7cm4"
Oct 30 02:54:54.063530 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:54.063104    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dkmxs\" (UniqueName: \"kubernetes.io/projected/3bd2c646-cebb-4e8a-b6c3-8b5b74049c11-kube-api-access-dkmxs\") pod \"network-metrics-daemon-mzk2s\" (UID: \"3bd2c646-cebb-4e8a-b6c3-8b5b74049c11\") " pod="openshift-multus/network-metrics-daemon-mzk2s"
Oct 30 02:54:54.063530 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:54.063130    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/4d8bf700-9e09-4741-954b-aba9ccd331c9-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-9l7k8\" (UID: \"4d8bf700-9e09-4741-954b-aba9ccd331c9\") " pod="openshift-multus/multus-additional-cni-plugins-9l7k8"
Oct 30 02:54:54.063530 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:54.063174    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/91a29ddf-25ca-40a8-b920-6d894360a050-etc-selinux\") pod \"aws-ebs-csi-driver-node-gqpp2\" (UID: \"91a29ddf-25ca-40a8-b920-6d894360a050\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gqpp2"
Oct 30 02:54:54.063530 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:54.063199    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/3e4730f0-d77e-4bff-b8af-b7fafd4525d8-lib-modules\") pod \"tuned-h7cm4\" (UID: \"3e4730f0-d77e-4bff-b8af-b7fafd4525d8\") " pod="openshift-cluster-node-tuning-operator/tuned-h7cm4"
Oct 30 02:54:54.063530 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:54.063225    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b4ce92fa-1612-4e46-ad6e-23049b2e279b-multus-conf-dir\") pod \"multus-9qcgk\" (UID: \"b4ce92fa-1612-4e46-ad6e-23049b2e279b\") " pod="openshift-multus/multus-9qcgk"
Oct 30 02:54:54.063530 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:54.063254    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/f425530a-8431-4c5b-8335-8cc1119745c1-run-ovn\") pod \"ovnkube-node-g6nqm\" (UID: \"f425530a-8431-4c5b-8335-8cc1119745c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-g6nqm"
Oct 30 02:54:54.063530 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:54.063281    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3e4730f0-d77e-4bff-b8af-b7fafd4525d8-etc-kubernetes\") pod \"tuned-h7cm4\" (UID: \"3e4730f0-d77e-4bff-b8af-b7fafd4525d8\") " pod="openshift-cluster-node-tuning-operator/tuned-h7cm4"
Oct 30 02:54:54.063530 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:54.063306    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xwjvv\" (UniqueName: \"kubernetes.io/projected/3e4730f0-d77e-4bff-b8af-b7fafd4525d8-kube-api-access-xwjvv\") pod \"tuned-h7cm4\" (UID: \"3e4730f0-d77e-4bff-b8af-b7fafd4525d8\") " pod="openshift-cluster-node-tuning-operator/tuned-h7cm4"
Oct 30 02:54:54.063530 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:54.063330    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/4d8bf700-9e09-4741-954b-aba9ccd331c9-cnibin\") pod \"multus-additional-cni-plugins-9l7k8\" (UID: \"4d8bf700-9e09-4741-954b-aba9ccd331c9\") " pod="openshift-multus/multus-additional-cni-plugins-9l7k8"
Oct 30 02:54:54.063530 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:54.063358    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/91a29ddf-25ca-40a8-b920-6d894360a050-kubelet-dir\") pod \"aws-ebs-csi-driver-node-gqpp2\" (UID: \"91a29ddf-25ca-40a8-b920-6d894360a050\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gqpp2"
Oct 30 02:54:54.063530 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:54.063387    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/b4ce92fa-1612-4e46-ad6e-23049b2e279b-multus-socket-dir-parent\") pod \"multus-9qcgk\" (UID: \"b4ce92fa-1612-4e46-ad6e-23049b2e279b\") " pod="openshift-multus/multus-9qcgk"
Oct 30 02:54:54.063530 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:54.063411    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b4ce92fa-1612-4e46-ad6e-23049b2e279b-host-run-netns\") pod \"multus-9qcgk\" (UID: \"b4ce92fa-1612-4e46-ad6e-23049b2e279b\") " pod="openshift-multus/multus-9qcgk"
Oct 30 02:54:54.063530 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:54.063452    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f425530a-8431-4c5b-8335-8cc1119745c1-host-slash\") pod \"ovnkube-node-g6nqm\" (UID: \"f425530a-8431-4c5b-8335-8cc1119745c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-g6nqm"
Oct 30 02:54:54.063530 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:54.063477    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/f425530a-8431-4c5b-8335-8cc1119745c1-run-systemd\") pod \"ovnkube-node-g6nqm\" (UID: \"f425530a-8431-4c5b-8335-8cc1119745c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-g6nqm"
Oct 30 02:54:54.063530 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:54.063514    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/f425530a-8431-4c5b-8335-8cc1119745c1-host-cni-netd\") pod \"ovnkube-node-g6nqm\" (UID: \"f425530a-8431-4c5b-8335-8cc1119745c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-g6nqm"
Oct 30 02:54:54.064285 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:54.063540    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/f425530a-8431-4c5b-8335-8cc1119745c1-ovn-node-metrics-cert\") pod \"ovnkube-node-g6nqm\" (UID: \"f425530a-8431-4c5b-8335-8cc1119745c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-g6nqm"
Oct 30 02:54:54.064285 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:54.063566    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0e6f015d-0a2d-4579-ad4d-a88b8c11c3b9-tmp-dir\") pod \"node-resolver-zhb7b\" (UID: \"0e6f015d-0a2d-4579-ad4d-a88b8c11c3b9\") " pod="openshift-dns/node-resolver-zhb7b"
Oct 30 02:54:54.064285 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:54.063591    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/b4ce92fa-1612-4e46-ad6e-23049b2e279b-host-run-k8s-cni-cncf-io\") pod \"multus-9qcgk\" (UID: \"b4ce92fa-1612-4e46-ad6e-23049b2e279b\") " pod="openshift-multus/multus-9qcgk"
Oct 30 02:54:54.064285 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:54.063616    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wmgr2\" (UniqueName: \"kubernetes.io/projected/b4ce92fa-1612-4e46-ad6e-23049b2e279b-kube-api-access-wmgr2\") pod \"multus-9qcgk\" (UID: \"b4ce92fa-1612-4e46-ad6e-23049b2e279b\") " pod="openshift-multus/multus-9qcgk"
Oct 30 02:54:54.064285 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:54.063641    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f425530a-8431-4c5b-8335-8cc1119745c1-host-cni-bin\") pod \"ovnkube-node-g6nqm\" (UID: \"f425530a-8431-4c5b-8335-8cc1119745c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-g6nqm"
Oct 30 02:54:54.064285 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:54.063666    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/98f9efe3-a732-4ba4-8543-941633d22170-dbus\") pod \"global-pull-secret-syncer-vbnwt\" (UID: \"98f9efe3-a732-4ba4-8543-941633d22170\") " pod="kube-system/global-pull-secret-syncer-vbnwt"
Oct 30 02:54:54.064285 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:54.063691    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b4ce92fa-1612-4e46-ad6e-23049b2e279b-multus-cni-dir\") pod \"multus-9qcgk\" (UID: \"b4ce92fa-1612-4e46-ad6e-23049b2e279b\") " pod="openshift-multus/multus-9qcgk"
Oct 30 02:54:54.064285 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:54.063726    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f425530a-8431-4c5b-8335-8cc1119745c1-host-run-ovn-kubernetes\") pod \"ovnkube-node-g6nqm\" (UID: \"f425530a-8431-4c5b-8335-8cc1119745c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-g6nqm"
Oct 30 02:54:54.064285 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:54.063751    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/f425530a-8431-4c5b-8335-8cc1119745c1-ovnkube-config\") pod \"ovnkube-node-g6nqm\" (UID: \"f425530a-8431-4c5b-8335-8cc1119745c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-g6nqm"
Oct 30 02:54:54.064285 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:54.063776    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/0e6f015d-0a2d-4579-ad4d-a88b8c11c3b9-hosts-file\") pod \"node-resolver-zhb7b\" (UID: \"0e6f015d-0a2d-4579-ad4d-a88b8c11c3b9\") " pod="openshift-dns/node-resolver-zhb7b"
Oct 30 02:54:54.064285 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:54.063802    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/91a29ddf-25ca-40a8-b920-6d894360a050-sys-fs\") pod \"aws-ebs-csi-driver-node-gqpp2\" (UID: \"91a29ddf-25ca-40a8-b920-6d894360a050\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gqpp2"
Oct 30 02:54:54.064285 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:54.063829    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f425530a-8431-4c5b-8335-8cc1119745c1-host-run-netns\") pod \"ovnkube-node-g6nqm\" (UID: \"f425530a-8431-4c5b-8335-8cc1119745c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-g6nqm"
Oct 30 02:54:54.064285 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:54.063854    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/f425530a-8431-4c5b-8335-8cc1119745c1-node-log\") pod \"ovnkube-node-g6nqm\" (UID: \"f425530a-8431-4c5b-8335-8cc1119745c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-g6nqm"
Oct 30 02:54:54.064285 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:54.063881    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3e4730f0-d77e-4bff-b8af-b7fafd4525d8-sys\") pod \"tuned-h7cm4\" (UID: \"3e4730f0-d77e-4bff-b8af-b7fafd4525d8\") " pod="openshift-cluster-node-tuning-operator/tuned-h7cm4"
Oct 30 02:54:54.064285 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:54.063908    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/4d8bf700-9e09-4741-954b-aba9ccd331c9-cni-binary-copy\") pod \"multus-additional-cni-plugins-9l7k8\" (UID: \"4d8bf700-9e09-4741-954b-aba9ccd331c9\") " pod="openshift-multus/multus-additional-cni-plugins-9l7k8"
Oct 30 02:54:54.064285 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:54.063935    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/91a29ddf-25ca-40a8-b920-6d894360a050-device-dir\") pod \"aws-ebs-csi-driver-node-gqpp2\" (UID: \"91a29ddf-25ca-40a8-b920-6d894360a050\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gqpp2"
Oct 30 02:54:54.064966 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:54.063961    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-62vqf\" (UniqueName: \"kubernetes.io/projected/75d9e1ec-cd1b-4aba-b911-5a7465c653cf-kube-api-access-62vqf\") pod \"network-check-target-7rmdj\" (UID: \"75d9e1ec-cd1b-4aba-b911-5a7465c653cf\") " pod="openshift-network-diagnostics/network-check-target-7rmdj"
Oct 30 02:54:54.064966 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:54.063991    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f425530a-8431-4c5b-8335-8cc1119745c1-run-openvswitch\") pod \"ovnkube-node-g6nqm\" (UID: \"f425530a-8431-4c5b-8335-8cc1119745c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-g6nqm"
Oct 30 02:54:54.064966 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:54.064017    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/3e4730f0-d77e-4bff-b8af-b7fafd4525d8-etc-systemd\") pod \"tuned-h7cm4\" (UID: \"3e4730f0-d77e-4bff-b8af-b7fafd4525d8\") " pod="openshift-cluster-node-tuning-operator/tuned-h7cm4"
Oct 30 02:54:54.064966 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:54.064047    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/4d8bf700-9e09-4741-954b-aba9ccd331c9-tuning-conf-dir\") pod \"multus-additional-cni-plugins-9l7k8\" (UID: \"4d8bf700-9e09-4741-954b-aba9ccd331c9\") " pod="openshift-multus/multus-additional-cni-plugins-9l7k8"
Oct 30 02:54:54.064966 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:54.064073    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/042af239-396c-4dae-8cf2-0c562af07c62-host-slash\") pod \"iptables-alerter-jhq2g\" (UID: \"042af239-396c-4dae-8cf2-0c562af07c62\") " pod="openshift-network-operator/iptables-alerter-jhq2g"
Oct 30 02:54:54.064966 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:54.064099    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/f425530a-8431-4c5b-8335-8cc1119745c1-systemd-units\") pod \"ovnkube-node-g6nqm\" (UID: \"f425530a-8431-4c5b-8335-8cc1119745c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-g6nqm"
Oct 30 02:54:54.064966 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:54.064126    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f425530a-8431-4c5b-8335-8cc1119745c1-var-lib-openvswitch\") pod \"ovnkube-node-g6nqm\" (UID: \"f425530a-8431-4c5b-8335-8cc1119745c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-g6nqm"
Oct 30 02:54:54.064966 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:54.064151    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cxnjk\" (UniqueName: \"kubernetes.io/projected/487caf0f-5f2a-4e93-9c06-de9af02c470b-kube-api-access-cxnjk\") pod \"node-ca-9mddj\" (UID: \"487caf0f-5f2a-4e93-9c06-de9af02c470b\") " pod="openshift-image-registry/node-ca-9mddj"
Oct 30 02:54:54.064966 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:54.064197    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/98f9efe3-a732-4ba4-8543-941633d22170-kubelet-config\") pod \"global-pull-secret-syncer-vbnwt\" (UID: \"98f9efe3-a732-4ba4-8543-941633d22170\") " pod="kube-system/global-pull-secret-syncer-vbnwt"
Oct 30 02:54:54.064966 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:54.064244    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-267l7\" (UniqueName: \"kubernetes.io/projected/042af239-396c-4dae-8cf2-0c562af07c62-kube-api-access-267l7\") pod \"iptables-alerter-jhq2g\" (UID: \"042af239-396c-4dae-8cf2-0c562af07c62\") " pod="openshift-network-operator/iptables-alerter-jhq2g"
Oct 30 02:54:54.064966 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:54.064272    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f425530a-8431-4c5b-8335-8cc1119745c1-etc-openvswitch\") pod \"ovnkube-node-g6nqm\" (UID: \"f425530a-8431-4c5b-8335-8cc1119745c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-g6nqm"
Oct 30 02:54:54.064966 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:54.064300    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/f593f4c2-de6c-42db-b451-d0ade01317f7-agent-certs\") pod \"konnectivity-agent-djpqs\" (UID: \"f593f4c2-de6c-42db-b451-d0ade01317f7\") " pod="kube-system/konnectivity-agent-djpqs"
Oct 30 02:54:54.064966 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:54.064326    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/3e4730f0-d77e-4bff-b8af-b7fafd4525d8-etc-tuned\") pod \"tuned-h7cm4\" (UID: \"3e4730f0-d77e-4bff-b8af-b7fafd4525d8\") " pod="openshift-cluster-node-tuning-operator/tuned-h7cm4"
Oct 30 02:54:54.064966 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:54.064354    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b4ce92fa-1612-4e46-ad6e-23049b2e279b-system-cni-dir\") pod \"multus-9qcgk\" (UID: \"b4ce92fa-1612-4e46-ad6e-23049b2e279b\") " pod="openshift-multus/multus-9qcgk"
Oct 30 02:54:54.064966 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:54.064381    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b4ce92fa-1612-4e46-ad6e-23049b2e279b-host-var-lib-kubelet\") pod \"multus-9qcgk\" (UID: \"b4ce92fa-1612-4e46-ad6e-23049b2e279b\") " pod="openshift-multus/multus-9qcgk"
Oct 30 02:54:54.064966 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:54.064421    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/b4ce92fa-1612-4e46-ad6e-23049b2e279b-hostroot\") pod \"multus-9qcgk\" (UID: \"b4ce92fa-1612-4e46-ad6e-23049b2e279b\") " pod="openshift-multus/multus-9qcgk"
Oct 30 02:54:54.065635 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:54.064447    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/b4ce92fa-1612-4e46-ad6e-23049b2e279b-multus-daemon-config\") pod \"multus-9qcgk\" (UID: \"b4ce92fa-1612-4e46-ad6e-23049b2e279b\") " pod="openshift-multus/multus-9qcgk"
Oct 30 02:54:54.065635 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:54.064474    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b4ce92fa-1612-4e46-ad6e-23049b2e279b-etc-kubernetes\") pod \"multus-9qcgk\" (UID: \"b4ce92fa-1612-4e46-ad6e-23049b2e279b\") " pod="openshift-multus/multus-9qcgk"
Oct 30 02:54:54.065635 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:54.064500    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/3e4730f0-d77e-4bff-b8af-b7fafd4525d8-etc-modprobe-d\") pod \"tuned-h7cm4\" (UID: \"3e4730f0-d77e-4bff-b8af-b7fafd4525d8\") " pod="openshift-cluster-node-tuning-operator/tuned-h7cm4"
Oct 30 02:54:54.065635 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:54.064527    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/3e4730f0-d77e-4bff-b8af-b7fafd4525d8-etc-sysconfig\") pod \"tuned-h7cm4\" (UID: \"3e4730f0-d77e-4bff-b8af-b7fafd4525d8\") " pod="openshift-cluster-node-tuning-operator/tuned-h7cm4"
Oct 30 02:54:54.095036 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:54.094878    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Oct 30 02:54:54.095036 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:54.094912    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Oct 30 02:54:54.095036 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:54.094916    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Oct 30 02:54:54.095036 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:54.095035    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Oct 30 02:54:54.095553 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:54.095512    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Oct 30 02:54:54.095663 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:54.095557    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Oct 30 02:54:54.095663 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:54.095631    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Oct 30 02:54:54.095778 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:54.095673    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Oct 30 02:54:54.095778 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:54.095678    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Oct 30 02:54:54.095778 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:54.095677    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-mk2hv\""
Oct 30 02:54:54.095778 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:54.095711    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Oct 30 02:54:54.095778 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:54.095718    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Oct 30 02:54:54.095778 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:54.095730    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Oct 30 02:54:54.095778 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:54.095736    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-6gpqv\""
Oct 30 02:54:54.095778 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:54.095746    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Oct 30 02:54:54.095778 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:54.095760    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Oct 30 02:54:54.096238 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:54.095790    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Oct 30 02:54:54.096238 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:54.095790    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-tpj88\""
Oct 30 02:54:54.096238 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:54.095823    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Oct 30 02:54:54.096238 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:54.095861    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-kn7z4\""
Oct 30 02:54:54.096238 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:54.095950    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Oct 30 02:54:54.096238 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:54.096021    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-fdxtg\""
Oct 30 02:54:54.096238 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:54.096138    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-chr82\""
Oct 30 02:54:54.165328 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:54.165298    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b4ce92fa-1612-4e46-ad6e-23049b2e279b-cnibin\") pod \"multus-9qcgk\" (UID: \"b4ce92fa-1612-4e46-ad6e-23049b2e279b\") " pod="openshift-multus/multus-9qcgk"
Oct 30 02:54:54.165522 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:54.165338    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/042af239-396c-4dae-8cf2-0c562af07c62-iptables-alerter-script\") pod \"iptables-alerter-jhq2g\" (UID: \"042af239-396c-4dae-8cf2-0c562af07c62\") " pod="openshift-network-operator/iptables-alerter-jhq2g"
Oct 30 02:54:54.165522 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:54.165365    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/f425530a-8431-4c5b-8335-8cc1119745c1-ovnkube-script-lib\") pod \"ovnkube-node-g6nqm\" (UID: \"f425530a-8431-4c5b-8335-8cc1119745c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-g6nqm"
Oct 30 02:54:54.165522 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:54.165465    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b4ce92fa-1612-4e46-ad6e-23049b2e279b-cnibin\") pod \"multus-9qcgk\" (UID: \"b4ce92fa-1612-4e46-ad6e-23049b2e279b\") " pod="openshift-multus/multus-9qcgk"
Oct 30 02:54:54.165522 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:54.165517    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/3e4730f0-d77e-4bff-b8af-b7fafd4525d8-run\") pod \"tuned-h7cm4\" (UID: \"3e4730f0-d77e-4bff-b8af-b7fafd4525d8\") " pod="openshift-cluster-node-tuning-operator/tuned-h7cm4"
Oct 30 02:54:54.165742 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:54.165552    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4d8bf700-9e09-4741-954b-aba9ccd331c9-system-cni-dir\") pod \"multus-additional-cni-plugins-9l7k8\" (UID: \"4d8bf700-9e09-4741-954b-aba9ccd331c9\") " pod="openshift-multus/multus-additional-cni-plugins-9l7k8"
Oct 30 02:54:54.165742 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:54.165585    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/91a29ddf-25ca-40a8-b920-6d894360a050-registration-dir\") pod \"aws-ebs-csi-driver-node-gqpp2\" (UID: \"91a29ddf-25ca-40a8-b920-6d894360a050\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gqpp2"
Oct 30 02:54:54.165742 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:54.165612    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/b4ce92fa-1612-4e46-ad6e-23049b2e279b-host-run-multus-certs\") pod \"multus-9qcgk\" (UID: \"b4ce92fa-1612-4e46-ad6e-23049b2e279b\") " pod="openshift-multus/multus-9qcgk"
Oct 30 02:54:54.165742 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:54.165638    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/f425530a-8431-4c5b-8335-8cc1119745c1-host-kubelet\") pod \"ovnkube-node-g6nqm\" (UID: \"f425530a-8431-4c5b-8335-8cc1119745c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-g6nqm"
Oct 30 02:54:54.165742 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:54.165664    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lrvbf\" (UniqueName: \"kubernetes.io/projected/f425530a-8431-4c5b-8335-8cc1119745c1-kube-api-access-lrvbf\") pod \"ovnkube-node-g6nqm\" (UID: \"f425530a-8431-4c5b-8335-8cc1119745c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-g6nqm"
Oct 30 02:54:54.165742 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:54.165722    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/487caf0f-5f2a-4e93-9c06-de9af02c470b-serviceca\") pod \"node-ca-9mddj\" (UID: \"487caf0f-5f2a-4e93-9c06-de9af02c470b\") " pod="openshift-image-registry/node-ca-9mddj"
Oct 30 02:54:54.166047 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:54.165748    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b4ce92fa-1612-4e46-ad6e-23049b2e279b-cni-binary-copy\") pod \"multus-9qcgk\" (UID: \"b4ce92fa-1612-4e46-ad6e-23049b2e279b\") " pod="openshift-multus/multus-9qcgk"
Oct 30 02:54:54.166047 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:54.165774    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b4ce92fa-1612-4e46-ad6e-23049b2e279b-host-var-lib-cni-bin\") pod \"multus-9qcgk\" (UID: \"b4ce92fa-1612-4e46-ad6e-23049b2e279b\") " pod="openshift-multus/multus-9qcgk"
Oct 30 02:54:54.166047 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:54.165801    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p764l\" (UniqueName: \"kubernetes.io/projected/0e6f015d-0a2d-4579-ad4d-a88b8c11c3b9-kube-api-access-p764l\") pod \"node-resolver-zhb7b\" (UID: \"0e6f015d-0a2d-4579-ad4d-a88b8c11c3b9\") " pod="openshift-dns/node-resolver-zhb7b"
Oct 30 02:54:54.166047 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:54.165829    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/4d8bf700-9e09-4741-954b-aba9ccd331c9-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-9l7k8\" (UID: \"4d8bf700-9e09-4741-954b-aba9ccd331c9\") " pod="openshift-multus/multus-additional-cni-plugins-9l7k8"
Oct 30 02:54:54.166047 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:54.165860    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pfgdh\" (UniqueName: \"kubernetes.io/projected/4d8bf700-9e09-4741-954b-aba9ccd331c9-kube-api-access-pfgdh\") pod \"multus-additional-cni-plugins-9l7k8\" (UID: \"4d8bf700-9e09-4741-954b-aba9ccd331c9\") " pod="openshift-multus/multus-additional-cni-plugins-9l7k8"
Oct 30 02:54:54.166047 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:54.165887    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/91a29ddf-25ca-40a8-b920-6d894360a050-socket-dir\") pod \"aws-ebs-csi-driver-node-gqpp2\" (UID: \"91a29ddf-25ca-40a8-b920-6d894360a050\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gqpp2"
Oct 30 02:54:54.166047 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:54.165914    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/3e4730f0-d77e-4bff-b8af-b7fafd4525d8-etc-sysctl-conf\") pod \"tuned-h7cm4\" (UID: \"3e4730f0-d77e-4bff-b8af-b7fafd4525d8\") " pod="openshift-cluster-node-tuning-operator/tuned-h7cm4"
Oct 30 02:54:54.166047 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:54.165939    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3e4730f0-d77e-4bff-b8af-b7fafd4525d8-var-lib-kubelet\") pod \"tuned-h7cm4\" (UID: \"3e4730f0-d77e-4bff-b8af-b7fafd4525d8\") " pod="openshift-cluster-node-tuning-operator/tuned-h7cm4"
Oct 30 02:54:54.166047 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:54.165972    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/042af239-396c-4dae-8cf2-0c562af07c62-iptables-alerter-script\") pod \"iptables-alerter-jhq2g\" (UID: \"042af239-396c-4dae-8cf2-0c562af07c62\") " pod="openshift-network-operator/iptables-alerter-jhq2g"
Oct 30 02:54:54.166047 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:54.165962    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/3e4730f0-d77e-4bff-b8af-b7fafd4525d8-host\") pod \"tuned-h7cm4\" (UID: \"3e4730f0-d77e-4bff-b8af-b7fafd4525d8\") " pod="openshift-cluster-node-tuning-operator/tuned-h7cm4"
Oct 30 02:54:54.166047 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:54.166008    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3bd2c646-cebb-4e8a-b6c3-8b5b74049c11-metrics-certs\") pod \"network-metrics-daemon-mzk2s\" (UID: \"3bd2c646-cebb-4e8a-b6c3-8b5b74049c11\") " pod="openshift-multus/network-metrics-daemon-mzk2s"
Oct 30 02:54:54.166047 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:54.166035    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g55cz\" (UniqueName: \"kubernetes.io/projected/91a29ddf-25ca-40a8-b920-6d894360a050-kube-api-access-g55cz\") pod \"aws-ebs-csi-driver-node-gqpp2\" (UID: \"91a29ddf-25ca-40a8-b920-6d894360a050\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gqpp2"
Oct 30 02:54:54.166670 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:54.166066    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/b4ce92fa-1612-4e46-ad6e-23049b2e279b-host-var-lib-cni-multus\") pod \"multus-9qcgk\" (UID: \"b4ce92fa-1612-4e46-ad6e-23049b2e279b\") " pod="openshift-multus/multus-9qcgk"
Oct 30 02:54:54.166670 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:54.166092    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/f425530a-8431-4c5b-8335-8cc1119745c1-log-socket\") pod \"ovnkube-node-g6nqm\" (UID: \"f425530a-8431-4c5b-8335-8cc1119745c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-g6nqm"
Oct 30 02:54:54.166670 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:54.166117    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/f425530a-8431-4c5b-8335-8cc1119745c1-env-overrides\") pod \"ovnkube-node-g6nqm\" (UID: \"f425530a-8431-4c5b-8335-8cc1119745c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-g6nqm"
Oct 30 02:54:54.166670 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:54.166142    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3e4730f0-d77e-4bff-b8af-b7fafd4525d8-tmp\") pod \"tuned-h7cm4\" (UID: \"3e4730f0-d77e-4bff-b8af-b7fafd4525d8\") " pod="openshift-cluster-node-tuning-operator/tuned-h7cm4"
Oct 30 02:54:54.166670 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:54.166189    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dkmxs\" (UniqueName: \"kubernetes.io/projected/3bd2c646-cebb-4e8a-b6c3-8b5b74049c11-kube-api-access-dkmxs\") pod \"network-metrics-daemon-mzk2s\" (UID: \"3bd2c646-cebb-4e8a-b6c3-8b5b74049c11\") " pod="openshift-multus/network-metrics-daemon-mzk2s"
Oct 30 02:54:54.166670 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:54.166221    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/4d8bf700-9e09-4741-954b-aba9ccd331c9-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-9l7k8\" (UID: \"4d8bf700-9e09-4741-954b-aba9ccd331c9\") " pod="openshift-multus/multus-additional-cni-plugins-9l7k8"
Oct 30 02:54:54.166670 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:54.166249    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/91a29ddf-25ca-40a8-b920-6d894360a050-etc-selinux\") pod \"aws-ebs-csi-driver-node-gqpp2\" (UID: \"91a29ddf-25ca-40a8-b920-6d894360a050\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gqpp2"
Oct 30 02:54:54.166670 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:54.166275    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/3e4730f0-d77e-4bff-b8af-b7fafd4525d8-lib-modules\") pod \"tuned-h7cm4\" (UID: \"3e4730f0-d77e-4bff-b8af-b7fafd4525d8\") " pod="openshift-cluster-node-tuning-operator/tuned-h7cm4"
Oct 30 02:54:54.166670 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:54.166301    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b4ce92fa-1612-4e46-ad6e-23049b2e279b-multus-conf-dir\") pod \"multus-9qcgk\" (UID: \"b4ce92fa-1612-4e46-ad6e-23049b2e279b\") " pod="openshift-multus/multus-9qcgk"
Oct 30 02:54:54.166670 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:54.166327    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/f425530a-8431-4c5b-8335-8cc1119745c1-run-ovn\") pod \"ovnkube-node-g6nqm\" (UID: \"f425530a-8431-4c5b-8335-8cc1119745c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-g6nqm"
Oct 30 02:54:54.166670 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:54.166355    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3e4730f0-d77e-4bff-b8af-b7fafd4525d8-etc-kubernetes\") pod \"tuned-h7cm4\" (UID: \"3e4730f0-d77e-4bff-b8af-b7fafd4525d8\") " pod="openshift-cluster-node-tuning-operator/tuned-h7cm4"
Oct 30 02:54:54.166670 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:54.166393    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xwjvv\" (UniqueName: \"kubernetes.io/projected/3e4730f0-d77e-4bff-b8af-b7fafd4525d8-kube-api-access-xwjvv\") pod \"tuned-h7cm4\" (UID: \"3e4730f0-d77e-4bff-b8af-b7fafd4525d8\") " pod="openshift-cluster-node-tuning-operator/tuned-h7cm4"
Oct 30 02:54:54.166670 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:54.166420    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/4d8bf700-9e09-4741-954b-aba9ccd331c9-cnibin\") pod \"multus-additional-cni-plugins-9l7k8\" (UID: \"4d8bf700-9e09-4741-954b-aba9ccd331c9\") " pod="openshift-multus/multus-additional-cni-plugins-9l7k8"
Oct 30 02:54:54.166670 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:54.166447    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/91a29ddf-25ca-40a8-b920-6d894360a050-kubelet-dir\") pod \"aws-ebs-csi-driver-node-gqpp2\" (UID: \"91a29ddf-25ca-40a8-b920-6d894360a050\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gqpp2"
Oct 30 02:54:54.166670 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:54.166466    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b4ce92fa-1612-4e46-ad6e-23049b2e279b-cni-binary-copy\") pod \"multus-9qcgk\" (UID: \"b4ce92fa-1612-4e46-ad6e-23049b2e279b\") " pod="openshift-multus/multus-9qcgk"
Oct 30 02:54:54.166670 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:54.166476    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/b4ce92fa-1612-4e46-ad6e-23049b2e279b-multus-socket-dir-parent\") pod \"multus-9qcgk\" (UID: \"b4ce92fa-1612-4e46-ad6e-23049b2e279b\") " pod="openshift-multus/multus-9qcgk"
Oct 30 02:54:54.166670 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:54.166504    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b4ce92fa-1612-4e46-ad6e-23049b2e279b-host-run-netns\") pod \"multus-9qcgk\" (UID: \"b4ce92fa-1612-4e46-ad6e-23049b2e279b\") " pod="openshift-multus/multus-9qcgk"
Oct 30 02:54:54.167494 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:54.166545    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f425530a-8431-4c5b-8335-8cc1119745c1-host-slash\") pod \"ovnkube-node-g6nqm\" (UID: \"f425530a-8431-4c5b-8335-8cc1119745c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-g6nqm"
Oct 30 02:54:54.167494 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:54.166572    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/f425530a-8431-4c5b-8335-8cc1119745c1-run-systemd\") pod \"ovnkube-node-g6nqm\" (UID: \"f425530a-8431-4c5b-8335-8cc1119745c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-g6nqm"
Oct 30 02:54:54.167494 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:54.166601    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/f425530a-8431-4c5b-8335-8cc1119745c1-host-cni-netd\") pod \"ovnkube-node-g6nqm\" (UID: \"f425530a-8431-4c5b-8335-8cc1119745c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-g6nqm"
Oct 30 02:54:54.167494 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:54.166630    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/f425530a-8431-4c5b-8335-8cc1119745c1-ovn-node-metrics-cert\") pod \"ovnkube-node-g6nqm\" (UID: \"f425530a-8431-4c5b-8335-8cc1119745c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-g6nqm"
Oct 30 02:54:54.167494 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:54.166657    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0e6f015d-0a2d-4579-ad4d-a88b8c11c3b9-tmp-dir\") pod \"node-resolver-zhb7b\" (UID: \"0e6f015d-0a2d-4579-ad4d-a88b8c11c3b9\") " pod="openshift-dns/node-resolver-zhb7b"
Oct 30 02:54:54.167494 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:54.166681    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/b4ce92fa-1612-4e46-ad6e-23049b2e279b-host-run-k8s-cni-cncf-io\") pod \"multus-9qcgk\" (UID: \"b4ce92fa-1612-4e46-ad6e-23049b2e279b\") " pod="openshift-multus/multus-9qcgk"
Oct 30 02:54:54.167494 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:54.166708    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wmgr2\" (UniqueName: \"kubernetes.io/projected/b4ce92fa-1612-4e46-ad6e-23049b2e279b-kube-api-access-wmgr2\") pod \"multus-9qcgk\" (UID: \"b4ce92fa-1612-4e46-ad6e-23049b2e279b\") " pod="openshift-multus/multus-9qcgk"
Oct 30 02:54:54.167494 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:54.166736    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f425530a-8431-4c5b-8335-8cc1119745c1-host-cni-bin\") pod \"ovnkube-node-g6nqm\" (UID: \"f425530a-8431-4c5b-8335-8cc1119745c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-g6nqm"
Oct 30 02:54:54.167494 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:54.166760    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/98f9efe3-a732-4ba4-8543-941633d22170-dbus\") pod \"global-pull-secret-syncer-vbnwt\" (UID: \"98f9efe3-a732-4ba4-8543-941633d22170\") " pod="kube-system/global-pull-secret-syncer-vbnwt"
Oct 30 02:54:54.167494 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:54.166786    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b4ce92fa-1612-4e46-ad6e-23049b2e279b-multus-cni-dir\") pod \"multus-9qcgk\" (UID: \"b4ce92fa-1612-4e46-ad6e-23049b2e279b\") " pod="openshift-multus/multus-9qcgk"
Oct 30 02:54:54.167494 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:54.166813    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f425530a-8431-4c5b-8335-8cc1119745c1-host-run-ovn-kubernetes\") pod \"ovnkube-node-g6nqm\" (UID: \"f425530a-8431-4c5b-8335-8cc1119745c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-g6nqm"
Oct 30 02:54:54.167494 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:54.166840    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/f425530a-8431-4c5b-8335-8cc1119745c1-ovnkube-config\") pod \"ovnkube-node-g6nqm\" (UID: \"f425530a-8431-4c5b-8335-8cc1119745c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-g6nqm"
Oct 30 02:54:54.167494 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:54.166869    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/0e6f015d-0a2d-4579-ad4d-a88b8c11c3b9-hosts-file\") pod \"node-resolver-zhb7b\" (UID: \"0e6f015d-0a2d-4579-ad4d-a88b8c11c3b9\") " pod="openshift-dns/node-resolver-zhb7b"
Oct 30 02:54:54.167494 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:54.166896    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/91a29ddf-25ca-40a8-b920-6d894360a050-sys-fs\") pod \"aws-ebs-csi-driver-node-gqpp2\" (UID: \"91a29ddf-25ca-40a8-b920-6d894360a050\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gqpp2"
Oct 30 02:54:54.167494 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:54.166915    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/f425530a-8431-4c5b-8335-8cc1119745c1-ovnkube-script-lib\") pod \"ovnkube-node-g6nqm\" (UID: \"f425530a-8431-4c5b-8335-8cc1119745c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-g6nqm"
Oct 30 02:54:54.167494 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:54.166922    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f425530a-8431-4c5b-8335-8cc1119745c1-host-run-netns\") pod \"ovnkube-node-g6nqm\" (UID: \"f425530a-8431-4c5b-8335-8cc1119745c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-g6nqm"
Oct 30 02:54:54.167494 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:54.166962    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/f425530a-8431-4c5b-8335-8cc1119745c1-node-log\") pod \"ovnkube-node-g6nqm\" (UID: \"f425530a-8431-4c5b-8335-8cc1119745c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-g6nqm"
Oct 30 02:54:54.167494 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:54.166991    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3e4730f0-d77e-4bff-b8af-b7fafd4525d8-sys\") pod \"tuned-h7cm4\" (UID: \"3e4730f0-d77e-4bff-b8af-b7fafd4525d8\") " pod="openshift-cluster-node-tuning-operator/tuned-h7cm4"
Oct 30 02:54:54.168416 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:54.167024    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/4d8bf700-9e09-4741-954b-aba9ccd331c9-cni-binary-copy\") pod \"multus-additional-cni-plugins-9l7k8\" (UID: \"4d8bf700-9e09-4741-954b-aba9ccd331c9\") " pod="openshift-multus/multus-additional-cni-plugins-9l7k8"
Oct 30 02:54:54.168416 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:54.167044    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/3e4730f0-d77e-4bff-b8af-b7fafd4525d8-run\") pod \"tuned-h7cm4\" (UID: \"3e4730f0-d77e-4bff-b8af-b7fafd4525d8\") " pod="openshift-cluster-node-tuning-operator/tuned-h7cm4"
Oct 30 02:54:54.168416 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:54.167052    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/91a29ddf-25ca-40a8-b920-6d894360a050-device-dir\") pod \"aws-ebs-csi-driver-node-gqpp2\" (UID: \"91a29ddf-25ca-40a8-b920-6d894360a050\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gqpp2"
Oct 30 02:54:54.168416 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:54.167079    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4d8bf700-9e09-4741-954b-aba9ccd331c9-system-cni-dir\") pod \"multus-additional-cni-plugins-9l7k8\" (UID: \"4d8bf700-9e09-4741-954b-aba9ccd331c9\") " pod="openshift-multus/multus-additional-cni-plugins-9l7k8"
Oct 30 02:54:54.168416 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:54.167081    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-62vqf\" (UniqueName: \"kubernetes.io/projected/75d9e1ec-cd1b-4aba-b911-5a7465c653cf-kube-api-access-62vqf\") pod \"network-check-target-7rmdj\" (UID: \"75d9e1ec-cd1b-4aba-b911-5a7465c653cf\") " pod="openshift-network-diagnostics/network-check-target-7rmdj"
Oct 30 02:54:54.168416 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:54.167133    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/b4ce92fa-1612-4e46-ad6e-23049b2e279b-host-run-multus-certs\") pod \"multus-9qcgk\" (UID: \"b4ce92fa-1612-4e46-ad6e-23049b2e279b\") " pod="openshift-multus/multus-9qcgk"
Oct 30 02:54:54.168416 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:54.167151    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b4ce92fa-1612-4e46-ad6e-23049b2e279b-host-var-lib-cni-bin\") pod \"multus-9qcgk\" (UID: \"b4ce92fa-1612-4e46-ad6e-23049b2e279b\") " pod="openshift-multus/multus-9qcgk"
Oct 30 02:54:54.168416 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:54.167193    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/91a29ddf-25ca-40a8-b920-6d894360a050-registration-dir\") pod \"aws-ebs-csi-driver-node-gqpp2\" (UID: \"91a29ddf-25ca-40a8-b920-6d894360a050\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gqpp2"
Oct 30 02:54:54.168416 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:54.167227    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/f425530a-8431-4c5b-8335-8cc1119745c1-host-kubelet\") pod \"ovnkube-node-g6nqm\" (UID: \"f425530a-8431-4c5b-8335-8cc1119745c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-g6nqm"
Oct 30 02:54:54.168416 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:54.167897    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/487caf0f-5f2a-4e93-9c06-de9af02c470b-serviceca\") pod \"node-ca-9mddj\" (UID: \"487caf0f-5f2a-4e93-9c06-de9af02c470b\") " pod="openshift-image-registry/node-ca-9mddj"
Oct 30 02:54:54.168416 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:54.167952    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b4ce92fa-1612-4e46-ad6e-23049b2e279b-host-run-netns\") pod \"multus-9qcgk\" (UID: \"b4ce92fa-1612-4e46-ad6e-23049b2e279b\") " pod="openshift-multus/multus-9qcgk"
Oct 30 02:54:54.168416 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:54.167956    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/4d8bf700-9e09-4741-954b-aba9ccd331c9-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-9l7k8\" (UID: \"4d8bf700-9e09-4741-954b-aba9ccd331c9\") " pod="openshift-multus/multus-additional-cni-plugins-9l7k8"
Oct 30 02:54:54.168416 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:54.168030    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f425530a-8431-4c5b-8335-8cc1119745c1-host-slash\") pod \"ovnkube-node-g6nqm\" (UID: \"f425530a-8431-4c5b-8335-8cc1119745c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-g6nqm"
Oct 30 02:54:54.168416 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:54.168083    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/f425530a-8431-4c5b-8335-8cc1119745c1-run-systemd\") pod \"ovnkube-node-g6nqm\" (UID: \"f425530a-8431-4c5b-8335-8cc1119745c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-g6nqm"
Oct 30 02:54:54.168416 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:54.168130    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/f425530a-8431-4c5b-8335-8cc1119745c1-host-cni-netd\") pod \"ovnkube-node-g6nqm\" (UID: \"f425530a-8431-4c5b-8335-8cc1119745c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-g6nqm"
Oct 30 02:54:54.168416 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:54.168248    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/91a29ddf-25ca-40a8-b920-6d894360a050-socket-dir\") pod \"aws-ebs-csi-driver-node-gqpp2\" (UID: \"91a29ddf-25ca-40a8-b920-6d894360a050\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gqpp2"
Oct 30 02:54:54.168416 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:54.168399    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/3e4730f0-d77e-4bff-b8af-b7fafd4525d8-etc-sysctl-conf\") pod \"tuned-h7cm4\" (UID: \"3e4730f0-d77e-4bff-b8af-b7fafd4525d8\") " pod="openshift-cluster-node-tuning-operator/tuned-h7cm4"
Oct 30 02:54:54.169225 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:54.168505    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/91a29ddf-25ca-40a8-b920-6d894360a050-etc-selinux\") pod \"aws-ebs-csi-driver-node-gqpp2\" (UID: \"91a29ddf-25ca-40a8-b920-6d894360a050\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gqpp2"
Oct 30 02:54:54.169225 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:54.168580    2574 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Oct 30 02:54:54.169225 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:54.168657    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/3e4730f0-d77e-4bff-b8af-b7fafd4525d8-lib-modules\") pod \"tuned-h7cm4\" (UID: \"3e4730f0-d77e-4bff-b8af-b7fafd4525d8\") " pod="openshift-cluster-node-tuning-operator/tuned-h7cm4"
Oct 30 02:54:54.169225 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:54.168715    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b4ce92fa-1612-4e46-ad6e-23049b2e279b-multus-conf-dir\") pod \"multus-9qcgk\" (UID: \"b4ce92fa-1612-4e46-ad6e-23049b2e279b\") " pod="openshift-multus/multus-9qcgk"
Oct 30 02:54:54.169225 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:54.168765    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/f425530a-8431-4c5b-8335-8cc1119745c1-run-ovn\") pod \"ovnkube-node-g6nqm\" (UID: \"f425530a-8431-4c5b-8335-8cc1119745c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-g6nqm"
Oct 30 02:54:54.169225 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:54.168779    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/4d8bf700-9e09-4741-954b-aba9ccd331c9-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-9l7k8\" (UID: \"4d8bf700-9e09-4741-954b-aba9ccd331c9\") " pod="openshift-multus/multus-additional-cni-plugins-9l7k8"
Oct 30 02:54:54.169225 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:54.168843    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3e4730f0-d77e-4bff-b8af-b7fafd4525d8-var-lib-kubelet\") pod \"tuned-h7cm4\" (UID: \"3e4730f0-d77e-4bff-b8af-b7fafd4525d8\") " pod="openshift-cluster-node-tuning-operator/tuned-h7cm4"
Oct 30 02:54:54.169225 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:54.168860    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3e4730f0-d77e-4bff-b8af-b7fafd4525d8-etc-kubernetes\") pod \"tuned-h7cm4\" (UID: \"3e4730f0-d77e-4bff-b8af-b7fafd4525d8\") " pod="openshift-cluster-node-tuning-operator/tuned-h7cm4"
Oct 30 02:54:54.169225 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:54.168895    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/3e4730f0-d77e-4bff-b8af-b7fafd4525d8-host\") pod \"tuned-h7cm4\" (UID: \"3e4730f0-d77e-4bff-b8af-b7fafd4525d8\") " pod="openshift-cluster-node-tuning-operator/tuned-h7cm4"
Oct 30 02:54:54.169225 ip-10-0-143-156 kubenswrapper[2574]: E1030 02:54:54.168977    2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 02:54:54.169225 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:54.169132    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/4d8bf700-9e09-4741-954b-aba9ccd331c9-cnibin\") pod \"multus-additional-cni-plugins-9l7k8\" (UID: \"4d8bf700-9e09-4741-954b-aba9ccd331c9\") " pod="openshift-multus/multus-additional-cni-plugins-9l7k8"
Oct 30 02:54:54.169225 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:54.169212    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/91a29ddf-25ca-40a8-b920-6d894360a050-kubelet-dir\") pod \"aws-ebs-csi-driver-node-gqpp2\" (UID: \"91a29ddf-25ca-40a8-b920-6d894360a050\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gqpp2"
Oct 30 02:54:54.169826 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:54.169276    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/b4ce92fa-1612-4e46-ad6e-23049b2e279b-multus-socket-dir-parent\") pod \"multus-9qcgk\" (UID: \"b4ce92fa-1612-4e46-ad6e-23049b2e279b\") " pod="openshift-multus/multus-9qcgk"
Oct 30 02:54:54.169826 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:54.169328    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/b4ce92fa-1612-4e46-ad6e-23049b2e279b-host-var-lib-cni-multus\") pod \"multus-9qcgk\" (UID: \"b4ce92fa-1612-4e46-ad6e-23049b2e279b\") " pod="openshift-multus/multus-9qcgk"
Oct 30 02:54:54.169826 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:54.169448    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/f425530a-8431-4c5b-8335-8cc1119745c1-env-overrides\") pod \"ovnkube-node-g6nqm\" (UID: \"f425530a-8431-4c5b-8335-8cc1119745c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-g6nqm"
Oct 30 02:54:54.169826 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:54.169520    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/f425530a-8431-4c5b-8335-8cc1119745c1-log-socket\") pod \"ovnkube-node-g6nqm\" (UID: \"f425530a-8431-4c5b-8335-8cc1119745c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-g6nqm"
Oct 30 02:54:54.169826 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:54.169751    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/0e6f015d-0a2d-4579-ad4d-a88b8c11c3b9-hosts-file\") pod \"node-resolver-zhb7b\" (UID: \"0e6f015d-0a2d-4579-ad4d-a88b8c11c3b9\") " pod="openshift-dns/node-resolver-zhb7b"
Oct 30 02:54:54.170041 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:54.170029    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0e6f015d-0a2d-4579-ad4d-a88b8c11c3b9-tmp-dir\") pod \"node-resolver-zhb7b\" (UID: \"0e6f015d-0a2d-4579-ad4d-a88b8c11c3b9\") " pod="openshift-dns/node-resolver-zhb7b"
Oct 30 02:54:54.170077 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:54.170034    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/91a29ddf-25ca-40a8-b920-6d894360a050-sys-fs\") pod \"aws-ebs-csi-driver-node-gqpp2\" (UID: \"91a29ddf-25ca-40a8-b920-6d894360a050\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gqpp2"
Oct 30 02:54:54.170114 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:54.170093    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f425530a-8431-4c5b-8335-8cc1119745c1-host-run-netns\") pod \"ovnkube-node-g6nqm\" (UID: \"f425530a-8431-4c5b-8335-8cc1119745c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-g6nqm"
Oct 30 02:54:54.170185 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:54.170117    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/b4ce92fa-1612-4e46-ad6e-23049b2e279b-host-run-k8s-cni-cncf-io\") pod \"multus-9qcgk\" (UID: \"b4ce92fa-1612-4e46-ad6e-23049b2e279b\") " pod="openshift-multus/multus-9qcgk"
Oct 30 02:54:54.170185 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:54.170140    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/f425530a-8431-4c5b-8335-8cc1119745c1-node-log\") pod \"ovnkube-node-g6nqm\" (UID: \"f425530a-8431-4c5b-8335-8cc1119745c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-g6nqm"
Oct 30 02:54:54.170292 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:54.170206    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/91a29ddf-25ca-40a8-b920-6d894360a050-device-dir\") pod \"aws-ebs-csi-driver-node-gqpp2\" (UID: \"91a29ddf-25ca-40a8-b920-6d894360a050\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gqpp2"
Oct 30 02:54:54.170292 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:54.170220    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3e4730f0-d77e-4bff-b8af-b7fafd4525d8-sys\") pod \"tuned-h7cm4\" (UID: \"3e4730f0-d77e-4bff-b8af-b7fafd4525d8\") " pod="openshift-cluster-node-tuning-operator/tuned-h7cm4"
Oct 30 02:54:54.170292 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:54.170278    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f425530a-8431-4c5b-8335-8cc1119745c1-run-openvswitch\") pod \"ovnkube-node-g6nqm\" (UID: \"f425530a-8431-4c5b-8335-8cc1119745c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-g6nqm"
Oct 30 02:54:54.170440 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:54.170318    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/3e4730f0-d77e-4bff-b8af-b7fafd4525d8-etc-systemd\") pod \"tuned-h7cm4\" (UID: \"3e4730f0-d77e-4bff-b8af-b7fafd4525d8\") " pod="openshift-cluster-node-tuning-operator/tuned-h7cm4"
Oct 30 02:54:54.170440 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:54.170349    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/4d8bf700-9e09-4741-954b-aba9ccd331c9-tuning-conf-dir\") pod \"multus-additional-cni-plugins-9l7k8\" (UID: \"4d8bf700-9e09-4741-954b-aba9ccd331c9\") " pod="openshift-multus/multus-additional-cni-plugins-9l7k8"
Oct 30 02:54:54.170440 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:54.170363    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b4ce92fa-1612-4e46-ad6e-23049b2e279b-multus-cni-dir\") pod \"multus-9qcgk\" (UID: \"b4ce92fa-1612-4e46-ad6e-23049b2e279b\") " pod="openshift-multus/multus-9qcgk"
Oct 30 02:54:54.170440 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:54.170424    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/3e4730f0-d77e-4bff-b8af-b7fafd4525d8-etc-systemd\") pod \"tuned-h7cm4\" (UID: \"3e4730f0-d77e-4bff-b8af-b7fafd4525d8\") " pod="openshift-cluster-node-tuning-operator/tuned-h7cm4"
Oct 30 02:54:54.170440 ip-10-0-143-156 kubenswrapper[2574]: E1030 02:54:54.170427    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3bd2c646-cebb-4e8a-b6c3-8b5b74049c11-metrics-certs podName:3bd2c646-cebb-4e8a-b6c3-8b5b74049c11 nodeName:}" failed. No retries permitted until 2026-10-30 02:54:54.67040498 +0000 UTC m=+3.089156539 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3bd2c646-cebb-4e8a-b6c3-8b5b74049c11-metrics-certs") pod "network-metrics-daemon-mzk2s" (UID: "3bd2c646-cebb-4e8a-b6c3-8b5b74049c11") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 02:54:54.170679 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:54.170454    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f425530a-8431-4c5b-8335-8cc1119745c1-host-run-ovn-kubernetes\") pod \"ovnkube-node-g6nqm\" (UID: \"f425530a-8431-4c5b-8335-8cc1119745c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-g6nqm"
Oct 30 02:54:54.170679 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:54.170466    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f425530a-8431-4c5b-8335-8cc1119745c1-host-cni-bin\") pod \"ovnkube-node-g6nqm\" (UID: \"f425530a-8431-4c5b-8335-8cc1119745c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-g6nqm"
Oct 30 02:54:54.170679 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:54.170493    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/042af239-396c-4dae-8cf2-0c562af07c62-host-slash\") pod \"iptables-alerter-jhq2g\" (UID: \"042af239-396c-4dae-8cf2-0c562af07c62\") " pod="openshift-network-operator/iptables-alerter-jhq2g"
Oct 30 02:54:54.170679 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:54.170519    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/f425530a-8431-4c5b-8335-8cc1119745c1-systemd-units\") pod \"ovnkube-node-g6nqm\" (UID: \"f425530a-8431-4c5b-8335-8cc1119745c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-g6nqm"
Oct 30 02:54:54.170679 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:54.170533    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/98f9efe3-a732-4ba4-8543-941633d22170-dbus\") pod \"global-pull-secret-syncer-vbnwt\" (UID: \"98f9efe3-a732-4ba4-8543-941633d22170\") " pod="kube-system/global-pull-secret-syncer-vbnwt"
Oct 30 02:54:54.170679 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:54.170547    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f425530a-8431-4c5b-8335-8cc1119745c1-var-lib-openvswitch\") pod \"ovnkube-node-g6nqm\" (UID: \"f425530a-8431-4c5b-8335-8cc1119745c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-g6nqm"
Oct 30 02:54:54.170679 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:54.170587    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cxnjk\" (UniqueName: \"kubernetes.io/projected/487caf0f-5f2a-4e93-9c06-de9af02c470b-kube-api-access-cxnjk\") pod \"node-ca-9mddj\" (UID: \"487caf0f-5f2a-4e93-9c06-de9af02c470b\") " pod="openshift-image-registry/node-ca-9mddj"
Oct 30 02:54:54.170679 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:54.170594    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f425530a-8431-4c5b-8335-8cc1119745c1-var-lib-openvswitch\") pod \"ovnkube-node-g6nqm\" (UID: \"f425530a-8431-4c5b-8335-8cc1119745c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-g6nqm"
Oct 30 02:54:54.170679 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:54.170621    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/98f9efe3-a732-4ba4-8543-941633d22170-kubelet-config\") pod \"global-pull-secret-syncer-vbnwt\" (UID: \"98f9efe3-a732-4ba4-8543-941633d22170\") " pod="kube-system/global-pull-secret-syncer-vbnwt"
Oct 30 02:54:54.170679 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:54.170653    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-267l7\" (UniqueName: \"kubernetes.io/projected/042af239-396c-4dae-8cf2-0c562af07c62-kube-api-access-267l7\") pod \"iptables-alerter-jhq2g\" (UID: \"042af239-396c-4dae-8cf2-0c562af07c62\") " pod="openshift-network-operator/iptables-alerter-jhq2g"
Oct 30 02:54:54.170679 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:54.170682    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f425530a-8431-4c5b-8335-8cc1119745c1-etc-openvswitch\") pod \"ovnkube-node-g6nqm\" (UID: \"f425530a-8431-4c5b-8335-8cc1119745c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-g6nqm"
Oct 30 02:54:54.171202 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:54.170728    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f425530a-8431-4c5b-8335-8cc1119745c1-etc-openvswitch\") pod \"ovnkube-node-g6nqm\" (UID: \"f425530a-8431-4c5b-8335-8cc1119745c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-g6nqm"
Oct 30 02:54:54.171202 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:54.170724    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/042af239-396c-4dae-8cf2-0c562af07c62-host-slash\") pod \"iptables-alerter-jhq2g\" (UID: \"042af239-396c-4dae-8cf2-0c562af07c62\") " pod="openshift-network-operator/iptables-alerter-jhq2g"
Oct 30 02:54:54.171202 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:54.170760    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/f425530a-8431-4c5b-8335-8cc1119745c1-systemd-units\") pod \"ovnkube-node-g6nqm\" (UID: \"f425530a-8431-4c5b-8335-8cc1119745c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-g6nqm"
Oct 30 02:54:54.171202 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:54.170797    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/98f9efe3-a732-4ba4-8543-941633d22170-kubelet-config\") pod \"global-pull-secret-syncer-vbnwt\" (UID: \"98f9efe3-a732-4ba4-8543-941633d22170\") " pod="kube-system/global-pull-secret-syncer-vbnwt"
Oct 30 02:54:54.171202 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:54.170816    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/f593f4c2-de6c-42db-b451-d0ade01317f7-agent-certs\") pod \"konnectivity-agent-djpqs\" (UID: \"f593f4c2-de6c-42db-b451-d0ade01317f7\") " pod="kube-system/konnectivity-agent-djpqs"
Oct 30 02:54:54.171202 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:54.170849    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/3e4730f0-d77e-4bff-b8af-b7fafd4525d8-etc-tuned\") pod \"tuned-h7cm4\" (UID: \"3e4730f0-d77e-4bff-b8af-b7fafd4525d8\") " pod="openshift-cluster-node-tuning-operator/tuned-h7cm4"
Oct 30 02:54:54.171202 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:54.170862    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f425530a-8431-4c5b-8335-8cc1119745c1-run-openvswitch\") pod \"ovnkube-node-g6nqm\" (UID: \"f425530a-8431-4c5b-8335-8cc1119745c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-g6nqm"
Oct 30 02:54:54.171202 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:54.170866    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b4ce92fa-1612-4e46-ad6e-23049b2e279b-system-cni-dir\") pod \"multus-9qcgk\" (UID: \"b4ce92fa-1612-4e46-ad6e-23049b2e279b\") " pod="openshift-multus/multus-9qcgk"
Oct 30 02:54:54.171202 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:54.170921    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b4ce92fa-1612-4e46-ad6e-23049b2e279b-system-cni-dir\") pod \"multus-9qcgk\" (UID: \"b4ce92fa-1612-4e46-ad6e-23049b2e279b\") " pod="openshift-multus/multus-9qcgk"
Oct 30 02:54:54.171202 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:54.170930    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b4ce92fa-1612-4e46-ad6e-23049b2e279b-host-var-lib-kubelet\") pod \"multus-9qcgk\" (UID: \"b4ce92fa-1612-4e46-ad6e-23049b2e279b\") " pod="openshift-multus/multus-9qcgk"
Oct 30 02:54:54.171202 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:54.170959    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/b4ce92fa-1612-4e46-ad6e-23049b2e279b-hostroot\") pod \"multus-9qcgk\" (UID: \"b4ce92fa-1612-4e46-ad6e-23049b2e279b\") " pod="openshift-multus/multus-9qcgk"
Oct 30 02:54:54.171202 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:54.170986    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/b4ce92fa-1612-4e46-ad6e-23049b2e279b-multus-daemon-config\") pod \"multus-9qcgk\" (UID: \"b4ce92fa-1612-4e46-ad6e-23049b2e279b\") " pod="openshift-multus/multus-9qcgk"
Oct 30 02:54:54.171202 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:54.170997    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b4ce92fa-1612-4e46-ad6e-23049b2e279b-host-var-lib-kubelet\") pod \"multus-9qcgk\" (UID: \"b4ce92fa-1612-4e46-ad6e-23049b2e279b\") " pod="openshift-multus/multus-9qcgk"
Oct 30 02:54:54.171202 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:54.171012    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b4ce92fa-1612-4e46-ad6e-23049b2e279b-etc-kubernetes\") pod \"multus-9qcgk\" (UID: \"b4ce92fa-1612-4e46-ad6e-23049b2e279b\") " pod="openshift-multus/multus-9qcgk"
Oct 30 02:54:54.171202 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:54.171036    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/b4ce92fa-1612-4e46-ad6e-23049b2e279b-hostroot\") pod \"multus-9qcgk\" (UID: \"b4ce92fa-1612-4e46-ad6e-23049b2e279b\") " pod="openshift-multus/multus-9qcgk"
Oct 30 02:54:54.171202 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:54.171041    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/3e4730f0-d77e-4bff-b8af-b7fafd4525d8-etc-modprobe-d\") pod \"tuned-h7cm4\" (UID: \"3e4730f0-d77e-4bff-b8af-b7fafd4525d8\") " pod="openshift-cluster-node-tuning-operator/tuned-h7cm4"
Oct 30 02:54:54.171202 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:54.171068    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/3e4730f0-d77e-4bff-b8af-b7fafd4525d8-etc-sysconfig\") pod \"tuned-h7cm4\" (UID: \"3e4730f0-d77e-4bff-b8af-b7fafd4525d8\") " pod="openshift-cluster-node-tuning-operator/tuned-h7cm4"
Oct 30 02:54:54.171202 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:54.171079    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b4ce92fa-1612-4e46-ad6e-23049b2e279b-etc-kubernetes\") pod \"multus-9qcgk\" (UID: \"b4ce92fa-1612-4e46-ad6e-23049b2e279b\") " pod="openshift-multus/multus-9qcgk"
Oct 30 02:54:54.172053 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:54.171094    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f425530a-8431-4c5b-8335-8cc1119745c1-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-g6nqm\" (UID: \"f425530a-8431-4c5b-8335-8cc1119745c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-g6nqm"
Oct 30 02:54:54.172053 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:54.171135    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f425530a-8431-4c5b-8335-8cc1119745c1-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-g6nqm\" (UID: \"f425530a-8431-4c5b-8335-8cc1119745c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-g6nqm"
Oct 30 02:54:54.172053 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:54.171098    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/3e4730f0-d77e-4bff-b8af-b7fafd4525d8-etc-modprobe-d\") pod \"tuned-h7cm4\" (UID: \"3e4730f0-d77e-4bff-b8af-b7fafd4525d8\") " pod="openshift-cluster-node-tuning-operator/tuned-h7cm4"
Oct 30 02:54:54.172053 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:54.171137    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/3e4730f0-d77e-4bff-b8af-b7fafd4525d8-etc-sysctl-d\") pod \"tuned-h7cm4\" (UID: \"3e4730f0-d77e-4bff-b8af-b7fafd4525d8\") " pod="openshift-cluster-node-tuning-operator/tuned-h7cm4"
Oct 30 02:54:54.172053 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:54.171157    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/3e4730f0-d77e-4bff-b8af-b7fafd4525d8-etc-sysconfig\") pod \"tuned-h7cm4\" (UID: \"3e4730f0-d77e-4bff-b8af-b7fafd4525d8\") " pod="openshift-cluster-node-tuning-operator/tuned-h7cm4"
Oct 30 02:54:54.172053 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:54.171181    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/487caf0f-5f2a-4e93-9c06-de9af02c470b-host\") pod \"node-ca-9mddj\" (UID: \"487caf0f-5f2a-4e93-9c06-de9af02c470b\") " pod="openshift-image-registry/node-ca-9mddj"
Oct 30 02:54:54.172053 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:54.171209    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/3e4730f0-d77e-4bff-b8af-b7fafd4525d8-etc-sysctl-d\") pod \"tuned-h7cm4\" (UID: \"3e4730f0-d77e-4bff-b8af-b7fafd4525d8\") " pod="openshift-cluster-node-tuning-operator/tuned-h7cm4"
Oct 30 02:54:54.172053 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:54.171210    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/98f9efe3-a732-4ba4-8543-941633d22170-original-pull-secret\") pod \"global-pull-secret-syncer-vbnwt\" (UID: \"98f9efe3-a732-4ba4-8543-941633d22170\") " pod="kube-system/global-pull-secret-syncer-vbnwt"
Oct 30 02:54:54.172053 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:54.171273    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/487caf0f-5f2a-4e93-9c06-de9af02c470b-host\") pod \"node-ca-9mddj\" (UID: \"487caf0f-5f2a-4e93-9c06-de9af02c470b\") " pod="openshift-image-registry/node-ca-9mddj"
Oct 30 02:54:54.172053 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:54.171296    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/f593f4c2-de6c-42db-b451-d0ade01317f7-konnectivity-ca\") pod \"konnectivity-agent-djpqs\" (UID: \"f593f4c2-de6c-42db-b451-d0ade01317f7\") " pod="kube-system/konnectivity-agent-djpqs"
Oct 30 02:54:54.172053 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:54.171371    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/4d8bf700-9e09-4741-954b-aba9ccd331c9-tuning-conf-dir\") pod \"multus-additional-cni-plugins-9l7k8\" (UID: \"4d8bf700-9e09-4741-954b-aba9ccd331c9\") " pod="openshift-multus/multus-additional-cni-plugins-9l7k8"
Oct 30 02:54:54.172053 ip-10-0-143-156 kubenswrapper[2574]: E1030 02:54:54.171383    2574 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 02:54:54.172053 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:54.171471    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/f425530a-8431-4c5b-8335-8cc1119745c1-ovnkube-config\") pod \"ovnkube-node-g6nqm\" (UID: \"f425530a-8431-4c5b-8335-8cc1119745c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-g6nqm"
Oct 30 02:54:54.172053 ip-10-0-143-156 kubenswrapper[2574]: E1030 02:54:54.171514    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/98f9efe3-a732-4ba4-8543-941633d22170-original-pull-secret podName:98f9efe3-a732-4ba4-8543-941633d22170 nodeName:}" failed. No retries permitted until 2026-10-30 02:54:54.671465304 +0000 UTC m=+3.090216871 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/98f9efe3-a732-4ba4-8543-941633d22170-original-pull-secret") pod "global-pull-secret-syncer-vbnwt" (UID: "98f9efe3-a732-4ba4-8543-941633d22170") : object "kube-system"/"original-pull-secret" not registered
Oct 30 02:54:54.172053 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:54.171637    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/4d8bf700-9e09-4741-954b-aba9ccd331c9-cni-binary-copy\") pod \"multus-additional-cni-plugins-9l7k8\" (UID: \"4d8bf700-9e09-4741-954b-aba9ccd331c9\") " pod="openshift-multus/multus-additional-cni-plugins-9l7k8"
Oct 30 02:54:54.172960 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:54.172926    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/b4ce92fa-1612-4e46-ad6e-23049b2e279b-multus-daemon-config\") pod \"multus-9qcgk\" (UID: \"b4ce92fa-1612-4e46-ad6e-23049b2e279b\") " pod="openshift-multus/multus-9qcgk"
Oct 30 02:54:54.173271 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:54.173247    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/f425530a-8431-4c5b-8335-8cc1119745c1-ovn-node-metrics-cert\") pod \"ovnkube-node-g6nqm\" (UID: \"f425530a-8431-4c5b-8335-8cc1119745c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-g6nqm"
Oct 30 02:54:54.173387 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:54.173250    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/f593f4c2-de6c-42db-b451-d0ade01317f7-konnectivity-ca\") pod \"konnectivity-agent-djpqs\" (UID: \"f593f4c2-de6c-42db-b451-d0ade01317f7\") " pod="kube-system/konnectivity-agent-djpqs"
Oct 30 02:54:54.173546 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:54.173432    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3e4730f0-d77e-4bff-b8af-b7fafd4525d8-tmp\") pod \"tuned-h7cm4\" (UID: \"3e4730f0-d77e-4bff-b8af-b7fafd4525d8\") " pod="openshift-cluster-node-tuning-operator/tuned-h7cm4"
Oct 30 02:54:54.174015 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:54.173995    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/3e4730f0-d77e-4bff-b8af-b7fafd4525d8-etc-tuned\") pod \"tuned-h7cm4\" (UID: \"3e4730f0-d77e-4bff-b8af-b7fafd4525d8\") " pod="openshift-cluster-node-tuning-operator/tuned-h7cm4"
Oct 30 02:54:54.174426 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:54.174406    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/f593f4c2-de6c-42db-b451-d0ade01317f7-agent-certs\") pod \"konnectivity-agent-djpqs\" (UID: \"f593f4c2-de6c-42db-b451-d0ade01317f7\") " pod="kube-system/konnectivity-agent-djpqs"
Oct 30 02:54:54.200734 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:54.200662    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pfgdh\" (UniqueName: \"kubernetes.io/projected/4d8bf700-9e09-4741-954b-aba9ccd331c9-kube-api-access-pfgdh\") pod \"multus-additional-cni-plugins-9l7k8\" (UID: \"4d8bf700-9e09-4741-954b-aba9ccd331c9\") " pod="openshift-multus/multus-additional-cni-plugins-9l7k8"
Oct 30 02:54:54.200734 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:54.200688    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p764l\" (UniqueName: \"kubernetes.io/projected/0e6f015d-0a2d-4579-ad4d-a88b8c11c3b9-kube-api-access-p764l\") pod \"node-resolver-zhb7b\" (UID: \"0e6f015d-0a2d-4579-ad4d-a88b8c11c3b9\") " pod="openshift-dns/node-resolver-zhb7b"
Oct 30 02:54:54.200734 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:54.200670    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lrvbf\" (UniqueName: \"kubernetes.io/projected/f425530a-8431-4c5b-8335-8cc1119745c1-kube-api-access-lrvbf\") pod \"ovnkube-node-g6nqm\" (UID: \"f425530a-8431-4c5b-8335-8cc1119745c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-g6nqm"
Oct 30 02:54:54.201146 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:54.201112    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xwjvv\" (UniqueName: \"kubernetes.io/projected/3e4730f0-d77e-4bff-b8af-b7fafd4525d8-kube-api-access-xwjvv\") pod \"tuned-h7cm4\" (UID: \"3e4730f0-d77e-4bff-b8af-b7fafd4525d8\") " pod="openshift-cluster-node-tuning-operator/tuned-h7cm4"
Oct 30 02:54:54.207541 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:54.207516    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g55cz\" (UniqueName: \"kubernetes.io/projected/91a29ddf-25ca-40a8-b920-6d894360a050-kube-api-access-g55cz\") pod \"aws-ebs-csi-driver-node-gqpp2\" (UID: \"91a29ddf-25ca-40a8-b920-6d894360a050\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gqpp2"
Oct 30 02:54:54.208959 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:54.208937    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cxnjk\" (UniqueName: \"kubernetes.io/projected/487caf0f-5f2a-4e93-9c06-de9af02c470b-kube-api-access-cxnjk\") pod \"node-ca-9mddj\" (UID: \"487caf0f-5f2a-4e93-9c06-de9af02c470b\") " pod="openshift-image-registry/node-ca-9mddj"
Oct 30 02:54:54.209424 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:54.209381    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dkmxs\" (UniqueName: \"kubernetes.io/projected/3bd2c646-cebb-4e8a-b6c3-8b5b74049c11-kube-api-access-dkmxs\") pod \"network-metrics-daemon-mzk2s\" (UID: \"3bd2c646-cebb-4e8a-b6c3-8b5b74049c11\") " pod="openshift-multus/network-metrics-daemon-mzk2s"
Oct 30 02:54:54.215204 ip-10-0-143-156 kubenswrapper[2574]: E1030 02:54:54.215184    2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 02:54:54.215291 ip-10-0-143-156 kubenswrapper[2574]: E1030 02:54:54.215210    2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 02:54:54.215291 ip-10-0-143-156 kubenswrapper[2574]: E1030 02:54:54.215224    2574 projected.go:196] Error preparing data for projected volume kube-api-access-62vqf for pod openshift-network-diagnostics/network-check-target-7rmdj: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 02:54:54.215291 ip-10-0-143-156 kubenswrapper[2574]: E1030 02:54:54.215287    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/75d9e1ec-cd1b-4aba-b911-5a7465c653cf-kube-api-access-62vqf podName:75d9e1ec-cd1b-4aba-b911-5a7465c653cf nodeName:}" failed. No retries permitted until 2026-10-30 02:54:54.715267416 +0000 UTC m=+3.134018985 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-62vqf" (UniqueName: "kubernetes.io/projected/75d9e1ec-cd1b-4aba-b911-5a7465c653cf-kube-api-access-62vqf") pod "network-check-target-7rmdj" (UID: "75d9e1ec-cd1b-4aba-b911-5a7465c653cf") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 02:54:54.217748 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:54.217723    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-267l7\" (UniqueName: \"kubernetes.io/projected/042af239-396c-4dae-8cf2-0c562af07c62-kube-api-access-267l7\") pod \"iptables-alerter-jhq2g\" (UID: \"042af239-396c-4dae-8cf2-0c562af07c62\") " pod="openshift-network-operator/iptables-alerter-jhq2g"
Oct 30 02:54:54.217861 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:54.217726    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wmgr2\" (UniqueName: \"kubernetes.io/projected/b4ce92fa-1612-4e46-ad6e-23049b2e279b-kube-api-access-wmgr2\") pod \"multus-9qcgk\" (UID: \"b4ce92fa-1612-4e46-ad6e-23049b2e279b\") " pod="openshift-multus/multus-9qcgk"
Oct 30 02:54:54.288328 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:54.288297    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gqpp2"
Oct 30 02:54:54.294868 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:54.294844    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-9qcgk"
Oct 30 02:54:54.301957 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:54.301932    2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod91a29ddf_25ca_40a8_b920_6d894360a050.slice/crio-93e392332e3ee47fd777cfeb398f91bde31f4aebfc88e480418d207a6966bee4 WatchSource:0}: Error finding container 93e392332e3ee47fd777cfeb398f91bde31f4aebfc88e480418d207a6966bee4: Status 404 returned error can't find the container with id 93e392332e3ee47fd777cfeb398f91bde31f4aebfc88e480418d207a6966bee4
Oct 30 02:54:54.303570 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:54.303393    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-jhq2g"
Oct 30 02:54:54.305746 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:54.305679    2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb4ce92fa_1612_4e46_ad6e_23049b2e279b.slice/crio-01245521c7a3ffefdfc2ae351f991af2ac42fc81f1927e05a8bdcb10a6c2e7b4 WatchSource:0}: Error finding container 01245521c7a3ffefdfc2ae351f991af2ac42fc81f1927e05a8bdcb10a6c2e7b4: Status 404 returned error can't find the container with id 01245521c7a3ffefdfc2ae351f991af2ac42fc81f1927e05a8bdcb10a6c2e7b4
Oct 30 02:54:54.314882 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:54.314854    2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod042af239_396c_4dae_8cf2_0c562af07c62.slice/crio-5e965e654016cb1c63687a11574c47db32be4910c883fd9274864b7a4daefd2e WatchSource:0}: Error finding container 5e965e654016cb1c63687a11574c47db32be4910c883fd9274864b7a4daefd2e: Status 404 returned error can't find the container with id 5e965e654016cb1c63687a11574c47db32be4910c883fd9274864b7a4daefd2e
Oct 30 02:54:54.327021 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:54.327000    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-zhb7b"
Oct 30 02:54:54.329778 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:54.329753    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-9mddj"
Oct 30 02:54:54.338917 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:54.338841    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-g6nqm"
Oct 30 02:54:54.341416 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:54.341373    2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e6f015d_0a2d_4579_ad4d_a88b8c11c3b9.slice/crio-983647bc364434314f920b45b26eed44790a22e8e35bf617694fe3d4cdbf5472 WatchSource:0}: Error finding container 983647bc364434314f920b45b26eed44790a22e8e35bf617694fe3d4cdbf5472: Status 404 returned error can't find the container with id 983647bc364434314f920b45b26eed44790a22e8e35bf617694fe3d4cdbf5472
Oct 30 02:54:54.342030 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:54.342005    2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod487caf0f_5f2a_4e93_9c06_de9af02c470b.slice/crio-9ea919af44be23142fcc6b67dcda63896ad4a5ce6c7b8bb5cc040d04b6bcfd89 WatchSource:0}: Error finding container 9ea919af44be23142fcc6b67dcda63896ad4a5ce6c7b8bb5cc040d04b6bcfd89: Status 404 returned error can't find the container with id 9ea919af44be23142fcc6b67dcda63896ad4a5ce6c7b8bb5cc040d04b6bcfd89
Oct 30 02:54:54.350148 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:54.350121    2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf425530a_8431_4c5b_8335_8cc1119745c1.slice/crio-1bae9e54a248d6ce344f3c1732f29332e4fdb208afd0e898287edc45c7c9db0d WatchSource:0}: Error finding container 1bae9e54a248d6ce344f3c1732f29332e4fdb208afd0e898287edc45c7c9db0d: Status 404 returned error can't find the container with id 1bae9e54a248d6ce344f3c1732f29332e4fdb208afd0e898287edc45c7c9db0d
Oct 30 02:54:54.355525 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:54.355338    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-h7cm4"
Oct 30 02:54:54.358202 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:54.358163    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-9l7k8"
Oct 30 02:54:54.360573 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:54.360524    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-djpqs"
Oct 30 02:54:54.375783 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:54.375760    2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3e4730f0_d77e_4bff_b8af_b7fafd4525d8.slice/crio-b22762b81ef676c76a4c60099db6f94c1ed533e25b371a4f6212e16b419fadd6 WatchSource:0}: Error finding container b22762b81ef676c76a4c60099db6f94c1ed533e25b371a4f6212e16b419fadd6: Status 404 returned error can't find the container with id b22762b81ef676c76a4c60099db6f94c1ed533e25b371a4f6212e16b419fadd6
Oct 30 02:54:54.381994 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:54.381970    2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4d8bf700_9e09_4741_954b_aba9ccd331c9.slice/crio-e31a65f04fc04593d238960000b7d26efdc16d169ec94e69691b59bfd0fab887 WatchSource:0}: Error finding container e31a65f04fc04593d238960000b7d26efdc16d169ec94e69691b59bfd0fab887: Status 404 returned error can't find the container with id e31a65f04fc04593d238960000b7d26efdc16d169ec94e69691b59bfd0fab887
Oct 30 02:54:54.382860 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:54:54.382837    2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf593f4c2_de6c_42db_b451_d0ade01317f7.slice/crio-5a05ec0ff1f0a8c904755e4c7aed0f098d5700a36d8f8911725fb313e2cbe6ca WatchSource:0}: Error finding container 5a05ec0ff1f0a8c904755e4c7aed0f098d5700a36d8f8911725fb313e2cbe6ca: Status 404 returned error can't find the container with id 5a05ec0ff1f0a8c904755e4c7aed0f098d5700a36d8f8911725fb313e2cbe6ca
Oct 30 02:54:54.437960 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:54.437933    2574 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 02:54:54.674385 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:54.674356    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3bd2c646-cebb-4e8a-b6c3-8b5b74049c11-metrics-certs\") pod \"network-metrics-daemon-mzk2s\" (UID: \"3bd2c646-cebb-4e8a-b6c3-8b5b74049c11\") " pod="openshift-multus/network-metrics-daemon-mzk2s"
Oct 30 02:54:54.674550 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:54.674458    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/98f9efe3-a732-4ba4-8543-941633d22170-original-pull-secret\") pod \"global-pull-secret-syncer-vbnwt\" (UID: \"98f9efe3-a732-4ba4-8543-941633d22170\") " pod="kube-system/global-pull-secret-syncer-vbnwt"
Oct 30 02:54:54.674550 ip-10-0-143-156 kubenswrapper[2574]: E1030 02:54:54.674521    2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 02:54:54.674687 ip-10-0-143-156 kubenswrapper[2574]: E1030 02:54:54.674598    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3bd2c646-cebb-4e8a-b6c3-8b5b74049c11-metrics-certs podName:3bd2c646-cebb-4e8a-b6c3-8b5b74049c11 nodeName:}" failed. No retries permitted until 2026-10-30 02:54:55.67458026 +0000 UTC m=+4.093331827 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3bd2c646-cebb-4e8a-b6c3-8b5b74049c11-metrics-certs") pod "network-metrics-daemon-mzk2s" (UID: "3bd2c646-cebb-4e8a-b6c3-8b5b74049c11") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 02:54:54.674687 ip-10-0-143-156 kubenswrapper[2574]: E1030 02:54:54.674629    2574 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 02:54:54.674774 ip-10-0-143-156 kubenswrapper[2574]: E1030 02:54:54.674694    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/98f9efe3-a732-4ba4-8543-941633d22170-original-pull-secret podName:98f9efe3-a732-4ba4-8543-941633d22170 nodeName:}" failed. No retries permitted until 2026-10-30 02:54:55.674669441 +0000 UTC m=+4.093420997 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/98f9efe3-a732-4ba4-8543-941633d22170-original-pull-secret") pod "global-pull-secret-syncer-vbnwt" (UID: "98f9efe3-a732-4ba4-8543-941633d22170") : object "kube-system"/"original-pull-secret" not registered
Oct 30 02:54:54.775310 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:54.774927    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-62vqf\" (UniqueName: \"kubernetes.io/projected/75d9e1ec-cd1b-4aba-b911-5a7465c653cf-kube-api-access-62vqf\") pod \"network-check-target-7rmdj\" (UID: \"75d9e1ec-cd1b-4aba-b911-5a7465c653cf\") " pod="openshift-network-diagnostics/network-check-target-7rmdj"
Oct 30 02:54:54.775493 ip-10-0-143-156 kubenswrapper[2574]: E1030 02:54:54.775378    2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 02:54:54.775493 ip-10-0-143-156 kubenswrapper[2574]: E1030 02:54:54.775398    2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 02:54:54.775493 ip-10-0-143-156 kubenswrapper[2574]: E1030 02:54:54.775412    2574 projected.go:196] Error preparing data for projected volume kube-api-access-62vqf for pod openshift-network-diagnostics/network-check-target-7rmdj: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 02:54:54.775663 ip-10-0-143-156 kubenswrapper[2574]: E1030 02:54:54.775499    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/75d9e1ec-cd1b-4aba-b911-5a7465c653cf-kube-api-access-62vqf podName:75d9e1ec-cd1b-4aba-b911-5a7465c653cf nodeName:}" failed. No retries permitted until 2026-10-30 02:54:55.77546896 +0000 UTC m=+4.194220531 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-62vqf" (UniqueName: "kubernetes.io/projected/75d9e1ec-cd1b-4aba-b911-5a7465c653cf-kube-api-access-62vqf") pod "network-check-target-7rmdj" (UID: "75d9e1ec-cd1b-4aba-b911-5a7465c653cf") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 02:54:55.035076 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:55.034950    2574 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 02:49:54 +0000 UTC" deadline="2028-05-01 05:28:47.264085877 +0000 UTC"
Oct 30 02:54:55.035076 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:55.034981    2574 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13178h33m52.229108905s"
Oct 30 02:54:55.038987 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:55.038954    2574 generic.go:356] "Generic (PLEG): container finished" podID="a7672f7a60a28c85d738430783c2cc68" containerID="4529048daffa3640c95d6c055dd0203665f53ee135946c50a4c94b98454d9d86" exitCode=0
Oct 30 02:54:55.039366 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:55.039150    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-156.ec2.internal" event={"ID":"a7672f7a60a28c85d738430783c2cc68","Type":"ContainerDied","Data":"4529048daffa3640c95d6c055dd0203665f53ee135946c50a4c94b98454d9d86"}
Oct 30 02:54:55.053497 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:55.053427    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9l7k8" event={"ID":"4d8bf700-9e09-4741-954b-aba9ccd331c9","Type":"ContainerStarted","Data":"e31a65f04fc04593d238960000b7d26efdc16d169ec94e69691b59bfd0fab887"}
Oct 30 02:54:55.060719 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:55.060645    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-h7cm4" event={"ID":"3e4730f0-d77e-4bff-b8af-b7fafd4525d8","Type":"ContainerStarted","Data":"b22762b81ef676c76a4c60099db6f94c1ed533e25b371a4f6212e16b419fadd6"}
Oct 30 02:54:55.068285 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:55.068253    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-g6nqm" event={"ID":"f425530a-8431-4c5b-8335-8cc1119745c1","Type":"ContainerStarted","Data":"1bae9e54a248d6ce344f3c1732f29332e4fdb208afd0e898287edc45c7c9db0d"}
Oct 30 02:54:55.083916 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:55.083879    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-zhb7b" event={"ID":"0e6f015d-0a2d-4579-ad4d-a88b8c11c3b9","Type":"ContainerStarted","Data":"983647bc364434314f920b45b26eed44790a22e8e35bf617694fe3d4cdbf5472"}
Oct 30 02:54:55.094875 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:55.094842    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-djpqs" event={"ID":"f593f4c2-de6c-42db-b451-d0ade01317f7","Type":"ContainerStarted","Data":"5a05ec0ff1f0a8c904755e4c7aed0f098d5700a36d8f8911725fb313e2cbe6ca"}
Oct 30 02:54:55.107025 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:55.106932    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-9mddj" event={"ID":"487caf0f-5f2a-4e93-9c06-de9af02c470b","Type":"ContainerStarted","Data":"9ea919af44be23142fcc6b67dcda63896ad4a5ce6c7b8bb5cc040d04b6bcfd89"}
Oct 30 02:54:55.137414 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:55.137340    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-jhq2g" event={"ID":"042af239-396c-4dae-8cf2-0c562af07c62","Type":"ContainerStarted","Data":"5e965e654016cb1c63687a11574c47db32be4910c883fd9274864b7a4daefd2e"}
Oct 30 02:54:55.148004 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:55.147934    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-9qcgk" event={"ID":"b4ce92fa-1612-4e46-ad6e-23049b2e279b","Type":"ContainerStarted","Data":"01245521c7a3ffefdfc2ae351f991af2ac42fc81f1927e05a8bdcb10a6c2e7b4"}
Oct 30 02:54:55.174130 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:55.174090    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gqpp2" event={"ID":"91a29ddf-25ca-40a8-b920-6d894360a050","Type":"ContainerStarted","Data":"93e392332e3ee47fd777cfeb398f91bde31f4aebfc88e480418d207a6966bee4"}
Oct 30 02:54:55.199063 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:55.198162    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-156.ec2.internal" event={"ID":"fee27b4a19dca64d5627dff27844f71f","Type":"ContainerStarted","Data":"3274dbbacadbe10e5b82128eb1fe384bfd6fad9641cd4f56785bb90c54ad93f2"}
Oct 30 02:54:55.245147 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:55.244416    2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-156.ec2.internal" podStartSLOduration=2.244395931 podStartE2EDuration="2.244395931s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 02:54:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:54:55.24382009 +0000 UTC m=+3.662571668" watchObservedRunningTime="2026-10-30 02:54:55.244395931 +0000 UTC m=+3.663147510"
Oct 30 02:54:55.686511 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:55.686422    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/98f9efe3-a732-4ba4-8543-941633d22170-original-pull-secret\") pod \"global-pull-secret-syncer-vbnwt\" (UID: \"98f9efe3-a732-4ba4-8543-941633d22170\") " pod="kube-system/global-pull-secret-syncer-vbnwt"
Oct 30 02:54:55.686511 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:55.686490    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3bd2c646-cebb-4e8a-b6c3-8b5b74049c11-metrics-certs\") pod \"network-metrics-daemon-mzk2s\" (UID: \"3bd2c646-cebb-4e8a-b6c3-8b5b74049c11\") " pod="openshift-multus/network-metrics-daemon-mzk2s"
Oct 30 02:54:55.686756 ip-10-0-143-156 kubenswrapper[2574]: E1030 02:54:55.686627    2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 02:54:55.686756 ip-10-0-143-156 kubenswrapper[2574]: E1030 02:54:55.686694    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3bd2c646-cebb-4e8a-b6c3-8b5b74049c11-metrics-certs podName:3bd2c646-cebb-4e8a-b6c3-8b5b74049c11 nodeName:}" failed. No retries permitted until 2026-10-30 02:54:57.68667237 +0000 UTC m=+6.105423927 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3bd2c646-cebb-4e8a-b6c3-8b5b74049c11-metrics-certs") pod "network-metrics-daemon-mzk2s" (UID: "3bd2c646-cebb-4e8a-b6c3-8b5b74049c11") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 02:54:55.687194 ip-10-0-143-156 kubenswrapper[2574]: E1030 02:54:55.687120    2574 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 02:54:55.687194 ip-10-0-143-156 kubenswrapper[2574]: E1030 02:54:55.687190    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/98f9efe3-a732-4ba4-8543-941633d22170-original-pull-secret podName:98f9efe3-a732-4ba4-8543-941633d22170 nodeName:}" failed. No retries permitted until 2026-10-30 02:54:57.687160722 +0000 UTC m=+6.105912279 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/98f9efe3-a732-4ba4-8543-941633d22170-original-pull-secret") pod "global-pull-secret-syncer-vbnwt" (UID: "98f9efe3-a732-4ba4-8543-941633d22170") : object "kube-system"/"original-pull-secret" not registered
Oct 30 02:54:55.787080 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:55.787044    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-62vqf\" (UniqueName: \"kubernetes.io/projected/75d9e1ec-cd1b-4aba-b911-5a7465c653cf-kube-api-access-62vqf\") pod \"network-check-target-7rmdj\" (UID: \"75d9e1ec-cd1b-4aba-b911-5a7465c653cf\") " pod="openshift-network-diagnostics/network-check-target-7rmdj"
Oct 30 02:54:55.787285 ip-10-0-143-156 kubenswrapper[2574]: E1030 02:54:55.787259    2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 02:54:55.787285 ip-10-0-143-156 kubenswrapper[2574]: E1030 02:54:55.787281    2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 02:54:55.787416 ip-10-0-143-156 kubenswrapper[2574]: E1030 02:54:55.787294    2574 projected.go:196] Error preparing data for projected volume kube-api-access-62vqf for pod openshift-network-diagnostics/network-check-target-7rmdj: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 02:54:55.787416 ip-10-0-143-156 kubenswrapper[2574]: E1030 02:54:55.787360    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/75d9e1ec-cd1b-4aba-b911-5a7465c653cf-kube-api-access-62vqf podName:75d9e1ec-cd1b-4aba-b911-5a7465c653cf nodeName:}" failed. No retries permitted until 2026-10-30 02:54:57.787339702 +0000 UTC m=+6.206091254 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-62vqf" (UniqueName: "kubernetes.io/projected/75d9e1ec-cd1b-4aba-b911-5a7465c653cf-kube-api-access-62vqf") pod "network-check-target-7rmdj" (UID: "75d9e1ec-cd1b-4aba-b911-5a7465c653cf") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 02:54:56.029309 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:56.026739    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vbnwt"
Oct 30 02:54:56.029309 ip-10-0-143-156 kubenswrapper[2574]: E1030 02:54:56.026878    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vbnwt" podUID="98f9efe3-a732-4ba4-8543-941633d22170"
Oct 30 02:54:56.029309 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:56.027333    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7rmdj"
Oct 30 02:54:56.029309 ip-10-0-143-156 kubenswrapper[2574]: E1030 02:54:56.027431    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7rmdj" podUID="75d9e1ec-cd1b-4aba-b911-5a7465c653cf"
Oct 30 02:54:56.029309 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:56.027500    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mzk2s"
Oct 30 02:54:56.029309 ip-10-0-143-156 kubenswrapper[2574]: E1030 02:54:56.027585    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mzk2s" podUID="3bd2c646-cebb-4e8a-b6c3-8b5b74049c11"
Oct 30 02:54:56.036012 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:56.035930    2574 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 02:49:54 +0000 UTC" deadline="2028-04-23 23:28:08.239879188 +0000 UTC"
Oct 30 02:54:56.036012 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:56.036000    2574 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13004h33m12.203883687s"
Oct 30 02:54:56.211256 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:56.211219    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-156.ec2.internal" event={"ID":"a7672f7a60a28c85d738430783c2cc68","Type":"ContainerStarted","Data":"ce428cb26f198f2072be833eeb1c3955424510dc0d3c628a99089634ae13057f"}
Oct 30 02:54:56.258436 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:56.258263    2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-156.ec2.internal" podStartSLOduration=3.258249548 podStartE2EDuration="3.258249548s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 02:54:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:54:56.257474345 +0000 UTC m=+4.676225921" watchObservedRunningTime="2026-10-30 02:54:56.258249548 +0000 UTC m=+4.677001124"
Oct 30 02:54:57.705716 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:57.705618    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3bd2c646-cebb-4e8a-b6c3-8b5b74049c11-metrics-certs\") pod \"network-metrics-daemon-mzk2s\" (UID: \"3bd2c646-cebb-4e8a-b6c3-8b5b74049c11\") " pod="openshift-multus/network-metrics-daemon-mzk2s"
Oct 30 02:54:57.705716 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:57.705712    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/98f9efe3-a732-4ba4-8543-941633d22170-original-pull-secret\") pod \"global-pull-secret-syncer-vbnwt\" (UID: \"98f9efe3-a732-4ba4-8543-941633d22170\") " pod="kube-system/global-pull-secret-syncer-vbnwt"
Oct 30 02:54:57.706298 ip-10-0-143-156 kubenswrapper[2574]: E1030 02:54:57.705844    2574 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 02:54:57.706298 ip-10-0-143-156 kubenswrapper[2574]: E1030 02:54:57.705906    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/98f9efe3-a732-4ba4-8543-941633d22170-original-pull-secret podName:98f9efe3-a732-4ba4-8543-941633d22170 nodeName:}" failed. No retries permitted until 2026-10-30 02:55:01.705886062 +0000 UTC m=+10.124637620 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/98f9efe3-a732-4ba4-8543-941633d22170-original-pull-secret") pod "global-pull-secret-syncer-vbnwt" (UID: "98f9efe3-a732-4ba4-8543-941633d22170") : object "kube-system"/"original-pull-secret" not registered
Oct 30 02:54:57.706298 ip-10-0-143-156 kubenswrapper[2574]: E1030 02:54:57.705972    2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 02:54:57.706298 ip-10-0-143-156 kubenswrapper[2574]: E1030 02:54:57.706008    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3bd2c646-cebb-4e8a-b6c3-8b5b74049c11-metrics-certs podName:3bd2c646-cebb-4e8a-b6c3-8b5b74049c11 nodeName:}" failed. No retries permitted until 2026-10-30 02:55:01.705994488 +0000 UTC m=+10.124746058 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3bd2c646-cebb-4e8a-b6c3-8b5b74049c11-metrics-certs") pod "network-metrics-daemon-mzk2s" (UID: "3bd2c646-cebb-4e8a-b6c3-8b5b74049c11") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 02:54:57.806471 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:57.806435    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-62vqf\" (UniqueName: \"kubernetes.io/projected/75d9e1ec-cd1b-4aba-b911-5a7465c653cf-kube-api-access-62vqf\") pod \"network-check-target-7rmdj\" (UID: \"75d9e1ec-cd1b-4aba-b911-5a7465c653cf\") " pod="openshift-network-diagnostics/network-check-target-7rmdj"
Oct 30 02:54:57.806656 ip-10-0-143-156 kubenswrapper[2574]: E1030 02:54:57.806643    2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 02:54:57.806720 ip-10-0-143-156 kubenswrapper[2574]: E1030 02:54:57.806681    2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 02:54:57.806720 ip-10-0-143-156 kubenswrapper[2574]: E1030 02:54:57.806705    2574 projected.go:196] Error preparing data for projected volume kube-api-access-62vqf for pod openshift-network-diagnostics/network-check-target-7rmdj: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 02:54:57.806800 ip-10-0-143-156 kubenswrapper[2574]: E1030 02:54:57.806762    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/75d9e1ec-cd1b-4aba-b911-5a7465c653cf-kube-api-access-62vqf podName:75d9e1ec-cd1b-4aba-b911-5a7465c653cf nodeName:}" failed. No retries permitted until 2026-10-30 02:55:01.8067423 +0000 UTC m=+10.225493858 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-62vqf" (UniqueName: "kubernetes.io/projected/75d9e1ec-cd1b-4aba-b911-5a7465c653cf-kube-api-access-62vqf") pod "network-check-target-7rmdj" (UID: "75d9e1ec-cd1b-4aba-b911-5a7465c653cf") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 02:54:58.027384 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:58.027308    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7rmdj"
Oct 30 02:54:58.027537 ip-10-0-143-156 kubenswrapper[2574]: E1030 02:54:58.027439    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7rmdj" podUID="75d9e1ec-cd1b-4aba-b911-5a7465c653cf"
Oct 30 02:54:58.027537 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:58.027508    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mzk2s"
Oct 30 02:54:58.027633 ip-10-0-143-156 kubenswrapper[2574]: E1030 02:54:58.027602    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mzk2s" podUID="3bd2c646-cebb-4e8a-b6c3-8b5b74049c11"
Oct 30 02:54:58.027671 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:58.027646    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vbnwt"
Oct 30 02:54:58.027738 ip-10-0-143-156 kubenswrapper[2574]: E1030 02:54:58.027717    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vbnwt" podUID="98f9efe3-a732-4ba4-8543-941633d22170"
Oct 30 02:54:58.236781 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:58.235837    2574 generic.go:356] "Generic (PLEG): container finished" podID="4d8bf700-9e09-4741-954b-aba9ccd331c9" containerID="c9d31b25c12fdd51cbfcc8da533c27d086b92dbf22b1d514734feddae17f3fe5" exitCode=0
Oct 30 02:54:58.236781 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:58.235890    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9l7k8" event={"ID":"4d8bf700-9e09-4741-954b-aba9ccd331c9","Type":"ContainerDied","Data":"c9d31b25c12fdd51cbfcc8da533c27d086b92dbf22b1d514734feddae17f3fe5"}
Oct 30 02:54:59.241864 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:54:59.241814    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-djpqs" event={"ID":"f593f4c2-de6c-42db-b451-d0ade01317f7","Type":"ContainerStarted","Data":"c5d698927f301bda9f05e988c689946f465fb8528dc6fc653aafc99e6eea696e"}
Oct 30 02:55:00.026257 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:55:00.026226    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7rmdj"
Oct 30 02:55:00.026419 ip-10-0-143-156 kubenswrapper[2574]: E1030 02:55:00.026354    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7rmdj" podUID="75d9e1ec-cd1b-4aba-b911-5a7465c653cf"
Oct 30 02:55:00.026776 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:55:00.026754    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mzk2s"
Oct 30 02:55:00.026884 ip-10-0-143-156 kubenswrapper[2574]: E1030 02:55:00.026863    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mzk2s" podUID="3bd2c646-cebb-4e8a-b6c3-8b5b74049c11"
Oct 30 02:55:00.026940 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:55:00.026927    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vbnwt"
Oct 30 02:55:00.027013 ip-10-0-143-156 kubenswrapper[2574]: E1030 02:55:00.026996    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vbnwt" podUID="98f9efe3-a732-4ba4-8543-941633d22170"
Oct 30 02:55:00.246043 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:55:00.245965    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-9mddj" event={"ID":"487caf0f-5f2a-4e93-9c06-de9af02c470b","Type":"ContainerStarted","Data":"93ff05c0ed294571af3af25321e5256358ac0d7782d293edb268de0a07cc6e66"}
Oct 30 02:55:00.250525 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:55:00.250450    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gqpp2" event={"ID":"91a29ddf-25ca-40a8-b920-6d894360a050","Type":"ContainerStarted","Data":"100a82e9cbba78c408f68e1490ffbf2272b60b559f63acf77b060a294bc75811"}
Oct 30 02:55:00.289165 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:55:00.288670    2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-djpqs" podStartSLOduration=4.560016758 podStartE2EDuration="8.288650993s" totalImagesPullingTime="3.728634235s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:54:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:54:59.290164446 +0000 UTC m=+7.708916024" watchObservedRunningTime="2026-10-30 02:55:00.288650993 +0000 UTC m=+8.707402569"
Oct 30 02:55:00.289165 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:55:00.288925    2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-9mddj" podStartSLOduration=3.20520724 podStartE2EDuration="8.288913166s" totalImagesPullingTime="5.083705926s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 02:54:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:55:00.288080523 +0000 UTC m=+8.706832099" watchObservedRunningTime="2026-10-30 02:55:00.288913166 +0000 UTC m=+8.707664740"
Oct 30 02:55:01.262741 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:55:01.262710    2574 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-djpqs"
Oct 30 02:55:01.263787 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:55:01.263762    2574 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-djpqs"
Oct 30 02:55:01.583748 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:55:01.583717    2574 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Oct 30 02:55:01.742739 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:55:01.742674    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/98f9efe3-a732-4ba4-8543-941633d22170-original-pull-secret\") pod \"global-pull-secret-syncer-vbnwt\" (UID: \"98f9efe3-a732-4ba4-8543-941633d22170\") " pod="kube-system/global-pull-secret-syncer-vbnwt"
Oct 30 02:55:01.742739 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:55:01.742739    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3bd2c646-cebb-4e8a-b6c3-8b5b74049c11-metrics-certs\") pod \"network-metrics-daemon-mzk2s\" (UID: \"3bd2c646-cebb-4e8a-b6c3-8b5b74049c11\") " pod="openshift-multus/network-metrics-daemon-mzk2s"
Oct 30 02:55:01.742984 ip-10-0-143-156 kubenswrapper[2574]: E1030 02:55:01.742869    2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 02:55:01.742984 ip-10-0-143-156 kubenswrapper[2574]: E1030 02:55:01.742934    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3bd2c646-cebb-4e8a-b6c3-8b5b74049c11-metrics-certs podName:3bd2c646-cebb-4e8a-b6c3-8b5b74049c11 nodeName:}" failed. No retries permitted until 2026-10-30 02:55:09.742915955 +0000 UTC m=+18.161667507 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3bd2c646-cebb-4e8a-b6c3-8b5b74049c11-metrics-certs") pod "network-metrics-daemon-mzk2s" (UID: "3bd2c646-cebb-4e8a-b6c3-8b5b74049c11") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 02:55:01.743395 ip-10-0-143-156 kubenswrapper[2574]: E1030 02:55:01.743369    2574 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 02:55:01.743502 ip-10-0-143-156 kubenswrapper[2574]: E1030 02:55:01.743443    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/98f9efe3-a732-4ba4-8543-941633d22170-original-pull-secret podName:98f9efe3-a732-4ba4-8543-941633d22170 nodeName:}" failed. No retries permitted until 2026-10-30 02:55:09.743424911 +0000 UTC m=+18.162176479 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/98f9efe3-a732-4ba4-8543-941633d22170-original-pull-secret") pod "global-pull-secret-syncer-vbnwt" (UID: "98f9efe3-a732-4ba4-8543-941633d22170") : object "kube-system"/"original-pull-secret" not registered
Oct 30 02:55:01.843930 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:55:01.843857    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-62vqf\" (UniqueName: \"kubernetes.io/projected/75d9e1ec-cd1b-4aba-b911-5a7465c653cf-kube-api-access-62vqf\") pod \"network-check-target-7rmdj\" (UID: \"75d9e1ec-cd1b-4aba-b911-5a7465c653cf\") " pod="openshift-network-diagnostics/network-check-target-7rmdj"
Oct 30 02:55:01.844082 ip-10-0-143-156 kubenswrapper[2574]: E1030 02:55:01.844024    2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 02:55:01.844082 ip-10-0-143-156 kubenswrapper[2574]: E1030 02:55:01.844044    2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 02:55:01.844082 ip-10-0-143-156 kubenswrapper[2574]: E1030 02:55:01.844055    2574 projected.go:196] Error preparing data for projected volume kube-api-access-62vqf for pod openshift-network-diagnostics/network-check-target-7rmdj: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 02:55:01.844274 ip-10-0-143-156 kubenswrapper[2574]: E1030 02:55:01.844128    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/75d9e1ec-cd1b-4aba-b911-5a7465c653cf-kube-api-access-62vqf podName:75d9e1ec-cd1b-4aba-b911-5a7465c653cf nodeName:}" failed. No retries permitted until 2026-10-30 02:55:09.844110944 +0000 UTC m=+18.262862509 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-62vqf" (UniqueName: "kubernetes.io/projected/75d9e1ec-cd1b-4aba-b911-5a7465c653cf-kube-api-access-62vqf") pod "network-check-target-7rmdj" (UID: "75d9e1ec-cd1b-4aba-b911-5a7465c653cf") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 02:55:02.028279 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:55:02.028251    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7rmdj"
Oct 30 02:55:02.040390 ip-10-0-143-156 kubenswrapper[2574]: E1030 02:55:02.038210    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7rmdj" podUID="75d9e1ec-cd1b-4aba-b911-5a7465c653cf"
Oct 30 02:55:02.040390 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:55:02.029388    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vbnwt"
Oct 30 02:55:02.040390 ip-10-0-143-156 kubenswrapper[2574]: E1030 02:55:02.038435    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vbnwt" podUID="98f9efe3-a732-4ba4-8543-941633d22170"
Oct 30 02:55:02.040390 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:55:02.032420    2574 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Oct 30 02:55:02.040390 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:55:02.038499    2574 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Oct 30 02:55:02.040390 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:55:02.029446    2574 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-30T02:55:01.583741903Z","UUID":"d1e74941-8199-4756-99b6-dfc98550a3c9","Handler":null,"Name":"","Endpoint":""}
Oct 30 02:55:02.040390 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:55:02.029354    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mzk2s"
Oct 30 02:55:02.040390 ip-10-0-143-156 kubenswrapper[2574]: E1030 02:55:02.039324    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mzk2s" podUID="3bd2c646-cebb-4e8a-b6c3-8b5b74049c11"
Oct 30 02:55:02.256841 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:55:02.256806    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-9qcgk" event={"ID":"b4ce92fa-1612-4e46-ad6e-23049b2e279b","Type":"ContainerStarted","Data":"938d60ed584bced3a881349da51d22d345848e920973fd5dd74832846aee402a"}
Oct 30 02:55:02.261042 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:55:02.261015    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gqpp2" event={"ID":"91a29ddf-25ca-40a8-b920-6d894360a050","Type":"ContainerStarted","Data":"8baa8d694650b47f2a4f8b77bb9a05f44a30f27a73bc6ca61ef20fb9f77d1f13"}
Oct 30 02:55:02.306727 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:55:02.306471    2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-9qcgk" podStartSLOduration=2.477626619 podStartE2EDuration="10.306453212s" totalImagesPullingTime="7.828826593s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 02:54:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:55:02.306205518 +0000 UTC m=+10.724957119" watchObservedRunningTime="2026-10-30 02:55:02.306453212 +0000 UTC m=+10.725204785"
Oct 30 02:55:03.265809 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:55:03.265582    2574 generic.go:356] "Generic (PLEG): container finished" podID="4d8bf700-9e09-4741-954b-aba9ccd331c9" containerID="5048d278d7f2999d742b1c724f68c695426de1f5a6ebb0eca5f6977ffaaae317" exitCode=0
Oct 30 02:55:03.265979 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:55:03.265661    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9l7k8" event={"ID":"4d8bf700-9e09-4741-954b-aba9ccd331c9","Type":"ContainerDied","Data":"5048d278d7f2999d742b1c724f68c695426de1f5a6ebb0eca5f6977ffaaae317"}
Oct 30 02:55:03.267619 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:55:03.267566    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-zhb7b" event={"ID":"0e6f015d-0a2d-4579-ad4d-a88b8c11c3b9","Type":"ContainerStarted","Data":"228f6e03eff0b1a9e095a186621e6febb4b86b18a8a12046cf55062b24ab3f98"}
Oct 30 02:55:03.270322 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:55:03.270294    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gqpp2" event={"ID":"91a29ddf-25ca-40a8-b920-6d894360a050","Type":"ContainerStarted","Data":"45943075bb342008f29e7dc267599464b54cdf2d84676fc459039a0880e3ca52"}
Oct 30 02:55:03.378928 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:55:03.378883    2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-zhb7b" podStartSLOduration=3.40669257 podStartE2EDuration="11.378869989s" totalImagesPullingTime="7.972177419s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 02:54:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:55:03.378213795 +0000 UTC m=+11.796965391" watchObservedRunningTime="2026-10-30 02:55:03.378869989 +0000 UTC m=+11.797621562"
Oct 30 02:55:03.423803 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:55:03.423752    2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gqpp2" podStartSLOduration=3.3694952320000002 podStartE2EDuration="11.423733359s" totalImagesPullingTime="8.054238127s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 02:54:52 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:55:03.423285266 +0000 UTC m=+11.842036840" watchObservedRunningTime="2026-10-30 02:55:03.423733359 +0000 UTC m=+11.842484933"
Oct 30 02:55:04.026873 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:55:04.026836    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vbnwt"
Oct 30 02:55:04.027085 ip-10-0-143-156 kubenswrapper[2574]: E1030 02:55:04.026966    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vbnwt" podUID="98f9efe3-a732-4ba4-8543-941633d22170"
Oct 30 02:55:04.027373 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:55:04.027351    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7rmdj"
Oct 30 02:55:04.027490 ip-10-0-143-156 kubenswrapper[2574]: E1030 02:55:04.027450    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7rmdj" podUID="75d9e1ec-cd1b-4aba-b911-5a7465c653cf"
Oct 30 02:55:04.027564 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:55:04.027511    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mzk2s"
Oct 30 02:55:04.027628 ip-10-0-143-156 kubenswrapper[2574]: E1030 02:55:04.027595    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mzk2s" podUID="3bd2c646-cebb-4e8a-b6c3-8b5b74049c11"
Oct 30 02:55:04.274636 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:55:04.274551    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-jhq2g" event={"ID":"042af239-396c-4dae-8cf2-0c562af07c62","Type":"ContainerStarted","Data":"95e169b16eb618769de810ab2b37d450491aa88398c11c83384f3d0042485041"}
Oct 30 02:55:04.276933 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:55:04.276908    2574 generic.go:356] "Generic (PLEG): container finished" podID="4d8bf700-9e09-4741-954b-aba9ccd331c9" containerID="b17f712c6e9fcdcb9c095a865bab449a1db53a73307f27689e97f763840c9f56" exitCode=0
Oct 30 02:55:04.277058 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:55:04.276996    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9l7k8" event={"ID":"4d8bf700-9e09-4741-954b-aba9ccd331c9","Type":"ContainerDied","Data":"b17f712c6e9fcdcb9c095a865bab449a1db53a73307f27689e97f763840c9f56"}
Oct 30 02:55:04.316204 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:55:04.316153    2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-jhq2g" podStartSLOduration=4.182928593 podStartE2EDuration="12.316136006s" totalImagesPullingTime="8.133207413s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 02:54:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:55:04.31612079 +0000 UTC m=+12.734872360" watchObservedRunningTime="2026-10-30 02:55:04.316136006 +0000 UTC m=+12.734887582"
Oct 30 02:55:05.281670 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:55:05.281630    2574 generic.go:356] "Generic (PLEG): container finished" podID="4d8bf700-9e09-4741-954b-aba9ccd331c9" containerID="063b68404f069111e2820f7efd973e509ba6381850dfa4714db55cf4093dfebd" exitCode=0
Oct 30 02:55:05.282377 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:55:05.281718    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9l7k8" event={"ID":"4d8bf700-9e09-4741-954b-aba9ccd331c9","Type":"ContainerDied","Data":"063b68404f069111e2820f7efd973e509ba6381850dfa4714db55cf4093dfebd"}
Oct 30 02:55:05.284234 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:55:05.283844    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-h7cm4" event={"ID":"3e4730f0-d77e-4bff-b8af-b7fafd4525d8","Type":"ContainerStarted","Data":"4b93e3afbcc95bd80f57cefb8c1931450fc184e643eeb1c80b9ddbff998d1893"}
Oct 30 02:55:05.392234 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:55:05.392128    2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-h7cm4" podStartSLOduration=3.188008464 podStartE2EDuration="13.39210887s" totalImagesPullingTime="10.204100406s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 02:54:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:55:05.391383568 +0000 UTC m=+13.810135142" watchObservedRunningTime="2026-10-30 02:55:05.39210887 +0000 UTC m=+13.810860438"
Oct 30 02:55:06.026229 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:55:06.026195    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mzk2s"
Oct 30 02:55:06.026424 ip-10-0-143-156 kubenswrapper[2574]: E1030 02:55:06.026337    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mzk2s" podUID="3bd2c646-cebb-4e8a-b6c3-8b5b74049c11"
Oct 30 02:55:06.026663 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:55:06.026648    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vbnwt"
Oct 30 02:55:06.026756 ip-10-0-143-156 kubenswrapper[2574]: E1030 02:55:06.026736    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vbnwt" podUID="98f9efe3-a732-4ba4-8543-941633d22170"
Oct 30 02:55:06.026811 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:55:06.026800    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7rmdj"
Oct 30 02:55:06.026887 ip-10-0-143-156 kubenswrapper[2574]: E1030 02:55:06.026868    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7rmdj" podUID="75d9e1ec-cd1b-4aba-b911-5a7465c653cf"
Oct 30 02:55:07.290958 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:55:07.290925    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9l7k8" event={"ID":"4d8bf700-9e09-4741-954b-aba9ccd331c9","Type":"ContainerStarted","Data":"4ea3df65ea09e24df426ac49c5be4e172918b496f7236197970bde9c7ac11c44"}
Oct 30 02:55:08.024502 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:55:08.024315    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mzk2s"
Oct 30 02:55:08.024668 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:55:08.024325    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7rmdj"
Oct 30 02:55:08.024668 ip-10-0-143-156 kubenswrapper[2574]: E1030 02:55:08.024596    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mzk2s" podUID="3bd2c646-cebb-4e8a-b6c3-8b5b74049c11"
Oct 30 02:55:08.024668 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:55:08.024337    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vbnwt"
Oct 30 02:55:08.024799 ip-10-0-143-156 kubenswrapper[2574]: E1030 02:55:08.024679    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7rmdj" podUID="75d9e1ec-cd1b-4aba-b911-5a7465c653cf"
Oct 30 02:55:08.024799 ip-10-0-143-156 kubenswrapper[2574]: E1030 02:55:08.024725    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vbnwt" podUID="98f9efe3-a732-4ba4-8543-941633d22170"
Oct 30 02:55:08.295721 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:55:08.295615    2574 generic.go:356] "Generic (PLEG): container finished" podID="4d8bf700-9e09-4741-954b-aba9ccd331c9" containerID="4ea3df65ea09e24df426ac49c5be4e172918b496f7236197970bde9c7ac11c44" exitCode=0
Oct 30 02:55:08.295721 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:55:08.295664    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9l7k8" event={"ID":"4d8bf700-9e09-4741-954b-aba9ccd331c9","Type":"ContainerDied","Data":"4ea3df65ea09e24df426ac49c5be4e172918b496f7236197970bde9c7ac11c44"}
Oct 30 02:55:09.300627 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:55:09.300381    2574 generic.go:356] "Generic (PLEG): container finished" podID="4d8bf700-9e09-4741-954b-aba9ccd331c9" containerID="ad660fba819dae6154bf330655b847dad7b0b3ec814b5d2ca732b2e1177f7a35" exitCode=0
Oct 30 02:55:09.300627 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:55:09.300462    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9l7k8" event={"ID":"4d8bf700-9e09-4741-954b-aba9ccd331c9","Type":"ContainerDied","Data":"ad660fba819dae6154bf330655b847dad7b0b3ec814b5d2ca732b2e1177f7a35"}
Oct 30 02:55:09.303728 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:55:09.303707    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-g6nqm" event={"ID":"f425530a-8431-4c5b-8335-8cc1119745c1","Type":"ContainerStarted","Data":"7959b955b653f9dd1e819a9589c3d5b1339a0e3d7fc2872c8bfceb30d17fac24"}
Oct 30 02:55:09.303784 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:55:09.303771    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-g6nqm" event={"ID":"f425530a-8431-4c5b-8335-8cc1119745c1","Type":"ContainerStarted","Data":"ccc7e12515a482eaefd6b19c15bebdf6749a670d3c862ca54bba4c5f680dd925"}
Oct 30 02:55:09.303828 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:55:09.303803    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-g6nqm" event={"ID":"f425530a-8431-4c5b-8335-8cc1119745c1","Type":"ContainerStarted","Data":"343026228ff4b3e1c245e73557fca4022896fcee706703af0f8b3878cd246d7b"}
Oct 30 02:55:09.303864 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:55:09.303831    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-g6nqm" event={"ID":"f425530a-8431-4c5b-8335-8cc1119745c1","Type":"ContainerStarted","Data":"2c7e452a52bb62dd1f2219d3bd458b403ce5f3e086b36cd200cf399b1c633016"}
Oct 30 02:55:09.303864 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:55:09.303858    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-g6nqm" event={"ID":"f425530a-8431-4c5b-8335-8cc1119745c1","Type":"ContainerStarted","Data":"506ef04e385dcee65c1f6930973fd48758fec988548a32307acf06a8a13839d5"}
Oct 30 02:55:09.303919 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:55:09.303883    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-g6nqm" event={"ID":"f425530a-8431-4c5b-8335-8cc1119745c1","Type":"ContainerStarted","Data":"0580419bb00bca22fbdb321960c8cc34c6d978aee951c8f702a3e35c57caf567"}
Oct 30 02:55:09.808165 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:55:09.808137    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/98f9efe3-a732-4ba4-8543-941633d22170-original-pull-secret\") pod \"global-pull-secret-syncer-vbnwt\" (UID: \"98f9efe3-a732-4ba4-8543-941633d22170\") " pod="kube-system/global-pull-secret-syncer-vbnwt"
Oct 30 02:55:09.808369 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:55:09.808180    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3bd2c646-cebb-4e8a-b6c3-8b5b74049c11-metrics-certs\") pod \"network-metrics-daemon-mzk2s\" (UID: \"3bd2c646-cebb-4e8a-b6c3-8b5b74049c11\") " pod="openshift-multus/network-metrics-daemon-mzk2s"
Oct 30 02:55:09.808369 ip-10-0-143-156 kubenswrapper[2574]: E1030 02:55:09.808282    2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 02:55:09.808369 ip-10-0-143-156 kubenswrapper[2574]: E1030 02:55:09.808333    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3bd2c646-cebb-4e8a-b6c3-8b5b74049c11-metrics-certs podName:3bd2c646-cebb-4e8a-b6c3-8b5b74049c11 nodeName:}" failed. No retries permitted until 2026-10-30 02:55:25.808320554 +0000 UTC m=+34.227072104 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3bd2c646-cebb-4e8a-b6c3-8b5b74049c11-metrics-certs") pod "network-metrics-daemon-mzk2s" (UID: "3bd2c646-cebb-4e8a-b6c3-8b5b74049c11") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 02:55:09.808369 ip-10-0-143-156 kubenswrapper[2574]: E1030 02:55:09.808282    2574 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 02:55:09.808501 ip-10-0-143-156 kubenswrapper[2574]: E1030 02:55:09.808407    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/98f9efe3-a732-4ba4-8543-941633d22170-original-pull-secret podName:98f9efe3-a732-4ba4-8543-941633d22170 nodeName:}" failed. No retries permitted until 2026-10-30 02:55:25.808394197 +0000 UTC m=+34.227145754 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/98f9efe3-a732-4ba4-8543-941633d22170-original-pull-secret") pod "global-pull-secret-syncer-vbnwt" (UID: "98f9efe3-a732-4ba4-8543-941633d22170") : object "kube-system"/"original-pull-secret" not registered
Oct 30 02:55:09.909180 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:55:09.909147    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-62vqf\" (UniqueName: \"kubernetes.io/projected/75d9e1ec-cd1b-4aba-b911-5a7465c653cf-kube-api-access-62vqf\") pod \"network-check-target-7rmdj\" (UID: \"75d9e1ec-cd1b-4aba-b911-5a7465c653cf\") " pod="openshift-network-diagnostics/network-check-target-7rmdj"
Oct 30 02:55:09.909409 ip-10-0-143-156 kubenswrapper[2574]: E1030 02:55:09.909268    2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 02:55:09.909409 ip-10-0-143-156 kubenswrapper[2574]: E1030 02:55:09.909283    2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 02:55:09.909409 ip-10-0-143-156 kubenswrapper[2574]: E1030 02:55:09.909290    2574 projected.go:196] Error preparing data for projected volume kube-api-access-62vqf for pod openshift-network-diagnostics/network-check-target-7rmdj: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 02:55:09.909409 ip-10-0-143-156 kubenswrapper[2574]: E1030 02:55:09.909334    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/75d9e1ec-cd1b-4aba-b911-5a7465c653cf-kube-api-access-62vqf podName:75d9e1ec-cd1b-4aba-b911-5a7465c653cf nodeName:}" failed. No retries permitted until 2026-10-30 02:55:25.909319705 +0000 UTC m=+34.328071260 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-62vqf" (UniqueName: "kubernetes.io/projected/75d9e1ec-cd1b-4aba-b911-5a7465c653cf-kube-api-access-62vqf") pod "network-check-target-7rmdj" (UID: "75d9e1ec-cd1b-4aba-b911-5a7465c653cf") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 02:55:10.024877 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:55:10.024848    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7rmdj"
Oct 30 02:55:10.025028 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:55:10.024898    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vbnwt"
Oct 30 02:55:10.025028 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:55:10.024908    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mzk2s"
Oct 30 02:55:10.025028 ip-10-0-143-156 kubenswrapper[2574]: E1030 02:55:10.024978    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vbnwt" podUID="98f9efe3-a732-4ba4-8543-941633d22170"
Oct 30 02:55:10.025190 ip-10-0-143-156 kubenswrapper[2574]: E1030 02:55:10.025038    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mzk2s" podUID="3bd2c646-cebb-4e8a-b6c3-8b5b74049c11"
Oct 30 02:55:10.025190 ip-10-0-143-156 kubenswrapper[2574]: E1030 02:55:10.025141    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7rmdj" podUID="75d9e1ec-cd1b-4aba-b911-5a7465c653cf"
Oct 30 02:55:10.308742 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:55:10.308713    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9l7k8" event={"ID":"4d8bf700-9e09-4741-954b-aba9ccd331c9","Type":"ContainerStarted","Data":"9042bc922e0eec88f8f15693b87ceeb08abd72a20aea1d72b69f75dd46c55f70"}
Oct 30 02:55:10.371407 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:55:10.371355    2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-9l7k8" podStartSLOduration=8.333143178 podStartE2EDuration="18.371342512s" totalImagesPullingTime="10.038199334s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 02:54:52 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:55:10.370980958 +0000 UTC m=+18.789732530" watchObservedRunningTime="2026-10-30 02:55:10.371342512 +0000 UTC m=+18.790094084"
Oct 30 02:55:11.314062 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:55:11.314033    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-g6nqm" event={"ID":"f425530a-8431-4c5b-8335-8cc1119745c1","Type":"ContainerStarted","Data":"e29a3afa6f28994a74679cde26bb3e32f783b9b16df546b30e85821ac0148671"}
Oct 30 02:55:12.024866 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:55:12.024830    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7rmdj"
Oct 30 02:55:12.025026 ip-10-0-143-156 kubenswrapper[2574]: E1030 02:55:12.024903    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7rmdj" podUID="75d9e1ec-cd1b-4aba-b911-5a7465c653cf"
Oct 30 02:55:12.025026 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:55:12.025012    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mzk2s"
Oct 30 02:55:12.025164 ip-10-0-143-156 kubenswrapper[2574]: E1030 02:55:12.025142    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mzk2s" podUID="3bd2c646-cebb-4e8a-b6c3-8b5b74049c11"
Oct 30 02:55:12.025208 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:55:12.025175    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vbnwt"
Oct 30 02:55:12.025249 ip-10-0-143-156 kubenswrapper[2574]: E1030 02:55:12.025236    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vbnwt" podUID="98f9efe3-a732-4ba4-8543-941633d22170"
Oct 30 02:55:12.046251 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:55:12.046227    2574 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-djpqs"
Oct 30 02:55:12.046387 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:55:12.046311    2574 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Oct 30 02:55:12.046789 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:55:12.046773    2574 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-djpqs"
Oct 30 02:55:13.322073 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:55:13.322040    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-g6nqm" event={"ID":"f425530a-8431-4c5b-8335-8cc1119745c1","Type":"ContainerStarted","Data":"0b403459f1cb29a5954f01e1a1e9e431b335ffb41126e48b8d7d73a6035ecd98"}
Oct 30 02:55:13.322778 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:55:13.322407    2574 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-g6nqm"
Oct 30 02:55:13.322778 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:55:13.322577    2574 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-g6nqm"
Oct 30 02:55:13.338664 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:55:13.338641    2574 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-g6nqm"
Oct 30 02:55:13.340709 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:55:13.340687    2574 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-g6nqm"
Oct 30 02:55:13.420197 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:55:13.420145    2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-g6nqm" podStartSLOduration=7.060610932 podStartE2EDuration="21.420130606s" totalImagesPullingTime="14.359519674s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 02:54:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:55:13.419780041 +0000 UTC m=+21.838531615" watchObservedRunningTime="2026-10-30 02:55:13.420130606 +0000 UTC m=+21.838882179"
Oct 30 02:55:14.024806 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:55:14.024768    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mzk2s"
Oct 30 02:55:14.024806 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:55:14.024792    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7rmdj"
Oct 30 02:55:14.025076 ip-10-0-143-156 kubenswrapper[2574]: E1030 02:55:14.024938    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mzk2s" podUID="3bd2c646-cebb-4e8a-b6c3-8b5b74049c11"
Oct 30 02:55:14.025076 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:55:14.024972    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vbnwt"
Oct 30 02:55:14.025076 ip-10-0-143-156 kubenswrapper[2574]: E1030 02:55:14.025064    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7rmdj" podUID="75d9e1ec-cd1b-4aba-b911-5a7465c653cf"
Oct 30 02:55:14.025193 ip-10-0-143-156 kubenswrapper[2574]: E1030 02:55:14.025160    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vbnwt" podUID="98f9efe3-a732-4ba4-8543-941633d22170"
Oct 30 02:55:14.324209 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:55:14.324145    2574 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Oct 30 02:55:15.326113 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:55:15.326031    2574 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Oct 30 02:55:15.337762 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:55:15.337739    2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-vbnwt"]
Oct 30 02:55:15.337877 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:55:15.337821    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vbnwt"
Oct 30 02:55:15.337915 ip-10-0-143-156 kubenswrapper[2574]: E1030 02:55:15.337894    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vbnwt" podUID="98f9efe3-a732-4ba4-8543-941633d22170"
Oct 30 02:55:15.341487 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:55:15.341462    2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-mzk2s"]
Oct 30 02:55:15.341608 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:55:15.341567    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mzk2s"
Oct 30 02:55:15.341699 ip-10-0-143-156 kubenswrapper[2574]: E1030 02:55:15.341678    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mzk2s" podUID="3bd2c646-cebb-4e8a-b6c3-8b5b74049c11"
Oct 30 02:55:15.351527 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:55:15.351501    2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-7rmdj"]
Oct 30 02:55:15.351589 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:55:15.351579    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7rmdj"
Oct 30 02:55:15.351657 ip-10-0-143-156 kubenswrapper[2574]: E1030 02:55:15.351643    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7rmdj" podUID="75d9e1ec-cd1b-4aba-b911-5a7465c653cf"
Oct 30 02:55:16.802812 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:55:16.802780    2574 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-g6nqm"
Oct 30 02:55:17.024478 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:55:17.024430    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vbnwt"
Oct 30 02:55:17.024687 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:55:17.024461    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7rmdj"
Oct 30 02:55:17.024687 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:55:17.024549    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mzk2s"
Oct 30 02:55:17.024687 ip-10-0-143-156 kubenswrapper[2574]: E1030 02:55:17.024557    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vbnwt" podUID="98f9efe3-a732-4ba4-8543-941633d22170"
Oct 30 02:55:17.024687 ip-10-0-143-156 kubenswrapper[2574]: E1030 02:55:17.024612    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7rmdj" podUID="75d9e1ec-cd1b-4aba-b911-5a7465c653cf"
Oct 30 02:55:17.024850 ip-10-0-143-156 kubenswrapper[2574]: E1030 02:55:17.024718    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mzk2s" podUID="3bd2c646-cebb-4e8a-b6c3-8b5b74049c11"
Oct 30 02:55:19.024605 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:55:19.024578    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mzk2s"
Oct 30 02:55:19.024605 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:55:19.024597    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7rmdj"
Oct 30 02:55:19.025118 ip-10-0-143-156 kubenswrapper[2574]: E1030 02:55:19.024695    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mzk2s" podUID="3bd2c646-cebb-4e8a-b6c3-8b5b74049c11"
Oct 30 02:55:19.025118 ip-10-0-143-156 kubenswrapper[2574]: E1030 02:55:19.024752    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7rmdj" podUID="75d9e1ec-cd1b-4aba-b911-5a7465c653cf"
Oct 30 02:55:19.025118 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:55:19.024799    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vbnwt"
Oct 30 02:55:19.025118 ip-10-0-143-156 kubenswrapper[2574]: E1030 02:55:19.024862    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vbnwt" podUID="98f9efe3-a732-4ba4-8543-941633d22170"
Oct 30 02:55:20.840709 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:55:20.840639    2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-156.ec2.internal" event="NodeReady"
Oct 30 02:55:20.841065 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:55:20.840776    2574 kubelet_node_status.go:477] "Fast updating node status as it just became ready"
Oct 30 02:55:20.900550 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:55:20.900517    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 02:55:20.903110 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:55:20.903074    2574 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-wjll7"]
Oct 30 02:55:20.906188 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:55:20.906170    2574 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-vwbj7"]
Oct 30 02:55:20.906322 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:55:20.906306    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-wjll7"
Oct 30 02:55:20.908827 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:55:20.908811    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-vwbj7"
Oct 30 02:55:20.923023 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:55:20.923005    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Oct 30 02:55:20.923327 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:55:20.923313    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Oct 30 02:55:20.923385 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:55:20.923372    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-r9wgn\""
Oct 30 02:55:20.927489 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:55:20.927470    2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-vwbj7"]
Oct 30 02:55:20.932009 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:55:20.931986    2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-wjll7"]
Oct 30 02:55:20.990170 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:55:20.990129    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/73752bb9-2a14-4582-a6d6-c1fd26524c3d-cert\") pod \"ingress-canary-vwbj7\" (UID: \"73752bb9-2a14-4582-a6d6-c1fd26524c3d\") " pod="openshift-ingress-canary/ingress-canary-vwbj7"
Oct 30 02:55:20.990341 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:55:20.990176    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/5f9fb000-9531-41f1-b3cf-7e79ca1a24d7-metrics-tls\") pod \"dns-default-wjll7\" (UID: \"5f9fb000-9531-41f1-b3cf-7e79ca1a24d7\") " pod="openshift-dns/dns-default-wjll7"
Oct 30 02:55:20.990341 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:55:20.990282    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gsgwj\" (UniqueName: \"kubernetes.io/projected/73752bb9-2a14-4582-a6d6-c1fd26524c3d-kube-api-access-gsgwj\") pod \"ingress-canary-vwbj7\" (UID: \"73752bb9-2a14-4582-a6d6-c1fd26524c3d\") " pod="openshift-ingress-canary/ingress-canary-vwbj7"
Oct 30 02:55:20.990341 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:55:20.990330    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/5f9fb000-9531-41f1-b3cf-7e79ca1a24d7-config-volume\") pod \"dns-default-wjll7\" (UID: \"5f9fb000-9531-41f1-b3cf-7e79ca1a24d7\") " pod="openshift-dns/dns-default-wjll7"
Oct 30 02:55:20.990490 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:55:20.990375    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5f9fb000-9531-41f1-b3cf-7e79ca1a24d7-tmp-dir\") pod \"dns-default-wjll7\" (UID: \"5f9fb000-9531-41f1-b3cf-7e79ca1a24d7\") " pod="openshift-dns/dns-default-wjll7"
Oct 30 02:55:20.990490 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:55:20.990407    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gwcwf\" (UniqueName: \"kubernetes.io/projected/5f9fb000-9531-41f1-b3cf-7e79ca1a24d7-kube-api-access-gwcwf\") pod \"dns-default-wjll7\" (UID: \"5f9fb000-9531-41f1-b3cf-7e79ca1a24d7\") " pod="openshift-dns/dns-default-wjll7"
Oct 30 02:55:21.023635 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:55:21.023606    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Oct 30 02:55:21.023691 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:55:21.023606    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Oct 30 02:55:21.023691 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:55:21.023666    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Oct 30 02:55:21.023745 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:55:21.023697    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-xzg75\""
Oct 30 02:55:21.023745 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:55:21.023720    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7rmdj"
Oct 30 02:55:21.023745 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:55:21.023736    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vbnwt"
Oct 30 02:55:21.023830 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:55:21.023729    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mzk2s"
Oct 30 02:55:21.090883 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:55:21.090796    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/73752bb9-2a14-4582-a6d6-c1fd26524c3d-cert\") pod \"ingress-canary-vwbj7\" (UID: \"73752bb9-2a14-4582-a6d6-c1fd26524c3d\") " pod="openshift-ingress-canary/ingress-canary-vwbj7"
Oct 30 02:55:21.090883 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:55:21.090835    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/5f9fb000-9531-41f1-b3cf-7e79ca1a24d7-metrics-tls\") pod \"dns-default-wjll7\" (UID: \"5f9fb000-9531-41f1-b3cf-7e79ca1a24d7\") " pod="openshift-dns/dns-default-wjll7"
Oct 30 02:55:21.090883 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:55:21.090866    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gsgwj\" (UniqueName: \"kubernetes.io/projected/73752bb9-2a14-4582-a6d6-c1fd26524c3d-kube-api-access-gsgwj\") pod \"ingress-canary-vwbj7\" (UID: \"73752bb9-2a14-4582-a6d6-c1fd26524c3d\") " pod="openshift-ingress-canary/ingress-canary-vwbj7"
Oct 30 02:55:21.091176 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:55:21.090897    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/5f9fb000-9531-41f1-b3cf-7e79ca1a24d7-config-volume\") pod \"dns-default-wjll7\" (UID: \"5f9fb000-9531-41f1-b3cf-7e79ca1a24d7\") " pod="openshift-dns/dns-default-wjll7"
Oct 30 02:55:21.091176 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:55:21.090911    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5f9fb000-9531-41f1-b3cf-7e79ca1a24d7-tmp-dir\") pod \"dns-default-wjll7\" (UID: \"5f9fb000-9531-41f1-b3cf-7e79ca1a24d7\") " pod="openshift-dns/dns-default-wjll7"
Oct 30 02:55:21.091176 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:55:21.090929    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gwcwf\" (UniqueName: \"kubernetes.io/projected/5f9fb000-9531-41f1-b3cf-7e79ca1a24d7-kube-api-access-gwcwf\") pod \"dns-default-wjll7\" (UID: \"5f9fb000-9531-41f1-b3cf-7e79ca1a24d7\") " pod="openshift-dns/dns-default-wjll7"
Oct 30 02:55:21.091176 ip-10-0-143-156 kubenswrapper[2574]: E1030 02:55:21.091038    2574 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 30 02:55:21.091176 ip-10-0-143-156 kubenswrapper[2574]: E1030 02:55:21.091132    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5f9fb000-9531-41f1-b3cf-7e79ca1a24d7-metrics-tls podName:5f9fb000-9531-41f1-b3cf-7e79ca1a24d7 nodeName:}" failed. No retries permitted until 2026-10-30 02:55:21.591087501 +0000 UTC m=+30.009839070 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/5f9fb000-9531-41f1-b3cf-7e79ca1a24d7-metrics-tls") pod "dns-default-wjll7" (UID: "5f9fb000-9531-41f1-b3cf-7e79ca1a24d7") : secret "dns-default-metrics-tls" not found
Oct 30 02:55:21.091355 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:55:21.091331    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5f9fb000-9531-41f1-b3cf-7e79ca1a24d7-tmp-dir\") pod \"dns-default-wjll7\" (UID: \"5f9fb000-9531-41f1-b3cf-7e79ca1a24d7\") " pod="openshift-dns/dns-default-wjll7"
Oct 30 02:55:21.091776 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:55:21.091752    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/5f9fb000-9531-41f1-b3cf-7e79ca1a24d7-config-volume\") pod \"dns-default-wjll7\" (UID: \"5f9fb000-9531-41f1-b3cf-7e79ca1a24d7\") " pod="openshift-dns/dns-default-wjll7"
Oct 30 02:55:21.094902 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:55:21.094778    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/73752bb9-2a14-4582-a6d6-c1fd26524c3d-cert\") pod \"ingress-canary-vwbj7\" (UID: \"73752bb9-2a14-4582-a6d6-c1fd26524c3d\") " pod="openshift-ingress-canary/ingress-canary-vwbj7"
Oct 30 02:55:21.119349 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:55:21.119312    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gwcwf\" (UniqueName: \"kubernetes.io/projected/5f9fb000-9531-41f1-b3cf-7e79ca1a24d7-kube-api-access-gwcwf\") pod \"dns-default-wjll7\" (UID: \"5f9fb000-9531-41f1-b3cf-7e79ca1a24d7\") " pod="openshift-dns/dns-default-wjll7"
Oct 30 02:55:21.119349 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:55:21.119329    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gsgwj\" (UniqueName: \"kubernetes.io/projected/73752bb9-2a14-4582-a6d6-c1fd26524c3d-kube-api-access-gsgwj\") pod \"ingress-canary-vwbj7\" (UID: \"73752bb9-2a14-4582-a6d6-c1fd26524c3d\") " pod="openshift-ingress-canary/ingress-canary-vwbj7"
Oct 30 02:55:21.121029 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:55:21.121002    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 02:55:21.124174 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:55:21.124153    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Oct 30 02:55:21.124267 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:55:21.124161    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Oct 30 02:55:21.124267 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:55:21.124185    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Oct 30 02:55:21.124267 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:55:21.124178    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Oct 30 02:55:21.124267 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:55:21.124219    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-bckpk\""
Oct 30 02:55:21.124405 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:55:21.124286    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-vzrlw\""
Oct 30 02:55:21.124405 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:55:21.124340    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-kdtjd\""
Oct 30 02:55:21.223717 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:55:21.223687    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-vwbj7"
Oct 30 02:55:21.356357 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:55:21.356281    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 02:55:21.531278 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:55:21.531243    2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-vwbj7"]
Oct 30 02:55:21.534929 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:55:21.534904    2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod73752bb9_2a14_4582_a6d6_c1fd26524c3d.slice/crio-e6dac9b7049e7ef1cd190b3786ce14164794eaa8d4d4277031c61a208cd1fea4 WatchSource:0}: Error finding container e6dac9b7049e7ef1cd190b3786ce14164794eaa8d4d4277031c61a208cd1fea4: Status 404 returned error can't find the container with id e6dac9b7049e7ef1cd190b3786ce14164794eaa8d4d4277031c61a208cd1fea4
Oct 30 02:55:21.594437 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:55:21.594408    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/5f9fb000-9531-41f1-b3cf-7e79ca1a24d7-metrics-tls\") pod \"dns-default-wjll7\" (UID: \"5f9fb000-9531-41f1-b3cf-7e79ca1a24d7\") " pod="openshift-dns/dns-default-wjll7"
Oct 30 02:55:21.594577 ip-10-0-143-156 kubenswrapper[2574]: E1030 02:55:21.594532    2574 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 30 02:55:21.594617 ip-10-0-143-156 kubenswrapper[2574]: E1030 02:55:21.594586    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5f9fb000-9531-41f1-b3cf-7e79ca1a24d7-metrics-tls podName:5f9fb000-9531-41f1-b3cf-7e79ca1a24d7 nodeName:}" failed. No retries permitted until 2026-10-30 02:55:22.594573883 +0000 UTC m=+31.013325435 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/5f9fb000-9531-41f1-b3cf-7e79ca1a24d7-metrics-tls") pod "dns-default-wjll7" (UID: "5f9fb000-9531-41f1-b3cf-7e79ca1a24d7") : secret "dns-default-metrics-tls" not found
Oct 30 02:55:22.344400 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:55:22.344364    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-vwbj7" event={"ID":"73752bb9-2a14-4582-a6d6-c1fd26524c3d","Type":"ContainerStarted","Data":"e6dac9b7049e7ef1cd190b3786ce14164794eaa8d4d4277031c61a208cd1fea4"}
Oct 30 02:55:22.603454 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:55:22.603381    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/5f9fb000-9531-41f1-b3cf-7e79ca1a24d7-metrics-tls\") pod \"dns-default-wjll7\" (UID: \"5f9fb000-9531-41f1-b3cf-7e79ca1a24d7\") " pod="openshift-dns/dns-default-wjll7"
Oct 30 02:55:22.603608 ip-10-0-143-156 kubenswrapper[2574]: E1030 02:55:22.603572    2574 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 30 02:55:22.603683 ip-10-0-143-156 kubenswrapper[2574]: E1030 02:55:22.603639    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5f9fb000-9531-41f1-b3cf-7e79ca1a24d7-metrics-tls podName:5f9fb000-9531-41f1-b3cf-7e79ca1a24d7 nodeName:}" failed. No retries permitted until 2026-10-30 02:55:24.603621792 +0000 UTC m=+33.022373343 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/5f9fb000-9531-41f1-b3cf-7e79ca1a24d7-metrics-tls") pod "dns-default-wjll7" (UID: "5f9fb000-9531-41f1-b3cf-7e79ca1a24d7") : secret "dns-default-metrics-tls" not found
Oct 30 02:55:22.736945 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:55:22.736905    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 02:55:22.906741 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:55:22.906662    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 02:55:24.348938 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:55:24.348906    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 02:55:24.350413 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:55:24.350382    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-vwbj7" event={"ID":"73752bb9-2a14-4582-a6d6-c1fd26524c3d","Type":"ContainerStarted","Data":"f14c9bc314b1f12f9d6e3df7be57f1918c27e4c0a283162a86f9e1fa2db9cbec"}
Oct 30 02:55:24.395200 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:55:24.395117    2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-vwbj7" podStartSLOduration=1.819093882 podStartE2EDuration="4.395089403s" totalImagesPullingTime="2.575995521s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:55:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:55:24.395011488 +0000 UTC m=+32.813763123" watchObservedRunningTime="2026-10-30 02:55:24.395089403 +0000 UTC m=+32.813840975"
Oct 30 02:55:24.519810 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:55:24.519784    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 02:55:24.618502 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:55:24.618471    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/5f9fb000-9531-41f1-b3cf-7e79ca1a24d7-metrics-tls\") pod \"dns-default-wjll7\" (UID: \"5f9fb000-9531-41f1-b3cf-7e79ca1a24d7\") " pod="openshift-dns/dns-default-wjll7"
Oct 30 02:55:24.618673 ip-10-0-143-156 kubenswrapper[2574]: E1030 02:55:24.618581    2574 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 30 02:55:24.618673 ip-10-0-143-156 kubenswrapper[2574]: E1030 02:55:24.618631    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5f9fb000-9531-41f1-b3cf-7e79ca1a24d7-metrics-tls podName:5f9fb000-9531-41f1-b3cf-7e79ca1a24d7 nodeName:}" failed. No retries permitted until 2026-10-30 02:55:28.618616653 +0000 UTC m=+37.037368205 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/5f9fb000-9531-41f1-b3cf-7e79ca1a24d7-metrics-tls") pod "dns-default-wjll7" (UID: "5f9fb000-9531-41f1-b3cf-7e79ca1a24d7") : secret "dns-default-metrics-tls" not found
Oct 30 02:55:25.826786 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:55:25.826748    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/98f9efe3-a732-4ba4-8543-941633d22170-original-pull-secret\") pod \"global-pull-secret-syncer-vbnwt\" (UID: \"98f9efe3-a732-4ba4-8543-941633d22170\") " pod="kube-system/global-pull-secret-syncer-vbnwt"
Oct 30 02:55:25.827293 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:55:25.826803    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3bd2c646-cebb-4e8a-b6c3-8b5b74049c11-metrics-certs\") pod \"network-metrics-daemon-mzk2s\" (UID: \"3bd2c646-cebb-4e8a-b6c3-8b5b74049c11\") " pod="openshift-multus/network-metrics-daemon-mzk2s"
Oct 30 02:55:25.827293 ip-10-0-143-156 kubenswrapper[2574]: E1030 02:55:25.826950    2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Oct 30 02:55:25.827293 ip-10-0-143-156 kubenswrapper[2574]: E1030 02:55:25.827058    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3bd2c646-cebb-4e8a-b6c3-8b5b74049c11-metrics-certs podName:3bd2c646-cebb-4e8a-b6c3-8b5b74049c11 nodeName:}" failed. No retries permitted until 2026-10-30 02:55:57.82700112 +0000 UTC m=+66.245752677 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3bd2c646-cebb-4e8a-b6c3-8b5b74049c11-metrics-certs") pod "network-metrics-daemon-mzk2s" (UID: "3bd2c646-cebb-4e8a-b6c3-8b5b74049c11") : secret "metrics-daemon-secret" not found
Oct 30 02:55:25.829378 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:55:25.829358    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/98f9efe3-a732-4ba4-8543-941633d22170-original-pull-secret\") pod \"global-pull-secret-syncer-vbnwt\" (UID: \"98f9efe3-a732-4ba4-8543-941633d22170\") " pod="kube-system/global-pull-secret-syncer-vbnwt"
Oct 30 02:55:25.847856 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:55:25.847832    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vbnwt"
Oct 30 02:55:25.927343 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:55:25.927316    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-62vqf\" (UniqueName: \"kubernetes.io/projected/75d9e1ec-cd1b-4aba-b911-5a7465c653cf-kube-api-access-62vqf\") pod \"network-check-target-7rmdj\" (UID: \"75d9e1ec-cd1b-4aba-b911-5a7465c653cf\") " pod="openshift-network-diagnostics/network-check-target-7rmdj"
Oct 30 02:55:25.929733 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:55:25.929714    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-62vqf\" (UniqueName: \"kubernetes.io/projected/75d9e1ec-cd1b-4aba-b911-5a7465c653cf-kube-api-access-62vqf\") pod \"network-check-target-7rmdj\" (UID: \"75d9e1ec-cd1b-4aba-b911-5a7465c653cf\") " pod="openshift-network-diagnostics/network-check-target-7rmdj"
Oct 30 02:55:26.135145 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:55:26.135033    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7rmdj"
Oct 30 02:55:26.220062 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:55:26.220033    2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-vbnwt"]
Oct 30 02:55:26.223024 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:55:26.222997    2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod98f9efe3_a732_4ba4_8543_941633d22170.slice/crio-2cc7ee155e2eb45b9d64e05c2e366ab7d8efbce84ad1c87a6a3026d836006ccf WatchSource:0}: Error finding container 2cc7ee155e2eb45b9d64e05c2e366ab7d8efbce84ad1c87a6a3026d836006ccf: Status 404 returned error can't find the container with id 2cc7ee155e2eb45b9d64e05c2e366ab7d8efbce84ad1c87a6a3026d836006ccf
Oct 30 02:55:26.356174 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:55:26.356140    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-vbnwt" event={"ID":"98f9efe3-a732-4ba4-8543-941633d22170","Type":"ContainerStarted","Data":"2cc7ee155e2eb45b9d64e05c2e366ab7d8efbce84ad1c87a6a3026d836006ccf"}
Oct 30 02:55:26.455832 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:55:26.455738    2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-7rmdj"]
Oct 30 02:55:26.459624 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:55:26.459598    2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod75d9e1ec_cd1b_4aba_b911_5a7465c653cf.slice/crio-e7d7991e2afe977864c6c9c3e20434a2864d1dfeb3fde173954ba46cf3d906b7 WatchSource:0}: Error finding container e7d7991e2afe977864c6c9c3e20434a2864d1dfeb3fde173954ba46cf3d906b7: Status 404 returned error can't find the container with id e7d7991e2afe977864c6c9c3e20434a2864d1dfeb3fde173954ba46cf3d906b7
Oct 30 02:55:26.526776 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:55:26.526744    2574 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-2k85b"]
Oct 30 02:55:26.561694 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:55:26.561661    2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-2k85b"]
Oct 30 02:55:26.561828 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:55:26.561784    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-2k85b"
Oct 30 02:55:26.625681 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:55:26.625639    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Oct 30 02:55:26.625681 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:55:26.625654    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Oct 30 02:55:26.625681 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:55:26.625653    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Oct 30 02:55:26.634973 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:55:26.634944    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Oct 30 02:55:26.635092 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:55:26.634983    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-d8b4w\""
Oct 30 02:55:26.665583 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:55:26.665532    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 02:55:26.707168 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:55:26.707078    2574 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-wx7pd"]
Oct 30 02:55:26.723041 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:55:26.723020    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-wx7pd"
Oct 30 02:55:26.728929 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:55:26.728903    2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-wx7pd"]
Oct 30 02:55:26.732688 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:55:26.732663    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/eee29ef7-4ad4-4ff9-8cb0-9d2a5aaabba3-crio-socket\") pod \"insights-runtime-extractor-2k85b\" (UID: \"eee29ef7-4ad4-4ff9-8cb0-9d2a5aaabba3\") " pod="openshift-insights/insights-runtime-extractor-2k85b"
Oct 30 02:55:26.732792 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:55:26.732702    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/eee29ef7-4ad4-4ff9-8cb0-9d2a5aaabba3-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2k85b\" (UID: \"eee29ef7-4ad4-4ff9-8cb0-9d2a5aaabba3\") " pod="openshift-insights/insights-runtime-extractor-2k85b"
Oct 30 02:55:26.732792 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:55:26.732746    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/eee29ef7-4ad4-4ff9-8cb0-9d2a5aaabba3-data-volume\") pod \"insights-runtime-extractor-2k85b\" (UID: \"eee29ef7-4ad4-4ff9-8cb0-9d2a5aaabba3\") " pod="openshift-insights/insights-runtime-extractor-2k85b"
Oct 30 02:55:26.732792 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:55:26.732767    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/eee29ef7-4ad4-4ff9-8cb0-9d2a5aaabba3-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-2k85b\" (UID: \"eee29ef7-4ad4-4ff9-8cb0-9d2a5aaabba3\") " pod="openshift-insights/insights-runtime-extractor-2k85b"
Oct 30 02:55:26.732943 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:55:26.732796    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wd8s8\" (UniqueName: \"kubernetes.io/projected/eee29ef7-4ad4-4ff9-8cb0-9d2a5aaabba3-kube-api-access-wd8s8\") pod \"insights-runtime-extractor-2k85b\" (UID: \"eee29ef7-4ad4-4ff9-8cb0-9d2a5aaabba3\") " pod="openshift-insights/insights-runtime-extractor-2k85b"
Oct 30 02:55:26.749849 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:55:26.749830    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Oct 30 02:55:26.750107 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:55:26.750067    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Oct 30 02:55:26.833489 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:55:26.833454    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wd8s8\" (UniqueName: \"kubernetes.io/projected/eee29ef7-4ad4-4ff9-8cb0-9d2a5aaabba3-kube-api-access-wd8s8\") pod \"insights-runtime-extractor-2k85b\" (UID: \"eee29ef7-4ad4-4ff9-8cb0-9d2a5aaabba3\") " pod="openshift-insights/insights-runtime-extractor-2k85b"
Oct 30 02:55:26.833912 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:55:26.833521    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/eee29ef7-4ad4-4ff9-8cb0-9d2a5aaabba3-crio-socket\") pod \"insights-runtime-extractor-2k85b\" (UID: \"eee29ef7-4ad4-4ff9-8cb0-9d2a5aaabba3\") " pod="openshift-insights/insights-runtime-extractor-2k85b"
Oct 30 02:55:26.833912 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:55:26.833550    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/eee29ef7-4ad4-4ff9-8cb0-9d2a5aaabba3-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2k85b\" (UID: \"eee29ef7-4ad4-4ff9-8cb0-9d2a5aaabba3\") " pod="openshift-insights/insights-runtime-extractor-2k85b"
Oct 30 02:55:26.833912 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:55:26.833582    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rkgpj\" (UniqueName: \"kubernetes.io/projected/f990bb17-3b0f-4bca-8939-089f79d6edef-kube-api-access-rkgpj\") pod \"migrator-66df5d496b-wx7pd\" (UID: \"f990bb17-3b0f-4bca-8939-089f79d6edef\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-wx7pd"
Oct 30 02:55:26.833912 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:55:26.833650    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/eee29ef7-4ad4-4ff9-8cb0-9d2a5aaabba3-data-volume\") pod \"insights-runtime-extractor-2k85b\" (UID: \"eee29ef7-4ad4-4ff9-8cb0-9d2a5aaabba3\") " pod="openshift-insights/insights-runtime-extractor-2k85b"
Oct 30 02:55:26.833912 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:55:26.833673    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/eee29ef7-4ad4-4ff9-8cb0-9d2a5aaabba3-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-2k85b\" (UID: \"eee29ef7-4ad4-4ff9-8cb0-9d2a5aaabba3\") " pod="openshift-insights/insights-runtime-extractor-2k85b"
Oct 30 02:55:26.833912 ip-10-0-143-156 kubenswrapper[2574]: E1030 02:55:26.833764    2574 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Oct 30 02:55:26.833912 ip-10-0-143-156 kubenswrapper[2574]: E1030 02:55:26.833842    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/eee29ef7-4ad4-4ff9-8cb0-9d2a5aaabba3-kube-rbac-proxy-cm podName:eee29ef7-4ad4-4ff9-8cb0-9d2a5aaabba3 nodeName:}" failed. No retries permitted until 2026-10-30 02:55:27.333820333 +0000 UTC m=+35.752571900 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/eee29ef7-4ad4-4ff9-8cb0-9d2a5aaabba3-kube-rbac-proxy-cm") pod "insights-runtime-extractor-2k85b" (UID: "eee29ef7-4ad4-4ff9-8cb0-9d2a5aaabba3") : configmap "kube-rbac-proxy" not found
Oct 30 02:55:26.833912 ip-10-0-143-156 kubenswrapper[2574]: E1030 02:55:26.833850    2574 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 30 02:55:26.833912 ip-10-0-143-156 kubenswrapper[2574]: E1030 02:55:26.833897    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/eee29ef7-4ad4-4ff9-8cb0-9d2a5aaabba3-insights-runtime-extractor-tls podName:eee29ef7-4ad4-4ff9-8cb0-9d2a5aaabba3 nodeName:}" failed. No retries permitted until 2026-10-30 02:55:27.333884711 +0000 UTC m=+35.752636263 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/eee29ef7-4ad4-4ff9-8cb0-9d2a5aaabba3-insights-runtime-extractor-tls") pod "insights-runtime-extractor-2k85b" (UID: "eee29ef7-4ad4-4ff9-8cb0-9d2a5aaabba3") : secret "insights-runtime-extractor-tls" not found
Oct 30 02:55:26.834311 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:55:26.834041    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/eee29ef7-4ad4-4ff9-8cb0-9d2a5aaabba3-crio-socket\") pod \"insights-runtime-extractor-2k85b\" (UID: \"eee29ef7-4ad4-4ff9-8cb0-9d2a5aaabba3\") " pod="openshift-insights/insights-runtime-extractor-2k85b"
Oct 30 02:55:26.834430 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:55:26.834409    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/eee29ef7-4ad4-4ff9-8cb0-9d2a5aaabba3-data-volume\") pod \"insights-runtime-extractor-2k85b\" (UID: \"eee29ef7-4ad4-4ff9-8cb0-9d2a5aaabba3\") " pod="openshift-insights/insights-runtime-extractor-2k85b"
Oct 30 02:55:26.835920 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:55:26.835888    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 02:55:26.836962 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:55:26.836941    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-chfb9\""
Oct 30 02:55:26.860849 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:55:26.860797    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wd8s8\" (UniqueName: \"kubernetes.io/projected/eee29ef7-4ad4-4ff9-8cb0-9d2a5aaabba3-kube-api-access-wd8s8\") pod \"insights-runtime-extractor-2k85b\" (UID: \"eee29ef7-4ad4-4ff9-8cb0-9d2a5aaabba3\") " pod="openshift-insights/insights-runtime-extractor-2k85b"
Oct 30 02:55:26.934460 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:55:26.934416    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rkgpj\" (UniqueName: \"kubernetes.io/projected/f990bb17-3b0f-4bca-8939-089f79d6edef-kube-api-access-rkgpj\") pod \"migrator-66df5d496b-wx7pd\" (UID: \"f990bb17-3b0f-4bca-8939-089f79d6edef\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-wx7pd"
Oct 30 02:55:26.960232 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:55:26.960170    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rkgpj\" (UniqueName: \"kubernetes.io/projected/f990bb17-3b0f-4bca-8939-089f79d6edef-kube-api-access-rkgpj\") pod \"migrator-66df5d496b-wx7pd\" (UID: \"f990bb17-3b0f-4bca-8939-089f79d6edef\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-wx7pd"
Oct 30 02:55:27.016845 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:55:27.016811    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 02:55:27.046122 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:55:27.046086    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-wx7pd"
Oct 30 02:55:27.336950 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:55:27.336885    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/eee29ef7-4ad4-4ff9-8cb0-9d2a5aaabba3-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2k85b\" (UID: \"eee29ef7-4ad4-4ff9-8cb0-9d2a5aaabba3\") " pod="openshift-insights/insights-runtime-extractor-2k85b"
Oct 30 02:55:27.337150 ip-10-0-143-156 kubenswrapper[2574]: E1030 02:55:27.337053    2574 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 30 02:55:27.337150 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:55:27.337055    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/eee29ef7-4ad4-4ff9-8cb0-9d2a5aaabba3-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-2k85b\" (UID: \"eee29ef7-4ad4-4ff9-8cb0-9d2a5aaabba3\") " pod="openshift-insights/insights-runtime-extractor-2k85b"
Oct 30 02:55:27.337278 ip-10-0-143-156 kubenswrapper[2574]: E1030 02:55:27.337151    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/eee29ef7-4ad4-4ff9-8cb0-9d2a5aaabba3-insights-runtime-extractor-tls podName:eee29ef7-4ad4-4ff9-8cb0-9d2a5aaabba3 nodeName:}" failed. No retries permitted until 2026-10-30 02:55:28.337124592 +0000 UTC m=+36.755876143 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/eee29ef7-4ad4-4ff9-8cb0-9d2a5aaabba3-insights-runtime-extractor-tls") pod "insights-runtime-extractor-2k85b" (UID: "eee29ef7-4ad4-4ff9-8cb0-9d2a5aaabba3") : secret "insights-runtime-extractor-tls" not found
Oct 30 02:55:27.337278 ip-10-0-143-156 kubenswrapper[2574]: E1030 02:55:27.337175    2574 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Oct 30 02:55:27.337278 ip-10-0-143-156 kubenswrapper[2574]: E1030 02:55:27.337228    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/eee29ef7-4ad4-4ff9-8cb0-9d2a5aaabba3-kube-rbac-proxy-cm podName:eee29ef7-4ad4-4ff9-8cb0-9d2a5aaabba3 nodeName:}" failed. No retries permitted until 2026-10-30 02:55:28.337213015 +0000 UTC m=+36.755964588 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/eee29ef7-4ad4-4ff9-8cb0-9d2a5aaabba3-kube-rbac-proxy-cm") pod "insights-runtime-extractor-2k85b" (UID: "eee29ef7-4ad4-4ff9-8cb0-9d2a5aaabba3") : configmap "kube-rbac-proxy" not found
Oct 30 02:55:27.361148 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:55:27.360705    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-7rmdj" event={"ID":"75d9e1ec-cd1b-4aba-b911-5a7465c653cf","Type":"ContainerStarted","Data":"e7d7991e2afe977864c6c9c3e20434a2864d1dfeb3fde173954ba46cf3d906b7"}
Oct 30 02:55:27.365222 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:55:27.365192    2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-wx7pd"]
Oct 30 02:55:27.368760 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:55:27.368729    2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf990bb17_3b0f_4bca_8939_089f79d6edef.slice/crio-276ad63052e23d43dd97ff88a9374214f5c7a3dd8ee648681c9783f6eba26749 WatchSource:0}: Error finding container 276ad63052e23d43dd97ff88a9374214f5c7a3dd8ee648681c9783f6eba26749: Status 404 returned error can't find the container with id 276ad63052e23d43dd97ff88a9374214f5c7a3dd8ee648681c9783f6eba26749
Oct 30 02:55:27.376844 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:55:27.376814    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 02:55:27.542663 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:55:27.542623    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 02:55:27.718465 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:55:27.718429    2574 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-rwq7h"]
Oct 30 02:55:27.753006 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:55:27.752967    2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-rwq7h"]
Oct 30 02:55:27.753168 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:55:27.753076    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-5fffc97bdf-rwq7h"
Oct 30 02:55:27.769696 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:55:27.769672    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\""
Oct 30 02:55:27.783076 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:55:27.783048    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\""
Oct 30 02:55:27.783252 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:55:27.783054    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\""
Oct 30 02:55:27.783252 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:55:27.783130    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\""
Oct 30 02:55:27.783372 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:55:27.783357    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-t6hqp\""
Oct 30 02:55:27.942111 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:55:27.942066    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/4a776e60-80b6-4ce0-921c-1884973b4d0b-signing-cabundle\") pod \"service-ca-5fffc97bdf-rwq7h\" (UID: \"4a776e60-80b6-4ce0-921c-1884973b4d0b\") " pod="openshift-service-ca/service-ca-5fffc97bdf-rwq7h"
Oct 30 02:55:27.942550 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:55:27.942221    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/4a776e60-80b6-4ce0-921c-1884973b4d0b-signing-key\") pod \"service-ca-5fffc97bdf-rwq7h\" (UID: \"4a776e60-80b6-4ce0-921c-1884973b4d0b\") " pod="openshift-service-ca/service-ca-5fffc97bdf-rwq7h"
Oct 30 02:55:27.942550 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:55:27.942311    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2289h\" (UniqueName: \"kubernetes.io/projected/4a776e60-80b6-4ce0-921c-1884973b4d0b-kube-api-access-2289h\") pod \"service-ca-5fffc97bdf-rwq7h\" (UID: \"4a776e60-80b6-4ce0-921c-1884973b4d0b\") " pod="openshift-service-ca/service-ca-5fffc97bdf-rwq7h"
Oct 30 02:55:28.043184 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:55:28.043105    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/4a776e60-80b6-4ce0-921c-1884973b4d0b-signing-cabundle\") pod \"service-ca-5fffc97bdf-rwq7h\" (UID: \"4a776e60-80b6-4ce0-921c-1884973b4d0b\") " pod="openshift-service-ca/service-ca-5fffc97bdf-rwq7h"
Oct 30 02:55:28.043341 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:55:28.043232    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/4a776e60-80b6-4ce0-921c-1884973b4d0b-signing-key\") pod \"service-ca-5fffc97bdf-rwq7h\" (UID: \"4a776e60-80b6-4ce0-921c-1884973b4d0b\") " pod="openshift-service-ca/service-ca-5fffc97bdf-rwq7h"
Oct 30 02:55:28.043341 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:55:28.043267    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2289h\" (UniqueName: \"kubernetes.io/projected/4a776e60-80b6-4ce0-921c-1884973b4d0b-kube-api-access-2289h\") pod \"service-ca-5fffc97bdf-rwq7h\" (UID: \"4a776e60-80b6-4ce0-921c-1884973b4d0b\") " pod="openshift-service-ca/service-ca-5fffc97bdf-rwq7h"
Oct 30 02:55:28.044457 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:55:28.044425    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/4a776e60-80b6-4ce0-921c-1884973b4d0b-signing-cabundle\") pod \"service-ca-5fffc97bdf-rwq7h\" (UID: \"4a776e60-80b6-4ce0-921c-1884973b4d0b\") " pod="openshift-service-ca/service-ca-5fffc97bdf-rwq7h"
Oct 30 02:55:28.047456 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:55:28.047431    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/4a776e60-80b6-4ce0-921c-1884973b4d0b-signing-key\") pod \"service-ca-5fffc97bdf-rwq7h\" (UID: \"4a776e60-80b6-4ce0-921c-1884973b4d0b\") " pod="openshift-service-ca/service-ca-5fffc97bdf-rwq7h"
Oct 30 02:55:28.068576 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:55:28.068525    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2289h\" (UniqueName: \"kubernetes.io/projected/4a776e60-80b6-4ce0-921c-1884973b4d0b-kube-api-access-2289h\") pod \"service-ca-5fffc97bdf-rwq7h\" (UID: \"4a776e60-80b6-4ce0-921c-1884973b4d0b\") " pod="openshift-service-ca/service-ca-5fffc97bdf-rwq7h"
Oct 30 02:55:28.078134 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:55:28.078075    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-5fffc97bdf-rwq7h"
Oct 30 02:55:28.179962 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:55:28.175856    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 02:55:28.346502 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:55:28.346420    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/eee29ef7-4ad4-4ff9-8cb0-9d2a5aaabba3-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2k85b\" (UID: \"eee29ef7-4ad4-4ff9-8cb0-9d2a5aaabba3\") " pod="openshift-insights/insights-runtime-extractor-2k85b"
Oct 30 02:55:28.346665 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:55:28.346504    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/eee29ef7-4ad4-4ff9-8cb0-9d2a5aaabba3-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-2k85b\" (UID: \"eee29ef7-4ad4-4ff9-8cb0-9d2a5aaabba3\") " pod="openshift-insights/insights-runtime-extractor-2k85b"
Oct 30 02:55:28.346665 ip-10-0-143-156 kubenswrapper[2574]: E1030 02:55:28.346618    2574 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Oct 30 02:55:28.346781 ip-10-0-143-156 kubenswrapper[2574]: E1030 02:55:28.346672    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/eee29ef7-4ad4-4ff9-8cb0-9d2a5aaabba3-kube-rbac-proxy-cm podName:eee29ef7-4ad4-4ff9-8cb0-9d2a5aaabba3 nodeName:}" failed. No retries permitted until 2026-10-30 02:55:30.34665671 +0000 UTC m=+38.765408291 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/eee29ef7-4ad4-4ff9-8cb0-9d2a5aaabba3-kube-rbac-proxy-cm") pod "insights-runtime-extractor-2k85b" (UID: "eee29ef7-4ad4-4ff9-8cb0-9d2a5aaabba3") : configmap "kube-rbac-proxy" not found
Oct 30 02:55:28.347074 ip-10-0-143-156 kubenswrapper[2574]: E1030 02:55:28.347048    2574 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 30 02:55:28.347226 ip-10-0-143-156 kubenswrapper[2574]: E1030 02:55:28.347159    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/eee29ef7-4ad4-4ff9-8cb0-9d2a5aaabba3-insights-runtime-extractor-tls podName:eee29ef7-4ad4-4ff9-8cb0-9d2a5aaabba3 nodeName:}" failed. No retries permitted until 2026-10-30 02:55:30.347109832 +0000 UTC m=+38.765861398 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/eee29ef7-4ad4-4ff9-8cb0-9d2a5aaabba3-insights-runtime-extractor-tls") pod "insights-runtime-extractor-2k85b" (UID: "eee29ef7-4ad4-4ff9-8cb0-9d2a5aaabba3") : secret "insights-runtime-extractor-tls" not found
Oct 30 02:55:28.350935 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:55:28.350904    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 02:55:28.367121 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:55:28.366458    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-wx7pd" event={"ID":"f990bb17-3b0f-4bca-8939-089f79d6edef","Type":"ContainerStarted","Data":"276ad63052e23d43dd97ff88a9374214f5c7a3dd8ee648681c9783f6eba26749"}
Oct 30 02:55:28.418332 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:55:28.418299    2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-rwq7h"]
Oct 30 02:55:28.422493 ip-10-0-143-156 kubenswrapper[2574]: W1030 02:55:28.422462    2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4a776e60_80b6_4ce0_921c_1884973b4d0b.slice/crio-86332d2a55a3de71d4696a8cc0d46285defd9952fa3b30ae59d98e1868590524 WatchSource:0}: Error finding container 86332d2a55a3de71d4696a8cc0d46285defd9952fa3b30ae59d98e1868590524: Status 404 returned error can't find the container with id 86332d2a55a3de71d4696a8cc0d46285defd9952fa3b30ae59d98e1868590524
Oct 30 02:55:28.649736 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:55:28.649649    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/5f9fb000-9531-41f1-b3cf-7e79ca1a24d7-metrics-tls\") pod \"dns-default-wjll7\" (UID: \"5f9fb000-9531-41f1-b3cf-7e79ca1a24d7\") " pod="openshift-dns/dns-default-wjll7"
Oct 30 02:55:28.649860 ip-10-0-143-156 kubenswrapper[2574]: E1030 02:55:28.649798    2574 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 30 02:55:28.649902 ip-10-0-143-156 kubenswrapper[2574]: E1030 02:55:28.649865    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5f9fb000-9531-41f1-b3cf-7e79ca1a24d7-metrics-tls podName:5f9fb000-9531-41f1-b3cf-7e79ca1a24d7 nodeName:}" failed. No retries permitted until 2026-10-30 02:55:36.649847876 +0000 UTC m=+45.068599428 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/5f9fb000-9531-41f1-b3cf-7e79ca1a24d7-metrics-tls") pod "dns-default-wjll7" (UID: "5f9fb000-9531-41f1-b3cf-7e79ca1a24d7") : secret "dns-default-metrics-tls" not found
Oct 30 02:55:29.025795 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:55:29.025761    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 02:55:29.370937 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:55:29.370905    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-5fffc97bdf-rwq7h" event={"ID":"4a776e60-80b6-4ce0-921c-1884973b4d0b","Type":"ContainerStarted","Data":"86332d2a55a3de71d4696a8cc0d46285defd9952fa3b30ae59d98e1868590524"}
Oct 30 02:55:29.373269 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:55:29.373239    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-wx7pd" event={"ID":"f990bb17-3b0f-4bca-8939-089f79d6edef","Type":"ContainerStarted","Data":"81ab4f3ee2661dc7681ea000520a52466e301bafc247c8caebd07eabcdb4838a"}
Oct 30 02:55:29.416514 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:55:29.416446    2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-wx7pd" podStartSLOduration=1.57474527 podStartE2EDuration="3.416425477s" totalImagesPullingTime="1.841680207s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:55:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:55:29.415387385 +0000 UTC m=+37.834138970" watchObservedRunningTime="2026-10-30 02:55:29.416425477 +0000 UTC m=+37.835177051"
Oct 30 02:55:30.363379 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:55:30.363336    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/eee29ef7-4ad4-4ff9-8cb0-9d2a5aaabba3-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2k85b\" (UID: \"eee29ef7-4ad4-4ff9-8cb0-9d2a5aaabba3\") " pod="openshift-insights/insights-runtime-extractor-2k85b"
Oct 30 02:55:30.363896 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:55:30.363424    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/eee29ef7-4ad4-4ff9-8cb0-9d2a5aaabba3-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-2k85b\" (UID: \"eee29ef7-4ad4-4ff9-8cb0-9d2a5aaabba3\") " pod="openshift-insights/insights-runtime-extractor-2k85b"
Oct 30 02:55:30.363896 ip-10-0-143-156 kubenswrapper[2574]: E1030 02:55:30.363507    2574 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 30 02:55:30.363896 ip-10-0-143-156 kubenswrapper[2574]: E1030 02:55:30.363546    2574 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Oct 30 02:55:30.363896 ip-10-0-143-156 kubenswrapper[2574]: E1030 02:55:30.363590    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/eee29ef7-4ad4-4ff9-8cb0-9d2a5aaabba3-insights-runtime-extractor-tls podName:eee29ef7-4ad4-4ff9-8cb0-9d2a5aaabba3 nodeName:}" failed. No retries permitted until 2026-10-30 02:55:34.363568629 +0000 UTC m=+42.782320197 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/eee29ef7-4ad4-4ff9-8cb0-9d2a5aaabba3-insights-runtime-extractor-tls") pod "insights-runtime-extractor-2k85b" (UID: "eee29ef7-4ad4-4ff9-8cb0-9d2a5aaabba3") : secret "insights-runtime-extractor-tls" not found
Oct 30 02:55:30.363896 ip-10-0-143-156 kubenswrapper[2574]: E1030 02:55:30.363614    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/eee29ef7-4ad4-4ff9-8cb0-9d2a5aaabba3-kube-rbac-proxy-cm podName:eee29ef7-4ad4-4ff9-8cb0-9d2a5aaabba3 nodeName:}" failed. No retries permitted until 2026-10-30 02:55:34.363603915 +0000 UTC m=+42.782355471 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/eee29ef7-4ad4-4ff9-8cb0-9d2a5aaabba3-kube-rbac-proxy-cm") pod "insights-runtime-extractor-2k85b" (UID: "eee29ef7-4ad4-4ff9-8cb0-9d2a5aaabba3") : configmap "kube-rbac-proxy" not found
Oct 30 02:55:30.377466 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:55:30.377419    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-7rmdj" event={"ID":"75d9e1ec-cd1b-4aba-b911-5a7465c653cf","Type":"ContainerStarted","Data":"4fde5b8b8094178ae5dbbdca7f2da30e6cde1e4360a4938e579e77bedbe9cfd8"}
Oct 30 02:55:30.421901 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:55:30.421834    2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-7rmdj" podStartSLOduration=34.676345004 podStartE2EDuration="38.421817932s" totalImagesPullingTime="3.745472928s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:54:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:55:30.421014979 +0000 UTC m=+38.839766553" watchObservedRunningTime="2026-10-30 02:55:30.421817932 +0000 UTC m=+38.840569505"
Oct 30 02:55:31.381590 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:55:31.381563    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-5fffc97bdf-rwq7h" event={"ID":"4a776e60-80b6-4ce0-921c-1884973b4d0b","Type":"ContainerStarted","Data":"2af542881307ba4264c03c7582b2be26da4909ac5966448ab49dedc7a6b22e97"}
Oct 30 02:55:31.382004 ip-10-0-143-156 kubenswrapper[2574]: I1030 02:55:31.381692    2574 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-7rmdj"
Oct 30 02:55:31.900274 ip-10-0-143-156 systemd[1]: Stopping Kubernetes Kubelet...
Oct 30 02:55:31.911442 ip-10-0-143-156 systemd[1]: kubelet.service: Deactivated successfully.
Oct 30 02:55:31.911615 ip-10-0-143-156 systemd[1]: Stopped Kubernetes Kubelet.
Oct 30 02:55:31.911681 ip-10-0-143-156 systemd[1]: kubelet.service: Consumed 2.285s CPU time.
Oct 30 02:55:31.919855 ip-10-0-143-156 systemd[1]: Starting Kubernetes Kubelet...
Oct 30 02:55:32.021261 ip-10-0-143-156 kubenswrapper[4461]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 02:55:32.021261 ip-10-0-143-156 kubenswrapper[4461]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Oct 30 02:55:32.021261 ip-10-0-143-156 kubenswrapper[4461]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 02:55:32.021261 ip-10-0-143-156 kubenswrapper[4461]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 02:55:32.023468 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.023378    4461 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 02:55:32.023468 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.023466    4461 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 02:55:32.023468 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.023470    4461 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 02:55:32.023468 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.023473    4461 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 02:55:32.023630 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.023477    4461 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 02:55:32.023630 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.023479    4461 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 02:55:32.023630 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.023482    4461 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 02:55:32.023630 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.023485    4461 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 02:55:32.023630 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.023488    4461 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 02:55:32.023630 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.023491    4461 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 02:55:32.023630 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.023493    4461 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 02:55:32.023630 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.023496    4461 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 02:55:32.023630 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.023499    4461 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 02:55:32.023630 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.023501    4461 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 02:55:32.023630 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.023504    4461 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 02:55:32.023630 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.023507    4461 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 02:55:32.023630 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.023510    4461 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 02:55:32.023630 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.023519    4461 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 02:55:32.023630 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.023522    4461 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 02:55:32.023630 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.023525    4461 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 02:55:32.023630 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.023527    4461 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 02:55:32.023630 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.023530    4461 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 02:55:32.023630 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.023533    4461 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 02:55:32.023630 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.023535    4461 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 02:55:32.024213 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.023538    4461 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 02:55:32.024213 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.023541    4461 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 02:55:32.024213 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.023543    4461 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 02:55:32.024213 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.023553    4461 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 02:55:32.024213 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.023556    4461 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 02:55:32.024213 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.023559    4461 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 02:55:32.024213 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.023561    4461 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 02:55:32.024213 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.023564    4461 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 02:55:32.024213 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.023567    4461 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 02:55:32.024213 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.023569    4461 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 02:55:32.024213 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.023573    4461 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 02:55:32.024213 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.023576    4461 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 02:55:32.024213 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.023579    4461 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 02:55:32.024213 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.023582    4461 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 02:55:32.024213 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.023585    4461 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 02:55:32.024213 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.023587    4461 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 02:55:32.024213 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.023590    4461 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 02:55:32.024213 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.023593    4461 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 02:55:32.024213 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.023595    4461 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 02:55:32.024213 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.023598    4461 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 02:55:32.024705 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.023600    4461 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 02:55:32.024705 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.023603    4461 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 02:55:32.024705 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.023607    4461 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 02:55:32.024705 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.023611    4461 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 02:55:32.024705 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.023615    4461 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 02:55:32.024705 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.023619    4461 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 02:55:32.024705 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.023626    4461 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 02:55:32.024705 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.023631    4461 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 02:55:32.024705 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.023636    4461 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 02:55:32.024705 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.023641    4461 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 02:55:32.024705 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.023646    4461 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 02:55:32.024705 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.023651    4461 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 02:55:32.024705 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.023659    4461 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 02:55:32.024705 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.023666    4461 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 02:55:32.024705 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.023673    4461 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 02:55:32.024705 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.023677    4461 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 02:55:32.024705 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.023681    4461 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 02:55:32.024705 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.023686    4461 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 02:55:32.024705 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.023690    4461 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 02:55:32.025198 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.023694    4461 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 02:55:32.025198 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.023698    4461 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 02:55:32.025198 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.023701    4461 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 02:55:32.025198 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.023705    4461 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 02:55:32.025198 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.023710    4461 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 02:55:32.025198 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.023715    4461 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 02:55:32.025198 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.023719    4461 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 02:55:32.025198 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.023723    4461 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 02:55:32.025198 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.023728    4461 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 02:55:32.025198 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.023732    4461 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 02:55:32.025198 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.023737    4461 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 02:55:32.025198 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.023741    4461 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 02:55:32.025198 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.023745    4461 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 02:55:32.025198 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.023749    4461 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 02:55:32.025198 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.023753    4461 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 02:55:32.025198 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.023756    4461 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 02:55:32.025198 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.023760    4461 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 02:55:32.025198 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.023763    4461 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 02:55:32.025198 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.023769    4461 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 02:55:32.025665 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.023776    4461 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 02:55:32.025665 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.023780    4461 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 02:55:32.025665 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.023785    4461 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 02:55:32.025665 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.023789    4461 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 02:55:32.025665 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.023793    4461 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 02:55:32.025665 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.023799    4461 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 02:55:32.025665 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.023805    4461 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 02:55:32.025665 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.023809    4461 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 02:55:32.025665 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.023814    4461 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 02:55:32.025665 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.023818    4461 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 02:55:32.025665 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.023822    4461 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 02:55:32.025665 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.023827    4461 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 02:55:32.025665 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.023831    4461 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 02:55:32.025665 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.023835    4461 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 02:55:32.025665 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.023839    4461 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 02:55:32.025665 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.023844    4461 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 02:55:32.025665 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.023848    4461 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 02:55:32.025665 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.023852    4461 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 02:55:32.025665 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.023857    4461 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 02:55:32.025665 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.023861    4461 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 02:55:32.026522 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.023865    4461 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 02:55:32.026522 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.023869    4461 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 02:55:32.026522 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.023873    4461 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 02:55:32.026522 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.023879    4461 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 02:55:32.026522 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.024672    4461 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 02:55:32.026522 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.024680    4461 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 02:55:32.026522 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.024683    4461 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 02:55:32.026522 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.024687    4461 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 02:55:32.026522 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.024690    4461 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 02:55:32.026522 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.024693    4461 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 02:55:32.026522 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.024697    4461 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 02:55:32.026522 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.024700    4461 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 02:55:32.026522 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.024703    4461 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 02:55:32.026522 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.024706    4461 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 02:55:32.026522 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.024709    4461 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 02:55:32.026522 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.024712    4461 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 02:55:32.026522 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.024715    4461 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 02:55:32.026522 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.024717    4461 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 02:55:32.026522 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.024722    4461 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 02:55:32.027281 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.024727    4461 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 02:55:32.027281 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.024730    4461 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 02:55:32.027281 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.024733    4461 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 02:55:32.027281 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.024735    4461 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 02:55:32.027281 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.024738    4461 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 02:55:32.027281 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.024741    4461 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 02:55:32.027281 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.024744    4461 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 02:55:32.027281 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.024746    4461 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 02:55:32.027281 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.024749    4461 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 02:55:32.027281 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.024752    4461 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 02:55:32.027281 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.024754    4461 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 02:55:32.027281 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.024757    4461 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 02:55:32.027281 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.024760    4461 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 02:55:32.027281 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.024763    4461 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 02:55:32.027281 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.024765    4461 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 02:55:32.027281 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.024768    4461 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 02:55:32.027281 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.024771    4461 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 02:55:32.027281 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.024774    4461 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 02:55:32.027281 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.024777    4461 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 02:55:32.027281 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.024779    4461 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 02:55:32.028006 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.024782    4461 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 02:55:32.028006 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.024785    4461 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 02:55:32.028006 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.024787    4461 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 02:55:32.028006 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.024790    4461 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 02:55:32.028006 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.024793    4461 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 02:55:32.028006 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.024795    4461 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 02:55:32.028006 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.024798    4461 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 02:55:32.028006 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.024800    4461 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 02:55:32.028006 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.024803    4461 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 02:55:32.028006 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.024806    4461 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 02:55:32.028006 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.024808    4461 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 02:55:32.028006 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.024811    4461 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 02:55:32.028006 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.024814    4461 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 02:55:32.028006 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.024817    4461 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 02:55:32.028006 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.024819    4461 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 02:55:32.028006 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.024822    4461 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 02:55:32.028006 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.024824    4461 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 02:55:32.028006 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.024827    4461 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 02:55:32.028006 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.024829    4461 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 02:55:32.028006 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.024832    4461 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 02:55:32.028546 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.024834    4461 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 02:55:32.028546 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.024837    4461 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 02:55:32.028546 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.024839    4461 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 02:55:32.028546 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.024841    4461 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 02:55:32.028546 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.024844    4461 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 02:55:32.028546 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.024848    4461 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 02:55:32.028546 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.024850    4461 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 02:55:32.028546 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.024853    4461 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 02:55:32.028546 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.024856    4461 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 02:55:32.028546 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.024859    4461 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 02:55:32.028546 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.024861    4461 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 02:55:32.028546 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.024864    4461 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 02:55:32.028546 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.024867    4461 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 02:55:32.028546 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.024869    4461 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 02:55:32.028546 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.024872    4461 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 02:55:32.028546 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.024874    4461 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 02:55:32.028546 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.024877    4461 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 02:55:32.028546 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.024879    4461 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 02:55:32.028546 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.024882    4461 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 02:55:32.028546 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.024884    4461 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 02:55:32.029037 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.024887    4461 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 02:55:32.029037 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.024890    4461 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 02:55:32.029037 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.024892    4461 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 02:55:32.029037 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.024895    4461 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 02:55:32.029037 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.024897    4461 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 02:55:32.029037 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.024902    4461 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 02:55:32.029037 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.024905    4461 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 02:55:32.029037 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.024908    4461 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 02:55:32.029037 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.024911    4461 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 02:55:32.029037 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.024914    4461 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 02:55:32.029037 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.024917    4461 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 02:55:32.029037 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.024920    4461 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 02:55:32.029037 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.024922    4461 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 02:55:32.029037 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.024925    4461 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 02:55:32.029037 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.024928    4461 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 02:55:32.029037 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.024930    4461 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 02:55:32.029037 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.024933    4461 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 02:55:32.029037 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.024936    4461 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 02:55:32.029037 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.024938    4461 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 02:55:32.029037 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.024945    4461 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 02:55:32.029544 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.024948    4461 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 02:55:32.029544 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.024951    4461 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 02:55:32.029544 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.024953    4461 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 02:55:32.029544 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.024956    4461 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 02:55:32.029544 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.024959    4461 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 02:55:32.029544 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.024962    4461 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 02:55:32.029544 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.024964    4461 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 02:55:32.029544 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.024967    4461 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 02:55:32.029544 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.024970    4461 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 02:55:32.029544 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.024973    4461 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 02:55:32.029544 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.024976    4461 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 02:55:32.029544 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.025528    4461 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 02:55:32.029544 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.025534    4461 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 02:55:32.029544 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.025537    4461 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 02:55:32.029544 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.025540    4461 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 02:55:32.029544 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.025543    4461 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 02:55:32.029544 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.025546    4461 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 02:55:32.029544 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.025549    4461 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 02:55:32.029544 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.025552    4461 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 02:55:32.030024 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.025555    4461 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 02:55:32.030024 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.025558    4461 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 02:55:32.030024 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.025560    4461 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 02:55:32.030024 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.025563    4461 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 02:55:32.030024 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.025566    4461 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 02:55:32.030024 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.025568    4461 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 02:55:32.030024 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.025571    4461 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 02:55:32.030024 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.025573    4461 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 02:55:32.030024 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.025576    4461 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 02:55:32.030024 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.025579    4461 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 02:55:32.030024 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.025582    4461 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 02:55:32.030024 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.025584    4461 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 02:55:32.030024 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.025589    4461 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 02:55:32.030024 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.025591    4461 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 02:55:32.030024 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.025594    4461 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 02:55:32.030024 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.025597    4461 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 02:55:32.030024 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.025599    4461 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 02:55:32.030024 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.025603    4461 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 02:55:32.030024 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.025605    4461 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 02:55:32.030024 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.025609    4461 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 02:55:32.030520 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.025611    4461 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 02:55:32.030520 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.025614    4461 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 02:55:32.030520 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.025617    4461 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 02:55:32.030520 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.025619    4461 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 02:55:32.030520 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.025622    4461 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 02:55:32.030520 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.025624    4461 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 02:55:32.030520 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.025627    4461 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 02:55:32.030520 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.025630    4461 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 02:55:32.030520 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.025632    4461 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 02:55:32.030520 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.025635    4461 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 02:55:32.030520 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.025638    4461 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 02:55:32.030520 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.025641    4461 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 02:55:32.030520 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.025644    4461 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 02:55:32.030520 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.025646    4461 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 02:55:32.030520 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.025649    4461 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 02:55:32.030520 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.025653    4461 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 02:55:32.030520 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.025656    4461 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 02:55:32.030520 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.025659    4461 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 02:55:32.030520 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.025662    4461 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 02:55:32.030972 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.025665    4461 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 02:55:32.030972 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.025669    4461 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 02:55:32.030972 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.025673    4461 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 02:55:32.030972 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.025676    4461 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 02:55:32.030972 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.025679    4461 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 02:55:32.030972 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.025683    4461 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 02:55:32.030972 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.025686    4461 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 02:55:32.030972 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.025688    4461 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 02:55:32.030972 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.025691    4461 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 02:55:32.030972 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.025693    4461 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 02:55:32.030972 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.025696    4461 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 02:55:32.030972 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.025698    4461 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 02:55:32.030972 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.025701    4461 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 02:55:32.030972 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.025704    4461 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 02:55:32.030972 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.025706    4461 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 02:55:32.030972 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.025709    4461 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 02:55:32.030972 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.025711    4461 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 02:55:32.030972 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.025714    4461 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 02:55:32.030972 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.025717    4461 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 02:55:32.031441 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.025720    4461 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 02:55:32.031441 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.025723    4461 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 02:55:32.031441 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.025725    4461 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 02:55:32.031441 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.025728    4461 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 02:55:32.031441 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.025730    4461 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 02:55:32.031441 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.025733    4461 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 02:55:32.031441 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.025736    4461 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 02:55:32.031441 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.025738    4461 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 02:55:32.031441 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.025741    4461 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 02:55:32.031441 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.025744    4461 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 02:55:32.031441 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.025746    4461 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 02:55:32.031441 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.025749    4461 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 02:55:32.031441 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.025751    4461 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 02:55:32.031441 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.025754    4461 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 02:55:32.031441 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.025761    4461 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 02:55:32.031441 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.025764    4461 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 02:55:32.031441 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.025766    4461 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 02:55:32.031441 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.025769    4461 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 02:55:32.031441 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.025773    4461 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 02:55:32.031441 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.025776    4461 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 02:55:32.031931 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.025778    4461 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 02:55:32.031931 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.025781    4461 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 02:55:32.031931 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.025784    4461 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 02:55:32.031931 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.025786    4461 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 02:55:32.031931 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.025789    4461 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 02:55:32.031931 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.025792    4461 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 02:55:32.031931 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.025794    4461 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 02:55:32.031931 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.025797    4461 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 02:55:32.031931 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.025800    4461 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 02:55:32.031931 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.025802    4461 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 02:55:32.031931 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.025805    4461 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 02:55:32.031931 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.025807    4461 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 02:55:32.031931 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.025810    4461 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 02:55:32.031931 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.025813    4461 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 02:55:32.031931 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.025815    4461 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 02:55:32.031931 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.025818    4461 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 02:55:32.031931 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.025821    4461 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 02:55:32.031931 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.025824    4461 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 02:55:32.031931 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.025827    4461 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 02:55:32.031931 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.025830    4461 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 02:55:32.032462 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:32.025940    4461 flags.go:64] FLAG: --address="0.0.0.0"
Oct 30 02:55:32.032462 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:32.025953    4461 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Oct 30 02:55:32.032462 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:32.025962    4461 flags.go:64] FLAG: --anonymous-auth="true"
Oct 30 02:55:32.032462 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:32.025967    4461 flags.go:64] FLAG: --application-metrics-count-limit="100"
Oct 30 02:55:32.032462 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:32.025972    4461 flags.go:64] FLAG: --authentication-token-webhook="false"
Oct 30 02:55:32.032462 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:32.025975    4461 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Oct 30 02:55:32.032462 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:32.025982    4461 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Oct 30 02:55:32.032462 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:32.025987    4461 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Oct 30 02:55:32.032462 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:32.025990    4461 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Oct 30 02:55:32.032462 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:32.025994    4461 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Oct 30 02:55:32.032462 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:32.026001    4461 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Oct 30 02:55:32.032462 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:32.026006    4461 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Oct 30 02:55:32.032462 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:32.026011    4461 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Oct 30 02:55:32.032462 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:32.026016    4461 flags.go:64] FLAG: --cgroup-root=""
Oct 30 02:55:32.032462 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:32.026019    4461 flags.go:64] FLAG: --cgroups-per-qos="true"
Oct 30 02:55:32.032462 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:32.026022    4461 flags.go:64] FLAG: --client-ca-file=""
Oct 30 02:55:32.032462 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:32.026025    4461 flags.go:64] FLAG: --cloud-provider="external"
Oct 30 02:55:32.032462 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:32.026028    4461 flags.go:64] FLAG: --cluster-dns="[]"
Oct 30 02:55:32.032462 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:32.026035    4461 flags.go:64] FLAG: --cluster-domain=""
Oct 30 02:55:32.032462 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:32.026039    4461 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Oct 30 02:55:32.032462 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:32.026044    4461 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Oct 30 02:55:32.032462 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:32.026049    4461 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Oct 30 02:55:32.032462 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:32.026053    4461 flags.go:64] FLAG: --container-log-max-files="5"
Oct 30 02:55:32.032462 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:32.026057    4461 flags.go:64] FLAG: --container-log-max-size="10Mi"
Oct 30 02:55:32.033284 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:32.026060    4461 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Oct 30 02:55:32.033284 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:32.026063    4461 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Oct 30 02:55:32.033284 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:32.026068    4461 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Oct 30 02:55:32.033284 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:32.026073    4461 flags.go:64] FLAG: --contention-profiling="false"
Oct 30 02:55:32.033284 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:32.026078    4461 flags.go:64] FLAG: --cpu-cfs-quota="true"
Oct 30 02:55:32.033284 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:32.026083    4461 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Oct 30 02:55:32.033284 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:32.026088    4461 flags.go:64] FLAG: --cpu-manager-policy="none"
Oct 30 02:55:32.033284 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:32.026106    4461 flags.go:64] FLAG: --cpu-manager-policy-options=""
Oct 30 02:55:32.033284 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:32.026114    4461 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Oct 30 02:55:32.033284 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:32.026119    4461 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Oct 30 02:55:32.033284 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:32.026124    4461 flags.go:64] FLAG: --enable-debugging-handlers="true"
Oct 30 02:55:32.033284 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:32.026129    4461 flags.go:64] FLAG: --enable-load-reader="false"
Oct 30 02:55:32.033284 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:32.026134    4461 flags.go:64] FLAG: --enable-server="true"
Oct 30 02:55:32.033284 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:32.026139    4461 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Oct 30 02:55:32.033284 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:32.026146    4461 flags.go:64] FLAG: --event-burst="100"
Oct 30 02:55:32.033284 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:32.026149    4461 flags.go:64] FLAG: --event-qps="50"
Oct 30 02:55:32.033284 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:32.026152    4461 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Oct 30 02:55:32.033284 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:32.026156    4461 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Oct 30 02:55:32.033284 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:32.026160    4461 flags.go:64] FLAG: --eviction-hard=""
Oct 30 02:55:32.033284 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:32.026164    4461 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Oct 30 02:55:32.033284 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:32.026168    4461 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Oct 30 02:55:32.033284 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:32.026171    4461 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Oct 30 02:55:32.033284 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:32.026174    4461 flags.go:64] FLAG: --eviction-soft=""
Oct 30 02:55:32.033284 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:32.026177    4461 flags.go:64] FLAG: --eviction-soft-grace-period=""
Oct 30 02:55:32.033284 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:32.026180    4461 flags.go:64] FLAG: --exit-on-lock-contention="false"
Oct 30 02:55:32.034500 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:32.026183    4461 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Oct 30 02:55:32.034500 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:32.026186    4461 flags.go:64] FLAG: --experimental-mounter-path=""
Oct 30 02:55:32.034500 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:32.026189    4461 flags.go:64] FLAG: --fail-cgroupv1="true"
Oct 30 02:55:32.034500 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:32.026192    4461 flags.go:64] FLAG: --fail-swap-on="true"
Oct 30 02:55:32.034500 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:32.026195    4461 flags.go:64] FLAG: --feature-gates=""
Oct 30 02:55:32.034500 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:32.026200    4461 flags.go:64] FLAG: --file-check-frequency="20s"
Oct 30 02:55:32.034500 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:32.026203    4461 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Oct 30 02:55:32.034500 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:32.026206    4461 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Oct 30 02:55:32.034500 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:32.026209    4461 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Oct 30 02:55:32.034500 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:32.026212    4461 flags.go:64] FLAG: --healthz-port="10248"
Oct 30 02:55:32.034500 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:32.026216    4461 flags.go:64] FLAG: --help="false"
Oct 30 02:55:32.034500 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:32.026219    4461 flags.go:64] FLAG: --hostname-override="ip-10-0-143-156.ec2.internal"
Oct 30 02:55:32.034500 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:32.026224    4461 flags.go:64] FLAG: --housekeeping-interval="10s"
Oct 30 02:55:32.034500 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:32.026229    4461 flags.go:64] FLAG: --http-check-frequency="20s"
Oct 30 02:55:32.034500 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:32.026233    4461 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Oct 30 02:55:32.034500 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:32.026239    4461 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Oct 30 02:55:32.034500 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:32.026246    4461 flags.go:64] FLAG: --image-gc-high-threshold="85"
Oct 30 02:55:32.034500 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:32.026251    4461 flags.go:64] FLAG: --image-gc-low-threshold="80"
Oct 30 02:55:32.034500 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:32.026256    4461 flags.go:64] FLAG: --image-service-endpoint=""
Oct 30 02:55:32.034500 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:32.026260    4461 flags.go:64] FLAG: --kernel-memcg-notification="false"
Oct 30 02:55:32.034500 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:32.026265    4461 flags.go:64] FLAG: --kube-api-burst="100"
Oct 30 02:55:32.034500 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:32.026270    4461 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Oct 30 02:55:32.034500 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:32.026275    4461 flags.go:64] FLAG: --kube-api-qps="50"
Oct 30 02:55:32.034500 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:32.026279    4461 flags.go:64] FLAG: --kube-reserved=""
Oct 30 02:55:32.035196 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:32.026285    4461 flags.go:64] FLAG: --kube-reserved-cgroup=""
Oct 30 02:55:32.035196 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:32.026291    4461 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Oct 30 02:55:32.035196 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:32.026296    4461 flags.go:64] FLAG: --kubelet-cgroups=""
Oct 30 02:55:32.035196 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:32.026300    4461 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Oct 30 02:55:32.035196 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:32.026305    4461 flags.go:64] FLAG: --lock-file=""
Oct 30 02:55:32.035196 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:32.026310    4461 flags.go:64] FLAG: --log-cadvisor-usage="false"
Oct 30 02:55:32.035196 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:32.026314    4461 flags.go:64] FLAG: --log-flush-frequency="5s"
Oct 30 02:55:32.035196 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:32.026319    4461 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Oct 30 02:55:32.035196 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:32.026328    4461 flags.go:64] FLAG: --log-json-split-stream="false"
Oct 30 02:55:32.035196 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:32.026333    4461 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Oct 30 02:55:32.035196 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:32.026337    4461 flags.go:64] FLAG: --log-text-split-stream="false"
Oct 30 02:55:32.035196 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:32.026343    4461 flags.go:64] FLAG: --logging-format="text"
Oct 30 02:55:32.035196 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:32.026349    4461 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Oct 30 02:55:32.035196 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:32.026356    4461 flags.go:64] FLAG: --make-iptables-util-chains="true"
Oct 30 02:55:32.035196 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:32.026364    4461 flags.go:64] FLAG: --manifest-url=""
Oct 30 02:55:32.035196 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:32.026371    4461 flags.go:64] FLAG: --manifest-url-header=""
Oct 30 02:55:32.035196 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:32.026382    4461 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Oct 30 02:55:32.035196 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:32.026390    4461 flags.go:64] FLAG: --max-open-files="1000000"
Oct 30 02:55:32.035196 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:32.026400    4461 flags.go:64] FLAG: --max-pods="110"
Oct 30 02:55:32.035196 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:32.026408    4461 flags.go:64] FLAG: --maximum-dead-containers="-1"
Oct 30 02:55:32.035196 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:32.026416    4461 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Oct 30 02:55:32.035196 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:32.026424    4461 flags.go:64] FLAG: --memory-manager-policy="None"
Oct 30 02:55:32.035196 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:32.026432    4461 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Oct 30 02:55:32.035196 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:32.026440    4461 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Oct 30 02:55:32.035196 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:32.026449    4461 flags.go:64] FLAG: --node-ip="0.0.0.0"
Oct 30 02:55:32.035868 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:32.026459    4461 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Oct 30 02:55:32.035868 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:32.026474    4461 flags.go:64] FLAG: --node-status-max-images="50"
Oct 30 02:55:32.035868 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:32.026482    4461 flags.go:64] FLAG: --node-status-update-frequency="10s"
Oct 30 02:55:32.035868 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:32.026490    4461 flags.go:64] FLAG: --oom-score-adj="-999"
Oct 30 02:55:32.035868 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:32.026502    4461 flags.go:64] FLAG: --pod-cidr=""
Oct 30 02:55:32.035868 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:32.026509    4461 flags.go:64] FLAG: --pod-manifest-path=""
Oct 30 02:55:32.035868 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:32.026517    4461 flags.go:64] FLAG: --pod-max-pids="-1"
Oct 30 02:55:32.035868 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:32.026524    4461 flags.go:64] FLAG: --pods-per-core="0"
Oct 30 02:55:32.035868 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:32.026537    4461 flags.go:64] FLAG: --port="10250"
Oct 30 02:55:32.035868 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:32.026545    4461 flags.go:64] FLAG: --protect-kernel-defaults="false"
Oct 30 02:55:32.035868 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:32.026553    4461 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0ab09b2952fca25ca"
Oct 30 02:55:32.035868 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:32.026562    4461 flags.go:64] FLAG: --qos-reserved=""
Oct 30 02:55:32.035868 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:32.026571    4461 flags.go:64] FLAG: --read-only-port="10255"
Oct 30 02:55:32.035868 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:32.026579    4461 flags.go:64] FLAG: --register-node="true"
Oct 30 02:55:32.035868 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:32.026587    4461 flags.go:64] FLAG: --register-with-taints=""
Oct 30 02:55:32.035868 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:32.026596    4461 flags.go:64] FLAG: --registry-burst="10"
Oct 30 02:55:32.035868 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:32.026605    4461 flags.go:64] FLAG: --registry-qps="5"
Oct 30 02:55:32.035868 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:32.026612    4461 flags.go:64] FLAG: --reserved-cpus=""
Oct 30 02:55:32.035868 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:32.026619    4461 flags.go:64] FLAG: --reserved-memory=""
Oct 30 02:55:32.035868 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:32.026628    4461 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Oct 30 02:55:32.035868 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:32.026637    4461 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Oct 30 02:55:32.035868 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:32.026645    4461 flags.go:64] FLAG: --rotate-certificates="false"
Oct 30 02:55:32.035868 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:32.026652    4461 flags.go:64] FLAG: --rotate-server-certificates="false"
Oct 30 02:55:32.035868 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:32.026659    4461 flags.go:64] FLAG: --runonce="false"
Oct 30 02:55:32.035868 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:32.026668    4461 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Oct 30 02:55:32.035868 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:32.026676    4461 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Oct 30 02:55:32.036619 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:32.026684    4461 flags.go:64] FLAG: --seccomp-default="false"
Oct 30 02:55:32.036619 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:32.026692    4461 flags.go:64] FLAG: --serialize-image-pulls="true"
Oct 30 02:55:32.036619 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:32.026700    4461 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Oct 30 02:55:32.036619 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:32.026707    4461 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Oct 30 02:55:32.036619 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:32.026714    4461 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Oct 30 02:55:32.036619 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:32.026722    4461 flags.go:64] FLAG: --storage-driver-password="root"
Oct 30 02:55:32.036619 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:32.026730    4461 flags.go:64] FLAG: --storage-driver-secure="false"
Oct 30 02:55:32.036619 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:32.026738    4461 flags.go:64] FLAG: --storage-driver-table="stats"
Oct 30 02:55:32.036619 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:32.026747    4461 flags.go:64] FLAG: --storage-driver-user="root"
Oct 30 02:55:32.036619 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:32.026754    4461 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Oct 30 02:55:32.036619 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:32.026765    4461 flags.go:64] FLAG: --sync-frequency="1m0s"
Oct 30 02:55:32.036619 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:32.026773    4461 flags.go:64] FLAG: --system-cgroups=""
Oct 30 02:55:32.036619 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:32.026781    4461 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Oct 30 02:55:32.036619 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:32.026790    4461 flags.go:64] FLAG: --system-reserved-cgroup=""
Oct 30 02:55:32.036619 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:32.026796    4461 flags.go:64] FLAG: --tls-cert-file=""
Oct 30 02:55:32.036619 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:32.026800    4461 flags.go:64] FLAG: --tls-cipher-suites="[]"
Oct 30 02:55:32.036619 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:32.026808    4461 flags.go:64] FLAG: --tls-min-version=""
Oct 30 02:55:32.036619 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:32.026812    4461 flags.go:64] FLAG: --tls-private-key-file=""
Oct 30 02:55:32.036619 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:32.026817    4461 flags.go:64] FLAG: --topology-manager-policy="none"
Oct 30 02:55:32.036619 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:32.026822    4461 flags.go:64] FLAG: --topology-manager-policy-options=""
Oct 30 02:55:32.036619 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:32.026826    4461 flags.go:64] FLAG: --topology-manager-scope="container"
Oct 30 02:55:32.036619 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:32.026831    4461 flags.go:64] FLAG: --v="2"
Oct 30 02:55:32.036619 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:32.026837    4461 flags.go:64] FLAG: --version="false"
Oct 30 02:55:32.036619 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:32.026841    4461 flags.go:64] FLAG: --vmodule=""
Oct 30 02:55:32.036619 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:32.026845    4461 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Oct 30 02:55:32.037260 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:32.026849    4461 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Oct 30 02:55:32.037260 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.026996    4461 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 02:55:32.037260 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.027000    4461 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 02:55:32.037260 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.027003    4461 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 02:55:32.037260 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.027007    4461 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 02:55:32.037260 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.027010    4461 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 02:55:32.037260 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.027012    4461 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 02:55:32.037260 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.027015    4461 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 02:55:32.037260 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.027017    4461 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 02:55:32.037260 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.027022    4461 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 02:55:32.037260 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.027025    4461 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 02:55:32.037260 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.027030    4461 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 02:55:32.037260 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.027034    4461 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 02:55:32.037260 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.027038    4461 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 02:55:32.037260 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.027042    4461 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 02:55:32.037260 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.027046    4461 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 02:55:32.037260 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.027050    4461 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 02:55:32.037260 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.027057    4461 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 02:55:32.037260 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.027061    4461 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 02:55:32.037260 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.027065    4461 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 02:55:32.037260 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.027069    4461 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 02:55:32.037964 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.027073    4461 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 02:55:32.037964 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.027077    4461 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 02:55:32.037964 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.027081    4461 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 02:55:32.037964 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.027085    4461 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 02:55:32.037964 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.027089    4461 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 02:55:32.037964 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.027106    4461 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 02:55:32.037964 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.027111    4461 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 02:55:32.037964 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.027116    4461 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 02:55:32.037964 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.027121    4461 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 02:55:32.037964 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.027124    4461 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 02:55:32.037964 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.027127    4461 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 02:55:32.037964 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.027129    4461 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 02:55:32.037964 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.027134    4461 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 02:55:32.037964 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.027138    4461 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 02:55:32.037964 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.027142    4461 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 02:55:32.037964 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.027146    4461 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 02:55:32.037964 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.027150    4461 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 02:55:32.037964 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.027154    4461 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 02:55:32.037964 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.027158    4461 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 02:55:32.037964 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.027162    4461 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 02:55:32.038755 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.027166    4461 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 02:55:32.038755 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.027170    4461 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 02:55:32.038755 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.027174    4461 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 02:55:32.038755 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.027178    4461 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 02:55:32.038755 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.027182    4461 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 02:55:32.038755 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.027187    4461 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 02:55:32.038755 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.027191    4461 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 02:55:32.038755 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.027196    4461 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 02:55:32.038755 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.027203    4461 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 02:55:32.038755 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.027207    4461 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 02:55:32.038755 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.027212    4461 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 02:55:32.038755 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.027216    4461 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 02:55:32.038755 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.027220    4461 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 02:55:32.038755 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.027224    4461 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 02:55:32.038755 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.027229    4461 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 02:55:32.038755 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.027233    4461 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 02:55:32.038755 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.027238    4461 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 02:55:32.038755 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.027242    4461 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 02:55:32.038755 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.027246    4461 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 02:55:32.039531 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.027251    4461 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 02:55:32.039531 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.027256    4461 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 02:55:32.039531 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.027260    4461 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 02:55:32.039531 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.027265    4461 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 02:55:32.039531 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.027269    4461 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 02:55:32.039531 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.027273    4461 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 02:55:32.039531 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.027277    4461 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 02:55:32.039531 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.027281    4461 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 02:55:32.039531 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.027286    4461 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 02:55:32.039531 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.027290    4461 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 02:55:32.039531 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.027295    4461 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 02:55:32.039531 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.027300    4461 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 02:55:32.039531 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.027304    4461 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 02:55:32.039531 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.027308    4461 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 02:55:32.039531 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.027312    4461 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 02:55:32.039531 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.027317    4461 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 02:55:32.039531 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.027320    4461 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 02:55:32.039531 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.027326    4461 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 02:55:32.039531 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.027330    4461 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 02:55:32.039531 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.027333    4461 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 02:55:32.040207 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.027336    4461 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 02:55:32.040207 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.027341    4461 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 02:55:32.040207 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.027344    4461 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 02:55:32.040207 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.027346    4461 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 02:55:32.040207 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.027349    4461 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 02:55:32.040207 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.027352    4461 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 02:55:32.040207 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.027355    4461 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 02:55:32.040207 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.027357    4461 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 02:55:32.040207 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.027361    4461 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 02:55:32.040207 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.027366    4461 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 02:55:32.040207 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.027369    4461 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 02:55:32.040207 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.027372    4461 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 02:55:32.040207 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.027375    4461 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 02:55:32.040207 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.027378    4461 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 02:55:32.040207 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.027380    4461 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 02:55:32.040207 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.027383    4461 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 02:55:32.040207 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.027385    4461 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 02:55:32.040207 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.027388    4461 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 02:55:32.040207 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.027393    4461 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 02:55:32.041224 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.027397    4461 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 02:55:32.041224 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.027402    4461 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 02:55:32.041224 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.027406    4461 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 02:55:32.041224 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.027410    4461 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 02:55:32.041224 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.027415    4461 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 02:55:32.041224 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.027419    4461 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 02:55:32.041224 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.027422    4461 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 02:55:32.041224 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.027425    4461 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 02:55:32.041224 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:32.027460    4461 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 02:55:32.041224 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.027497    4461 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 02:55:32.041224 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.027502    4461 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 02:55:32.041224 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.027505    4461 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 02:55:32.041224 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.027508    4461 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 02:55:32.041224 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.027511    4461 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 02:55:32.041224 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.027523    4461 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 02:55:32.041224 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.027527    4461 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 02:55:32.041870 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.027530    4461 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 02:55:32.041870 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.027532    4461 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 02:55:32.041870 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.027535    4461 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 02:55:32.041870 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.027538    4461 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 02:55:32.041870 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.027540    4461 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 02:55:32.041870 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.027543    4461 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 02:55:32.041870 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.027546    4461 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 02:55:32.041870 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.027548    4461 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 02:55:32.041870 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.027551    4461 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 02:55:32.041870 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.027554    4461 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 02:55:32.041870 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.027557    4461 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 02:55:32.041870 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.027559    4461 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 02:55:32.041870 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.027562    4461 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 02:55:32.041870 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.027565    4461 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 02:55:32.041870 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.027568    4461 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 02:55:32.041870 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.027572    4461 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 02:55:32.041870 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.027579    4461 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 02:55:32.041870 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.027583    4461 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 02:55:32.041870 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.027588    4461 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 02:55:32.042650 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.027590    4461 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 02:55:32.042650 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.027593    4461 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 02:55:32.042650 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.027596    4461 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 02:55:32.042650 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.027599    4461 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 02:55:32.042650 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.027602    4461 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 02:55:32.042650 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.027604    4461 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 02:55:32.042650 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.027607    4461 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 02:55:32.042650 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.027610    4461 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 02:55:32.042650 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.027612    4461 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 02:55:32.042650 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.027615    4461 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 02:55:32.042650 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.027617    4461 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 02:55:32.042650 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.027621    4461 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 02:55:32.042650 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.027624    4461 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 02:55:32.042650 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.027628    4461 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 02:55:32.042650 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.027632    4461 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 02:55:32.042650 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.027636    4461 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 02:55:32.042650 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.027640    4461 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 02:55:32.042650 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.027644    4461 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 02:55:32.042650 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.027648    4461 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 02:55:32.043404 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.027652    4461 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 02:55:32.043404 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.027656    4461 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 02:55:32.043404 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.027660    4461 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 02:55:32.043404 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.027664    4461 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 02:55:32.043404 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.027668    4461 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 02:55:32.043404 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.027672    4461 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 02:55:32.043404 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.027676    4461 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 02:55:32.043404 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.027680    4461 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 02:55:32.043404 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.027684    4461 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 02:55:32.043404 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.027689    4461 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 02:55:32.043404 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.027693    4461 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 02:55:32.043404 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.027696    4461 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 02:55:32.043404 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.027700    4461 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 02:55:32.043404 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.027704    4461 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 02:55:32.043404 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.027707    4461 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 02:55:32.043404 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.027710    4461 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 02:55:32.043404 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.027713    4461 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 02:55:32.043404 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.027716    4461 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 02:55:32.043404 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.027718    4461 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 02:55:32.043885 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.027721    4461 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 02:55:32.043885 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.027723    4461 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 02:55:32.043885 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.027726    4461 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 02:55:32.043885 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.027728    4461 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 02:55:32.043885 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.027731    4461 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 02:55:32.043885 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.027735    4461 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 02:55:32.043885 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.027738    4461 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 02:55:32.043885 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.027740    4461 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 02:55:32.043885 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.027750    4461 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 02:55:32.043885 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.027754    4461 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 02:55:32.043885 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.027757    4461 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 02:55:32.043885 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.027762    4461 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 02:55:32.043885 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.027767    4461 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 02:55:32.043885 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.027771    4461 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 02:55:32.043885 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.027775    4461 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 02:55:32.043885 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.027779    4461 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 02:55:32.043885 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.027781    4461 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 02:55:32.043885 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.027784    4461 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 02:55:32.043885 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.027787    4461 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 02:55:32.044445 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.027789    4461 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 02:55:32.044445 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.027792    4461 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 02:55:32.044445 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.027794    4461 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 02:55:32.044445 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.027797    4461 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 02:55:32.044445 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.027801    4461 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 02:55:32.044445 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.027803    4461 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 02:55:32.044445 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.027806    4461 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 02:55:32.044445 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.027809    4461 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 02:55:32.044445 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.027811    4461 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 02:55:32.044445 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.027814    4461 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 02:55:32.044445 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.027817    4461 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 02:55:32.044445 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.027820    4461 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 02:55:32.044445 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.027824    4461 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 02:55:32.044445 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.027828    4461 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 02:55:32.044445 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.027833    4461 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 02:55:32.044445 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.027837    4461 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 02:55:32.044445 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.027841    4461 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 02:55:32.044445 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.027845    4461 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 02:55:32.044445 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.027851    4461 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 02:55:32.044445 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.027857    4461 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 02:55:32.045169 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.027861    4461 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 02:55:32.045169 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.027865    4461 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 02:55:32.045169 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.027870    4461 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 02:55:32.045169 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:32.032059    4461 server.go:525] "Kubelet version" kubeletVersion="v1.35.9"
Oct 30 02:55:32.045169 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:32.032076    4461 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Oct 30 02:55:32.045169 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:32.032109    4461 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Oct 30 02:55:32.045169 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:32.032116    4461 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Oct 30 02:55:32.045169 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.032184    4461 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 02:55:32.045169 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.032190    4461 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 02:55:32.045169 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.032195    4461 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 02:55:32.045169 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.032199    4461 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 02:55:32.045169 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.032202    4461 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 02:55:32.045169 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.032205    4461 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 02:55:32.045169 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.032207    4461 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 02:55:32.045169 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.032210    4461 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 02:55:32.045169 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.032213    4461 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 02:55:32.045169 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.032216    4461 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 02:55:32.045169 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.032219    4461 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 02:55:32.045169 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.032222    4461 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 02:55:32.045963 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.032224    4461 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 02:55:32.045963 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.032227    4461 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 02:55:32.045963 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.032235    4461 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 02:55:32.045963 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.032239    4461 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 02:55:32.045963 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.032241    4461 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 02:55:32.045963 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.032244    4461 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 02:55:32.045963 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.032247    4461 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 02:55:32.045963 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.032249    4461 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 02:55:32.045963 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.032252    4461 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 02:55:32.045963 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.032254    4461 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 02:55:32.045963 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.032257    4461 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 02:55:32.045963 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.032260    4461 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 02:55:32.045963 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.032263    4461 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 02:55:32.045963 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.032266    4461 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 02:55:32.045963 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.032269    4461 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 02:55:32.045963 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.032272    4461 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 02:55:32.045963 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.032274    4461 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 02:55:32.045963 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.032277    4461 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 02:55:32.045963 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.032279    4461 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 02:55:32.046746 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.032282    4461 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 02:55:32.046746 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.032284    4461 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 02:55:32.046746 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.032287    4461 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 02:55:32.046746 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.032290    4461 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 02:55:32.046746 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.032292    4461 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 02:55:32.046746 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.032295    4461 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 02:55:32.046746 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.032298    4461 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 02:55:32.046746 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.032300    4461 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 02:55:32.046746 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.032303    4461 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 02:55:32.046746 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.032311    4461 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 02:55:32.046746 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.032314    4461 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 02:55:32.046746 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.032317    4461 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 02:55:32.046746 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.032320    4461 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 02:55:32.046746 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.032322    4461 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 02:55:32.046746 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.032325    4461 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 02:55:32.046746 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.032327    4461 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 02:55:32.046746 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.032338    4461 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 02:55:32.046746 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.032342    4461 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 02:55:32.046746 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.032346    4461 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 02:55:32.047563 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.032349    4461 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 02:55:32.047563 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.032352    4461 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 02:55:32.047563 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.032354    4461 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 02:55:32.047563 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.032357    4461 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 02:55:32.047563 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.032360    4461 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 02:55:32.047563 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.032362    4461 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 02:55:32.047563 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.032365    4461 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 02:55:32.047563 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.032368    4461 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 02:55:32.047563 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.032370    4461 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 02:55:32.047563 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.032373    4461 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 02:55:32.047563 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.032376    4461 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 02:55:32.047563 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.032378    4461 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 02:55:32.047563 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.032381    4461 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 02:55:32.047563 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.032384    4461 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 02:55:32.047563 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.032386    4461 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 02:55:32.047563 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.032389    4461 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 02:55:32.047563 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.032391    4461 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 02:55:32.047563 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.032394    4461 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 02:55:32.047563 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.032396    4461 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 02:55:32.047563 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.032399    4461 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 02:55:32.048065 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.032402    4461 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 02:55:32.048065 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.032404    4461 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 02:55:32.048065 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.032407    4461 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 02:55:32.048065 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.032409    4461 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 02:55:32.048065 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.032412    4461 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 02:55:32.048065 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.032414    4461 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 02:55:32.048065 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.032417    4461 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 02:55:32.048065 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.032419    4461 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 02:55:32.048065 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.032422    4461 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 02:55:32.048065 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.032425    4461 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 02:55:32.048065 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.032433    4461 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 02:55:32.048065 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.032435    4461 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 02:55:32.048065 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.032438    4461 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 02:55:32.048065 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.032441    4461 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 02:55:32.048065 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.032443    4461 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 02:55:32.048065 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.032446    4461 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 02:55:32.048065 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.032448    4461 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 02:55:32.048065 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.032451    4461 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 02:55:32.048065 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.032454    4461 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 02:55:32.048065 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.032457    4461 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 02:55:32.048624 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.032459    4461 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 02:55:32.048624 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.032462    4461 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 02:55:32.048624 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.032465    4461 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 02:55:32.048624 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.032467    4461 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 02:55:32.048624 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.032470    4461 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 02:55:32.048624 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.032473    4461 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 02:55:32.048624 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.032476    4461 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 02:55:32.048624 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.032478    4461 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 02:55:32.048624 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.032481    4461 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 02:55:32.048624 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.032484    4461 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 02:55:32.048624 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.032486    4461 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 02:55:32.048624 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.032489    4461 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 02:55:32.048624 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.032492    4461 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 02:55:32.048624 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.032494    4461 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 02:55:32.048624 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.032497    4461 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 02:55:32.048624 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.032499    4461 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 02:55:32.049022 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:32.032526    4461 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 02:55:32.049022 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.033442    4461 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 02:55:32.049022 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.033458    4461 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 02:55:32.049022 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.033464    4461 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 02:55:32.049022 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.033474    4461 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 02:55:32.049022 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.033478    4461 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 02:55:32.049022 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.033483    4461 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 02:55:32.049022 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.033512    4461 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 02:55:32.049022 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.033517    4461 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 02:55:32.049022 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.033522    4461 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 02:55:32.049022 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.033526    4461 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 02:55:32.049022 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.033531    4461 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 02:55:32.049022 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.033535    4461 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 02:55:32.049022 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.033539    4461 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 02:55:32.049022 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.033544    4461 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 02:55:32.049022 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.033548    4461 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 02:55:32.049442 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.033553    4461 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 02:55:32.049442 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.033563    4461 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 02:55:32.049442 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.033569    4461 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 02:55:32.049442 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.033575    4461 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 02:55:32.049442 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.033579    4461 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 02:55:32.049442 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.033584    4461 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 02:55:32.049442 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.033589    4461 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 02:55:32.049442 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.033593    4461 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 02:55:32.049442 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.033597    4461 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 02:55:32.049442 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.033602    4461 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 02:55:32.049442 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.033606    4461 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 02:55:32.049442 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.033611    4461 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 02:55:32.049442 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.033615    4461 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 02:55:32.049442 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.033625    4461 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 02:55:32.049442 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.033629    4461 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 02:55:32.049442 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.033633    4461 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 02:55:32.049442 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.033637    4461 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 02:55:32.049442 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.033641    4461 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 02:55:32.049442 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.033645    4461 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 02:55:32.049896 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.033650    4461 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 02:55:32.049896 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.033654    4461 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 02:55:32.049896 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.033658    4461 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 02:55:32.049896 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.033662    4461 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 02:55:32.049896 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.033666    4461 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 02:55:32.049896 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.033697    4461 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 02:55:32.049896 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.033701    4461 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 02:55:32.049896 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.033710    4461 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 02:55:32.049896 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.033714    4461 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 02:55:32.049896 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.033719    4461 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 02:55:32.049896 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.033724    4461 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 02:55:32.049896 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.033729    4461 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 02:55:32.049896 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.033733    4461 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 02:55:32.049896 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.033737    4461 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 02:55:32.049896 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.033741    4461 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 02:55:32.049896 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.033745    4461 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 02:55:32.049896 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.033749    4461 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 02:55:32.049896 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.033753    4461 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 02:55:32.049896 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.033757    4461 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 02:55:32.050369 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.033761    4461 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 02:55:32.050369 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.033771    4461 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 02:55:32.050369 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.033776    4461 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 02:55:32.050369 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.033780    4461 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 02:55:32.050369 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.033784    4461 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 02:55:32.050369 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.033788    4461 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 02:55:32.050369 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.033792    4461 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 02:55:32.050369 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.033797    4461 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 02:55:32.050369 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.033801    4461 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 02:55:32.050369 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.033806    4461 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 02:55:32.050369 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.033810    4461 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 02:55:32.050369 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.033814    4461 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 02:55:32.050369 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.033818    4461 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 02:55:32.050369 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.033824    4461 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 02:55:32.050369 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.033826    4461 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 02:55:32.050369 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.033830    4461 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 02:55:32.050369 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.033832    4461 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 02:55:32.050369 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.033835    4461 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 02:55:32.050369 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.033838    4461 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 02:55:32.050369 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.033887    4461 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 02:55:32.050843 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.033989    4461 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 02:55:32.050843 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.033998    4461 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 02:55:32.050843 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.034005    4461 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 02:55:32.050843 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.034010    4461 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 02:55:32.050843 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.034068    4461 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 02:55:32.050843 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.034075    4461 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 02:55:32.050843 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.034080    4461 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 02:55:32.050843 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.034085    4461 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 02:55:32.050843 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.034090    4461 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 02:55:32.050843 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.034113    4461 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 02:55:32.050843 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.034118    4461 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 02:55:32.050843 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.034122    4461 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 02:55:32.050843 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.034127    4461 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 02:55:32.050843 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.034132    4461 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 02:55:32.050843 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.034137    4461 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 02:55:32.050843 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.034142    4461 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 02:55:32.050843 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.034146    4461 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 02:55:32.050843 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.034155    4461 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 02:55:32.050843 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.034159    4461 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 02:55:32.050843 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.034163    4461 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 02:55:32.051369 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.034167    4461 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 02:55:32.051369 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.034171    4461 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 02:55:32.051369 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.034175    4461 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 02:55:32.051369 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.034179    4461 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 02:55:32.051369 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.034187    4461 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 02:55:32.051369 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.034194    4461 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 02:55:32.051369 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.034223    4461 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 02:55:32.051369 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.034370    4461 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 02:55:32.051369 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.034375    4461 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 02:55:32.051369 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.034378    4461 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 02:55:32.051369 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.034381    4461 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 02:55:32.051369 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.034385    4461 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 02:55:32.051369 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.034387    4461 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 02:55:32.051369 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:32.034425    4461 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 02:55:32.051369 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.034487    4461 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 02:55:32.051832 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.034494    4461 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 02:55:32.051832 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.034497    4461 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 02:55:32.051832 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.034500    4461 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 02:55:32.051832 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.034502    4461 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 02:55:32.051832 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.034505    4461 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 02:55:32.051832 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.034508    4461 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 02:55:32.051832 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.034513    4461 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 02:55:32.051832 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.034516    4461 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 02:55:32.051832 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.034518    4461 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 02:55:32.051832 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.034521    4461 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 02:55:32.051832 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.034524    4461 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 02:55:32.051832 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.034527    4461 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 02:55:32.051832 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.034530    4461 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 02:55:32.051832 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.034532    4461 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 02:55:32.051832 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.034535    4461 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 02:55:32.051832 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.034538    4461 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 02:55:32.051832 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.034541    4461 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 02:55:32.051832 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.034544    4461 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 02:55:32.051832 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.034547    4461 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 02:55:32.051832 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.034549    4461 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 02:55:32.052425 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.034552    4461 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 02:55:32.052425 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.034555    4461 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 02:55:32.052425 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.034557    4461 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 02:55:32.052425 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.034560    4461 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 02:55:32.052425 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.034564    4461 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 02:55:32.052425 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.034566    4461 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 02:55:32.052425 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.034569    4461 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 02:55:32.052425 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.034571    4461 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 02:55:32.052425 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.034574    4461 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 02:55:32.052425 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.034577    4461 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 02:55:32.052425 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.034580    4461 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 02:55:32.052425 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.034583    4461 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 02:55:32.052425 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.034586    4461 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 02:55:32.052425 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.034588    4461 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 02:55:32.052425 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.034591    4461 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 02:55:32.052425 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.034593    4461 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 02:55:32.052425 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.034596    4461 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 02:55:32.052425 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.034598    4461 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 02:55:32.052425 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.034601    4461 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 02:55:32.052425 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.034603    4461 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 02:55:32.053328 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.034606    4461 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 02:55:32.053328 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.034609    4461 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 02:55:32.053328 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.034612    4461 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 02:55:32.053328 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.034617    4461 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 02:55:32.053328 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.034620    4461 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 02:55:32.053328 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.034623    4461 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 02:55:32.053328 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.034626    4461 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 02:55:32.053328 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.034629    4461 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 02:55:32.053328 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.034631    4461 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 02:55:32.053328 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.034634    4461 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 02:55:32.053328 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.034637    4461 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 02:55:32.053328 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.034640    4461 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 02:55:32.053328 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.034644    4461 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 02:55:32.053328 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.034647    4461 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 02:55:32.053328 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.034650    4461 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 02:55:32.053328 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.034652    4461 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 02:55:32.053328 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.034655    4461 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 02:55:32.053328 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.034657    4461 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 02:55:32.053328 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.034660    4461 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 02:55:32.053829 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.034662    4461 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 02:55:32.053829 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.034665    4461 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 02:55:32.053829 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.034668    4461 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 02:55:32.053829 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.034671    4461 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 02:55:32.053829 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.034674    4461 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 02:55:32.053829 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.034676    4461 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 02:55:32.053829 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.034679    4461 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 02:55:32.053829 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.034682    4461 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 02:55:32.053829 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.034685    4461 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 02:55:32.053829 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.034689    4461 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 02:55:32.053829 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.034693    4461 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 02:55:32.053829 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.034696    4461 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 02:55:32.053829 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.034699    4461 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 02:55:32.053829 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.034701    4461 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 02:55:32.053829 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.034704    4461 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 02:55:32.053829 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.034707    4461 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 02:55:32.053829 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.034710    4461 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 02:55:32.053829 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.034712    4461 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 02:55:32.053829 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.034715    4461 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 02:55:32.054496 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.034717    4461 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 02:55:32.054496 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.034720    4461 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 02:55:32.054496 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.034723    4461 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 02:55:32.054496 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.034725    4461 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 02:55:32.054496 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.034728    4461 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 02:55:32.054496 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.034731    4461 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 02:55:32.054496 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.034733    4461 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 02:55:32.054496 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.034736    4461 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 02:55:32.054496 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.034739    4461 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 02:55:32.054496 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.034741    4461 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 02:55:32.054496 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.034744    4461 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 02:55:32.054496 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.034746    4461 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 02:55:32.054496 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.034748    4461 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 02:55:32.054496 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.034751    4461 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 02:55:32.054496 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.034754    4461 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 02:55:32.054496 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.034757    4461 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 02:55:32.054496 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.034759    4461 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 02:55:32.054496 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.034763    4461 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 02:55:32.054496 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.034765    4461 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 02:55:32.055243 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.034768    4461 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 02:55:32.055243 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.034770    4461 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 02:55:32.055243 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.034773    4461 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 02:55:32.055243 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.034775    4461 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 02:55:32.055243 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.034778    4461 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 02:55:32.055243 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.034780    4461 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 02:55:32.055243 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.034783    4461 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 02:55:32.055243 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:32.034785    4461 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 02:55:32.055243 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:32.034847    4461 server.go:957] "Client rotation is on, will bootstrap in background"
Oct 30 02:55:32.055243 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:32.036043    4461 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary"
Oct 30 02:55:32.055243 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:32.036121    4461 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem"
Oct 30 02:55:32.055243 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:32.036317    4461 server.go:1014] "Starting client certificate rotation"
Oct 30 02:55:32.055243 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:32.036467    4461 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Oct 30 02:55:32.055243 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:32.036537    4461 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-29 02:49:52 +0000 UTC" deadline="2028-08-10 12:59:56.53734617 +0000 UTC"
Oct 30 02:55:32.055243 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:32.036588    4461 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15610h4m24.50076318s"
Oct 30 02:55:32.055243 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:32.036923    4461 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 30 02:55:32.055943 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:32.040151    4461 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 30 02:55:32.055943 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:32.042504    4461 log.go:25] "Validated CRI v1 runtime API"
Oct 30 02:55:32.055943 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:32.044330    4461 log.go:25] "Validated CRI v1 image API"
Oct 30 02:55:32.055943 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:32.045161    4461 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Oct 30 02:55:32.055943 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:32.048728    4461 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 8bf05c95-fcd3-46e0-b571-1a0371d90777:/dev/nvme0n1p3 eab5acd2-acc1-4e27-a6b9-91c13694e430:/dev/nvme0n1p4]
Oct 30 02:55:32.056194 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:32.048749    4461 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/01245521c7a3ffefdfc2ae351f991af2ac42fc81f1927e05a8bdcb10a6c2e7b4/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/01245521c7a3ffefdfc2ae351f991af2ac42fc81f1927e05a8bdcb10a6c2e7b4/userdata/shm major:0 minor:72 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/1bae9e54a248d6ce344f3c1732f29332e4fdb208afd0e898287edc45c7c9db0d/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/1bae9e54a248d6ce344f3c1732f29332e4fdb208afd0e898287edc45c7c9db0d/userdata/shm major:0 minor:88 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/276ad63052e23d43dd97ff88a9374214f5c7a3dd8ee648681c9783f6eba26749/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/276ad63052e23d43dd97ff88a9374214f5c7a3dd8ee648681c9783f6eba26749/userdata/shm major:0 minor:224 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/2cc7ee155e2eb45b9d64e05c2e366ab7d8efbce84ad1c87a6a3026d836006ccf/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/2cc7ee155e2eb45b9d64e05c2e366ab7d8efbce84ad1c87a6a3026d836006ccf/userdata/shm major:0 minor:214 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/5a05ec0ff1f0a8c904755e4c7aed0f098d5700a36d8f8911725fb313e2cbe6ca/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/5a05ec0ff1f0a8c904755e4c7aed0f098d5700a36d8f8911725fb313e2cbe6ca/userdata/shm major:0 minor:95 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/5e965e654016cb1c63687a11574c47db32be4910c883fd9274864b7a4daefd2e/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/5e965e654016cb1c63687a11574c47db32be4910c883fd9274864b7a4daefd2e/userdata/shm major:0 minor:76 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/86332d2a55a3de71d4696a8cc0d46285defd9952fa3b30ae59d98e1868590524/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/86332d2a55a3de71d4696a8cc0d46285defd9952fa3b30ae59d98e1868590524/userdata/shm major:0 minor:230 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/8e25068db3f97685e5f4f9df47b334f09abff36314e4419220b21069b574efb3/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/8e25068db3f97685e5f4f9df47b334f09abff36314e4419220b21069b574efb3/userdata/shm major:0 minor:48 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/93e392332e3ee47fd777cfeb398f91bde31f4aebfc88e480418d207a6966bee4/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/93e392332e3ee47fd777cfeb398f91bde31f4aebfc88e480418d207a6966bee4/userdata/shm major:0 minor:68 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/983647bc364434314f920b45b26eed44790a22e8e35bf617694fe3d4cdbf5472/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/983647bc364434314f920b45b26eed44790a22e8e35bf617694fe3d4cdbf5472/userdata/shm major:0 minor:80 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/9ea919af44be23142fcc6b67dcda63896ad4a5ce6c7b8bb5cc040d04b6bcfd89/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/9ea919af44be23142fcc6b67dcda63896ad4a5ce6c7b8bb5cc040d04b6bcfd89/userdata/shm major:0 minor:81 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/c45974cae891b83781f22b2ed8a3e9c6dd117bb07816519904afa6a549a61b93/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/c45974cae891b83781f22b2ed8a3e9c6dd117bb07816519904afa6a549a61b93/userdata/shm major:0 minor:47 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/e31a65f04fc04593d238960000b7d26efdc16d169ec94e69691b59bfd0fab887/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/e31a65f04fc04593d238960000b7d26efdc16d169ec94e69691b59bfd0fab887/userdata/shm major:0 minor:94 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/e6dac9b7049e7ef1cd190b3786ce14164794eaa8d4d4277031c61a208cd1fea4/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/e6dac9b7049e7ef1cd190b3786ce14164794eaa8d4d4277031c61a208cd1fea4/userdata/shm major:0 minor:202 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/e7d7991e2afe977864c6c9c3e20434a2864d1dfeb3fde173954ba46cf3d906b7/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/e7d7991e2afe977864c6c9c3e20434a2864d1dfeb3fde173954ba46cf3d906b7/userdata/shm major:0 minor:216 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/042af239-396c-4dae-8cf2-0c562af07c62/volumes/kubernetes.io~projected/kube-api-access-267l7:{mountpoint:/var/lib/kubelet/pods/042af239-396c-4dae-8cf2-0c562af07c62/volumes/kubernetes.io~projected/kube-api-access-267l7 major:0 minor:66 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0e6f015d-0a2d-4579-ad4d-a88b8c11c3b9/volumes/kubernetes.io~projected/kube-api-access-p764l:{mountpoint:/var/lib/kubelet/pods/0e6f015d-0a2d-4579-ad4d-a88b8c11c3b9/volumes/kubernetes.io~projected/kube-api-access-p764l major:0 minor:59 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3bd2c646-cebb-4e8a-b6c3-8b5b74049c11/volumes/kubernetes.io~projected/kube-api-access-dkmxs:{mountpoint:/var/lib/kubelet/pods/3bd2c646-cebb-4e8a-b6c3-8b5b74049c11/volumes/kubernetes.io~projected/kube-api-access-dkmxs major:0 minor:65 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3e4730f0-d77e-4bff-b8af-b7fafd4525d8/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/3e4730f0-d77e-4bff-b8af-b7fafd4525d8/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:57 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3e4730f0-d77e-4bff-b8af-b7fafd4525d8/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/3e4730f0-d77e-4bff-b8af-b7fafd4525d8/volumes/kubernetes.io~empty-dir/tmp major:0 minor:56 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3e4730f0-d77e-4bff-b8af-b7fafd4525d8/volumes/kubernetes.io~projected/kube-api-access-xwjvv:{mountpoint:/var/lib/kubelet/pods/3e4730f0-d77e-4bff-b8af-b7fafd4525d8/volumes/kubernetes.io~projected/kube-api-access-xwjvv major:0 minor:62 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/487caf0f-5f2a-4e93-9c06-de9af02c470b/volumes/kubernetes.io~projected/kube-api-access-cxnjk:{mountpoint:/var/lib/kubelet/pods/487caf0f-5f2a-4e93-9c06-de9af02c470b/volumes/kubernetes.io~projected/kube-api-access-cxnjk major:0 minor:64 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4a776e60-80b6-4ce0-921c-1884973b4d0b/volumes/kubernetes.io~projected/kube-api-access-2289h:{mountpoint:/var/lib/kubelet/pods/4a776e60-80b6-4ce0-921c-1884973b4d0b/volumes/kubernetes.io~projected/kube-api-access-2289h major:0 minor:229 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4a776e60-80b6-4ce0-921c-1884973b4d0b/volumes/kubernetes.io~secret/signing-key:{mountpoint:/var/lib/kubelet/pods/4a776e60-80b6-4ce0-921c-1884973b4d0b/volumes/kubernetes.io~secret/signing-key major:0 minor:228 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4d8bf700-9e09-4741-954b-aba9ccd331c9/volumes/kubernetes.io~projected/kube-api-access-pfgdh:{mountpoint:/var/lib/kubelet/pods/4d8bf700-9e09-4741-954b-aba9ccd331c9/volumes/kubernetes.io~projected/kube-api-access-pfgdh major:0 minor:61 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5f9fb000-9531-41f1-b3cf-7e79ca1a24d7/volumes/kubernetes.io~projected/kube-api-access-gwcwf:{mountpoint:/var/lib/kubelet/pods/5f9fb000-9531-41f1-b3cf-7e79ca1a24d7/volumes/kubernetes.io~projected/kube-api-access-gwcwf major:0 minor:197 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/73752bb9-2a14-4582-a6d6-c1fd26524c3d/volumes/kubernetes.io~projected/kube-api-access-gsgwj:{mountpoint:/var/lib/kubelet/pods/73752bb9-2a14-4582-a6d6-c1fd26524c3d/volumes/kubernetes.io~projected/kube-api-access-gsgwj major:0 minor:201 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/73752bb9-2a14-4582-a6d6-c1fd26524c3d/volumes/kubernetes.io~secret/cert:{mountpoint:/var/lib/kubelet/pods/73752bb9-2a14-4582-a6d6-c1fd26524c3d/volumes/kubernetes.io~secret/cert major:0 minor:45 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/75d9e1ec-cd1b-4aba-b911-5a7465c653cf/volumes/kubernetes.io~projected/kube-api-access-62vqf:{mountpoint:/var/lib/kubelet/pods/75d9e1ec-cd1b-4aba-b911-5a7465c653cf/volumes/kubernetes.io~projected/kube-api-access-62vqf major:0 minor:213 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/91a29ddf-25ca-40a8-b920-6d894360a050/volumes/kubernetes.io~projected/kube-api-access-g55cz:{mountpoint:/var/lib/kubelet/pods/91a29ddf-25ca-40a8-b920-6d894360a050/volumes/kubernetes.io~projected/kube-api-access-g55cz major:0 minor:63 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/98f9efe3-a732-4ba4-8543-941633d22170/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/98f9efe3-a732-4ba4-8543-941633d22170/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:208 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b4ce92fa-1612-4e46-ad6e-23049b2e279b/volumes/kubernetes.io~projected/kube-api-access-wmgr2:{mountpoint:/var/lib/kubelet/pods/b4ce92fa-1612-4e46-ad6e-23049b2e279b/volumes/kubernetes.io~projected/kube-api-access-wmgr2 major:0 minor:67 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/eee29ef7-4ad4-4ff9-8cb0-9d2a5aaabba3/volumes/kubernetes.io~projected/kube-api-access-wd8s8:{mountpoint:/var/lib/kubelet/pods/eee29ef7-4ad4-4ff9-8cb0-9d2a5aaabba3/volumes/kubernetes.io~projected/kube-api-access-wd8s8 major:0 minor:222 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f425530a-8431-4c5b-8335-8cc1119745c1/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/f425530a-8431-4c5b-8335-8cc1119745c1/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f425530a-8431-4c5b-8335-8cc1119745c1/volumes/kubernetes.io~projected/kube-api-access-lrvbf:{mountpoint:/var/lib/kubelet/pods/f425530a-8431-4c5b-8335-8cc1119745c1/volumes/kubernetes.io~projected/kube-api-access-lrvbf major:0 minor:60 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f425530a-8431-4c5b-8335-8cc1119745c1/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/f425530a-8431-4c5b-8335-8cc1119745c1/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:55 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f593f4c2-de6c-42db-b451-d0ade01317f7/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/f593f4c2-de6c-42db-b451-d0ade01317f7/volumes/kubernetes.io~secret/agent-certs major:0 minor:58 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f990bb17-3b0f-4bca-8939-089f79d6edef/volumes/kubernetes.io~projected/kube-api-access-rkgpj:{mountpoint:/var/lib/kubelet/pods/f990bb17-3b0f-4bca-8939-089f79d6edef/volumes/kubernetes.io~projected/kube-api-access-rkgpj major:0 minor:223 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-100:{mountpoint:/var/lib/containers/storage/overlay/644f26a444fd3391a21a7184aa2d94a36dda697bca3af8bdb30bc2a2c6c7e159/merged major:0 minor:100 fsType:overlay blockSize:0} overlay_0-102:{mountpoint:/var/lib/containers/storage/overlay/1db375840b65437c81b36a54a0bddd0d88cafb7636ed93f8abe63b7a6e397721/merged major:0 minor:102 fsType:overlay blockSize:0} overlay_0-104:{mountpoint:/var/lib/containers/storage/overlay/ad286491116fb2ea68328f04cc62711c6da19d2af0ab7d5b928273e2c3686a9d/merged major:0 minor:104 fsType:overlay blockSize:0} overlay_0-106:{mountpoint:/var/lib/containers/storage/overlay/11168c1939145aa0cb2f0e01e0c82f997d8fa5a4c30be0a4dd53bb7039f0dbcf/merged major:0 minor:106 fsType:overlay blockSize:0} overlay_0-114:{mountpoint:/var/lib/containers/storage/overlay/7f3ddfb34ffde380fc8d36b0e86e7c67a55e0e75a3c582f7b78bd8678101ff4d/merged major:0 minor:114 fsType:overlay blockSize:0} overlay_0-119:{mountpoint:/var/lib/containers/storage/overlay/9bafd895d8b419c373d552868732184e0ba3bcb7722e2d8f5e2d7eb11123544f/merged major:0 minor:119 fsType:overlay blockSize:0} overlay_0-124:{mountpoint:/var/lib/containers/storage/overlay/f618ac73abd7d2a768ebb8254492083705296b491a3460937b4f7a275ef86a38/merged major:0 minor:124 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/959ca9f86fd51bc889e4e3b156277e2841f926bca53343a222b237c368ac3e95/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-132:{mountpoint:/var/lib/containers/storage/overlay/7cb1cda35a3619cb8446bc1fdbae6b42241f3c1fa584ca7d9ed5a79f218d48b6/merged major:0 minor:132 fsType:overlay blockSize:0} overlay_0-137:{mountpoint:/var/lib/containers/storage/overlay/e802cff5cd1336b688fd35df15d6fd98833634e50c1afb43ed4ca50f0b8a1f52/merged major:0 minor:137 fsType:overlay blockSize:0} overlay_0-139:{mountpoint:/var/lib/containers/storage/overlay/bd367df9a93bf7974fa236ce8ebb3809c137fcf926bc45b86611637db92942c8/merged major:0 minor:139 fsType:overlay blockSize:0} overlay_0-140:{mountpoint:/var/lib/containers/storage/overlay/7aed79a34acdde62cfc4667c662ea36174fb6706b67327bf0fc00a2193297405/merged major:0 minor:140 fsType:overlay blockSize:0} overlay_0-152:{mountpoint:/var/lib/containers/storage/overlay/9eb8e30d01a5e38046da40a4ddd29a9e9af8c7ec2f65ad9a803fa2c4934e5cf6/merged major:0 minor:152 fsType:overlay blockSize:0} overlay_0-157:{mountpoint:/var/lib/containers/storage/overlay/d93d3ef9b9fcbefaf8ee425ceca7b89cd4635703990b0bfeaae763c6275a1849/merged major:0 minor:157 fsType:overlay blockSize:0} overlay_0-162:{mountpoint:/var/lib/containers/storage/overlay/ba602806ea9cb2c54cac4ba6deb33cf399882f48a580c75d0b217b3db6223982/merged major:0 minor:162 fsType:overlay blockSize:0} overlay_0-167:{mountpoint:/var/lib/containers/storage/overlay/09edfc9afbd0383941fcfd8216e8f26fe22e12de57432de857347fb9c7b45525/merged major:0 minor:167 fsType:overlay blockSize:0} overlay_0-172:{mountpoint:/var/lib/containers/storage/overlay/3cc95d5c50b07fabb168785e13f0b22bad9c87426fa6ff17bd0e435e2d68e873/merged major:0 minor:172 fsType:overlay blockSize:0} overlay_0-177:{mountpoint:/var/lib/containers/storage/overlay/81183adfe700f48e98609f059ab4d406b2481c07d6d6ed5cfe5a1521aa8b855f/merged major:0 minor:177 fsType:overlay blockSize:0} overlay_0-182:{mountpoint:/var/lib/containers/storage/overlay/41c5b5366d9981d014127e4ed1408d1a461f538614bf26e93bc41dffe7de36dc/merged major:0 minor:182 fsType:overlay blockSize:0} overlay_0-187:{mountpoint:/var/lib/containers/storage/overlay/c79b369aade936ebae20b12afe6ffa061a9d12e1b352da40fdbb3385b19923dc/merged major:0 minor:187 fsType:overlay blockSize:0} overlay_0-192:{mountpoint:/var/lib/containers/storage/overlay/0d61af33195510ab2f7d8b986031ecdf4dd4233d3407afdbcaec0bc8d4fd6a2c/merged major:0 minor:192 fsType:overlay blockSize:0} overlay_0-204:{mountpoint:/var/lib/containers/storage/overlay/e43396357f839bbec600c0f77d10abddba3f19ed8d4723cabec1b2d94c71825c/merged major:0 minor:204 fsType:overlay blockSize:0} overlay_0-206:{mountpoint:/var/lib/containers/storage/overlay/aabbfd1ce158fe1f1039451aeea6089b9ed0eb3e22270c44579bf34697d5a31d/merged major:0 minor:206 fsType:overlay blockSize:0} overlay_0-218:{mountpoint:/var/lib/containers/storage/overlay/0ec463a8295eaf4e577caecf0dfb5a7aaaffccb1e54c96f5abc68396c83e5014/merged major:0 minor:218 fsType:overlay blockSize:0} overlay_0-220:{mountpoint:/var/lib/containers/storage/overlay/d1f243739c4bb1e54929f0112c421831e876195eb16c838e195fb8313a7fdf48/merged major:0 minor:220 fsType:overlay blockSize:0} overlay_0-226:{mountpoint:/var/lib/containers/storage/overlay/e1b4b5a2a8be186a896adade08ac19e39557531759ef11ef5f90f48c5d1f5112/merged major:0 minor:226 fsType:overlay blockSize:0} overlay_0-232:{mountpoint:/var/lib/containers/storage/overlay/4236dd029449ca06382346cfe96bbf6794a8538a753c6bc06e9791e80a11b13b/merged major:0 minor:232 fsType:overlay blockSize:0} overlay_0-234:{mountpoint:/var/lib/containers/storage/overlay/5eb5f45f50e06acef7dea714e127cb2d81dfbc9bea3003a2532d2b397913ab83/merged major:0 minor:234 fsType:overlay blockSize:0} overlay_0-236:{mountpoint:/var/lib/containers/storage/overlay/fbdeea0108b57fd57038d88306d4187e79355fec2b42115eac0a28c5837256a8/merged major:0 minor:236 fsType:overlay blockSize:0} overlay_0-242:{mountpoint:/var/lib/containers/storage/overlay/4cc5db945b66968ceb80632ac069687ea325272b622bdea356a8647e3e7a0cb1/merged major:0 minor:242 fsType:overlay blockSize:0} overlay_0-248:{mountpoint:/var/lib/containers/storage/overlay/2494a25134f32e54d3b08a7c36f7cf4382c3930b7118c9b121a5ddaa99e3f374/merged major:0 minor:248 fsType:overlay blockSize:0} overlay_0-51:{mountpoint:/var/lib/containers/storage/overlay/87b8407f267894a108e5e005af5b0b49b5dfd05a7839803428d40e02f1cb88ba/merged major:0 minor:51 fsType:overlay blockSize:0} overlay_0-53:{mountpoint:/var/lib/containers/storage/overlay/f71b22d69127635d97b77ca59cead9dd5b4b71cc1e39bad6082cf4abe45e9d69/merged major:0 minor:53 fsType:overlay blockSize:0} overlay_0-70:{mountpoint:/var/lib/containers/storage/overlay/a28e6a6c558e5e75917ccb3ac12e96b4a77b6201bdf77baca2d8e5333a047d3f/merged major:0 minor:70 fsType:overlay blockSize:0} overlay_0-74:{mountpoint:/var/lib/containers/storage/overlay/95c17d11f187b6397efbde04cb500c5eb33b92e0cb7c7bc0d6c0c7496b537cea/merged major:0 minor:74 fsType:overlay blockSize:0} overlay_0-78:{mountpoint:/var/lib/containers/storage/overlay/46ec699b6aa4dd4c1670d6f1fba81d462cec3cb6379e5dabc7b3ad23bfbf7f67/merged major:0 minor:78 fsType:overlay blockSize:0} overlay_0-84:{mountpoint:/var/lib/containers/storage/overlay/13dc8e36cc164ea82ab4a0a1a32ee699bf3b273c80a9cdf54abf2d76df7a9f31/merged major:0 minor:84 fsType:overlay blockSize:0} overlay_0-86:{mountpoint:/var/lib/containers/storage/overlay/52c97549a9499433d1b8b2fab4afe11c464fba20af476986b48c3b69b7625f8f/merged major:0 minor:86 fsType:overlay blockSize:0} overlay_0-90:{mountpoint:/var/lib/containers/storage/overlay/2958879501fb5cab8002757696369527c3741f1da8b52aec10134995f8810a9d/merged major:0 minor:90 fsType:overlay blockSize:0} overlay_0-92:{mountpoint:/var/lib/containers/storage/overlay/95f3ad9f90fab2a45fa01612d243608f76c5da3dacb31ffcfd38b720b094d7ef/merged major:0 minor:92 fsType:overlay blockSize:0} overlay_0-98:{mountpoint:/var/lib/containers/storage/overlay/b56dbd9ebcdf5236ba6d2061cfce218dbce313ca7c84f26c874715e16ed87a8b/merged major:0 minor:98 fsType:overlay blockSize:0}]
Oct 30 02:55:32.057111 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:32.056638    4461 manager.go:217] Machine: {Timestamp:2026-10-30 02:55:32.055947361 +0000 UTC m=+0.105752529 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3099573 MemoryCapacity:33164337152 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec24b99a3425169b54576a8e645a4a9b SystemUUID:ec24b99a-3425-169b-5457-6a8e645a4a9b BootID:05b7e5fc-b59f-440e-ac9b-b26ec188cc4d Filesystems:[{Device:overlay_0-182 DeviceMajor:0 DeviceMinor:182 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-236 DeviceMajor:0 DeviceMinor:236 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/5a05ec0ff1f0a8c904755e4c7aed0f098d5700a36d8f8911725fb313e2cbe6ca/userdata/shm DeviceMajor:0 DeviceMinor:95 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/3bd2c646-cebb-4e8a-b6c3-8b5b74049c11/volumes/kubernetes.io~projected/kube-api-access-dkmxs DeviceMajor:0 DeviceMinor:65 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/042af239-396c-4dae-8cf2-0c562af07c62/volumes/kubernetes.io~projected/kube-api-access-267l7 DeviceMajor:0 DeviceMinor:66 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/93e392332e3ee47fd777cfeb398f91bde31f4aebfc88e480418d207a6966bee4/userdata/shm DeviceMajor:0 DeviceMinor:68 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/8e25068db3f97685e5f4f9df47b334f09abff36314e4419220b21069b574efb3/userdata/shm DeviceMajor:0 DeviceMinor:48 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-102 DeviceMajor:0 DeviceMinor:102 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-177 DeviceMajor:0 DeviceMinor:177 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/98f9efe3-a732-4ba4-8543-941633d22170/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:208 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/2cc7ee155e2eb45b9d64e05c2e366ab7d8efbce84ad1c87a6a3026d836006ccf/userdata/shm DeviceMajor:0 DeviceMinor:214 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582168576 Type:vfs Inodes:1048576 HasInodes:true} {Device:overlay_0-104 DeviceMajor:0 DeviceMinor:104 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-132 DeviceMajor:0 DeviceMinor:132 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-232 DeviceMajor:0 DeviceMinor:232 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-162 DeviceMajor:0 DeviceMinor:162 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-74 DeviceMajor:0 DeviceMinor:74 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-114 DeviceMajor:0 DeviceMinor:114 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/75d9e1ec-cd1b-4aba-b911-5a7465c653cf/volumes/kubernetes.io~projected/kube-api-access-62vqf DeviceMajor:0 DeviceMinor:213 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-124 DeviceMajor:0 DeviceMinor:124 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-139 DeviceMajor:0 DeviceMinor:139 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-157 DeviceMajor:0 DeviceMinor:157 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/c45974cae891b83781f22b2ed8a3e9c6dd117bb07816519904afa6a549a61b93/userdata/shm DeviceMajor:0 DeviceMinor:47 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/f425530a-8431-4c5b-8335-8cc1119745c1/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:55 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/3e4730f0-d77e-4bff-b8af-b7fafd4525d8/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:57 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/4d8bf700-9e09-4741-954b-aba9ccd331c9/volumes/kubernetes.io~projected/kube-api-access-pfgdh DeviceMajor:0 DeviceMinor:61 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-206 DeviceMajor:0 DeviceMinor:206 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/276ad63052e23d43dd97ff88a9374214f5c7a3dd8ee648681c9783f6eba26749/userdata/shm DeviceMajor:0 DeviceMinor:224 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-192 DeviceMajor:0 DeviceMinor:192 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/e6dac9b7049e7ef1cd190b3786ce14164794eaa8d4d4277031c61a208cd1fea4/userdata/shm DeviceMajor:0 DeviceMinor:202 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-234 DeviceMajor:0 DeviceMinor:234 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-242 DeviceMajor:0 DeviceMinor:242 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:overlay_0-78 DeviceMajor:0 DeviceMinor:78 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-86 DeviceMajor:0 DeviceMinor:86 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/73752bb9-2a14-4582-a6d6-c1fd26524c3d/volumes/kubernetes.io~secret/cert DeviceMajor:0 DeviceMinor:45 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/73752bb9-2a14-4582-a6d6-c1fd26524c3d/volumes/kubernetes.io~projected/kube-api-access-gsgwj DeviceMajor:0 DeviceMinor:201 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582168576 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-51 DeviceMajor:0 DeviceMinor:51 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/9ea919af44be23142fcc6b67dcda63896ad4a5ce6c7b8bb5cc040d04b6bcfd89/userdata/shm DeviceMajor:0 DeviceMinor:81 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/f425530a-8431-4c5b-8335-8cc1119745c1/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:overlay_0-220 DeviceMajor:0 DeviceMinor:220 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-152 DeviceMajor:0 DeviceMinor:152 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/487caf0f-5f2a-4e93-9c06-de9af02c470b/volumes/kubernetes.io~projected/kube-api-access-cxnjk DeviceMajor:0 DeviceMinor:64 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-70 DeviceMajor:0 DeviceMinor:70 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/f593f4c2-de6c-42db-b451-d0ade01317f7/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:58 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/e31a65f04fc04593d238960000b7d26efdc16d169ec94e69691b59bfd0fab887/userdata/shm DeviceMajor:0 DeviceMinor:94 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-98 DeviceMajor:0 DeviceMinor:98 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-119 DeviceMajor:0 DeviceMinor:119 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:overlay_0-53 DeviceMajor:0 DeviceMinor:53 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/f425530a-8431-4c5b-8335-8cc1119745c1/volumes/kubernetes.io~projected/kube-api-access-lrvbf DeviceMajor:0 DeviceMinor:60 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/0e6f015d-0a2d-4579-ad4d-a88b8c11c3b9/volumes/kubernetes.io~projected/kube-api-access-p764l DeviceMajor:0 DeviceMinor:59 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/b4ce92fa-1612-4e46-ad6e-23049b2e279b/volumes/kubernetes.io~projected/kube-api-access-wmgr2 DeviceMajor:0 DeviceMinor:67 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-84 DeviceMajor:0 DeviceMinor:84 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/1bae9e54a248d6ce344f3c1732f29332e4fdb208afd0e898287edc45c7c9db0d/userdata/shm DeviceMajor:0 DeviceMinor:88 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/f990bb17-3b0f-4bca-8939-089f79d6edef/volumes/kubernetes.io~projected/kube-api-access-rkgpj DeviceMajor:0 DeviceMinor:223 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/var/lib/kubelet/pods/3e4730f0-d77e-4bff-b8af-b7fafd4525d8/volumes/kubernetes.io~projected/kube-api-access-xwjvv DeviceMajor:0 DeviceMinor:62 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/5e965e654016cb1c63687a11574c47db32be4910c883fd9274864b7a4daefd2e/userdata/shm DeviceMajor:0 DeviceMinor:76 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-100 DeviceMajor:0 DeviceMinor:100 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-137 DeviceMajor:0 DeviceMinor:137 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/5f9fb000-9531-41f1-b3cf-7e79ca1a24d7/volumes/kubernetes.io~projected/kube-api-access-gwcwf DeviceMajor:0 DeviceMinor:197 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/4a776e60-80b6-4ce0-921c-1884973b4d0b/volumes/kubernetes.io~projected/kube-api-access-2289h DeviceMajor:0 DeviceMinor:229 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-167 DeviceMajor:0 DeviceMinor:167 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-92 DeviceMajor:0 DeviceMinor:92 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-140 DeviceMajor:0 DeviceMinor:140 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-204 DeviceMajor:0 DeviceMinor:204 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-218 DeviceMajor:0 DeviceMinor:218 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-226 DeviceMajor:0 DeviceMinor:226 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/01245521c7a3ffefdfc2ae351f991af2ac42fc81f1927e05a8bdcb10a6c2e7b4/userdata/shm DeviceMajor:0 DeviceMinor:72 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/91a29ddf-25ca-40a8-b920-6d894360a050/volumes/kubernetes.io~projected/kube-api-access-g55cz DeviceMajor:0 DeviceMinor:63 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/983647bc364434314f920b45b26eed44790a22e8e35bf617694fe3d4cdbf5472/userdata/shm DeviceMajor:0 DeviceMinor:80 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-187 DeviceMajor:0 DeviceMinor:187 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/e7d7991e2afe977864c6c9c3e20434a2864d1dfeb3fde173954ba46cf3d906b7/userdata/shm DeviceMajor:0 DeviceMinor:216 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/eee29ef7-4ad4-4ff9-8cb0-9d2a5aaabba3/volumes/kubernetes.io~projected/kube-api-access-wd8s8 DeviceMajor:0 DeviceMinor:222 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/4a776e60-80b6-4ce0-921c-1884973b4d0b/volumes/kubernetes.io~secret/signing-key DeviceMajor:0 DeviceMinor:228 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/86332d2a55a3de71d4696a8cc0d46285defd9952fa3b30ae59d98e1868590524/userdata/shm DeviceMajor:0 DeviceMinor:230 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/3e4730f0-d77e-4bff-b8af-b7fafd4525d8/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:56 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-90 DeviceMajor:0 DeviceMinor:90 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-172 DeviceMajor:0 DeviceMinor:172 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-248 DeviceMajor:0 DeviceMinor:248 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-106 DeviceMajor:0 DeviceMinor:106 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:276ad63052e23d4 MacAddress:62:1d:db:83:ab:ce Speed:10000 Mtu:8901} {Name:2cc7ee155e2eb45 MacAddress:26:4c:b3:0b:8b:6f Speed:10000 Mtu:8901} {Name:86332d2a55a3de7 MacAddress:36:89:ac:94:bf:06 Speed:10000 Mtu:8901} {Name:br-ex MacAddress:02:ff:e5:84:0a:c9 Speed:0 Mtu:9001} {Name:br-int MacAddress:82:57:9b:d2:34:5d Speed:0 Mtu:8901} {Name:e6dac9b7049e7ef MacAddress:0a:42:7b:75:33:c8 Speed:10000 Mtu:8901} {Name:e7d7991e2afe977 MacAddress:2e:49:97:2c:e8:bc Speed:10000 Mtu:8901} {Name:ens5 MacAddress:02:ff:e5:84:0a:c9 Speed:0 Mtu:9001} {Name:genev_sys_6081 MacAddress:ce:d9:6d:18:3a:e6 Speed:0 Mtu:65000} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:85:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:ae:13:91:b2:51:44 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164337152 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Oct 30 02:55:32.057111 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:32.057110    4461 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Oct 30 02:55:32.057379 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:32.057178    4461 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Oct 30 02:55:32.057619 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:32.057592    4461 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Oct 30 02:55:32.057791 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:32.057621    4461 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-143-156.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Oct 30 02:55:32.057861 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:32.057795    4461 topology_manager.go:143] "Creating topology manager with none policy"
Oct 30 02:55:32.057861 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:32.057808    4461 container_manager_linux.go:311] "Creating device plugin manager"
Oct 30 02:55:32.057861 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:32.057827    4461 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 30 02:55:32.057861 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:32.057853    4461 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 30 02:55:32.058017 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:32.058004    4461 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager"
Oct 30 02:55:32.058514 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:32.058273    4461 state_mem.go:41] "Initialized" logger="CPUManager state memory"
Oct 30 02:55:32.058514 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:32.058400    4461 server.go:1235] "Using root directory" path="/var/lib/kubelet"
Oct 30 02:55:32.058514 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:32.058443    4461 kubelet.go:493] "Attempting to sync node with API server"
Oct 30 02:55:32.058514 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:32.058462    4461 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests"
Oct 30 02:55:32.058514 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:32.058473    4461 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Oct 30 02:55:32.058514 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:32.058484    4461 kubelet.go:405] "Adding apiserver pod source"
Oct 30 02:55:32.058514 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:32.058491    4461 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Oct 30 02:55:32.062423 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:32.060002    4461 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint"
Oct 30 02:55:32.062423 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:32.060025    4461 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 30 02:55:32.064959 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:32.064934    4461 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1"
Oct 30 02:55:32.067379 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:32.067363    4461 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint"
Oct 30 02:55:32.067379 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:32.067377    4461 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 30 02:55:32.067504 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:32.067492    4461 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem"
Oct 30 02:55:32.067799 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:32.067782    4461 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Oct 30 02:55:32.067878 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:32.067831    4461 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Oct 30 02:55:32.067993 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:32.067956    4461 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Oct 30 02:55:32.067993 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:32.067981    4461 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Oct 30 02:55:32.067993 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:32.067991    4461 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Oct 30 02:55:32.068204 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:32.068001    4461 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Oct 30 02:55:32.068204 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:32.068011    4461 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Oct 30 02:55:32.068204 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:32.068020    4461 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Oct 30 02:55:32.068204 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:32.068029    4461 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Oct 30 02:55:32.068204 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:32.068038    4461 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Oct 30 02:55:32.068204 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:32.068050    4461 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Oct 30 02:55:32.068204 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:32.068059    4461 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Oct 30 02:55:32.068204 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:32.068076    4461 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Oct 30 02:55:32.068204 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:32.068086    4461 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Oct 30 02:55:32.068204 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:32.068134    4461 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Oct 30 02:55:32.068204 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:32.068141    4461 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Oct 30 02:55:32.072325 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:32.072162    4461 server.go:1263] "Started kubelet"
Oct 30 02:55:32.072741 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:32.072576    4461 server.go:182] "Starting to listen" address="0.0.0.0" port=10250
Oct 30 02:55:32.072689 ip-10-0-143-156 systemd[1]: Started Kubernetes Kubelet.
Oct 30 02:55:32.075310 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:32.073204    4461 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Oct 30 02:55:32.075310 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:32.073279    4461 server_v1.go:49] "podresources" method="list" useActivePods=true
Oct 30 02:55:32.075310 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:32.073729    4461 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Oct 30 02:55:32.075310 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:32.074871    4461 server.go:317] "Adding debug handlers to kubelet server"
Oct 30 02:55:32.075842 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:32.075825    4461 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer"
Oct 30 02:55:32.076373 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:32.076253    4461 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Oct 30 02:55:32.076465 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:32.076456    4461 volume_manager.go:309] "The desired_state_of_world populator starts"
Oct 30 02:55:32.076519 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:32.076469    4461 volume_manager.go:311] "Starting Kubelet Volume Manager"
Oct 30 02:55:32.076568 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:32.076498    4461 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 02:49:54 +0000 UTC" deadline="2028-06-05 05:30:36.338963327 +0000 UTC"
Oct 30 02:55:32.076568 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:32.076560    4461 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Oct 30 02:55:32.078627 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:32.076725    4461 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14018h35m4.262429285s"
Oct 30 02:55:32.078627 ip-10-0-143-156 kubenswrapper[4461]: E1030 02:55:32.076780    4461 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-143-156.ec2.internal\" not found"
Oct 30 02:55:32.078762 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:32.078175    4461 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f425530a-8431-4c5b-8335-8cc1119745c1" volumeName="kubernetes.io/configmap/f425530a-8431-4c5b-8335-8cc1119745c1-ovnkube-config" seLinuxMountContext=""
Oct 30 02:55:32.078762 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:32.078717    4461 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f425530a-8431-4c5b-8335-8cc1119745c1" volumeName="kubernetes.io/projected/f425530a-8431-4c5b-8335-8cc1119745c1-kube-api-access-lrvbf" seLinuxMountContext=""
Oct 30 02:55:32.078762 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:32.078735    4461 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f425530a-8431-4c5b-8335-8cc1119745c1" volumeName="kubernetes.io/secret/f425530a-8431-4c5b-8335-8cc1119745c1-ovn-node-metrics-cert" seLinuxMountContext=""
Oct 30 02:55:32.078762 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:32.078747    4461 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f593f4c2-de6c-42db-b451-d0ade01317f7" volumeName="kubernetes.io/secret/f593f4c2-de6c-42db-b451-d0ade01317f7-agent-certs" seLinuxMountContext=""
Oct 30 02:55:32.078762 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:32.078762    4461 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3e4730f0-d77e-4bff-b8af-b7fafd4525d8" volumeName="kubernetes.io/empty-dir/3e4730f0-d77e-4bff-b8af-b7fafd4525d8-tmp" seLinuxMountContext=""
Oct 30 02:55:32.078762 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:32.078776    4461 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4a776e60-80b6-4ce0-921c-1884973b4d0b" volumeName="kubernetes.io/projected/4a776e60-80b6-4ce0-921c-1884973b4d0b-kube-api-access-2289h" seLinuxMountContext=""
Oct 30 02:55:32.078762 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:32.078788    4461 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5f9fb000-9531-41f1-b3cf-7e79ca1a24d7" volumeName="kubernetes.io/configmap/5f9fb000-9531-41f1-b3cf-7e79ca1a24d7-config-volume" seLinuxMountContext=""
Oct 30 02:55:32.084423 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:32.078802    4461 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="eee29ef7-4ad4-4ff9-8cb0-9d2a5aaabba3" volumeName="kubernetes.io/projected/eee29ef7-4ad4-4ff9-8cb0-9d2a5aaabba3-kube-api-access-wd8s8" seLinuxMountContext=""
Oct 30 02:55:32.084423 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:32.078817    4461 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0e6f015d-0a2d-4579-ad4d-a88b8c11c3b9" volumeName="kubernetes.io/empty-dir/0e6f015d-0a2d-4579-ad4d-a88b8c11c3b9-tmp-dir" seLinuxMountContext=""
Oct 30 02:55:32.084423 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:32.078829    4461 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4d8bf700-9e09-4741-954b-aba9ccd331c9" volumeName="kubernetes.io/projected/4d8bf700-9e09-4741-954b-aba9ccd331c9-kube-api-access-pfgdh" seLinuxMountContext=""
Oct 30 02:55:32.084423 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:32.078842    4461 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="042af239-396c-4dae-8cf2-0c562af07c62" volumeName="kubernetes.io/projected/042af239-396c-4dae-8cf2-0c562af07c62-kube-api-access-267l7" seLinuxMountContext=""
Oct 30 02:55:32.084423 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:32.078854    4461 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5f9fb000-9531-41f1-b3cf-7e79ca1a24d7" volumeName="kubernetes.io/empty-dir/5f9fb000-9531-41f1-b3cf-7e79ca1a24d7-tmp-dir" seLinuxMountContext=""
Oct 30 02:55:32.084423 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:32.078866    4461 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f425530a-8431-4c5b-8335-8cc1119745c1" volumeName="kubernetes.io/configmap/f425530a-8431-4c5b-8335-8cc1119745c1-env-overrides" seLinuxMountContext=""
Oct 30 02:55:32.084423 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:32.078877    4461 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f990bb17-3b0f-4bca-8939-089f79d6edef" volumeName="kubernetes.io/projected/f990bb17-3b0f-4bca-8939-089f79d6edef-kube-api-access-rkgpj" seLinuxMountContext=""
Oct 30 02:55:32.084423 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:32.078894    4461 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3bd2c646-cebb-4e8a-b6c3-8b5b74049c11" volumeName="kubernetes.io/projected/3bd2c646-cebb-4e8a-b6c3-8b5b74049c11-kube-api-access-dkmxs" seLinuxMountContext=""
Oct 30 02:55:32.084423 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:32.078907    4461 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="487caf0f-5f2a-4e93-9c06-de9af02c470b" volumeName="kubernetes.io/configmap/487caf0f-5f2a-4e93-9c06-de9af02c470b-serviceca" seLinuxMountContext=""
Oct 30 02:55:32.084423 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:32.078920    4461 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4d8bf700-9e09-4741-954b-aba9ccd331c9" volumeName="kubernetes.io/configmap/4d8bf700-9e09-4741-954b-aba9ccd331c9-whereabouts-flatfile-configmap" seLinuxMountContext=""
Oct 30 02:55:32.084423 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:32.078933    4461 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f425530a-8431-4c5b-8335-8cc1119745c1" volumeName="kubernetes.io/configmap/f425530a-8431-4c5b-8335-8cc1119745c1-ovnkube-script-lib" seLinuxMountContext=""
Oct 30 02:55:32.084423 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:32.078945    4461 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f593f4c2-de6c-42db-b451-d0ade01317f7" volumeName="kubernetes.io/configmap/f593f4c2-de6c-42db-b451-d0ade01317f7-konnectivity-ca" seLinuxMountContext=""
Oct 30 02:55:32.084423 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:32.078957    4461 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="75d9e1ec-cd1b-4aba-b911-5a7465c653cf" volumeName="kubernetes.io/projected/75d9e1ec-cd1b-4aba-b911-5a7465c653cf-kube-api-access-62vqf" seLinuxMountContext=""
Oct 30 02:55:32.084423 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:32.078969    4461 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="91a29ddf-25ca-40a8-b920-6d894360a050" volumeName="kubernetes.io/projected/91a29ddf-25ca-40a8-b920-6d894360a050-kube-api-access-g55cz" seLinuxMountContext=""
Oct 30 02:55:32.084423 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:32.078981    4461 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="eee29ef7-4ad4-4ff9-8cb0-9d2a5aaabba3" volumeName="kubernetes.io/empty-dir/eee29ef7-4ad4-4ff9-8cb0-9d2a5aaabba3-data-volume" seLinuxMountContext=""
Oct 30 02:55:32.084423 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:32.078993    4461 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4a776e60-80b6-4ce0-921c-1884973b4d0b" volumeName="kubernetes.io/secret/4a776e60-80b6-4ce0-921c-1884973b4d0b-signing-key" seLinuxMountContext=""
Oct 30 02:55:32.084423 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:32.079005    4461 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="042af239-396c-4dae-8cf2-0c562af07c62" volumeName="kubernetes.io/configmap/042af239-396c-4dae-8cf2-0c562af07c62-iptables-alerter-script" seLinuxMountContext=""
Oct 30 02:55:32.084423 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:32.079017    4461 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="487caf0f-5f2a-4e93-9c06-de9af02c470b" volumeName="kubernetes.io/projected/487caf0f-5f2a-4e93-9c06-de9af02c470b-kube-api-access-cxnjk" seLinuxMountContext=""
Oct 30 02:55:32.084423 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:32.079031    4461 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5f9fb000-9531-41f1-b3cf-7e79ca1a24d7" volumeName="kubernetes.io/projected/5f9fb000-9531-41f1-b3cf-7e79ca1a24d7-kube-api-access-gwcwf" seLinuxMountContext=""
Oct 30 02:55:32.084423 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:32.079043    4461 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b4ce92fa-1612-4e46-ad6e-23049b2e279b" volumeName="kubernetes.io/projected/b4ce92fa-1612-4e46-ad6e-23049b2e279b-kube-api-access-wmgr2" seLinuxMountContext=""
Oct 30 02:55:32.084423 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:32.079054    4461 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3e4730f0-d77e-4bff-b8af-b7fafd4525d8" volumeName="kubernetes.io/empty-dir/3e4730f0-d77e-4bff-b8af-b7fafd4525d8-etc-tuned" seLinuxMountContext=""
Oct 30 02:55:32.084423 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:32.079109    4461 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4d8bf700-9e09-4741-954b-aba9ccd331c9" volumeName="kubernetes.io/configmap/4d8bf700-9e09-4741-954b-aba9ccd331c9-cni-sysctl-allowlist" seLinuxMountContext=""
Oct 30 02:55:32.084423 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:32.079122    4461 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4d8bf700-9e09-4741-954b-aba9ccd331c9" volumeName="kubernetes.io/configmap/4d8bf700-9e09-4741-954b-aba9ccd331c9-cni-binary-copy" seLinuxMountContext=""
Oct 30 02:55:32.084423 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:32.079134    4461 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="73752bb9-2a14-4582-a6d6-c1fd26524c3d" volumeName="kubernetes.io/projected/73752bb9-2a14-4582-a6d6-c1fd26524c3d-kube-api-access-gsgwj" seLinuxMountContext=""
Oct 30 02:55:32.084423 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:32.079145    4461 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="98f9efe3-a732-4ba4-8543-941633d22170" volumeName="kubernetes.io/secret/98f9efe3-a732-4ba4-8543-941633d22170-original-pull-secret" seLinuxMountContext=""
Oct 30 02:55:32.084423 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:32.079156    4461 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b4ce92fa-1612-4e46-ad6e-23049b2e279b" volumeName="kubernetes.io/configmap/b4ce92fa-1612-4e46-ad6e-23049b2e279b-cni-binary-copy" seLinuxMountContext=""
Oct 30 02:55:32.084423 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:32.079169    4461 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b4ce92fa-1612-4e46-ad6e-23049b2e279b" volumeName="kubernetes.io/configmap/b4ce92fa-1612-4e46-ad6e-23049b2e279b-multus-daemon-config" seLinuxMountContext=""
Oct 30 02:55:32.084423 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:32.079181    4461 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0e6f015d-0a2d-4579-ad4d-a88b8c11c3b9" volumeName="kubernetes.io/projected/0e6f015d-0a2d-4579-ad4d-a88b8c11c3b9-kube-api-access-p764l" seLinuxMountContext=""
Oct 30 02:55:32.084423 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:32.079192    4461 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3e4730f0-d77e-4bff-b8af-b7fafd4525d8" volumeName="kubernetes.io/projected/3e4730f0-d77e-4bff-b8af-b7fafd4525d8-kube-api-access-xwjvv" seLinuxMountContext=""
Oct 30 02:55:32.084423 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:32.079203    4461 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4a776e60-80b6-4ce0-921c-1884973b4d0b" volumeName="kubernetes.io/configmap/4a776e60-80b6-4ce0-921c-1884973b4d0b-signing-cabundle" seLinuxMountContext=""
Oct 30 02:55:32.084423 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:32.079214    4461 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="73752bb9-2a14-4582-a6d6-c1fd26524c3d" volumeName="kubernetes.io/secret/73752bb9-2a14-4582-a6d6-c1fd26524c3d-cert" seLinuxMountContext=""
Oct 30 02:55:32.084423 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:32.079228    4461 reconstruct.go:97] "Volume reconstruction finished"
Oct 30 02:55:32.084423 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:32.079235    4461 reconciler.go:29] "Reconciler: start to sync state"
Oct 30 02:55:32.084423 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:32.079737    4461 factory.go:55] Registering systemd factory
Oct 30 02:55:32.084423 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:32.079754    4461 factory.go:223] Registration of the systemd container factory successfully
Oct 30 02:55:32.087022 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:32.086998    4461 factory.go:153] Registering CRI-O factory
Oct 30 02:55:32.087123 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:32.087044    4461 factory.go:223] Registration of the crio container factory successfully
Oct 30 02:55:32.087161 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:32.087146    4461 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Oct 30 02:55:32.087192 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:32.087173    4461 factory.go:103] Registering Raw factory
Oct 30 02:55:32.087237 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:32.087203    4461 manager.go:1199] Started watching for new ooms in manager
Oct 30 02:55:32.087570 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:32.087557    4461 manager.go:319] Starting recovery of all containers
Oct 30 02:55:32.107413 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:32.107375    4461 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Oct 30 02:55:32.108993 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:32.108967    4461 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Oct 30 02:55:32.108993 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:32.108996    4461 status_manager.go:249] "Starting to sync pod status with apiserver"
Oct 30 02:55:32.110273 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:32.109019    4461 kubelet.go:2521] "Starting kubelet main sync loop"
Oct 30 02:55:32.110273 ip-10-0-143-156 kubenswrapper[4461]: E1030 02:55:32.109065    4461 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]"
Oct 30 02:55:32.119712 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:32.119684    4461 generic.go:356] "Generic (PLEG): container finished" podID="4d8bf700-9e09-4741-954b-aba9ccd331c9" containerID="ad660fba819dae6154bf330655b847dad7b0b3ec814b5d2ca732b2e1177f7a35" exitCode=0
Oct 30 02:55:32.119712 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:32.119707    4461 generic.go:356] "Generic (PLEG): container finished" podID="4d8bf700-9e09-4741-954b-aba9ccd331c9" containerID="4ea3df65ea09e24df426ac49c5be4e172918b496f7236197970bde9c7ac11c44" exitCode=0
Oct 30 02:55:32.119712 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:32.119714    4461 generic.go:356] "Generic (PLEG): container finished" podID="4d8bf700-9e09-4741-954b-aba9ccd331c9" containerID="063b68404f069111e2820f7efd973e509ba6381850dfa4714db55cf4093dfebd" exitCode=0
Oct 30 02:55:32.119941 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:32.119719    4461 generic.go:356] "Generic (PLEG): container finished" podID="4d8bf700-9e09-4741-954b-aba9ccd331c9" containerID="b17f712c6e9fcdcb9c095a865bab449a1db53a73307f27689e97f763840c9f56" exitCode=0
Oct 30 02:55:32.119941 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:32.119725    4461 generic.go:356] "Generic (PLEG): container finished" podID="4d8bf700-9e09-4741-954b-aba9ccd331c9" containerID="5048d278d7f2999d742b1c724f68c695426de1f5a6ebb0eca5f6977ffaaae317" exitCode=0
Oct 30 02:55:32.119941 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:32.119731    4461 generic.go:356] "Generic (PLEG): container finished" podID="4d8bf700-9e09-4741-954b-aba9ccd331c9" containerID="c9d31b25c12fdd51cbfcc8da533c27d086b92dbf22b1d514734feddae17f3fe5" exitCode=0
Oct 30 02:55:32.124018 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:32.123997    4461 reconstruct.go:205] "DevicePaths of reconstructed volumes updated"
Oct 30 02:55:32.125471 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:32.125428    4461 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 02:55:32.125619 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:32.125585    4461 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 02:55:32.134046 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:32.134022    4461 generic.go:356] "Generic (PLEG): container finished" podID="a7672f7a60a28c85d738430783c2cc68" containerID="4529048daffa3640c95d6c055dd0203665f53ee135946c50a4c94b98454d9d86" exitCode=0
Oct 30 02:55:32.138169 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:32.138143    4461 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 02:55:32.138433 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:32.138412    4461 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 02:55:32.176175 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:32.172781    4461 manager.go:324] Recovery completed
Oct 30 02:55:32.204384 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:32.204353    4461 cpu_manager.go:226] "Starting" policy="none"
Oct 30 02:55:32.204384 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:32.204375    4461 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s"
Oct 30 02:55:32.204605 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:32.204397    4461 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory"
Oct 30 02:55:32.204605 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:32.204556    4461 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet=""
Oct 30 02:55:32.204605 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:32.204570    4461 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={}
Oct 30 02:55:32.204605 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:32.204593    4461 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint"
Oct 30 02:55:32.204605 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:32.204601    4461 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet=""
Oct 30 02:55:32.204605 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:32.204609    4461 policy_none.go:50] "Start"
Oct 30 02:55:32.204847 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:32.204620    4461 memory_manager.go:187] "Starting memorymanager" policy="None"
Oct 30 02:55:32.204847 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:32.204631    4461 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Oct 30 02:55:32.204847 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:32.204793    4461 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint"
Oct 30 02:55:32.204847 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:32.204802    4461 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint"
Oct 30 02:55:32.204847 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:32.204809    4461 policy_none.go:44] "Start"
Oct 30 02:55:32.209735 ip-10-0-143-156 kubenswrapper[4461]: E1030 02:55:32.209712    4461 kubelet.go:2545] "Skipping pod synchronization" err="container runtime status check may not have completed yet"
Oct 30 02:55:32.210304 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:32.210271    4461 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"}
Oct 30 02:55:32.219642 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:32.219602    4461 manager.go:341] "Starting Device Plugin manager"
Oct 30 02:55:32.220717 ip-10-0-143-156 kubenswrapper[4461]: E1030 02:55:32.219635    4461 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Oct 30 02:55:32.220717 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:32.220339    4461 server.go:87] "Starting device plugin registration server"
Oct 30 02:55:32.220717 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:32.220637    4461 eviction_manager.go:194] "Eviction manager: starting control loop"
Oct 30 02:55:32.220717 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:32.220649    4461 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Oct 30 02:55:32.221875 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:32.221038    4461 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Oct 30 02:55:32.221875 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:32.221158    4461 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Oct 30 02:55:32.221875 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:32.221193    4461 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts"
Oct 30 02:55:32.221875 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:32.221200    4461 plugin_manager.go:121] "Starting Kubelet Plugin Manager"
Oct 30 02:55:32.221875 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:32.221250    4461 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-30T02:55:32.221184159Z","UUID":"5f162626-c0f6-421f-a6fc-5ebbfe8606d9","Handler":null,"Name":"","Endpoint":""}
Oct 30 02:55:32.223185 ip-10-0-143-156 kubenswrapper[4461]: E1030 02:55:32.223166    4461 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Oct 30 02:55:32.225336 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:32.225318    4461 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Oct 30 02:55:32.225336 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:32.225346    4461 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Oct 30 02:55:32.321719 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:32.321686    4461 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 02:55:32.322960 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:32.322941    4461 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-156.ec2.internal" event="NodeHasSufficientMemory"
Oct 30 02:55:32.323064 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:32.322977    4461 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-156.ec2.internal" event="NodeHasNoDiskPressure"
Oct 30 02:55:32.323064 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:32.322991    4461 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-156.ec2.internal" event="NodeHasSufficientPID"
Oct 30 02:55:32.323064 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:32.323031    4461 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-143-156.ec2.internal"
Oct 30 02:55:32.361786 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:32.361756    4461 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-143-156.ec2.internal"
Oct 30 02:55:32.361915 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:32.361845    4461 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-143-156.ec2.internal"
Oct 30 02:55:32.410387 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:32.410335    4461 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-156.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-143-156.ec2.internal"]
Oct 30 02:55:32.410905 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:32.410858    4461 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-156.ec2.internal" event={"ID":"fee27b4a19dca64d5627dff27844f71f","Type":"ContainerStarted","Data":"3274dbbacadbe10e5b82128eb1fe384bfd6fad9641cd4f56785bb90c54ad93f2"}
Oct 30 02:55:32.410957 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:32.410924    4461 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-156.ec2.internal" event={"ID":"fee27b4a19dca64d5627dff27844f71f","Type":"ContainerStarted","Data":"8e25068db3f97685e5f4f9df47b334f09abff36314e4419220b21069b574efb3"}
Oct 30 02:55:32.410957 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:32.410947    4461 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-156.ec2.internal" event={"ID":"a7672f7a60a28c85d738430783c2cc68","Type":"ContainerStarted","Data":"ce428cb26f198f2072be833eeb1c3955424510dc0d3c628a99089634ae13057f"}
Oct 30 02:55:32.411046 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:32.410965    4461 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-156.ec2.internal" event={"ID":"a7672f7a60a28c85d738430783c2cc68","Type":"ContainerDied","Data":"4529048daffa3640c95d6c055dd0203665f53ee135946c50a4c94b98454d9d86"}
Oct 30 02:55:32.411046 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:32.410980    4461 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-156.ec2.internal"
Oct 30 02:55:32.411156 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:32.410983    4461 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-156.ec2.internal" event={"ID":"a7672f7a60a28c85d738430783c2cc68","Type":"ContainerStarted","Data":"c45974cae891b83781f22b2ed8a3e9c6dd117bb07816519904afa6a549a61b93"}
Oct 30 02:55:32.411201 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:32.411113    4461 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-156.ec2.internal"
Oct 30 02:55:32.435369 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:32.435306    4461 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 02:55:32.435369 ip-10-0-143-156 kubenswrapper[4461]: E1030 02:55:32.435349    4461 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-143-156.ec2.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-156.ec2.internal"
Oct 30 02:55:32.435369 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:32.435307    4461 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 02:55:32.435575 ip-10-0-143-156 kubenswrapper[4461]: E1030 02:55:32.435414    4461 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-143-156.ec2.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-156.ec2.internal"
Oct 30 02:55:32.526280 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:32.526243    4461 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/a7672f7a60a28c85d738430783c2cc68-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-143-156.ec2.internal\" (UID: \"a7672f7a60a28c85d738430783c2cc68\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-156.ec2.internal"
Oct 30 02:55:32.526280 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:32.526278    4461 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a7672f7a60a28c85d738430783c2cc68-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-143-156.ec2.internal\" (UID: \"a7672f7a60a28c85d738430783c2cc68\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-156.ec2.internal"
Oct 30 02:55:32.526470 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:32.526296    4461 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/fee27b4a19dca64d5627dff27844f71f-config\") pod \"kube-apiserver-proxy-ip-10-0-143-156.ec2.internal\" (UID: \"fee27b4a19dca64d5627dff27844f71f\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-143-156.ec2.internal"
Oct 30 02:55:32.627205 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:32.627174    4461 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/fee27b4a19dca64d5627dff27844f71f-config\") pod \"kube-apiserver-proxy-ip-10-0-143-156.ec2.internal\" (UID: \"fee27b4a19dca64d5627dff27844f71f\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-143-156.ec2.internal"
Oct 30 02:55:32.627205 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:32.627207    4461 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/a7672f7a60a28c85d738430783c2cc68-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-143-156.ec2.internal\" (UID: \"a7672f7a60a28c85d738430783c2cc68\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-156.ec2.internal"
Oct 30 02:55:32.627386 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:32.627226    4461 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a7672f7a60a28c85d738430783c2cc68-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-143-156.ec2.internal\" (UID: \"a7672f7a60a28c85d738430783c2cc68\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-156.ec2.internal"
Oct 30 02:55:32.627386 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:32.627273    4461 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a7672f7a60a28c85d738430783c2cc68-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-143-156.ec2.internal\" (UID: \"a7672f7a60a28c85d738430783c2cc68\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-156.ec2.internal"
Oct 30 02:55:32.627386 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:32.627286    4461 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/fee27b4a19dca64d5627dff27844f71f-config\") pod \"kube-apiserver-proxy-ip-10-0-143-156.ec2.internal\" (UID: \"fee27b4a19dca64d5627dff27844f71f\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-143-156.ec2.internal"
Oct 30 02:55:32.627386 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:32.627286    4461 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/a7672f7a60a28c85d738430783c2cc68-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-143-156.ec2.internal\" (UID: \"a7672f7a60a28c85d738430783c2cc68\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-156.ec2.internal"
Oct 30 02:55:33.059991 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:33.059959    4461 apiserver.go:52] "Watching apiserver"
Oct 30 02:55:33.100552 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:33.100526    4461 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Oct 30 02:55:33.100980 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:33.100961    4461 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-wjll7","openshift-multus/network-metrics-daemon-mzk2s","openshift-dns/node-resolver-zhb7b","kube-system/global-pull-secret-syncer-vbnwt","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gqpp2","openshift-ingress-canary/ingress-canary-vwbj7","openshift-network-operator/iptables-alerter-jhq2g","openshift-insights/insights-runtime-extractor-2k85b","kube-system/konnectivity-agent-djpqs","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-156.ec2.internal","openshift-multus/multus-additional-cni-plugins-9l7k8","openshift-kube-storage-version-migrator/migrator-66df5d496b-wx7pd","openshift-multus/multus-9qcgk","kube-system/kube-apiserver-proxy-ip-10-0-143-156.ec2.internal","openshift-image-registry/node-ca-9mddj","openshift-network-diagnostics/network-check-target-7rmdj","openshift-ovn-kubernetes/ovnkube-node-g6nqm","openshift-service-ca/service-ca-5fffc97bdf-rwq7h","openshift-cluster-node-tuning-operator/tuned-h7cm4"]
Oct 30 02:55:33.103680 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:33.103660    4461 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mzk2s"
Oct 30 02:55:33.107387 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:33.107366    4461 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-wjll7"
Oct 30 02:55:33.108571 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:33.108549    4461 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-2k85b"
Oct 30 02:55:33.116513 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:33.116489    4461 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-mk2hv\""
Oct 30 02:55:33.116777 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:33.116762    4461 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-6gpqv\""
Oct 30 02:55:33.116855 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:33.116813    4461 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Oct 30 02:55:33.116963 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:33.116951    4461 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-ft4zm\""
Oct 30 02:55:33.117037 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:33.116974    4461 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Oct 30 02:55:33.117467 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:33.117450    4461 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-kn7z4\""
Oct 30 02:55:33.117747 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:33.117732    4461 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-8ljf8\""
Oct 30 02:55:33.117962 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:33.117950    4461 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Oct 30 02:55:33.118019 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:33.117950    4461 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-fdxtg\""
Oct 30 02:55:33.118068 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:33.118036    4461 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-9qpgh\""
Oct 30 02:55:33.118575 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:33.118556    4461 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Oct 30 02:55:33.118868 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:33.118850    4461 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Oct 30 02:55:33.118953 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:33.118868    4461 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Oct 30 02:55:33.118953 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:33.118885    4461 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Oct 30 02:55:33.118953 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:33.118899    4461 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Oct 30 02:55:33.118953 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:33.118859    4461 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Oct 30 02:55:33.118953 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:33.118922    4461 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Oct 30 02:55:33.118953 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:33.118929    4461 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Oct 30 02:55:33.118953 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:33.118929    4461 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Oct 30 02:55:33.118953 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:33.118948    4461 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Oct 30 02:55:33.119271 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:33.119051    4461 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Oct 30 02:55:33.119271 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:33.119056    4461 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Oct 30 02:55:33.119271 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:33.119072    4461 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Oct 30 02:55:33.119271 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:33.119085    4461 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Oct 30 02:55:33.119271 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:33.119086    4461 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Oct 30 02:55:33.119271 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:33.119130    4461 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Oct 30 02:55:33.119271 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:33.119200    4461 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Oct 30 02:55:33.119542 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:33.119522    4461 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Oct 30 02:55:33.150534 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:33.150508    4461 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-156.ec2.internal"
Oct 30 02:55:33.150634 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:33.150589    4461 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-156.ec2.internal"
Oct 30 02:55:33.174492 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:33.174470    4461 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 02:55:33.174585 ip-10-0-143-156 kubenswrapper[4461]: E1030 02:55:33.174518    4461 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-143-156.ec2.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-156.ec2.internal"
Oct 30 02:55:33.177045 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:33.177006    4461 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Oct 30 02:55:33.177045 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:33.177049    4461 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Oct 30 02:55:33.177298 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:33.177276    4461 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Oct 30 02:55:33.177402 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:33.177292    4461 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Oct 30 02:55:33.177402 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:33.177323    4461 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Oct 30 02:55:33.177402 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:33.177377    4461 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Oct 30 02:55:33.177402 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:33.177386    4461 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Oct 30 02:55:33.177402 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:33.177391    4461 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\""
Oct 30 02:55:33.177668 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:33.177458    4461 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Oct 30 02:55:33.177668 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:33.177464    4461 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Oct 30 02:55:33.177668 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:33.177617    4461 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\""
Oct 30 02:55:33.177668 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:33.177633    4461 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Oct 30 02:55:33.177668 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:33.177657    4461 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Oct 30 02:55:33.177844 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:33.177658    4461 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Oct 30 02:55:33.177844 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:33.177685    4461 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Oct 30 02:55:33.177844 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:33.177662    4461 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\""
Oct 30 02:55:33.177844 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:33.177693    4461 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 02:55:33.177844 ip-10-0-143-156 kubenswrapper[4461]: E1030 02:55:33.177745    4461 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-143-156.ec2.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-156.ec2.internal"
Oct 30 02:55:33.177844 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:33.177693    4461 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Oct 30 02:55:33.177844 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:33.177803    4461 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Oct 30 02:55:33.178023 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:33.177898    4461 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Oct 30 02:55:33.178023 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:33.177976    4461 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Oct 30 02:55:33.216906 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:33.216881    4461 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Oct 30 02:55:33.217050 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:33.216941    4461 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Oct 30 02:55:33.217050 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:33.216988    4461 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-chfb9\""
Oct 30 02:55:33.217050 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:33.217017    4461 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\""
Oct 30 02:55:33.217337 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:33.217161    4461 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Oct 30 02:55:33.217337 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:33.217174    4461 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-kdtjd\""
Oct 30 02:55:33.217337 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:33.217203    4461 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-xzg75\""
Oct 30 02:55:33.217337 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:33.217245    4461 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Oct 30 02:55:33.217565 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:33.217399    4461 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-d8b4w\""
Oct 30 02:55:33.217565 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:33.217447    4461 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-chr82\""
Oct 30 02:55:33.217565 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:33.217459    4461 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-bckpk\""
Oct 30 02:55:33.217565 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:33.217465    4461 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-r9wgn\""
Oct 30 02:55:33.217565 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:33.217467    4461 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-vzrlw\""
Oct 30 02:55:33.217565 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:33.217519    4461 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-tpj88\""
Oct 30 02:55:33.217565 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:33.217528    4461 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-t6hqp\""
Oct 30 02:55:33.217565 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:33.217538    4461 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Oct 30 02:55:33.231605 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:33.231180    4461 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/b4ce92fa-1612-4e46-ad6e-23049b2e279b-host-run-multus-certs\") pod \"multus-9qcgk\" (UID: \"b4ce92fa-1612-4e46-ad6e-23049b2e279b\") " pod="openshift-multus/multus-9qcgk"
Oct 30 02:55:33.231687 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:33.231617    4461 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xwjvv\" (UniqueName: \"kubernetes.io/projected/3e4730f0-d77e-4bff-b8af-b7fafd4525d8-kube-api-access-xwjvv\") pod \"tuned-h7cm4\" (UID: \"3e4730f0-d77e-4bff-b8af-b7fafd4525d8\") " pod="openshift-cluster-node-tuning-operator/tuned-h7cm4"
Oct 30 02:55:33.231687 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:33.231634    4461 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g55cz\" (UniqueName: \"kubernetes.io/projected/91a29ddf-25ca-40a8-b920-6d894360a050-kube-api-access-g55cz\") pod \"aws-ebs-csi-driver-node-gqpp2\" (UID: \"91a29ddf-25ca-40a8-b920-6d894360a050\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gqpp2"
Oct 30 02:55:33.231687 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:33.231651    4461 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/f425530a-8431-4c5b-8335-8cc1119745c1-systemd-units\") pod \"ovnkube-node-g6nqm\" (UID: \"f425530a-8431-4c5b-8335-8cc1119745c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-g6nqm"
Oct 30 02:55:33.231687 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:33.231666    4461 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f425530a-8431-4c5b-8335-8cc1119745c1-run-openvswitch\") pod \"ovnkube-node-g6nqm\" (UID: \"f425530a-8431-4c5b-8335-8cc1119745c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-g6nqm"
Oct 30 02:55:33.231687 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:33.231684    4461 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/f425530a-8431-4c5b-8335-8cc1119745c1-run-ovn\") pod \"ovnkube-node-g6nqm\" (UID: \"f425530a-8431-4c5b-8335-8cc1119745c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-g6nqm"
Oct 30 02:55:33.231874 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:33.231707    4461 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/3e4730f0-d77e-4bff-b8af-b7fafd4525d8-run\") pod \"tuned-h7cm4\" (UID: \"3e4730f0-d77e-4bff-b8af-b7fafd4525d8\") " pod="openshift-cluster-node-tuning-operator/tuned-h7cm4"
Oct 30 02:55:33.231874 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:33.231774    4461 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3e4730f0-d77e-4bff-b8af-b7fafd4525d8-var-lib-kubelet\") pod \"tuned-h7cm4\" (UID: \"3e4730f0-d77e-4bff-b8af-b7fafd4525d8\") " pod="openshift-cluster-node-tuning-operator/tuned-h7cm4"
Oct 30 02:55:33.231874 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:33.231818    4461 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/f593f4c2-de6c-42db-b451-d0ade01317f7-konnectivity-ca\") pod \"konnectivity-agent-djpqs\" (UID: \"f593f4c2-de6c-42db-b451-d0ade01317f7\") " pod="kube-system/konnectivity-agent-djpqs"
Oct 30 02:55:33.231874 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:33.231854    4461 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/4a776e60-80b6-4ce0-921c-1884973b4d0b-signing-cabundle\") pod \"service-ca-5fffc97bdf-rwq7h\" (UID: \"4a776e60-80b6-4ce0-921c-1884973b4d0b\") " pod="openshift-service-ca/service-ca-5fffc97bdf-rwq7h"
Oct 30 02:55:33.231874 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:33.231870    4461 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/3e4730f0-d77e-4bff-b8af-b7fafd4525d8-etc-sysctl-conf\") pod \"tuned-h7cm4\" (UID: \"3e4730f0-d77e-4bff-b8af-b7fafd4525d8\") " pod="openshift-cluster-node-tuning-operator/tuned-h7cm4"
Oct 30 02:55:33.232028 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:33.231889    4461 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3e4730f0-d77e-4bff-b8af-b7fafd4525d8-tmp\") pod \"tuned-h7cm4\" (UID: \"3e4730f0-d77e-4bff-b8af-b7fafd4525d8\") " pod="openshift-cluster-node-tuning-operator/tuned-h7cm4"
Oct 30 02:55:33.232028 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:33.231930    4461 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/98f9efe3-a732-4ba4-8543-941633d22170-original-pull-secret\") pod \"global-pull-secret-syncer-vbnwt\" (UID: \"98f9efe3-a732-4ba4-8543-941633d22170\") " pod="kube-system/global-pull-secret-syncer-vbnwt"
Oct 30 02:55:33.232028 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:33.231952    4461 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3e4730f0-d77e-4bff-b8af-b7fafd4525d8-tmp\") pod \"tuned-h7cm4\" (UID: \"3e4730f0-d77e-4bff-b8af-b7fafd4525d8\") " pod="openshift-cluster-node-tuning-operator/tuned-h7cm4"
Oct 30 02:55:33.232028 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:33.231962    4461 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/0e6f015d-0a2d-4579-ad4d-a88b8c11c3b9-hosts-file\") pod \"node-resolver-zhb7b\" (UID: \"0e6f015d-0a2d-4579-ad4d-a88b8c11c3b9\") " pod="openshift-dns/node-resolver-zhb7b"
Oct 30 02:55:33.232028 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:33.231988    4461 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/91a29ddf-25ca-40a8-b920-6d894360a050-kubelet-dir\") pod \"aws-ebs-csi-driver-node-gqpp2\" (UID: \"91a29ddf-25ca-40a8-b920-6d894360a050\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gqpp2"
Oct 30 02:55:33.232203 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:33.232036    4461 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/91a29ddf-25ca-40a8-b920-6d894360a050-socket-dir\") pod \"aws-ebs-csi-driver-node-gqpp2\" (UID: \"91a29ddf-25ca-40a8-b920-6d894360a050\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gqpp2"
Oct 30 02:55:33.232203 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:33.232062    4461 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4d8bf700-9e09-4741-954b-aba9ccd331c9-system-cni-dir\") pod \"multus-additional-cni-plugins-9l7k8\" (UID: \"4d8bf700-9e09-4741-954b-aba9ccd331c9\") " pod="openshift-multus/multus-additional-cni-plugins-9l7k8"
Oct 30 02:55:33.232203 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:33.232082    4461 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/f425530a-8431-4c5b-8335-8cc1119745c1-host-kubelet\") pod \"ovnkube-node-g6nqm\" (UID: \"f425530a-8431-4c5b-8335-8cc1119745c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-g6nqm"
Oct 30 02:55:33.232203 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:33.232148    4461 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/f593f4c2-de6c-42db-b451-d0ade01317f7-agent-certs\") pod \"konnectivity-agent-djpqs\" (UID: \"f593f4c2-de6c-42db-b451-d0ade01317f7\") " pod="kube-system/konnectivity-agent-djpqs"
Oct 30 02:55:33.232203 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:33.232159    4461 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/f593f4c2-de6c-42db-b451-d0ade01317f7-konnectivity-ca\") pod \"konnectivity-agent-djpqs\" (UID: \"f593f4c2-de6c-42db-b451-d0ade01317f7\") " pod="kube-system/konnectivity-agent-djpqs"
Oct 30 02:55:33.232432 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:33.232188    4461 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2289h\" (UniqueName: \"kubernetes.io/projected/4a776e60-80b6-4ce0-921c-1884973b4d0b-kube-api-access-2289h\") pod \"service-ca-5fffc97bdf-rwq7h\" (UID: \"4a776e60-80b6-4ce0-921c-1884973b4d0b\") " pod="openshift-service-ca/service-ca-5fffc97bdf-rwq7h"
Oct 30 02:55:33.232432 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:33.232229    4461 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/98f9efe3-a732-4ba4-8543-941633d22170-original-pull-secret\") pod \"global-pull-secret-syncer-vbnwt\" (UID: \"98f9efe3-a732-4ba4-8543-941633d22170\") " pod="kube-system/global-pull-secret-syncer-vbnwt"
Oct 30 02:55:33.232432 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:33.232239    4461 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/4a776e60-80b6-4ce0-921c-1884973b4d0b-signing-cabundle\") pod \"service-ca-5fffc97bdf-rwq7h\" (UID: \"4a776e60-80b6-4ce0-921c-1884973b4d0b\") " pod="openshift-service-ca/service-ca-5fffc97bdf-rwq7h"
Oct 30 02:55:33.232432 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:33.232297    4461 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/4d8bf700-9e09-4741-954b-aba9ccd331c9-cni-binary-copy\") pod \"multus-additional-cni-plugins-9l7k8\" (UID: \"4d8bf700-9e09-4741-954b-aba9ccd331c9\") " pod="openshift-multus/multus-additional-cni-plugins-9l7k8"
Oct 30 02:55:33.232432 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:33.232342    4461 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/f425530a-8431-4c5b-8335-8cc1119745c1-log-socket\") pod \"ovnkube-node-g6nqm\" (UID: \"f425530a-8431-4c5b-8335-8cc1119745c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-g6nqm"
Oct 30 02:55:33.232432 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:33.232387    4461 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f425530a-8431-4c5b-8335-8cc1119745c1-host-cni-bin\") pod \"ovnkube-node-g6nqm\" (UID: \"f425530a-8431-4c5b-8335-8cc1119745c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-g6nqm"
Oct 30 02:55:33.232432 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:33.232431    4461 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/f593f4c2-de6c-42db-b451-d0ade01317f7-agent-certs\") pod \"konnectivity-agent-djpqs\" (UID: \"f593f4c2-de6c-42db-b451-d0ade01317f7\") " pod="kube-system/konnectivity-agent-djpqs"
Oct 30 02:55:33.232636 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:33.232432    4461 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/042af239-396c-4dae-8cf2-0c562af07c62-iptables-alerter-script\") pod \"iptables-alerter-jhq2g\" (UID: \"042af239-396c-4dae-8cf2-0c562af07c62\") " pod="openshift-network-operator/iptables-alerter-jhq2g"
Oct 30 02:55:33.232636 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:33.232480    4461 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/b4ce92fa-1612-4e46-ad6e-23049b2e279b-host-var-lib-cni-multus\") pod \"multus-9qcgk\" (UID: \"b4ce92fa-1612-4e46-ad6e-23049b2e279b\") " pod="openshift-multus/multus-9qcgk"
Oct 30 02:55:33.232636 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:33.232508    4461 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/4d8bf700-9e09-4741-954b-aba9ccd331c9-cni-binary-copy\") pod \"multus-additional-cni-plugins-9l7k8\" (UID: \"4d8bf700-9e09-4741-954b-aba9ccd331c9\") " pod="openshift-multus/multus-additional-cni-plugins-9l7k8"
Oct 30 02:55:33.232636 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:33.232549    4461 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f425530a-8431-4c5b-8335-8cc1119745c1-host-run-netns\") pod \"ovnkube-node-g6nqm\" (UID: \"f425530a-8431-4c5b-8335-8cc1119745c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-g6nqm"
Oct 30 02:55:33.232636 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:33.232602    4461 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/b4ce92fa-1612-4e46-ad6e-23049b2e279b-multus-daemon-config\") pod \"multus-9qcgk\" (UID: \"b4ce92fa-1612-4e46-ad6e-23049b2e279b\") " pod="openshift-multus/multus-9qcgk"
Oct 30 02:55:33.232636 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:33.232627    4461 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/042af239-396c-4dae-8cf2-0c562af07c62-iptables-alerter-script\") pod \"iptables-alerter-jhq2g\" (UID: \"042af239-396c-4dae-8cf2-0c562af07c62\") " pod="openshift-network-operator/iptables-alerter-jhq2g"
Oct 30 02:55:33.232636 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:33.232634    4461 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/3e4730f0-d77e-4bff-b8af-b7fafd4525d8-etc-sysctl-d\") pod \"tuned-h7cm4\" (UID: \"3e4730f0-d77e-4bff-b8af-b7fafd4525d8\") " pod="openshift-cluster-node-tuning-operator/tuned-h7cm4"
Oct 30 02:55:33.232827 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:33.232657    4461 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wd8s8\" (UniqueName: \"kubernetes.io/projected/eee29ef7-4ad4-4ff9-8cb0-9d2a5aaabba3-kube-api-access-wd8s8\") pod \"insights-runtime-extractor-2k85b\" (UID: \"eee29ef7-4ad4-4ff9-8cb0-9d2a5aaabba3\") " pod="openshift-insights/insights-runtime-extractor-2k85b"
Oct 30 02:55:33.232827 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:33.232726    4461 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/5f9fb000-9531-41f1-b3cf-7e79ca1a24d7-config-volume\") pod \"dns-default-wjll7\" (UID: \"5f9fb000-9531-41f1-b3cf-7e79ca1a24d7\") " pod="openshift-dns/dns-default-wjll7"
Oct 30 02:55:33.232827 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:33.232750    4461 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5f9fb000-9531-41f1-b3cf-7e79ca1a24d7-tmp-dir\") pod \"dns-default-wjll7\" (UID: \"5f9fb000-9531-41f1-b3cf-7e79ca1a24d7\") " pod="openshift-dns/dns-default-wjll7"
Oct 30 02:55:33.232827 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:33.232803    4461 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3e4730f0-d77e-4bff-b8af-b7fafd4525d8-sys\") pod \"tuned-h7cm4\" (UID: \"3e4730f0-d77e-4bff-b8af-b7fafd4525d8\") " pod="openshift-cluster-node-tuning-operator/tuned-h7cm4"
Oct 30 02:55:33.232960 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:33.232856    4461 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5f9fb000-9531-41f1-b3cf-7e79ca1a24d7-tmp-dir\") pod \"dns-default-wjll7\" (UID: \"5f9fb000-9531-41f1-b3cf-7e79ca1a24d7\") " pod="openshift-dns/dns-default-wjll7"
Oct 30 02:55:33.232960 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:33.232889    4461 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/b4ce92fa-1612-4e46-ad6e-23049b2e279b-multus-socket-dir-parent\") pod \"multus-9qcgk\" (UID: \"b4ce92fa-1612-4e46-ad6e-23049b2e279b\") " pod="openshift-multus/multus-9qcgk"
Oct 30 02:55:33.232960 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:33.232930    4461 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/5f9fb000-9531-41f1-b3cf-7e79ca1a24d7-config-volume\") pod \"dns-default-wjll7\" (UID: \"5f9fb000-9531-41f1-b3cf-7e79ca1a24d7\") " pod="openshift-dns/dns-default-wjll7"
Oct 30 02:55:33.232960 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:33.232930    4461 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/b4ce92fa-1612-4e46-ad6e-23049b2e279b-host-run-k8s-cni-cncf-io\") pod \"multus-9qcgk\" (UID: \"b4ce92fa-1612-4e46-ad6e-23049b2e279b\") " pod="openshift-multus/multus-9qcgk"
Oct 30 02:55:33.232960 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:33.232952    4461 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/b4ce92fa-1612-4e46-ad6e-23049b2e279b-multus-daemon-config\") pod \"multus-9qcgk\" (UID: \"b4ce92fa-1612-4e46-ad6e-23049b2e279b\") " pod="openshift-multus/multus-9qcgk"
Oct 30 02:55:33.233149 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:33.232967    4461 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b4ce92fa-1612-4e46-ad6e-23049b2e279b-host-var-lib-cni-bin\") pod \"multus-9qcgk\" (UID: \"b4ce92fa-1612-4e46-ad6e-23049b2e279b\") " pod="openshift-multus/multus-9qcgk"
Oct 30 02:55:33.233149 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:33.232984    4461 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/3e4730f0-d77e-4bff-b8af-b7fafd4525d8-host\") pod \"tuned-h7cm4\" (UID: \"3e4730f0-d77e-4bff-b8af-b7fafd4525d8\") " pod="openshift-cluster-node-tuning-operator/tuned-h7cm4"
Oct 30 02:55:33.233149 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:33.232999    4461 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/eee29ef7-4ad4-4ff9-8cb0-9d2a5aaabba3-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2k85b\" (UID: \"eee29ef7-4ad4-4ff9-8cb0-9d2a5aaabba3\") " pod="openshift-insights/insights-runtime-extractor-2k85b"
Oct 30 02:55:33.233149 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:33.233034    4461 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/4d8bf700-9e09-4741-954b-aba9ccd331c9-cnibin\") pod \"multus-additional-cni-plugins-9l7k8\" (UID: \"4d8bf700-9e09-4741-954b-aba9ccd331c9\") " pod="openshift-multus/multus-additional-cni-plugins-9l7k8"
Oct 30 02:55:33.233149 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:33.233064    4461 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/f425530a-8431-4c5b-8335-8cc1119745c1-host-cni-netd\") pod \"ovnkube-node-g6nqm\" (UID: \"f425530a-8431-4c5b-8335-8cc1119745c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-g6nqm"
Oct 30 02:55:33.233149 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:33.233143    4461 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/4d8bf700-9e09-4741-954b-aba9ccd331c9-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-9l7k8\" (UID: \"4d8bf700-9e09-4741-954b-aba9ccd331c9\") " pod="openshift-multus/multus-additional-cni-plugins-9l7k8"
Oct 30 02:55:33.233311 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:33.233193    4461 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pfgdh\" (UniqueName: \"kubernetes.io/projected/4d8bf700-9e09-4741-954b-aba9ccd331c9-kube-api-access-pfgdh\") pod \"multus-additional-cni-plugins-9l7k8\" (UID: \"4d8bf700-9e09-4741-954b-aba9ccd331c9\") " pod="openshift-multus/multus-additional-cni-plugins-9l7k8"
Oct 30 02:55:33.233311 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:33.233266    4461 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/f425530a-8431-4c5b-8335-8cc1119745c1-run-systemd\") pod \"ovnkube-node-g6nqm\" (UID: \"f425530a-8431-4c5b-8335-8cc1119745c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-g6nqm"
Oct 30 02:55:33.233311 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:33.233300    4461 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/f425530a-8431-4c5b-8335-8cc1119745c1-ovnkube-config\") pod \"ovnkube-node-g6nqm\" (UID: \"f425530a-8431-4c5b-8335-8cc1119745c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-g6nqm"
Oct 30 02:55:33.233394 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:33.233328    4461 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/4d8bf700-9e09-4741-954b-aba9ccd331c9-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-9l7k8\" (UID: \"4d8bf700-9e09-4741-954b-aba9ccd331c9\") " pod="openshift-multus/multus-additional-cni-plugins-9l7k8"
Oct 30 02:55:33.233394 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:33.233333    4461 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/f425530a-8431-4c5b-8335-8cc1119745c1-env-overrides\") pod \"ovnkube-node-g6nqm\" (UID: \"f425530a-8431-4c5b-8335-8cc1119745c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-g6nqm"
Oct 30 02:55:33.233394 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:33.233374    4461 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/f425530a-8431-4c5b-8335-8cc1119745c1-ovnkube-script-lib\") pod \"ovnkube-node-g6nqm\" (UID: \"f425530a-8431-4c5b-8335-8cc1119745c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-g6nqm"
Oct 30 02:55:33.233491 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:33.233410    4461 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/487caf0f-5f2a-4e93-9c06-de9af02c470b-host\") pod \"node-ca-9mddj\" (UID: \"487caf0f-5f2a-4e93-9c06-de9af02c470b\") " pod="openshift-image-registry/node-ca-9mddj"
Oct 30 02:55:33.233491 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:33.233437    4461 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cxnjk\" (UniqueName: \"kubernetes.io/projected/487caf0f-5f2a-4e93-9c06-de9af02c470b-kube-api-access-cxnjk\") pod \"node-ca-9mddj\" (UID: \"487caf0f-5f2a-4e93-9c06-de9af02c470b\") " pod="openshift-image-registry/node-ca-9mddj"
Oct 30 02:55:33.233491 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:33.233441    4461 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/f425530a-8431-4c5b-8335-8cc1119745c1-env-overrides\") pod \"ovnkube-node-g6nqm\" (UID: \"f425530a-8431-4c5b-8335-8cc1119745c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-g6nqm"
Oct 30 02:55:33.233491 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:33.233460    4461 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b4ce92fa-1612-4e46-ad6e-23049b2e279b-host-var-lib-kubelet\") pod \"multus-9qcgk\" (UID: \"b4ce92fa-1612-4e46-ad6e-23049b2e279b\") " pod="openshift-multus/multus-9qcgk"
Oct 30 02:55:33.233491 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:33.233472    4461 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/f425530a-8431-4c5b-8335-8cc1119745c1-ovnkube-config\") pod \"ovnkube-node-g6nqm\" (UID: \"f425530a-8431-4c5b-8335-8cc1119745c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-g6nqm"
Oct 30 02:55:33.233491 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:33.233490    4461 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b4ce92fa-1612-4e46-ad6e-23049b2e279b-etc-kubernetes\") pod \"multus-9qcgk\" (UID: \"b4ce92fa-1612-4e46-ad6e-23049b2e279b\") " pod="openshift-multus/multus-9qcgk"
Oct 30 02:55:33.233713 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:33.233515    4461 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/3e4730f0-d77e-4bff-b8af-b7fafd4525d8-etc-tuned\") pod \"tuned-h7cm4\" (UID: \"3e4730f0-d77e-4bff-b8af-b7fafd4525d8\") " pod="openshift-cluster-node-tuning-operator/tuned-h7cm4"
Oct 30 02:55:33.233713 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:33.233553    4461 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/91a29ddf-25ca-40a8-b920-6d894360a050-device-dir\") pod \"aws-ebs-csi-driver-node-gqpp2\" (UID: \"91a29ddf-25ca-40a8-b920-6d894360a050\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gqpp2"
Oct 30 02:55:33.233713 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:33.233587    4461 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f425530a-8431-4c5b-8335-8cc1119745c1-etc-openvswitch\") pod \"ovnkube-node-g6nqm\" (UID: \"f425530a-8431-4c5b-8335-8cc1119745c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-g6nqm"
Oct 30 02:55:33.233713 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:33.233588    4461 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/3e4730f0-d77e-4bff-b8af-b7fafd4525d8-etc-tuned\") pod \"tuned-h7cm4\" (UID: \"3e4730f0-d77e-4bff-b8af-b7fafd4525d8\") " pod="openshift-cluster-node-tuning-operator/tuned-h7cm4"
Oct 30 02:55:33.233713 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:33.233586    4461 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/f425530a-8431-4c5b-8335-8cc1119745c1-ovnkube-script-lib\") pod \"ovnkube-node-g6nqm\" (UID: \"f425530a-8431-4c5b-8335-8cc1119745c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-g6nqm"
Oct 30 02:55:33.233713 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:33.233619    4461 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/f425530a-8431-4c5b-8335-8cc1119745c1-node-log\") pod \"ovnkube-node-g6nqm\" (UID: \"f425530a-8431-4c5b-8335-8cc1119745c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-g6nqm"
Oct 30 02:55:33.233713 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:33.233640    4461 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b4ce92fa-1612-4e46-ad6e-23049b2e279b-multus-cni-dir\") pod \"multus-9qcgk\" (UID: \"b4ce92fa-1612-4e46-ad6e-23049b2e279b\") " pod="openshift-multus/multus-9qcgk"
Oct 30 02:55:33.233713 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:33.233663    4461 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/3e4730f0-d77e-4bff-b8af-b7fafd4525d8-etc-modprobe-d\") pod \"tuned-h7cm4\" (UID: \"3e4730f0-d77e-4bff-b8af-b7fafd4525d8\") " pod="openshift-cluster-node-tuning-operator/tuned-h7cm4"
Oct 30 02:55:33.233713 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:33.233688    4461 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/98f9efe3-a732-4ba4-8543-941633d22170-kubelet-config\") pod \"global-pull-secret-syncer-vbnwt\" (UID: \"98f9efe3-a732-4ba4-8543-941633d22170\") " pod="kube-system/global-pull-secret-syncer-vbnwt"
Oct 30 02:55:33.233713 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:33.233707    4461 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/73752bb9-2a14-4582-a6d6-c1fd26524c3d-cert\") pod \"ingress-canary-vwbj7\" (UID: \"73752bb9-2a14-4582-a6d6-c1fd26524c3d\") " pod="openshift-ingress-canary/ingress-canary-vwbj7"
Oct 30 02:55:33.233968 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:33.233731    4461 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gsgwj\" (UniqueName: \"kubernetes.io/projected/73752bb9-2a14-4582-a6d6-c1fd26524c3d-kube-api-access-gsgwj\") pod \"ingress-canary-vwbj7\" (UID: \"73752bb9-2a14-4582-a6d6-c1fd26524c3d\") " pod="openshift-ingress-canary/ingress-canary-vwbj7"
Oct 30 02:55:33.233968 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:33.233831    4461 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p764l\" (UniqueName: \"kubernetes.io/projected/0e6f015d-0a2d-4579-ad4d-a88b8c11c3b9-kube-api-access-p764l\") pod \"node-resolver-zhb7b\" (UID: \"0e6f015d-0a2d-4579-ad4d-a88b8c11c3b9\") " pod="openshift-dns/node-resolver-zhb7b"
Oct 30 02:55:33.233968 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:33.233865    4461 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/91a29ddf-25ca-40a8-b920-6d894360a050-registration-dir\") pod \"aws-ebs-csi-driver-node-gqpp2\" (UID: \"91a29ddf-25ca-40a8-b920-6d894360a050\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gqpp2"
Oct 30 02:55:33.233968 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:33.233906    4461 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lrvbf\" (UniqueName: \"kubernetes.io/projected/f425530a-8431-4c5b-8335-8cc1119745c1-kube-api-access-lrvbf\") pod \"ovnkube-node-g6nqm\" (UID: \"f425530a-8431-4c5b-8335-8cc1119745c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-g6nqm"
Oct 30 02:55:33.233968 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:33.233910    4461 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/73752bb9-2a14-4582-a6d6-c1fd26524c3d-cert\") pod \"ingress-canary-vwbj7\" (UID: \"73752bb9-2a14-4582-a6d6-c1fd26524c3d\") " pod="openshift-ingress-canary/ingress-canary-vwbj7"
Oct 30 02:55:33.233968 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:33.233945    4461 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b4ce92fa-1612-4e46-ad6e-23049b2e279b-system-cni-dir\") pod \"multus-9qcgk\" (UID: \"b4ce92fa-1612-4e46-ad6e-23049b2e279b\") " pod="openshift-multus/multus-9qcgk"
Oct 30 02:55:33.234143 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:33.234063    4461 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b4ce92fa-1612-4e46-ad6e-23049b2e279b-cnibin\") pod \"multus-9qcgk\" (UID: \"b4ce92fa-1612-4e46-ad6e-23049b2e279b\") " pod="openshift-multus/multus-9qcgk"
Oct 30 02:55:33.234143 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:33.234106    4461 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3e4730f0-d77e-4bff-b8af-b7fafd4525d8-etc-kubernetes\") pod \"tuned-h7cm4\" (UID: \"3e4730f0-d77e-4bff-b8af-b7fafd4525d8\") " pod="openshift-cluster-node-tuning-operator/tuned-h7cm4"
Oct 30 02:55:33.234143 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:33.234133    4461 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/98f9efe3-a732-4ba4-8543-941633d22170-dbus\") pod \"global-pull-secret-syncer-vbnwt\" (UID: \"98f9efe3-a732-4ba4-8543-941633d22170\") " pod="kube-system/global-pull-secret-syncer-vbnwt"
Oct 30 02:55:33.234257 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:33.234158    4461 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/eee29ef7-4ad4-4ff9-8cb0-9d2a5aaabba3-data-volume\") pod \"insights-runtime-extractor-2k85b\" (UID: \"eee29ef7-4ad4-4ff9-8cb0-9d2a5aaabba3\") " pod="openshift-insights/insights-runtime-extractor-2k85b"
Oct 30 02:55:33.234257 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:33.234220    4461 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/eee29ef7-4ad4-4ff9-8cb0-9d2a5aaabba3-data-volume\") pod \"insights-runtime-extractor-2k85b\" (UID: \"eee29ef7-4ad4-4ff9-8cb0-9d2a5aaabba3\") " pod="openshift-insights/insights-runtime-extractor-2k85b"
Oct 30 02:55:33.234257 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:33.234233    4461 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/4a776e60-80b6-4ce0-921c-1884973b4d0b-signing-key\") pod \"service-ca-5fffc97bdf-rwq7h\" (UID: \"4a776e60-80b6-4ce0-921c-1884973b4d0b\") " pod="openshift-service-ca/service-ca-5fffc97bdf-rwq7h"
Oct 30 02:55:33.234333 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:33.234264    4461 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/4d8bf700-9e09-4741-954b-aba9ccd331c9-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-9l7k8\" (UID: \"4d8bf700-9e09-4741-954b-aba9ccd331c9\") " pod="openshift-multus/multus-additional-cni-plugins-9l7k8"
Oct 30 02:55:33.234333 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:33.234290    4461 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f425530a-8431-4c5b-8335-8cc1119745c1-var-lib-openvswitch\") pod \"ovnkube-node-g6nqm\" (UID: \"f425530a-8431-4c5b-8335-8cc1119745c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-g6nqm"
Oct 30 02:55:33.234333 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:33.234315    4461 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/eee29ef7-4ad4-4ff9-8cb0-9d2a5aaabba3-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-2k85b\" (UID: \"eee29ef7-4ad4-4ff9-8cb0-9d2a5aaabba3\") " pod="openshift-insights/insights-runtime-extractor-2k85b"
Oct 30 02:55:33.234413 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:33.234340    4461 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/91a29ddf-25ca-40a8-b920-6d894360a050-etc-selinux\") pod \"aws-ebs-csi-driver-node-gqpp2\" (UID: \"91a29ddf-25ca-40a8-b920-6d894360a050\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gqpp2"
Oct 30 02:55:33.234441 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:33.234431    4461 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/4a776e60-80b6-4ce0-921c-1884973b4d0b-signing-key\") pod \"service-ca-5fffc97bdf-rwq7h\" (UID: \"4a776e60-80b6-4ce0-921c-1884973b4d0b\") " pod="openshift-service-ca/service-ca-5fffc97bdf-rwq7h"
Oct 30 02:55:33.234509 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:33.234495    4461 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/042af239-396c-4dae-8cf2-0c562af07c62-host-slash\") pod \"iptables-alerter-jhq2g\" (UID: \"042af239-396c-4dae-8cf2-0c562af07c62\") " pod="openshift-network-operator/iptables-alerter-jhq2g"
Oct 30 02:55:33.234542 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:33.234519    4461 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3bd2c646-cebb-4e8a-b6c3-8b5b74049c11-metrics-certs\") pod \"network-metrics-daemon-mzk2s\" (UID: \"3bd2c646-cebb-4e8a-b6c3-8b5b74049c11\") " pod="openshift-multus/network-metrics-daemon-mzk2s"
Oct 30 02:55:33.234542 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:33.234509    4461 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/4d8bf700-9e09-4741-954b-aba9ccd331c9-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-9l7k8\" (UID: \"4d8bf700-9e09-4741-954b-aba9ccd331c9\") " pod="openshift-multus/multus-additional-cni-plugins-9l7k8"
Oct 30 02:55:33.234594 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:33.234543    4461 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dkmxs\" (UniqueName: \"kubernetes.io/projected/3bd2c646-cebb-4e8a-b6c3-8b5b74049c11-kube-api-access-dkmxs\") pod \"network-metrics-daemon-mzk2s\" (UID: \"3bd2c646-cebb-4e8a-b6c3-8b5b74049c11\") " pod="openshift-multus/network-metrics-daemon-mzk2s"
Oct 30 02:55:33.234594 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:33.234569    4461 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gwcwf\" (UniqueName: \"kubernetes.io/projected/5f9fb000-9531-41f1-b3cf-7e79ca1a24d7-kube-api-access-gwcwf\") pod \"dns-default-wjll7\" (UID: \"5f9fb000-9531-41f1-b3cf-7e79ca1a24d7\") " pod="openshift-dns/dns-default-wjll7"
Oct 30 02:55:33.234658 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:33.234594    4461 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rkgpj\" (UniqueName: \"kubernetes.io/projected/f990bb17-3b0f-4bca-8939-089f79d6edef-kube-api-access-rkgpj\") pod \"migrator-66df5d496b-wx7pd\" (UID: \"f990bb17-3b0f-4bca-8939-089f79d6edef\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-wx7pd"
Oct 30 02:55:33.234658 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:33.234610    4461 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/b4ce92fa-1612-4e46-ad6e-23049b2e279b-hostroot\") pod \"multus-9qcgk\" (UID: \"b4ce92fa-1612-4e46-ad6e-23049b2e279b\") " pod="openshift-multus/multus-9qcgk"
Oct 30 02:55:33.234658 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:33.234634    4461 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b4ce92fa-1612-4e46-ad6e-23049b2e279b-multus-conf-dir\") pod \"multus-9qcgk\" (UID: \"b4ce92fa-1612-4e46-ad6e-23049b2e279b\") " pod="openshift-multus/multus-9qcgk"
Oct 30 02:55:33.234774 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:33.234753    4461 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wmgr2\" (UniqueName: \"kubernetes.io/projected/b4ce92fa-1612-4e46-ad6e-23049b2e279b-kube-api-access-wmgr2\") pod \"multus-9qcgk\" (UID: \"b4ce92fa-1612-4e46-ad6e-23049b2e279b\") " pod="openshift-multus/multus-9qcgk"
Oct 30 02:55:33.234817 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:33.234783    4461 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/3e4730f0-d77e-4bff-b8af-b7fafd4525d8-etc-sysconfig\") pod \"tuned-h7cm4\" (UID: \"3e4730f0-d77e-4bff-b8af-b7fafd4525d8\") " pod="openshift-cluster-node-tuning-operator/tuned-h7cm4"
Oct 30 02:55:33.234851 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:33.234836    4461 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-62vqf\" (UniqueName: \"kubernetes.io/projected/75d9e1ec-cd1b-4aba-b911-5a7465c653cf-kube-api-access-62vqf\") pod \"network-check-target-7rmdj\" (UID: \"75d9e1ec-cd1b-4aba-b911-5a7465c653cf\") " pod="openshift-network-diagnostics/network-check-target-7rmdj"
Oct 30 02:55:33.234974 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:33.234956    4461 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/487caf0f-5f2a-4e93-9c06-de9af02c470b-serviceca\") pod \"node-ca-9mddj\" (UID: \"487caf0f-5f2a-4e93-9c06-de9af02c470b\") " pod="openshift-image-registry/node-ca-9mddj"
Oct 30 02:55:33.235003 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:33.234990    4461 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/5f9fb000-9531-41f1-b3cf-7e79ca1a24d7-metrics-tls\") pod \"dns-default-wjll7\" (UID: \"5f9fb000-9531-41f1-b3cf-7e79ca1a24d7\") " pod="openshift-dns/dns-default-wjll7"
Oct 30 02:55:33.235038 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:33.235016    4461 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b4ce92fa-1612-4e46-ad6e-23049b2e279b-host-run-netns\") pod \"multus-9qcgk\" (UID: \"b4ce92fa-1612-4e46-ad6e-23049b2e279b\") " pod="openshift-multus/multus-9qcgk"
Oct 30 02:55:33.235076 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:33.235062    4461 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f425530a-8431-4c5b-8335-8cc1119745c1-host-run-ovn-kubernetes\") pod \"ovnkube-node-g6nqm\" (UID: \"f425530a-8431-4c5b-8335-8cc1119745c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-g6nqm"
Oct 30 02:55:33.235158 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:33.235142    4461 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f425530a-8431-4c5b-8335-8cc1119745c1-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-g6nqm\" (UID: \"f425530a-8431-4c5b-8335-8cc1119745c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-g6nqm"
Oct 30 02:55:33.235192 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:33.235174    4461 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/f425530a-8431-4c5b-8335-8cc1119745c1-ovn-node-metrics-cert\") pod \"ovnkube-node-g6nqm\" (UID: \"f425530a-8431-4c5b-8335-8cc1119745c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-g6nqm"
Oct 30 02:55:33.235233 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:33.235199    4461 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-267l7\" (UniqueName: \"kubernetes.io/projected/042af239-396c-4dae-8cf2-0c562af07c62-kube-api-access-267l7\") pod \"iptables-alerter-jhq2g\" (UID: \"042af239-396c-4dae-8cf2-0c562af07c62\") " pod="openshift-network-operator/iptables-alerter-jhq2g"
Oct 30 02:55:33.235328 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:33.235311    4461 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/3e4730f0-d77e-4bff-b8af-b7fafd4525d8-lib-modules\") pod \"tuned-h7cm4\" (UID: \"3e4730f0-d77e-4bff-b8af-b7fafd4525d8\") " pod="openshift-cluster-node-tuning-operator/tuned-h7cm4"
Oct 30 02:55:33.235378 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:33.235335    4461 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/91a29ddf-25ca-40a8-b920-6d894360a050-sys-fs\") pod \"aws-ebs-csi-driver-node-gqpp2\" (UID: \"91a29ddf-25ca-40a8-b920-6d894360a050\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gqpp2"
Oct 30 02:55:33.235378 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:33.235360    4461 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b4ce92fa-1612-4e46-ad6e-23049b2e279b-cni-binary-copy\") pod \"multus-9qcgk\" (UID: \"b4ce92fa-1612-4e46-ad6e-23049b2e279b\") " pod="openshift-multus/multus-9qcgk"
Oct 30 02:55:33.235476 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:33.235436    4461 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/3e4730f0-d77e-4bff-b8af-b7fafd4525d8-etc-systemd\") pod \"tuned-h7cm4\" (UID: \"3e4730f0-d77e-4bff-b8af-b7fafd4525d8\") " pod="openshift-cluster-node-tuning-operator/tuned-h7cm4"
Oct 30 02:55:33.235537 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:33.235475    4461 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/eee29ef7-4ad4-4ff9-8cb0-9d2a5aaabba3-crio-socket\") pod \"insights-runtime-extractor-2k85b\" (UID: \"eee29ef7-4ad4-4ff9-8cb0-9d2a5aaabba3\") " pod="openshift-insights/insights-runtime-extractor-2k85b"
Oct 30 02:55:33.235537 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:33.235506    4461 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0e6f015d-0a2d-4579-ad4d-a88b8c11c3b9-tmp-dir\") pod \"node-resolver-zhb7b\" (UID: \"0e6f015d-0a2d-4579-ad4d-a88b8c11c3b9\") " pod="openshift-dns/node-resolver-zhb7b"
Oct 30 02:55:33.235537 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:33.235522    4461 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b4ce92fa-1612-4e46-ad6e-23049b2e279b-cni-binary-copy\") pod \"multus-9qcgk\" (UID: \"b4ce92fa-1612-4e46-ad6e-23049b2e279b\") " pod="openshift-multus/multus-9qcgk"
Oct 30 02:55:33.235670 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:33.235540    4461 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/4d8bf700-9e09-4741-954b-aba9ccd331c9-tuning-conf-dir\") pod \"multus-additional-cni-plugins-9l7k8\" (UID: \"4d8bf700-9e09-4741-954b-aba9ccd331c9\") " pod="openshift-multus/multus-additional-cni-plugins-9l7k8"
Oct 30 02:55:33.235670 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:33.235567    4461 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f425530a-8431-4c5b-8335-8cc1119745c1-host-slash\") pod \"ovnkube-node-g6nqm\" (UID: \"f425530a-8431-4c5b-8335-8cc1119745c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-g6nqm"
Oct 30 02:55:33.235670 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:33.235579    4461 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0e6f015d-0a2d-4579-ad4d-a88b8c11c3b9-tmp-dir\") pod \"node-resolver-zhb7b\" (UID: \"0e6f015d-0a2d-4579-ad4d-a88b8c11c3b9\") " pod="openshift-dns/node-resolver-zhb7b"
Oct 30 02:55:33.235670 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:33.235592    4461 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/f425530a-8431-4c5b-8335-8cc1119745c1-ovn-node-metrics-cert\") pod \"ovnkube-node-g6nqm\" (UID: \"f425530a-8431-4c5b-8335-8cc1119745c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-g6nqm"
Oct 30 02:55:33.235774 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:33.235692    4461 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/487caf0f-5f2a-4e93-9c06-de9af02c470b-serviceca\") pod \"node-ca-9mddj\" (UID: \"487caf0f-5f2a-4e93-9c06-de9af02c470b\") " pod="openshift-image-registry/node-ca-9mddj"
Oct 30 02:55:33.256212 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:33.256192    4461 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g55cz\" (UniqueName: \"kubernetes.io/projected/91a29ddf-25ca-40a8-b920-6d894360a050-kube-api-access-g55cz\") pod \"aws-ebs-csi-driver-node-gqpp2\" (UID: \"91a29ddf-25ca-40a8-b920-6d894360a050\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gqpp2"
Oct 30 02:55:33.256369 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:33.256338    4461 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lrvbf\" (UniqueName: \"kubernetes.io/projected/f425530a-8431-4c5b-8335-8cc1119745c1-kube-api-access-lrvbf\") pod \"ovnkube-node-g6nqm\" (UID: \"f425530a-8431-4c5b-8335-8cc1119745c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-g6nqm"
Oct 30 02:55:33.256711 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:33.256687    4461 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gwcwf\" (UniqueName: \"kubernetes.io/projected/5f9fb000-9531-41f1-b3cf-7e79ca1a24d7-kube-api-access-gwcwf\") pod \"dns-default-wjll7\" (UID: \"5f9fb000-9531-41f1-b3cf-7e79ca1a24d7\") " pod="openshift-dns/dns-default-wjll7"
Oct 30 02:55:33.256849 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:33.256828    4461 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cxnjk\" (UniqueName: \"kubernetes.io/projected/487caf0f-5f2a-4e93-9c06-de9af02c470b-kube-api-access-cxnjk\") pod \"node-ca-9mddj\" (UID: \"487caf0f-5f2a-4e93-9c06-de9af02c470b\") " pod="openshift-image-registry/node-ca-9mddj"
Oct 30 02:55:33.256902 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:33.256839    4461 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wd8s8\" (UniqueName: \"kubernetes.io/projected/eee29ef7-4ad4-4ff9-8cb0-9d2a5aaabba3-kube-api-access-wd8s8\") pod \"insights-runtime-extractor-2k85b\" (UID: \"eee29ef7-4ad4-4ff9-8cb0-9d2a5aaabba3\") " pod="openshift-insights/insights-runtime-extractor-2k85b"
Oct 30 02:55:33.256940 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:33.256909    4461 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xwjvv\" (UniqueName: \"kubernetes.io/projected/3e4730f0-d77e-4bff-b8af-b7fafd4525d8-kube-api-access-xwjvv\") pod \"tuned-h7cm4\" (UID: \"3e4730f0-d77e-4bff-b8af-b7fafd4525d8\") " pod="openshift-cluster-node-tuning-operator/tuned-h7cm4"
Oct 30 02:55:33.256940 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:33.256911    4461 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pfgdh\" (UniqueName: \"kubernetes.io/projected/4d8bf700-9e09-4741-954b-aba9ccd331c9-kube-api-access-pfgdh\") pod \"multus-additional-cni-plugins-9l7k8\" (UID: \"4d8bf700-9e09-4741-954b-aba9ccd331c9\") " pod="openshift-multus/multus-additional-cni-plugins-9l7k8"
Oct 30 02:55:33.257124 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:33.257089    4461 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2289h\" (UniqueName: \"kubernetes.io/projected/4a776e60-80b6-4ce0-921c-1884973b4d0b-kube-api-access-2289h\") pod \"service-ca-5fffc97bdf-rwq7h\" (UID: \"4a776e60-80b6-4ce0-921c-1884973b4d0b\") " pod="openshift-service-ca/service-ca-5fffc97bdf-rwq7h"
Oct 30 02:55:33.261463 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:33.261444    4461 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-267l7\" (UniqueName: \"kubernetes.io/projected/042af239-396c-4dae-8cf2-0c562af07c62-kube-api-access-267l7\") pod \"iptables-alerter-jhq2g\" (UID: \"042af239-396c-4dae-8cf2-0c562af07c62\") " pod="openshift-network-operator/iptables-alerter-jhq2g"
Oct 30 02:55:33.264727 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:33.262051    4461 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-62vqf\" (UniqueName: \"kubernetes.io/projected/75d9e1ec-cd1b-4aba-b911-5a7465c653cf-kube-api-access-62vqf\") pod \"network-check-target-7rmdj\" (UID: \"75d9e1ec-cd1b-4aba-b911-5a7465c653cf\") " pod="openshift-network-diagnostics/network-check-target-7rmdj"
Oct 30 02:55:33.264727 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:33.262228    4461 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dkmxs\" (UniqueName: \"kubernetes.io/projected/3bd2c646-cebb-4e8a-b6c3-8b5b74049c11-kube-api-access-dkmxs\") pod \"network-metrics-daemon-mzk2s\" (UID: \"3bd2c646-cebb-4e8a-b6c3-8b5b74049c11\") " pod="openshift-multus/network-metrics-daemon-mzk2s"
Oct 30 02:55:33.264727 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:33.262314    4461 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gsgwj\" (UniqueName: \"kubernetes.io/projected/73752bb9-2a14-4582-a6d6-c1fd26524c3d-kube-api-access-gsgwj\") pod \"ingress-canary-vwbj7\" (UID: \"73752bb9-2a14-4582-a6d6-c1fd26524c3d\") " pod="openshift-ingress-canary/ingress-canary-vwbj7"
Oct 30 02:55:33.264727 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:33.262346    4461 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rkgpj\" (UniqueName: \"kubernetes.io/projected/f990bb17-3b0f-4bca-8939-089f79d6edef-kube-api-access-rkgpj\") pod \"migrator-66df5d496b-wx7pd\" (UID: \"f990bb17-3b0f-4bca-8939-089f79d6edef\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-wx7pd"
Oct 30 02:55:33.264727 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:33.262358    4461 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wmgr2\" (UniqueName: \"kubernetes.io/projected/b4ce92fa-1612-4e46-ad6e-23049b2e279b-kube-api-access-wmgr2\") pod \"multus-9qcgk\" (UID: \"b4ce92fa-1612-4e46-ad6e-23049b2e279b\") " pod="openshift-multus/multus-9qcgk"
Oct 30 02:55:33.264727 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:33.262526    4461 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p764l\" (UniqueName: \"kubernetes.io/projected/0e6f015d-0a2d-4579-ad4d-a88b8c11c3b9-kube-api-access-p764l\") pod \"node-resolver-zhb7b\" (UID: \"0e6f015d-0a2d-4579-ad4d-a88b8c11c3b9\") " pod="openshift-dns/node-resolver-zhb7b"
Oct 30 02:55:33.336376 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:33.336350    4461 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f425530a-8431-4c5b-8335-8cc1119745c1-var-lib-openvswitch\") pod \"ovnkube-node-g6nqm\" (UID: \"f425530a-8431-4c5b-8335-8cc1119745c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-g6nqm"
Oct 30 02:55:33.336450 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:33.336378    4461 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/eee29ef7-4ad4-4ff9-8cb0-9d2a5aaabba3-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-2k85b\" (UID: \"eee29ef7-4ad4-4ff9-8cb0-9d2a5aaabba3\") " pod="openshift-insights/insights-runtime-extractor-2k85b"
Oct 30 02:55:33.336450 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:33.336394    4461 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/91a29ddf-25ca-40a8-b920-6d894360a050-etc-selinux\") pod \"aws-ebs-csi-driver-node-gqpp2\" (UID: \"91a29ddf-25ca-40a8-b920-6d894360a050\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gqpp2"
Oct 30 02:55:33.336450 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:33.336408    4461 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/042af239-396c-4dae-8cf2-0c562af07c62-host-slash\") pod \"iptables-alerter-jhq2g\" (UID: \"042af239-396c-4dae-8cf2-0c562af07c62\") " pod="openshift-network-operator/iptables-alerter-jhq2g"
Oct 30 02:55:33.336450 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:33.336429    4461 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3bd2c646-cebb-4e8a-b6c3-8b5b74049c11-metrics-certs\") pod \"network-metrics-daemon-mzk2s\" (UID: \"3bd2c646-cebb-4e8a-b6c3-8b5b74049c11\") " pod="openshift-multus/network-metrics-daemon-mzk2s"
Oct 30 02:55:33.336617 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:33.336456    4461 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/b4ce92fa-1612-4e46-ad6e-23049b2e279b-hostroot\") pod \"multus-9qcgk\" (UID: \"b4ce92fa-1612-4e46-ad6e-23049b2e279b\") " pod="openshift-multus/multus-9qcgk"
Oct 30 02:55:33.336617 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:33.336476    4461 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f425530a-8431-4c5b-8335-8cc1119745c1-var-lib-openvswitch\") pod \"ovnkube-node-g6nqm\" (UID: \"f425530a-8431-4c5b-8335-8cc1119745c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-g6nqm"
Oct 30 02:55:33.336617 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:33.336478    4461 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b4ce92fa-1612-4e46-ad6e-23049b2e279b-multus-conf-dir\") pod \"multus-9qcgk\" (UID: \"b4ce92fa-1612-4e46-ad6e-23049b2e279b\") " pod="openshift-multus/multus-9qcgk"
Oct 30 02:55:33.336617 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:33.336523    4461 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/b4ce92fa-1612-4e46-ad6e-23049b2e279b-hostroot\") pod \"multus-9qcgk\" (UID: \"b4ce92fa-1612-4e46-ad6e-23049b2e279b\") " pod="openshift-multus/multus-9qcgk"
Oct 30 02:55:33.336617 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:33.336527    4461 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/3e4730f0-d77e-4bff-b8af-b7fafd4525d8-etc-sysconfig\") pod \"tuned-h7cm4\" (UID: \"3e4730f0-d77e-4bff-b8af-b7fafd4525d8\") " pod="openshift-cluster-node-tuning-operator/tuned-h7cm4"
Oct 30 02:55:33.336617 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:33.336514    4461 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/91a29ddf-25ca-40a8-b920-6d894360a050-etc-selinux\") pod \"aws-ebs-csi-driver-node-gqpp2\" (UID: \"91a29ddf-25ca-40a8-b920-6d894360a050\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gqpp2"
Oct 30 02:55:33.336617 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:33.336555    4461 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/5f9fb000-9531-41f1-b3cf-7e79ca1a24d7-metrics-tls\") pod \"dns-default-wjll7\" (UID: \"5f9fb000-9531-41f1-b3cf-7e79ca1a24d7\") " pod="openshift-dns/dns-default-wjll7"
Oct 30 02:55:33.336617 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:33.336573    4461 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b4ce92fa-1612-4e46-ad6e-23049b2e279b-host-run-netns\") pod \"multus-9qcgk\" (UID: \"b4ce92fa-1612-4e46-ad6e-23049b2e279b\") " pod="openshift-multus/multus-9qcgk"
Oct 30 02:55:33.336617 ip-10-0-143-156 kubenswrapper[4461]: E1030 02:55:33.336556    4461 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Oct 30 02:55:33.336617 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:33.336588    4461 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f425530a-8431-4c5b-8335-8cc1119745c1-host-run-ovn-kubernetes\") pod \"ovnkube-node-g6nqm\" (UID: \"f425530a-8431-4c5b-8335-8cc1119745c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-g6nqm"
Oct 30 02:55:33.336617 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:33.336611    4461 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b4ce92fa-1612-4e46-ad6e-23049b2e279b-host-run-netns\") pod \"multus-9qcgk\" (UID: \"b4ce92fa-1612-4e46-ad6e-23049b2e279b\") " pod="openshift-multus/multus-9qcgk"
Oct 30 02:55:33.337025 ip-10-0-143-156 kubenswrapper[4461]: E1030 02:55:33.336629    4461 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 30 02:55:33.337025 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:33.336635    4461 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b4ce92fa-1612-4e46-ad6e-23049b2e279b-multus-conf-dir\") pod \"multus-9qcgk\" (UID: \"b4ce92fa-1612-4e46-ad6e-23049b2e279b\") " pod="openshift-multus/multus-9qcgk"
Oct 30 02:55:33.337025 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:33.336634    4461 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/3e4730f0-d77e-4bff-b8af-b7fafd4525d8-etc-sysconfig\") pod \"tuned-h7cm4\" (UID: \"3e4730f0-d77e-4bff-b8af-b7fafd4525d8\") " pod="openshift-cluster-node-tuning-operator/tuned-h7cm4"
Oct 30 02:55:33.337025 ip-10-0-143-156 kubenswrapper[4461]: E1030 02:55:33.336662    4461 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3bd2c646-cebb-4e8a-b6c3-8b5b74049c11-metrics-certs podName:3bd2c646-cebb-4e8a-b6c3-8b5b74049c11 nodeName:}" failed. No retries permitted until 2026-10-30 02:55:33.836639353 +0000 UTC m=+1.886444529 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3bd2c646-cebb-4e8a-b6c3-8b5b74049c11-metrics-certs") pod "network-metrics-daemon-mzk2s" (UID: "3bd2c646-cebb-4e8a-b6c3-8b5b74049c11") : secret "metrics-daemon-secret" not found
Oct 30 02:55:33.337025 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:33.336661    4461 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f425530a-8431-4c5b-8335-8cc1119745c1-host-run-ovn-kubernetes\") pod \"ovnkube-node-g6nqm\" (UID: \"f425530a-8431-4c5b-8335-8cc1119745c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-g6nqm"
Oct 30 02:55:33.337025 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:33.336571    4461 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/042af239-396c-4dae-8cf2-0c562af07c62-host-slash\") pod \"iptables-alerter-jhq2g\" (UID: \"042af239-396c-4dae-8cf2-0c562af07c62\") " pod="openshift-network-operator/iptables-alerter-jhq2g"
Oct 30 02:55:33.337025 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:33.336611    4461 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f425530a-8431-4c5b-8335-8cc1119745c1-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-g6nqm\" (UID: \"f425530a-8431-4c5b-8335-8cc1119745c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-g6nqm"
Oct 30 02:55:33.337025 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:33.336643    4461 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f425530a-8431-4c5b-8335-8cc1119745c1-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-g6nqm\" (UID: \"f425530a-8431-4c5b-8335-8cc1119745c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-g6nqm"
Oct 30 02:55:33.337025 ip-10-0-143-156 kubenswrapper[4461]: E1030 02:55:33.336692    4461 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5f9fb000-9531-41f1-b3cf-7e79ca1a24d7-metrics-tls podName:5f9fb000-9531-41f1-b3cf-7e79ca1a24d7 nodeName:}" failed. No retries permitted until 2026-10-30 02:55:33.83667458 +0000 UTC m=+1.886479748 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/5f9fb000-9531-41f1-b3cf-7e79ca1a24d7-metrics-tls") pod "dns-default-wjll7" (UID: "5f9fb000-9531-41f1-b3cf-7e79ca1a24d7") : secret "dns-default-metrics-tls" not found
Oct 30 02:55:33.337025 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:33.336755    4461 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/3e4730f0-d77e-4bff-b8af-b7fafd4525d8-lib-modules\") pod \"tuned-h7cm4\" (UID: \"3e4730f0-d77e-4bff-b8af-b7fafd4525d8\") " pod="openshift-cluster-node-tuning-operator/tuned-h7cm4"
Oct 30 02:55:33.337025 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:33.336775    4461 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/91a29ddf-25ca-40a8-b920-6d894360a050-sys-fs\") pod \"aws-ebs-csi-driver-node-gqpp2\" (UID: \"91a29ddf-25ca-40a8-b920-6d894360a050\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gqpp2"
Oct 30 02:55:33.337025 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:33.336800    4461 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/3e4730f0-d77e-4bff-b8af-b7fafd4525d8-etc-systemd\") pod \"tuned-h7cm4\" (UID: \"3e4730f0-d77e-4bff-b8af-b7fafd4525d8\") " pod="openshift-cluster-node-tuning-operator/tuned-h7cm4"
Oct 30 02:55:33.337025 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:33.336826    4461 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/eee29ef7-4ad4-4ff9-8cb0-9d2a5aaabba3-crio-socket\") pod \"insights-runtime-extractor-2k85b\" (UID: \"eee29ef7-4ad4-4ff9-8cb0-9d2a5aaabba3\") " pod="openshift-insights/insights-runtime-extractor-2k85b"
Oct 30 02:55:33.337025 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:33.336848    4461 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/91a29ddf-25ca-40a8-b920-6d894360a050-sys-fs\") pod \"aws-ebs-csi-driver-node-gqpp2\" (UID: \"91a29ddf-25ca-40a8-b920-6d894360a050\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gqpp2"
Oct 30 02:55:33.337025 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:33.336852    4461 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/4d8bf700-9e09-4741-954b-aba9ccd331c9-tuning-conf-dir\") pod \"multus-additional-cni-plugins-9l7k8\" (UID: \"4d8bf700-9e09-4741-954b-aba9ccd331c9\") " pod="openshift-multus/multus-additional-cni-plugins-9l7k8"
Oct 30 02:55:33.337025 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:33.336861    4461 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/3e4730f0-d77e-4bff-b8af-b7fafd4525d8-etc-systemd\") pod \"tuned-h7cm4\" (UID: \"3e4730f0-d77e-4bff-b8af-b7fafd4525d8\") " pod="openshift-cluster-node-tuning-operator/tuned-h7cm4"
Oct 30 02:55:33.337025 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:33.336887    4461 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f425530a-8431-4c5b-8335-8cc1119745c1-host-slash\") pod \"ovnkube-node-g6nqm\" (UID: \"f425530a-8431-4c5b-8335-8cc1119745c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-g6nqm"
Oct 30 02:55:33.337025 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:33.336901    4461 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/3e4730f0-d77e-4bff-b8af-b7fafd4525d8-lib-modules\") pod \"tuned-h7cm4\" (UID: \"3e4730f0-d77e-4bff-b8af-b7fafd4525d8\") " pod="openshift-cluster-node-tuning-operator/tuned-h7cm4"
Oct 30 02:55:33.337025 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:33.336902    4461 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/4d8bf700-9e09-4741-954b-aba9ccd331c9-tuning-conf-dir\") pod \"multus-additional-cni-plugins-9l7k8\" (UID: \"4d8bf700-9e09-4741-954b-aba9ccd331c9\") " pod="openshift-multus/multus-additional-cni-plugins-9l7k8"
Oct 30 02:55:33.337025 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:33.336915    4461 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/eee29ef7-4ad4-4ff9-8cb0-9d2a5aaabba3-crio-socket\") pod \"insights-runtime-extractor-2k85b\" (UID: \"eee29ef7-4ad4-4ff9-8cb0-9d2a5aaabba3\") " pod="openshift-insights/insights-runtime-extractor-2k85b"
Oct 30 02:55:33.337025 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:33.336925    4461 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/eee29ef7-4ad4-4ff9-8cb0-9d2a5aaabba3-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-2k85b\" (UID: \"eee29ef7-4ad4-4ff9-8cb0-9d2a5aaabba3\") " pod="openshift-insights/insights-runtime-extractor-2k85b"
Oct 30 02:55:33.337025 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:33.336920    4461 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f425530a-8431-4c5b-8335-8cc1119745c1-host-slash\") pod \"ovnkube-node-g6nqm\" (UID: \"f425530a-8431-4c5b-8335-8cc1119745c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-g6nqm"
Oct 30 02:55:33.337025 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:33.336937    4461 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/b4ce92fa-1612-4e46-ad6e-23049b2e279b-host-run-multus-certs\") pod \"multus-9qcgk\" (UID: \"b4ce92fa-1612-4e46-ad6e-23049b2e279b\") " pod="openshift-multus/multus-9qcgk"
Oct 30 02:55:33.337025 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:33.336980    4461 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/f425530a-8431-4c5b-8335-8cc1119745c1-systemd-units\") pod \"ovnkube-node-g6nqm\" (UID: \"f425530a-8431-4c5b-8335-8cc1119745c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-g6nqm"
Oct 30 02:55:33.337025 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:33.337005    4461 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/b4ce92fa-1612-4e46-ad6e-23049b2e279b-host-run-multus-certs\") pod \"multus-9qcgk\" (UID: \"b4ce92fa-1612-4e46-ad6e-23049b2e279b\") " pod="openshift-multus/multus-9qcgk"
Oct 30 02:55:33.337025 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:33.337005    4461 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f425530a-8431-4c5b-8335-8cc1119745c1-run-openvswitch\") pod \"ovnkube-node-g6nqm\" (UID: \"f425530a-8431-4c5b-8335-8cc1119745c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-g6nqm"
Oct 30 02:55:33.337025 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:33.337022    4461 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/f425530a-8431-4c5b-8335-8cc1119745c1-systemd-units\") pod \"ovnkube-node-g6nqm\" (UID: \"f425530a-8431-4c5b-8335-8cc1119745c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-g6nqm"
Oct 30 02:55:33.337025 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:33.337036    4461 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f425530a-8431-4c5b-8335-8cc1119745c1-run-openvswitch\") pod \"ovnkube-node-g6nqm\" (UID: \"f425530a-8431-4c5b-8335-8cc1119745c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-g6nqm"
Oct 30 02:55:33.337025 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:33.337037    4461 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/f425530a-8431-4c5b-8335-8cc1119745c1-run-ovn\") pod \"ovnkube-node-g6nqm\" (UID: \"f425530a-8431-4c5b-8335-8cc1119745c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-g6nqm"
Oct 30 02:55:33.338179 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:33.337064    4461 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/3e4730f0-d77e-4bff-b8af-b7fafd4525d8-run\") pod \"tuned-h7cm4\" (UID: \"3e4730f0-d77e-4bff-b8af-b7fafd4525d8\") " pod="openshift-cluster-node-tuning-operator/tuned-h7cm4"
Oct 30 02:55:33.338179 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:33.337073    4461 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/f425530a-8431-4c5b-8335-8cc1119745c1-run-ovn\") pod \"ovnkube-node-g6nqm\" (UID: \"f425530a-8431-4c5b-8335-8cc1119745c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-g6nqm"
Oct 30 02:55:33.338179 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:33.337078    4461 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3e4730f0-d77e-4bff-b8af-b7fafd4525d8-var-lib-kubelet\") pod \"tuned-h7cm4\" (UID: \"3e4730f0-d77e-4bff-b8af-b7fafd4525d8\") " pod="openshift-cluster-node-tuning-operator/tuned-h7cm4"
Oct 30 02:55:33.338179 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:33.337128    4461 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/3e4730f0-d77e-4bff-b8af-b7fafd4525d8-run\") pod \"tuned-h7cm4\" (UID: \"3e4730f0-d77e-4bff-b8af-b7fafd4525d8\") " pod="openshift-cluster-node-tuning-operator/tuned-h7cm4"
Oct 30 02:55:33.338179 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:33.337132    4461 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/3e4730f0-d77e-4bff-b8af-b7fafd4525d8-etc-sysctl-conf\") pod \"tuned-h7cm4\" (UID: \"3e4730f0-d77e-4bff-b8af-b7fafd4525d8\") " pod="openshift-cluster-node-tuning-operator/tuned-h7cm4"
Oct 30 02:55:33.338179 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:33.337144    4461 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3e4730f0-d77e-4bff-b8af-b7fafd4525d8-var-lib-kubelet\") pod \"tuned-h7cm4\" (UID: \"3e4730f0-d77e-4bff-b8af-b7fafd4525d8\") " pod="openshift-cluster-node-tuning-operator/tuned-h7cm4"
Oct 30 02:55:33.338179 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:33.337157    4461 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/0e6f015d-0a2d-4579-ad4d-a88b8c11c3b9-hosts-file\") pod \"node-resolver-zhb7b\" (UID: \"0e6f015d-0a2d-4579-ad4d-a88b8c11c3b9\") " pod="openshift-dns/node-resolver-zhb7b"
Oct 30 02:55:33.338179 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:33.337172    4461 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/91a29ddf-25ca-40a8-b920-6d894360a050-kubelet-dir\") pod \"aws-ebs-csi-driver-node-gqpp2\" (UID: \"91a29ddf-25ca-40a8-b920-6d894360a050\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gqpp2"
Oct 30 02:55:33.338179 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:33.337187    4461 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/91a29ddf-25ca-40a8-b920-6d894360a050-socket-dir\") pod \"aws-ebs-csi-driver-node-gqpp2\" (UID: \"91a29ddf-25ca-40a8-b920-6d894360a050\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gqpp2"
Oct 30 02:55:33.338179 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:33.337219    4461 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/0e6f015d-0a2d-4579-ad4d-a88b8c11c3b9-hosts-file\") pod \"node-resolver-zhb7b\" (UID: \"0e6f015d-0a2d-4579-ad4d-a88b8c11c3b9\") " pod="openshift-dns/node-resolver-zhb7b"
Oct 30 02:55:33.338179 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:33.337239    4461 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/91a29ddf-25ca-40a8-b920-6d894360a050-socket-dir\") pod \"aws-ebs-csi-driver-node-gqpp2\" (UID: \"91a29ddf-25ca-40a8-b920-6d894360a050\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gqpp2"
Oct 30 02:55:33.338179 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:33.337247    4461 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4d8bf700-9e09-4741-954b-aba9ccd331c9-system-cni-dir\") pod \"multus-additional-cni-plugins-9l7k8\" (UID: \"4d8bf700-9e09-4741-954b-aba9ccd331c9\") " pod="openshift-multus/multus-additional-cni-plugins-9l7k8"
Oct 30 02:55:33.338179 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:33.337266    4461 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/f425530a-8431-4c5b-8335-8cc1119745c1-host-kubelet\") pod \"ovnkube-node-g6nqm\" (UID: \"f425530a-8431-4c5b-8335-8cc1119745c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-g6nqm"
Oct 30 02:55:33.338179 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:33.337272    4461 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4d8bf700-9e09-4741-954b-aba9ccd331c9-system-cni-dir\") pod \"multus-additional-cni-plugins-9l7k8\" (UID: \"4d8bf700-9e09-4741-954b-aba9ccd331c9\") " pod="openshift-multus/multus-additional-cni-plugins-9l7k8"
Oct 30 02:55:33.338179 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:33.337248    4461 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/91a29ddf-25ca-40a8-b920-6d894360a050-kubelet-dir\") pod \"aws-ebs-csi-driver-node-gqpp2\" (UID: \"91a29ddf-25ca-40a8-b920-6d894360a050\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gqpp2"
Oct 30 02:55:33.338179 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:33.337282    4461 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/f425530a-8431-4c5b-8335-8cc1119745c1-log-socket\") pod \"ovnkube-node-g6nqm\" (UID: \"f425530a-8431-4c5b-8335-8cc1119745c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-g6nqm"
Oct 30 02:55:33.338179 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:33.337302    4461 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f425530a-8431-4c5b-8335-8cc1119745c1-host-cni-bin\") pod \"ovnkube-node-g6nqm\" (UID: \"f425530a-8431-4c5b-8335-8cc1119745c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-g6nqm"
Oct 30 02:55:33.338179 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:33.337326    4461 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/b4ce92fa-1612-4e46-ad6e-23049b2e279b-host-var-lib-cni-multus\") pod \"multus-9qcgk\" (UID: \"b4ce92fa-1612-4e46-ad6e-23049b2e279b\") " pod="openshift-multus/multus-9qcgk"
Oct 30 02:55:33.338179 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:33.337331    4461 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/f425530a-8431-4c5b-8335-8cc1119745c1-host-kubelet\") pod \"ovnkube-node-g6nqm\" (UID: \"f425530a-8431-4c5b-8335-8cc1119745c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-g6nqm"
Oct 30 02:55:33.338179 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:33.337302    4461 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/f425530a-8431-4c5b-8335-8cc1119745c1-log-socket\") pod \"ovnkube-node-g6nqm\" (UID: \"f425530a-8431-4c5b-8335-8cc1119745c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-g6nqm"
Oct 30 02:55:33.338179 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:33.337341    4461 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f425530a-8431-4c5b-8335-8cc1119745c1-host-run-netns\") pod \"ovnkube-node-g6nqm\" (UID: \"f425530a-8431-4c5b-8335-8cc1119745c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-g6nqm"
Oct 30 02:55:33.338179 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:33.337335    4461 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f425530a-8431-4c5b-8335-8cc1119745c1-host-cni-bin\") pod \"ovnkube-node-g6nqm\" (UID: \"f425530a-8431-4c5b-8335-8cc1119745c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-g6nqm"
Oct 30 02:55:33.338179 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:33.337340    4461 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/3e4730f0-d77e-4bff-b8af-b7fafd4525d8-etc-sysctl-conf\") pod \"tuned-h7cm4\" (UID: \"3e4730f0-d77e-4bff-b8af-b7fafd4525d8\") " pod="openshift-cluster-node-tuning-operator/tuned-h7cm4"
Oct 30 02:55:33.338179 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:33.337364    4461 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f425530a-8431-4c5b-8335-8cc1119745c1-host-run-netns\") pod \"ovnkube-node-g6nqm\" (UID: \"f425530a-8431-4c5b-8335-8cc1119745c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-g6nqm"
Oct 30 02:55:33.338179 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:33.337369    4461 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/3e4730f0-d77e-4bff-b8af-b7fafd4525d8-etc-sysctl-d\") pod \"tuned-h7cm4\" (UID: \"3e4730f0-d77e-4bff-b8af-b7fafd4525d8\") " pod="openshift-cluster-node-tuning-operator/tuned-h7cm4"
Oct 30 02:55:33.338179 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:33.337371    4461 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/b4ce92fa-1612-4e46-ad6e-23049b2e279b-host-var-lib-cni-multus\") pod \"multus-9qcgk\" (UID: \"b4ce92fa-1612-4e46-ad6e-23049b2e279b\") " pod="openshift-multus/multus-9qcgk"
Oct 30 02:55:33.338179 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:33.337392    4461 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3e4730f0-d77e-4bff-b8af-b7fafd4525d8-sys\") pod \"tuned-h7cm4\" (UID: \"3e4730f0-d77e-4bff-b8af-b7fafd4525d8\") " pod="openshift-cluster-node-tuning-operator/tuned-h7cm4"
Oct 30 02:55:33.338179 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:33.337408    4461 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/b4ce92fa-1612-4e46-ad6e-23049b2e279b-multus-socket-dir-parent\") pod \"multus-9qcgk\" (UID: \"b4ce92fa-1612-4e46-ad6e-23049b2e279b\") " pod="openshift-multus/multus-9qcgk"
Oct 30 02:55:33.338179 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:33.337427    4461 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/b4ce92fa-1612-4e46-ad6e-23049b2e279b-host-run-k8s-cni-cncf-io\") pod \"multus-9qcgk\" (UID: \"b4ce92fa-1612-4e46-ad6e-23049b2e279b\") " pod="openshift-multus/multus-9qcgk"
Oct 30 02:55:33.338179 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:33.337447    4461 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b4ce92fa-1612-4e46-ad6e-23049b2e279b-host-var-lib-cni-bin\") pod \"multus-9qcgk\" (UID: \"b4ce92fa-1612-4e46-ad6e-23049b2e279b\") " pod="openshift-multus/multus-9qcgk"
Oct 30 02:55:33.338179 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:33.337458    4461 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3e4730f0-d77e-4bff-b8af-b7fafd4525d8-sys\") pod \"tuned-h7cm4\" (UID: \"3e4730f0-d77e-4bff-b8af-b7fafd4525d8\") " pod="openshift-cluster-node-tuning-operator/tuned-h7cm4"
Oct 30 02:55:33.338179 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:33.337460    4461 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/3e4730f0-d77e-4bff-b8af-b7fafd4525d8-etc-sysctl-d\") pod \"tuned-h7cm4\" (UID: \"3e4730f0-d77e-4bff-b8af-b7fafd4525d8\") " pod="openshift-cluster-node-tuning-operator/tuned-h7cm4"
Oct 30 02:55:33.338179 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:33.337469    4461 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/3e4730f0-d77e-4bff-b8af-b7fafd4525d8-host\") pod \"tuned-h7cm4\" (UID: \"3e4730f0-d77e-4bff-b8af-b7fafd4525d8\") " pod="openshift-cluster-node-tuning-operator/tuned-h7cm4"
Oct 30 02:55:33.338179 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:33.337477    4461 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/b4ce92fa-1612-4e46-ad6e-23049b2e279b-multus-socket-dir-parent\") pod \"multus-9qcgk\" (UID: \"b4ce92fa-1612-4e46-ad6e-23049b2e279b\") " pod="openshift-multus/multus-9qcgk"
Oct 30 02:55:33.338179 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:33.337492    4461 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/eee29ef7-4ad4-4ff9-8cb0-9d2a5aaabba3-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2k85b\" (UID: \"eee29ef7-4ad4-4ff9-8cb0-9d2a5aaabba3\") " pod="openshift-insights/insights-runtime-extractor-2k85b"
Oct 30 02:55:33.338179 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:33.337497    4461 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b4ce92fa-1612-4e46-ad6e-23049b2e279b-host-var-lib-cni-bin\") pod \"multus-9qcgk\" (UID: \"b4ce92fa-1612-4e46-ad6e-23049b2e279b\") " pod="openshift-multus/multus-9qcgk"
Oct 30 02:55:33.338179 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:33.337493    4461 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/b4ce92fa-1612-4e46-ad6e-23049b2e279b-host-run-k8s-cni-cncf-io\") pod \"multus-9qcgk\" (UID: \"b4ce92fa-1612-4e46-ad6e-23049b2e279b\") " pod="openshift-multus/multus-9qcgk"
Oct 30 02:55:33.338179 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:33.337510    4461 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/3e4730f0-d77e-4bff-b8af-b7fafd4525d8-host\") pod \"tuned-h7cm4\" (UID: \"3e4730f0-d77e-4bff-b8af-b7fafd4525d8\") " pod="openshift-cluster-node-tuning-operator/tuned-h7cm4"
Oct 30 02:55:33.338179 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:33.337509    4461 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/4d8bf700-9e09-4741-954b-aba9ccd331c9-cnibin\") pod \"multus-additional-cni-plugins-9l7k8\" (UID: \"4d8bf700-9e09-4741-954b-aba9ccd331c9\") " pod="openshift-multus/multus-additional-cni-plugins-9l7k8"
Oct 30 02:55:33.338179 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:33.337535    4461 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/f425530a-8431-4c5b-8335-8cc1119745c1-host-cni-netd\") pod \"ovnkube-node-g6nqm\" (UID: \"f425530a-8431-4c5b-8335-8cc1119745c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-g6nqm"
Oct 30 02:55:33.338179 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:33.337556    4461 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/f425530a-8431-4c5b-8335-8cc1119745c1-host-cni-netd\") pod \"ovnkube-node-g6nqm\" (UID: \"f425530a-8431-4c5b-8335-8cc1119745c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-g6nqm"
Oct 30 02:55:33.338179 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:33.337536    4461 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/4d8bf700-9e09-4741-954b-aba9ccd331c9-cnibin\") pod \"multus-additional-cni-plugins-9l7k8\" (UID: \"4d8bf700-9e09-4741-954b-aba9ccd331c9\") " pod="openshift-multus/multus-additional-cni-plugins-9l7k8"
Oct 30 02:55:33.338179 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:33.337558    4461 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/f425530a-8431-4c5b-8335-8cc1119745c1-run-systemd\") pod \"ovnkube-node-g6nqm\" (UID: \"f425530a-8431-4c5b-8335-8cc1119745c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-g6nqm"
Oct 30 02:55:33.338179 ip-10-0-143-156 kubenswrapper[4461]: E1030 02:55:33.337578    4461 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 30 02:55:33.338179 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:33.337590    4461 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/487caf0f-5f2a-4e93-9c06-de9af02c470b-host\") pod \"node-ca-9mddj\" (UID: \"487caf0f-5f2a-4e93-9c06-de9af02c470b\") " pod="openshift-image-registry/node-ca-9mddj"
Oct 30 02:55:33.338179 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:33.337604    4461 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b4ce92fa-1612-4e46-ad6e-23049b2e279b-host-var-lib-kubelet\") pod \"multus-9qcgk\" (UID: \"b4ce92fa-1612-4e46-ad6e-23049b2e279b\") " pod="openshift-multus/multus-9qcgk"
Oct 30 02:55:33.338179 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:33.337580    4461 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/f425530a-8431-4c5b-8335-8cc1119745c1-run-systemd\") pod \"ovnkube-node-g6nqm\" (UID: \"f425530a-8431-4c5b-8335-8cc1119745c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-g6nqm"
Oct 30 02:55:33.338179 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:33.337620    4461 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b4ce92fa-1612-4e46-ad6e-23049b2e279b-etc-kubernetes\") pod \"multus-9qcgk\" (UID: \"b4ce92fa-1612-4e46-ad6e-23049b2e279b\") " pod="openshift-multus/multus-9qcgk"
Oct 30 02:55:33.338179 ip-10-0-143-156 kubenswrapper[4461]: E1030 02:55:33.337634    4461 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/eee29ef7-4ad4-4ff9-8cb0-9d2a5aaabba3-insights-runtime-extractor-tls podName:eee29ef7-4ad4-4ff9-8cb0-9d2a5aaabba3 nodeName:}" failed. No retries permitted until 2026-10-30 02:55:33.837618058 +0000 UTC m=+1.887423214 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/eee29ef7-4ad4-4ff9-8cb0-9d2a5aaabba3-insights-runtime-extractor-tls") pod "insights-runtime-extractor-2k85b" (UID: "eee29ef7-4ad4-4ff9-8cb0-9d2a5aaabba3") : secret "insights-runtime-extractor-tls" not found
Oct 30 02:55:33.338179 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:33.337656    4461 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b4ce92fa-1612-4e46-ad6e-23049b2e279b-etc-kubernetes\") pod \"multus-9qcgk\" (UID: \"b4ce92fa-1612-4e46-ad6e-23049b2e279b\") " pod="openshift-multus/multus-9qcgk"
Oct 30 02:55:33.338179 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:33.337673    4461 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/91a29ddf-25ca-40a8-b920-6d894360a050-device-dir\") pod \"aws-ebs-csi-driver-node-gqpp2\" (UID: \"91a29ddf-25ca-40a8-b920-6d894360a050\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gqpp2"
Oct 30 02:55:33.338179 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:33.337680    4461 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/487caf0f-5f2a-4e93-9c06-de9af02c470b-host\") pod \"node-ca-9mddj\" (UID: \"487caf0f-5f2a-4e93-9c06-de9af02c470b\") " pod="openshift-image-registry/node-ca-9mddj"
Oct 30 02:55:33.338179 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:33.337708    4461 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b4ce92fa-1612-4e46-ad6e-23049b2e279b-host-var-lib-kubelet\") pod \"multus-9qcgk\" (UID: \"b4ce92fa-1612-4e46-ad6e-23049b2e279b\") " pod="openshift-multus/multus-9qcgk"
Oct 30 02:55:33.338179 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:33.337728    4461 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f425530a-8431-4c5b-8335-8cc1119745c1-etc-openvswitch\") pod \"ovnkube-node-g6nqm\" (UID: \"f425530a-8431-4c5b-8335-8cc1119745c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-g6nqm"
Oct 30 02:55:33.338179 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:33.337727    4461 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/91a29ddf-25ca-40a8-b920-6d894360a050-device-dir\") pod \"aws-ebs-csi-driver-node-gqpp2\" (UID: \"91a29ddf-25ca-40a8-b920-6d894360a050\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gqpp2"
Oct 30 02:55:33.338179 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:33.337751    4461 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f425530a-8431-4c5b-8335-8cc1119745c1-etc-openvswitch\") pod \"ovnkube-node-g6nqm\" (UID: \"f425530a-8431-4c5b-8335-8cc1119745c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-g6nqm"
Oct 30 02:55:33.338179 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:33.337755    4461 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/f425530a-8431-4c5b-8335-8cc1119745c1-node-log\") pod \"ovnkube-node-g6nqm\" (UID: \"f425530a-8431-4c5b-8335-8cc1119745c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-g6nqm"
Oct 30 02:55:33.338179 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:33.337779    4461 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b4ce92fa-1612-4e46-ad6e-23049b2e279b-multus-cni-dir\") pod \"multus-9qcgk\" (UID: \"b4ce92fa-1612-4e46-ad6e-23049b2e279b\") " pod="openshift-multus/multus-9qcgk"
Oct 30 02:55:33.338179 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:33.337801    4461 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/3e4730f0-d77e-4bff-b8af-b7fafd4525d8-etc-modprobe-d\") pod \"tuned-h7cm4\" (UID: \"3e4730f0-d77e-4bff-b8af-b7fafd4525d8\") " pod="openshift-cluster-node-tuning-operator/tuned-h7cm4"
Oct 30 02:55:33.338179 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:33.337825    4461 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/98f9efe3-a732-4ba4-8543-941633d22170-kubelet-config\") pod \"global-pull-secret-syncer-vbnwt\" (UID: \"98f9efe3-a732-4ba4-8543-941633d22170\") " pod="kube-system/global-pull-secret-syncer-vbnwt"
Oct 30 02:55:33.338179 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:33.337834    4461 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/f425530a-8431-4c5b-8335-8cc1119745c1-node-log\") pod \"ovnkube-node-g6nqm\" (UID: \"f425530a-8431-4c5b-8335-8cc1119745c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-g6nqm"
Oct 30 02:55:33.338179 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:33.337856    4461 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/91a29ddf-25ca-40a8-b920-6d894360a050-registration-dir\") pod \"aws-ebs-csi-driver-node-gqpp2\" (UID: \"91a29ddf-25ca-40a8-b920-6d894360a050\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gqpp2"
Oct 30 02:55:33.338179 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:33.337877    4461 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/98f9efe3-a732-4ba4-8543-941633d22170-kubelet-config\") pod \"global-pull-secret-syncer-vbnwt\" (UID: \"98f9efe3-a732-4ba4-8543-941633d22170\") " pod="kube-system/global-pull-secret-syncer-vbnwt"
Oct 30 02:55:33.338179 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:33.337879    4461 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b4ce92fa-1612-4e46-ad6e-23049b2e279b-system-cni-dir\") pod \"multus-9qcgk\" (UID: \"b4ce92fa-1612-4e46-ad6e-23049b2e279b\") " pod="openshift-multus/multus-9qcgk"
Oct 30 02:55:33.338179 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:33.337890    4461 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/91a29ddf-25ca-40a8-b920-6d894360a050-registration-dir\") pod \"aws-ebs-csi-driver-node-gqpp2\" (UID: \"91a29ddf-25ca-40a8-b920-6d894360a050\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gqpp2"
Oct 30 02:55:33.338179 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:33.337879    4461 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b4ce92fa-1612-4e46-ad6e-23049b2e279b-multus-cni-dir\") pod \"multus-9qcgk\" (UID: \"b4ce92fa-1612-4e46-ad6e-23049b2e279b\") " pod="openshift-multus/multus-9qcgk"
Oct 30 02:55:33.338179 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:33.337903    4461 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b4ce92fa-1612-4e46-ad6e-23049b2e279b-cnibin\") pod \"multus-9qcgk\" (UID: \"b4ce92fa-1612-4e46-ad6e-23049b2e279b\") " pod="openshift-multus/multus-9qcgk"
Oct 30 02:55:33.338179 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:33.337909    4461 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/3e4730f0-d77e-4bff-b8af-b7fafd4525d8-etc-modprobe-d\") pod \"tuned-h7cm4\" (UID: \"3e4730f0-d77e-4bff-b8af-b7fafd4525d8\") " pod="openshift-cluster-node-tuning-operator/tuned-h7cm4"
Oct 30 02:55:33.338179 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:33.337908    4461 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b4ce92fa-1612-4e46-ad6e-23049b2e279b-system-cni-dir\") pod \"multus-9qcgk\" (UID: \"b4ce92fa-1612-4e46-ad6e-23049b2e279b\") " pod="openshift-multus/multus-9qcgk"
Oct 30 02:55:33.338179 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:33.337941    4461 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b4ce92fa-1612-4e46-ad6e-23049b2e279b-cnibin\") pod \"multus-9qcgk\" (UID: \"b4ce92fa-1612-4e46-ad6e-23049b2e279b\") " pod="openshift-multus/multus-9qcgk"
Oct 30 02:55:33.338179 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:33.337960    4461 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3e4730f0-d77e-4bff-b8af-b7fafd4525d8-etc-kubernetes\") pod \"tuned-h7cm4\" (UID: \"3e4730f0-d77e-4bff-b8af-b7fafd4525d8\") " pod="openshift-cluster-node-tuning-operator/tuned-h7cm4"
Oct 30 02:55:33.338179 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:33.337981    4461 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/98f9efe3-a732-4ba4-8543-941633d22170-dbus\") pod \"global-pull-secret-syncer-vbnwt\" (UID: \"98f9efe3-a732-4ba4-8543-941633d22170\") " pod="kube-system/global-pull-secret-syncer-vbnwt"
Oct 30 02:55:33.338179 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:33.338007    4461 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3e4730f0-d77e-4bff-b8af-b7fafd4525d8-etc-kubernetes\") pod \"tuned-h7cm4\" (UID: \"3e4730f0-d77e-4bff-b8af-b7fafd4525d8\") " pod="openshift-cluster-node-tuning-operator/tuned-h7cm4"
Oct 30 02:55:33.338179 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:33.338040    4461 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/98f9efe3-a732-4ba4-8543-941633d22170-dbus\") pod \"global-pull-secret-syncer-vbnwt\" (UID: \"98f9efe3-a732-4ba4-8543-941633d22170\") " pod="kube-system/global-pull-secret-syncer-vbnwt"
Oct 30 02:55:33.679020 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:33.678925    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 02:55:33.840647 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:33.840616    4461 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/eee29ef7-4ad4-4ff9-8cb0-9d2a5aaabba3-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2k85b\" (UID: \"eee29ef7-4ad4-4ff9-8cb0-9d2a5aaabba3\") " pod="openshift-insights/insights-runtime-extractor-2k85b"
Oct 30 02:55:33.840824 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:33.840670    4461 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3bd2c646-cebb-4e8a-b6c3-8b5b74049c11-metrics-certs\") pod \"network-metrics-daemon-mzk2s\" (UID: \"3bd2c646-cebb-4e8a-b6c3-8b5b74049c11\") " pod="openshift-multus/network-metrics-daemon-mzk2s"
Oct 30 02:55:33.840824 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:33.840691    4461 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/5f9fb000-9531-41f1-b3cf-7e79ca1a24d7-metrics-tls\") pod \"dns-default-wjll7\" (UID: \"5f9fb000-9531-41f1-b3cf-7e79ca1a24d7\") " pod="openshift-dns/dns-default-wjll7"
Oct 30 02:55:33.840824 ip-10-0-143-156 kubenswrapper[4461]: E1030 02:55:33.840765    4461 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 30 02:55:33.840824 ip-10-0-143-156 kubenswrapper[4461]: E1030 02:55:33.840778    4461 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 30 02:55:33.840950 ip-10-0-143-156 kubenswrapper[4461]: E1030 02:55:33.840826    4461 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Oct 30 02:55:33.840950 ip-10-0-143-156 kubenswrapper[4461]: E1030 02:55:33.840829    4461 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/eee29ef7-4ad4-4ff9-8cb0-9d2a5aaabba3-insights-runtime-extractor-tls podName:eee29ef7-4ad4-4ff9-8cb0-9d2a5aaabba3 nodeName:}" failed. No retries permitted until 2026-10-30 02:55:34.840816002 +0000 UTC m=+2.890621155 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/eee29ef7-4ad4-4ff9-8cb0-9d2a5aaabba3-insights-runtime-extractor-tls") pod "insights-runtime-extractor-2k85b" (UID: "eee29ef7-4ad4-4ff9-8cb0-9d2a5aaabba3") : secret "insights-runtime-extractor-tls" not found
Oct 30 02:55:33.840950 ip-10-0-143-156 kubenswrapper[4461]: E1030 02:55:33.840890    4461 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5f9fb000-9531-41f1-b3cf-7e79ca1a24d7-metrics-tls podName:5f9fb000-9531-41f1-b3cf-7e79ca1a24d7 nodeName:}" failed. No retries permitted until 2026-10-30 02:55:34.840874656 +0000 UTC m=+2.890679821 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/5f9fb000-9531-41f1-b3cf-7e79ca1a24d7-metrics-tls") pod "dns-default-wjll7" (UID: "5f9fb000-9531-41f1-b3cf-7e79ca1a24d7") : secret "dns-default-metrics-tls" not found
Oct 30 02:55:33.840950 ip-10-0-143-156 kubenswrapper[4461]: E1030 02:55:33.840911    4461 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3bd2c646-cebb-4e8a-b6c3-8b5b74049c11-metrics-certs podName:3bd2c646-cebb-4e8a-b6c3-8b5b74049c11 nodeName:}" failed. No retries permitted until 2026-10-30 02:55:34.840901769 +0000 UTC m=+2.890706927 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3bd2c646-cebb-4e8a-b6c3-8b5b74049c11-metrics-certs") pod "network-metrics-daemon-mzk2s" (UID: "3bd2c646-cebb-4e8a-b6c3-8b5b74049c11") : secret "metrics-daemon-secret" not found
Oct 30 02:55:33.961050 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:33.960959    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 02:55:34.169443 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:34.169413    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 02:55:34.363307 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:34.363247    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 02:55:34.567312 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:34.567280    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 02:55:34.681495 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:34.681435    4461 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-g6nqm"
Oct 30 02:55:34.763413 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:34.763390    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 02:55:34.846539 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:34.846511    4461 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3bd2c646-cebb-4e8a-b6c3-8b5b74049c11-metrics-certs\") pod \"network-metrics-daemon-mzk2s\" (UID: \"3bd2c646-cebb-4e8a-b6c3-8b5b74049c11\") " pod="openshift-multus/network-metrics-daemon-mzk2s"
Oct 30 02:55:34.846539 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:34.846541    4461 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/5f9fb000-9531-41f1-b3cf-7e79ca1a24d7-metrics-tls\") pod \"dns-default-wjll7\" (UID: \"5f9fb000-9531-41f1-b3cf-7e79ca1a24d7\") " pod="openshift-dns/dns-default-wjll7"
Oct 30 02:55:34.846693 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:34.846574    4461 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/eee29ef7-4ad4-4ff9-8cb0-9d2a5aaabba3-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2k85b\" (UID: \"eee29ef7-4ad4-4ff9-8cb0-9d2a5aaabba3\") " pod="openshift-insights/insights-runtime-extractor-2k85b"
Oct 30 02:55:34.846693 ip-10-0-143-156 kubenswrapper[4461]: E1030 02:55:34.846664    4461 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Oct 30 02:55:34.846763 ip-10-0-143-156 kubenswrapper[4461]: E1030 02:55:34.846752    4461 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3bd2c646-cebb-4e8a-b6c3-8b5b74049c11-metrics-certs podName:3bd2c646-cebb-4e8a-b6c3-8b5b74049c11 nodeName:}" failed. No retries permitted until 2026-10-30 02:55:36.846738768 +0000 UTC m=+4.896543923 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3bd2c646-cebb-4e8a-b6c3-8b5b74049c11-metrics-certs") pod "network-metrics-daemon-mzk2s" (UID: "3bd2c646-cebb-4e8a-b6c3-8b5b74049c11") : secret "metrics-daemon-secret" not found
Oct 30 02:55:34.846763 ip-10-0-143-156 kubenswrapper[4461]: E1030 02:55:34.846667    4461 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 30 02:55:34.846825 ip-10-0-143-156 kubenswrapper[4461]: E1030 02:55:34.846783    4461 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/eee29ef7-4ad4-4ff9-8cb0-9d2a5aaabba3-insights-runtime-extractor-tls podName:eee29ef7-4ad4-4ff9-8cb0-9d2a5aaabba3 nodeName:}" failed. No retries permitted until 2026-10-30 02:55:36.846776875 +0000 UTC m=+4.896582028 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/eee29ef7-4ad4-4ff9-8cb0-9d2a5aaabba3-insights-runtime-extractor-tls") pod "insights-runtime-extractor-2k85b" (UID: "eee29ef7-4ad4-4ff9-8cb0-9d2a5aaabba3") : secret "insights-runtime-extractor-tls" not found
Oct 30 02:55:34.846825 ip-10-0-143-156 kubenswrapper[4461]: E1030 02:55:34.846667    4461 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 30 02:55:34.846825 ip-10-0-143-156 kubenswrapper[4461]: E1030 02:55:34.846815    4461 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5f9fb000-9531-41f1-b3cf-7e79ca1a24d7-metrics-tls podName:5f9fb000-9531-41f1-b3cf-7e79ca1a24d7 nodeName:}" failed. No retries permitted until 2026-10-30 02:55:36.846809816 +0000 UTC m=+4.896614970 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/5f9fb000-9531-41f1-b3cf-7e79ca1a24d7-metrics-tls") pod "dns-default-wjll7" (UID: "5f9fb000-9531-41f1-b3cf-7e79ca1a24d7") : secret "dns-default-metrics-tls" not found
Oct 30 02:55:34.942317 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:34.942242    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 02:55:35.292530 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:35.292503    4461 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-zhb7b_0e6f015d-0a2d-4579-ad4d-a88b8c11c3b9/dns-node-resolver/0.log"
Oct 30 02:55:35.892891 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:35.892865    4461 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-9mddj_487caf0f-5f2a-4e93-9c06-de9af02c470b/node-ca/0.log"
Oct 30 02:55:35.962828 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:35.962796    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 02:55:36.144732 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:36.144654    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 02:55:36.309014 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:36.308978    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 02:55:36.474638 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:36.474608    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 02:55:36.637220 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:36.637182    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 02:55:36.811237 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:36.811163    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 02:55:36.860212 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:36.860179    4461 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/eee29ef7-4ad4-4ff9-8cb0-9d2a5aaabba3-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2k85b\" (UID: \"eee29ef7-4ad4-4ff9-8cb0-9d2a5aaabba3\") " pod="openshift-insights/insights-runtime-extractor-2k85b"
Oct 30 02:55:36.860389 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:36.860223    4461 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3bd2c646-cebb-4e8a-b6c3-8b5b74049c11-metrics-certs\") pod \"network-metrics-daemon-mzk2s\" (UID: \"3bd2c646-cebb-4e8a-b6c3-8b5b74049c11\") " pod="openshift-multus/network-metrics-daemon-mzk2s"
Oct 30 02:55:36.860389 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:36.860245    4461 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/5f9fb000-9531-41f1-b3cf-7e79ca1a24d7-metrics-tls\") pod \"dns-default-wjll7\" (UID: \"5f9fb000-9531-41f1-b3cf-7e79ca1a24d7\") " pod="openshift-dns/dns-default-wjll7"
Oct 30 02:55:36.860389 ip-10-0-143-156 kubenswrapper[4461]: E1030 02:55:36.860335    4461 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 30 02:55:36.860389 ip-10-0-143-156 kubenswrapper[4461]: E1030 02:55:36.860341    4461 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Oct 30 02:55:36.860389 ip-10-0-143-156 kubenswrapper[4461]: E1030 02:55:36.860390    4461 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5f9fb000-9531-41f1-b3cf-7e79ca1a24d7-metrics-tls podName:5f9fb000-9531-41f1-b3cf-7e79ca1a24d7 nodeName:}" failed. No retries permitted until 2026-10-30 02:55:40.860377752 +0000 UTC m=+8.910182907 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/5f9fb000-9531-41f1-b3cf-7e79ca1a24d7-metrics-tls") pod "dns-default-wjll7" (UID: "5f9fb000-9531-41f1-b3cf-7e79ca1a24d7") : secret "dns-default-metrics-tls" not found
Oct 30 02:55:36.860571 ip-10-0-143-156 kubenswrapper[4461]: E1030 02:55:36.860341    4461 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 30 02:55:36.860571 ip-10-0-143-156 kubenswrapper[4461]: E1030 02:55:36.860402    4461 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3bd2c646-cebb-4e8a-b6c3-8b5b74049c11-metrics-certs podName:3bd2c646-cebb-4e8a-b6c3-8b5b74049c11 nodeName:}" failed. No retries permitted until 2026-10-30 02:55:40.8603966 +0000 UTC m=+8.910201763 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3bd2c646-cebb-4e8a-b6c3-8b5b74049c11-metrics-certs") pod "network-metrics-daemon-mzk2s" (UID: "3bd2c646-cebb-4e8a-b6c3-8b5b74049c11") : secret "metrics-daemon-secret" not found
Oct 30 02:55:36.860571 ip-10-0-143-156 kubenswrapper[4461]: E1030 02:55:36.860422    4461 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/eee29ef7-4ad4-4ff9-8cb0-9d2a5aaabba3-insights-runtime-extractor-tls podName:eee29ef7-4ad4-4ff9-8cb0-9d2a5aaabba3 nodeName:}" failed. No retries permitted until 2026-10-30 02:55:40.860410212 +0000 UTC m=+8.910215365 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/eee29ef7-4ad4-4ff9-8cb0-9d2a5aaabba3-insights-runtime-extractor-tls") pod "insights-runtime-extractor-2k85b" (UID: "eee29ef7-4ad4-4ff9-8cb0-9d2a5aaabba3") : secret "insights-runtime-extractor-tls" not found
Oct 30 02:55:36.926682 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:36.926657    4461 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-djpqs"
Oct 30 02:55:37.093493 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:37.093421    4461 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-vwbj7_73752bb9-2a14-4582-a6d6-c1fd26524c3d/serve-healthcheck-canary/0.log"
Oct 30 02:55:37.493008 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:37.492978    4461 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-66df5d496b-wx7pd_f990bb17-3b0f-4bca-8939-089f79d6edef/migrator/0.log"
Oct 30 02:55:37.960606 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:37.960528    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 02:55:38.157256 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:38.157225    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 02:55:38.360704 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:38.360674    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 02:55:38.564364 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:38.564333    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 02:55:38.763301 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:38.763270    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 02:55:39.634061 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:39.634028    4461 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-7rmdj"
Oct 30 02:55:39.998935 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:39.998900    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 02:55:40.052943 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:40.052907    4461 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-g6nqm"
Oct 30 02:55:40.053147 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:40.053139    4461 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Oct 30 02:55:40.180605 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:40.180575    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 02:55:40.361903 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:40.361823    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 02:55:40.566925 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:40.566890    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 02:55:40.763497 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:40.763466    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 02:55:40.890907 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:40.890867    4461 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3bd2c646-cebb-4e8a-b6c3-8b5b74049c11-metrics-certs\") pod \"network-metrics-daemon-mzk2s\" (UID: \"3bd2c646-cebb-4e8a-b6c3-8b5b74049c11\") " pod="openshift-multus/network-metrics-daemon-mzk2s"
Oct 30 02:55:40.890907 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:40.890908    4461 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/5f9fb000-9531-41f1-b3cf-7e79ca1a24d7-metrics-tls\") pod \"dns-default-wjll7\" (UID: \"5f9fb000-9531-41f1-b3cf-7e79ca1a24d7\") " pod="openshift-dns/dns-default-wjll7"
Oct 30 02:55:40.891133 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:40.890946    4461 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/eee29ef7-4ad4-4ff9-8cb0-9d2a5aaabba3-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2k85b\" (UID: \"eee29ef7-4ad4-4ff9-8cb0-9d2a5aaabba3\") " pod="openshift-insights/insights-runtime-extractor-2k85b"
Oct 30 02:55:40.891133 ip-10-0-143-156 kubenswrapper[4461]: E1030 02:55:40.891038    4461 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 30 02:55:40.891216 ip-10-0-143-156 kubenswrapper[4461]: E1030 02:55:40.891135    4461 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5f9fb000-9531-41f1-b3cf-7e79ca1a24d7-metrics-tls podName:5f9fb000-9531-41f1-b3cf-7e79ca1a24d7 nodeName:}" failed. No retries permitted until 2026-10-30 02:55:48.891118757 +0000 UTC m=+16.940923924 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/5f9fb000-9531-41f1-b3cf-7e79ca1a24d7-metrics-tls") pod "dns-default-wjll7" (UID: "5f9fb000-9531-41f1-b3cf-7e79ca1a24d7") : secret "dns-default-metrics-tls" not found
Oct 30 02:55:40.891216 ip-10-0-143-156 kubenswrapper[4461]: E1030 02:55:40.891046    4461 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 30 02:55:40.891216 ip-10-0-143-156 kubenswrapper[4461]: E1030 02:55:40.891167    4461 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/eee29ef7-4ad4-4ff9-8cb0-9d2a5aaabba3-insights-runtime-extractor-tls podName:eee29ef7-4ad4-4ff9-8cb0-9d2a5aaabba3 nodeName:}" failed. No retries permitted until 2026-10-30 02:55:48.891160977 +0000 UTC m=+16.940966130 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/eee29ef7-4ad4-4ff9-8cb0-9d2a5aaabba3-insights-runtime-extractor-tls") pod "insights-runtime-extractor-2k85b" (UID: "eee29ef7-4ad4-4ff9-8cb0-9d2a5aaabba3") : secret "insights-runtime-extractor-tls" not found
Oct 30 02:55:40.891368 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:40.891350    4461 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Oct 30 02:55:40.893359 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:40.893343    4461 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3bd2c646-cebb-4e8a-b6c3-8b5b74049c11-metrics-certs\") pod \"network-metrics-daemon-mzk2s\" (UID: \"3bd2c646-cebb-4e8a-b6c3-8b5b74049c11\") " pod="openshift-multus/network-metrics-daemon-mzk2s"
Oct 30 02:55:40.904269 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:40.904242    4461 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mzk2s"
Oct 30 02:55:40.932598 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:40.932572    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 02:55:40.965250 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:40.959762    4461 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-g6nqm"
Oct 30 02:55:41.202813 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:41.202735    4461 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-mzk2s"]
Oct 30 02:55:41.206621 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:41.206597    4461 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3bd2c646_cebb_4e8a_b6c3_8b5b74049c11.slice/crio-641439f12d885dcc073bba988338e33ccd515f3765ef44a0c0ba18bfb0160b5c WatchSource:0}: Error finding container 641439f12d885dcc073bba988338e33ccd515f3765ef44a0c0ba18bfb0160b5c: Status 404 returned error can't find the container with id 641439f12d885dcc073bba988338e33ccd515f3765ef44a0c0ba18bfb0160b5c
Oct 30 02:55:41.209004 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:41.208988    4461 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 02:55:41.980921 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:41.980892    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 02:55:42.163164 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:42.163140    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 02:55:42.176141 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:42.176112    4461 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-mzk2s" event={"ID":"3bd2c646-cebb-4e8a-b6c3-8b5b74049c11","Type":"ContainerStarted","Data":"6e321656d43c0d7250e2d8d7b1b0502793048c25d0db53a21bb4e951dedfd349"}
Oct 30 02:55:42.176271 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:42.176168    4461 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-mzk2s" event={"ID":"3bd2c646-cebb-4e8a-b6c3-8b5b74049c11","Type":"ContainerStarted","Data":"7febafceefb6960165b5f12f5c96625bfb30b82b05bc591fa9c94f3252c5fae4"}
Oct 30 02:55:42.176271 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:42.176202    4461 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-mzk2s" event={"ID":"3bd2c646-cebb-4e8a-b6c3-8b5b74049c11","Type":"ContainerStarted","Data":"641439f12d885dcc073bba988338e33ccd515f3765ef44a0c0ba18bfb0160b5c"}
Oct 30 02:55:42.358812 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:42.358735    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 02:55:42.561741 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:42.561709    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 02:55:42.761082 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:42.761044    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 02:55:43.503825 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:43.503793    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 02:55:43.695427 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:43.695388    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 02:55:43.870905 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:43.870822    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 02:55:44.054342 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:44.054305    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 02:55:45.524534 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:45.524500    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 02:55:48.945265 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:48.945220    4461 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/eee29ef7-4ad4-4ff9-8cb0-9d2a5aaabba3-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2k85b\" (UID: \"eee29ef7-4ad4-4ff9-8cb0-9d2a5aaabba3\") " pod="openshift-insights/insights-runtime-extractor-2k85b"
Oct 30 02:55:48.945840 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:48.945302    4461 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/5f9fb000-9531-41f1-b3cf-7e79ca1a24d7-metrics-tls\") pod \"dns-default-wjll7\" (UID: \"5f9fb000-9531-41f1-b3cf-7e79ca1a24d7\") " pod="openshift-dns/dns-default-wjll7"
Oct 30 02:55:48.947614 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:48.947591    4461 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/5f9fb000-9531-41f1-b3cf-7e79ca1a24d7-metrics-tls\") pod \"dns-default-wjll7\" (UID: \"5f9fb000-9531-41f1-b3cf-7e79ca1a24d7\") " pod="openshift-dns/dns-default-wjll7"
Oct 30 02:55:48.947719 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:48.947638    4461 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/eee29ef7-4ad4-4ff9-8cb0-9d2a5aaabba3-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2k85b\" (UID: \"eee29ef7-4ad4-4ff9-8cb0-9d2a5aaabba3\") " pod="openshift-insights/insights-runtime-extractor-2k85b"
Oct 30 02:55:49.007702 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:49.007657    4461 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-wjll7"
Oct 30 02:55:49.009226 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:49.009201    4461 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-2k85b"
Oct 30 02:55:49.330389 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:49.330355    4461 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-wjll7"]
Oct 30 02:55:49.333596 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:49.333539    4461 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5f9fb000_9531_41f1_b3cf_7e79ca1a24d7.slice/crio-8ec3d9ef4d0e8e89621f0c3652d85f3df4badeab6e5a8d7508f98abd0dbc1d73 WatchSource:0}: Error finding container 8ec3d9ef4d0e8e89621f0c3652d85f3df4badeab6e5a8d7508f98abd0dbc1d73: Status 404 returned error can't find the container with id 8ec3d9ef4d0e8e89621f0c3652d85f3df4badeab6e5a8d7508f98abd0dbc1d73
Oct 30 02:55:49.335889 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:49.335859    4461 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeee29ef7_4ad4_4ff9_8cb0_9d2a5aaabba3.slice/crio-736c044810b32466f7a1ba0ff9184d237a4d03a1c662c9ce86fea62846a43bc1 WatchSource:0}: Error finding container 736c044810b32466f7a1ba0ff9184d237a4d03a1c662c9ce86fea62846a43bc1: Status 404 returned error can't find the container with id 736c044810b32466f7a1ba0ff9184d237a4d03a1c662c9ce86fea62846a43bc1
Oct 30 02:55:49.342861 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:49.342836    4461 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-2k85b"]
Oct 30 02:55:49.835784 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:49.835746    4461 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-7fbbd597bb-twtnh"]
Oct 30 02:55:49.839812 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:49.839180    4461 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7fbbd597bb-twtnh"
Oct 30 02:55:49.880495 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:49.880292    4461 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Oct 30 02:55:49.880495 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:49.880352    4461 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Oct 30 02:55:49.880495 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:49.880387    4461 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Oct 30 02:55:49.880495 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:49.880459    4461 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-vvtsk\""
Oct 30 02:55:49.881754 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:49.881735    4461 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Oct 30 02:55:49.885108 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:49.885077    4461 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-7fbbd597bb-twtnh"]
Oct 30 02:55:49.911612 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:49.911584    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 02:55:49.950372 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:49.950347    4461 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/21d9fcc4-833e-4196-a646-c1f0d0c38313-ca-trust-extracted\") pod \"image-registry-7fbbd597bb-twtnh\" (UID: \"21d9fcc4-833e-4196-a646-c1f0d0c38313\") " pod="openshift-image-registry/image-registry-7fbbd597bb-twtnh"
Oct 30 02:55:49.950712 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:49.950392    4461 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/21d9fcc4-833e-4196-a646-c1f0d0c38313-registry-tls\") pod \"image-registry-7fbbd597bb-twtnh\" (UID: \"21d9fcc4-833e-4196-a646-c1f0d0c38313\") " pod="openshift-image-registry/image-registry-7fbbd597bb-twtnh"
Oct 30 02:55:49.950712 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:49.950548    4461 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/21d9fcc4-833e-4196-a646-c1f0d0c38313-registry-certificates\") pod \"image-registry-7fbbd597bb-twtnh\" (UID: \"21d9fcc4-833e-4196-a646-c1f0d0c38313\") " pod="openshift-image-registry/image-registry-7fbbd597bb-twtnh"
Oct 30 02:55:49.950712 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:49.950593    4461 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/21d9fcc4-833e-4196-a646-c1f0d0c38313-trusted-ca\") pod \"image-registry-7fbbd597bb-twtnh\" (UID: \"21d9fcc4-833e-4196-a646-c1f0d0c38313\") " pod="openshift-image-registry/image-registry-7fbbd597bb-twtnh"
Oct 30 02:55:49.950712 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:49.950621    4461 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/21d9fcc4-833e-4196-a646-c1f0d0c38313-installation-pull-secrets\") pod \"image-registry-7fbbd597bb-twtnh\" (UID: \"21d9fcc4-833e-4196-a646-c1f0d0c38313\") " pod="openshift-image-registry/image-registry-7fbbd597bb-twtnh"
Oct 30 02:55:49.950712 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:49.950645    4461 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/21d9fcc4-833e-4196-a646-c1f0d0c38313-bound-sa-token\") pod \"image-registry-7fbbd597bb-twtnh\" (UID: \"21d9fcc4-833e-4196-a646-c1f0d0c38313\") " pod="openshift-image-registry/image-registry-7fbbd597bb-twtnh"
Oct 30 02:55:49.950712 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:49.950692    4461 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9fv8m\" (UniqueName: \"kubernetes.io/projected/21d9fcc4-833e-4196-a646-c1f0d0c38313-kube-api-access-9fv8m\") pod \"image-registry-7fbbd597bb-twtnh\" (UID: \"21d9fcc4-833e-4196-a646-c1f0d0c38313\") " pod="openshift-image-registry/image-registry-7fbbd597bb-twtnh"
Oct 30 02:55:49.950972 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:49.950718    4461 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/21d9fcc4-833e-4196-a646-c1f0d0c38313-image-registry-private-configuration\") pod \"image-registry-7fbbd597bb-twtnh\" (UID: \"21d9fcc4-833e-4196-a646-c1f0d0c38313\") " pod="openshift-image-registry/image-registry-7fbbd597bb-twtnh"
Oct 30 02:55:50.051539 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:50.051509    4461 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/21d9fcc4-833e-4196-a646-c1f0d0c38313-registry-tls\") pod \"image-registry-7fbbd597bb-twtnh\" (UID: \"21d9fcc4-833e-4196-a646-c1f0d0c38313\") " pod="openshift-image-registry/image-registry-7fbbd597bb-twtnh"
Oct 30 02:55:50.051539 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:50.051545    4461 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/21d9fcc4-833e-4196-a646-c1f0d0c38313-registry-certificates\") pod \"image-registry-7fbbd597bb-twtnh\" (UID: \"21d9fcc4-833e-4196-a646-c1f0d0c38313\") " pod="openshift-image-registry/image-registry-7fbbd597bb-twtnh"
Oct 30 02:55:50.051732 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:50.051574    4461 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/21d9fcc4-833e-4196-a646-c1f0d0c38313-trusted-ca\") pod \"image-registry-7fbbd597bb-twtnh\" (UID: \"21d9fcc4-833e-4196-a646-c1f0d0c38313\") " pod="openshift-image-registry/image-registry-7fbbd597bb-twtnh"
Oct 30 02:55:50.051732 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:50.051592    4461 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/21d9fcc4-833e-4196-a646-c1f0d0c38313-installation-pull-secrets\") pod \"image-registry-7fbbd597bb-twtnh\" (UID: \"21d9fcc4-833e-4196-a646-c1f0d0c38313\") " pod="openshift-image-registry/image-registry-7fbbd597bb-twtnh"
Oct 30 02:55:50.051732 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:50.051614    4461 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/21d9fcc4-833e-4196-a646-c1f0d0c38313-bound-sa-token\") pod \"image-registry-7fbbd597bb-twtnh\" (UID: \"21d9fcc4-833e-4196-a646-c1f0d0c38313\") " pod="openshift-image-registry/image-registry-7fbbd597bb-twtnh"
Oct 30 02:55:50.051732 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:50.051662    4461 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9fv8m\" (UniqueName: \"kubernetes.io/projected/21d9fcc4-833e-4196-a646-c1f0d0c38313-kube-api-access-9fv8m\") pod \"image-registry-7fbbd597bb-twtnh\" (UID: \"21d9fcc4-833e-4196-a646-c1f0d0c38313\") " pod="openshift-image-registry/image-registry-7fbbd597bb-twtnh"
Oct 30 02:55:50.051732 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:50.051688    4461 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/21d9fcc4-833e-4196-a646-c1f0d0c38313-image-registry-private-configuration\") pod \"image-registry-7fbbd597bb-twtnh\" (UID: \"21d9fcc4-833e-4196-a646-c1f0d0c38313\") " pod="openshift-image-registry/image-registry-7fbbd597bb-twtnh"
Oct 30 02:55:50.051887 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:50.051734    4461 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/21d9fcc4-833e-4196-a646-c1f0d0c38313-ca-trust-extracted\") pod \"image-registry-7fbbd597bb-twtnh\" (UID: \"21d9fcc4-833e-4196-a646-c1f0d0c38313\") " pod="openshift-image-registry/image-registry-7fbbd597bb-twtnh"
Oct 30 02:55:50.052175 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:50.052148    4461 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/21d9fcc4-833e-4196-a646-c1f0d0c38313-ca-trust-extracted\") pod \"image-registry-7fbbd597bb-twtnh\" (UID: \"21d9fcc4-833e-4196-a646-c1f0d0c38313\") " pod="openshift-image-registry/image-registry-7fbbd597bb-twtnh"
Oct 30 02:55:50.053313 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:50.053283    4461 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/21d9fcc4-833e-4196-a646-c1f0d0c38313-registry-certificates\") pod \"image-registry-7fbbd597bb-twtnh\" (UID: \"21d9fcc4-833e-4196-a646-c1f0d0c38313\") " pod="openshift-image-registry/image-registry-7fbbd597bb-twtnh"
Oct 30 02:55:50.053567 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:50.053526    4461 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/21d9fcc4-833e-4196-a646-c1f0d0c38313-trusted-ca\") pod \"image-registry-7fbbd597bb-twtnh\" (UID: \"21d9fcc4-833e-4196-a646-c1f0d0c38313\") " pod="openshift-image-registry/image-registry-7fbbd597bb-twtnh"
Oct 30 02:55:50.055244 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:50.055221    4461 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/21d9fcc4-833e-4196-a646-c1f0d0c38313-installation-pull-secrets\") pod \"image-registry-7fbbd597bb-twtnh\" (UID: \"21d9fcc4-833e-4196-a646-c1f0d0c38313\") " pod="openshift-image-registry/image-registry-7fbbd597bb-twtnh"
Oct 30 02:55:50.057928 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:50.057907    4461 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/21d9fcc4-833e-4196-a646-c1f0d0c38313-image-registry-private-configuration\") pod \"image-registry-7fbbd597bb-twtnh\" (UID: \"21d9fcc4-833e-4196-a646-c1f0d0c38313\") " pod="openshift-image-registry/image-registry-7fbbd597bb-twtnh"
Oct 30 02:55:50.058443 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:50.058423    4461 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/21d9fcc4-833e-4196-a646-c1f0d0c38313-registry-tls\") pod \"image-registry-7fbbd597bb-twtnh\" (UID: \"21d9fcc4-833e-4196-a646-c1f0d0c38313\") " pod="openshift-image-registry/image-registry-7fbbd597bb-twtnh"
Oct 30 02:55:50.082815 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:50.082787    4461 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/21d9fcc4-833e-4196-a646-c1f0d0c38313-bound-sa-token\") pod \"image-registry-7fbbd597bb-twtnh\" (UID: \"21d9fcc4-833e-4196-a646-c1f0d0c38313\") " pod="openshift-image-registry/image-registry-7fbbd597bb-twtnh"
Oct 30 02:55:50.082935 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:50.082899    4461 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9fv8m\" (UniqueName: \"kubernetes.io/projected/21d9fcc4-833e-4196-a646-c1f0d0c38313-kube-api-access-9fv8m\") pod \"image-registry-7fbbd597bb-twtnh\" (UID: \"21d9fcc4-833e-4196-a646-c1f0d0c38313\") " pod="openshift-image-registry/image-registry-7fbbd597bb-twtnh"
Oct 30 02:55:50.084575 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:50.084553    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 02:55:50.152772 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:50.152696    4461 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7fbbd597bb-twtnh"
Oct 30 02:55:50.204382 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:50.204349    4461 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-2k85b" event={"ID":"eee29ef7-4ad4-4ff9-8cb0-9d2a5aaabba3","Type":"ContainerStarted","Data":"ce3d266bd456d53962c280c898231b455624bf5fc3824b38e72f3e2208e104a7"}
Oct 30 02:55:50.204523 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:50.204415    4461 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-2k85b" event={"ID":"eee29ef7-4ad4-4ff9-8cb0-9d2a5aaabba3","Type":"ContainerStarted","Data":"847dfb20fff4c3c218d05b3a7d1ab0f9a767ebfd6349fa03cb335bd06c0af281"}
Oct 30 02:55:50.204523 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:50.204442    4461 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-2k85b" event={"ID":"eee29ef7-4ad4-4ff9-8cb0-9d2a5aaabba3","Type":"ContainerStarted","Data":"736c044810b32466f7a1ba0ff9184d237a4d03a1c662c9ce86fea62846a43bc1"}
Oct 30 02:55:50.208568 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:50.208542    4461 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-wjll7" event={"ID":"5f9fb000-9531-41f1-b3cf-7e79ca1a24d7","Type":"ContainerStarted","Data":"8ec3d9ef4d0e8e89621f0c3652d85f3df4badeab6e5a8d7508f98abd0dbc1d73"}
Oct 30 02:55:50.257863 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:50.257818    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 02:55:50.495649 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:50.495615    4461 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-7fbbd597bb-twtnh"]
Oct 30 02:55:50.513526 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:50.513483    4461 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod21d9fcc4_833e_4196_a646_c1f0d0c38313.slice/crio-23ccb17bda6d2184fd424f67adbfa56816450cb900df0a4f431d97d5e85e5e57 WatchSource:0}: Error finding container 23ccb17bda6d2184fd424f67adbfa56816450cb900df0a4f431d97d5e85e5e57: Status 404 returned error can't find the container with id 23ccb17bda6d2184fd424f67adbfa56816450cb900df0a4f431d97d5e85e5e57
Oct 30 02:55:50.522428 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:50.522402    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 02:55:50.698443 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:50.698385    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 02:55:50.913799 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:50.913711    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 02:55:51.091496 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:51.091462    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 02:55:51.213762 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:51.213726    4461 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7fbbd597bb-twtnh" event={"ID":"21d9fcc4-833e-4196-a646-c1f0d0c38313","Type":"ContainerStarted","Data":"c18f82d9b67a8facfd69b9b5805971a6a7103bb684783469bd3010d78e99fa57"}
Oct 30 02:55:51.213932 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:51.213775    4461 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7fbbd597bb-twtnh" event={"ID":"21d9fcc4-833e-4196-a646-c1f0d0c38313","Type":"ContainerStarted","Data":"23ccb17bda6d2184fd424f67adbfa56816450cb900df0a4f431d97d5e85e5e57"}
Oct 30 02:55:51.214342 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:51.214320    4461 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-7fbbd597bb-twtnh"
Oct 30 02:55:51.216283 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:51.216262    4461 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-wjll7" event={"ID":"5f9fb000-9531-41f1-b3cf-7e79ca1a24d7","Type":"ContainerStarted","Data":"6c0a588b9eeb8c5e482e323f26b5041136d007b2d0fa1f02922fff9ba1bda7ce"}
Oct 30 02:55:51.216368 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:51.216298    4461 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-wjll7" event={"ID":"5f9fb000-9531-41f1-b3cf-7e79ca1a24d7","Type":"ContainerStarted","Data":"fd56fa79b20bd9de5fa400c3a634f60c44e248d735a8f4e6d8eaa53150d257b2"}
Oct 30 02:55:51.216720 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:51.216705    4461 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-wjll7"
Oct 30 02:55:51.259798 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:51.259733    4461 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/image-registry-7fbbd597bb-twtnh" podStartSLOduration=2.259715888 podStartE2EDuration="2.259715888s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:55:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:55:51.258815185 +0000 UTC m=+19.308620373" watchObservedRunningTime="2026-10-30 02:55:51.259715888 +0000 UTC m=+19.309521055"
Oct 30 02:55:51.261273 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:51.261244    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 02:55:52.224603 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:52.224527    4461 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-2k85b" event={"ID":"eee29ef7-4ad4-4ff9-8cb0-9d2a5aaabba3","Type":"ContainerStarted","Data":"f5944863f1b6b00b7cff2fd75b820abd5fd04878ca797d2a6311b2aad4244bf6"}
Oct 30 02:55:54.003481 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:54.003436    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 02:55:54.044896 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:54.044864    4461 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-jrs99"]
Oct 30 02:55:54.046827 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:54.046811    4461 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-9cd89b74c-jrs99"
Oct 30 02:55:54.061578 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:54.061553    4461 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\""
Oct 30 02:55:54.074110 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:54.074066    4461 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\""
Oct 30 02:55:54.074110 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:54.074078    4461 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-mm5q5\""
Oct 30 02:55:54.074280 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:54.074202    4461 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Oct 30 02:55:54.074481 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:54.074466    4461 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Oct 30 02:55:54.074520 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:54.074468    4461 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Oct 30 02:55:54.087343 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:54.087316    4461 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-jrs99"]
Oct 30 02:55:54.178505 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:54.178462    4461 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bs8nh\" (UniqueName: \"kubernetes.io/projected/759f6f70-c39d-4607-92df-1ef50553b00f-kube-api-access-bs8nh\") pod \"prometheus-operator-9cd89b74c-jrs99\" (UID: \"759f6f70-c39d-4607-92df-1ef50553b00f\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-jrs99"
Oct 30 02:55:54.178505 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:54.178507    4461 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/759f6f70-c39d-4607-92df-1ef50553b00f-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-jrs99\" (UID: \"759f6f70-c39d-4607-92df-1ef50553b00f\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-jrs99"
Oct 30 02:55:54.178719 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:54.178527    4461 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/759f6f70-c39d-4607-92df-1ef50553b00f-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-jrs99\" (UID: \"759f6f70-c39d-4607-92df-1ef50553b00f\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-jrs99"
Oct 30 02:55:54.178719 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:54.178635    4461 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/759f6f70-c39d-4607-92df-1ef50553b00f-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-jrs99\" (UID: \"759f6f70-c39d-4607-92df-1ef50553b00f\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-jrs99"
Oct 30 02:55:54.187949 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:54.187908    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 02:55:54.279078 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:54.278978    4461 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bs8nh\" (UniqueName: \"kubernetes.io/projected/759f6f70-c39d-4607-92df-1ef50553b00f-kube-api-access-bs8nh\") pod \"prometheus-operator-9cd89b74c-jrs99\" (UID: \"759f6f70-c39d-4607-92df-1ef50553b00f\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-jrs99"
Oct 30 02:55:54.279078 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:54.279026    4461 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/759f6f70-c39d-4607-92df-1ef50553b00f-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-jrs99\" (UID: \"759f6f70-c39d-4607-92df-1ef50553b00f\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-jrs99"
Oct 30 02:55:54.279078 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:54.279045    4461 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/759f6f70-c39d-4607-92df-1ef50553b00f-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-jrs99\" (UID: \"759f6f70-c39d-4607-92df-1ef50553b00f\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-jrs99"
Oct 30 02:55:54.279078 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:54.279064    4461 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/759f6f70-c39d-4607-92df-1ef50553b00f-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-jrs99\" (UID: \"759f6f70-c39d-4607-92df-1ef50553b00f\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-jrs99"
Oct 30 02:55:54.279808 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:54.279770    4461 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/759f6f70-c39d-4607-92df-1ef50553b00f-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-jrs99\" (UID: \"759f6f70-c39d-4607-92df-1ef50553b00f\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-jrs99"
Oct 30 02:55:54.281715 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:54.281688    4461 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/759f6f70-c39d-4607-92df-1ef50553b00f-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-jrs99\" (UID: \"759f6f70-c39d-4607-92df-1ef50553b00f\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-jrs99"
Oct 30 02:55:54.281814 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:54.281714    4461 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/759f6f70-c39d-4607-92df-1ef50553b00f-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-jrs99\" (UID: \"759f6f70-c39d-4607-92df-1ef50553b00f\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-jrs99"
Oct 30 02:55:54.302672 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:54.302645    4461 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bs8nh\" (UniqueName: \"kubernetes.io/projected/759f6f70-c39d-4607-92df-1ef50553b00f-kube-api-access-bs8nh\") pod \"prometheus-operator-9cd89b74c-jrs99\" (UID: \"759f6f70-c39d-4607-92df-1ef50553b00f\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-jrs99"
Oct 30 02:55:54.357201 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:54.357172    4461 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-9cd89b74c-jrs99"
Oct 30 02:55:54.361964 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:54.361935    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 02:55:54.502587 ip-10-0-143-156 kubenswrapper[4461]: E1030 02:55:54.502535    4461 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2k85b" containerName="extractor"
Oct 30 02:55:54.649367 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:54.649294    4461 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-jrs99"]
Oct 30 02:55:54.653251 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:54.653227    4461 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod759f6f70_c39d_4607_92df_1ef50553b00f.slice/crio-5589b4648ef42c76d1031708e1fa23a83177861e74734a980ed343aa88739014 WatchSource:0}: Error finding container 5589b4648ef42c76d1031708e1fa23a83177861e74734a980ed343aa88739014: Status 404 returned error can't find the container with id 5589b4648ef42c76d1031708e1fa23a83177861e74734a980ed343aa88739014
Oct 30 02:55:55.237394 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:55.237357    4461 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-jrs99" event={"ID":"759f6f70-c39d-4607-92df-1ef50553b00f","Type":"ContainerStarted","Data":"5589b4648ef42c76d1031708e1fa23a83177861e74734a980ed343aa88739014"}
Oct 30 02:55:56.242836 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:56.242796    4461 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-jrs99" event={"ID":"759f6f70-c39d-4607-92df-1ef50553b00f","Type":"ContainerStarted","Data":"5d0a1846b1e9cb36838201cbcb9eab0aed854155104d181e9b692ec90f10e4b8"}
Oct 30 02:55:56.243314 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:56.242848    4461 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-jrs99" event={"ID":"759f6f70-c39d-4607-92df-1ef50553b00f","Type":"ContainerStarted","Data":"c39c4dbcae3e7352cb0a4a22693586a3e844d83a0a552820b2ae274f5786936e"}
Oct 30 02:55:56.284667 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:56.284619    4461 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-9cd89b74c-jrs99" podStartSLOduration=1.232609801 podStartE2EDuration="2.284604488s" totalImagesPullingTime="1.051994687s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:55:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:55:56.283835486 +0000 UTC m=+24.333640681" watchObservedRunningTime="2026-10-30 02:55:56.284604488 +0000 UTC m=+24.334409679"
Oct 30 02:55:56.667289 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:56.667212    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 02:55:58.422670 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:58.422638    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 02:55:58.651327 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:58.651286    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 02:55:58.686240 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:58.686170    4461 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-n769x"]
Oct 30 02:55:58.688899 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:58.688881    4461 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-n769x"
Oct 30 02:55:58.724348 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:58.724315    4461 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Oct 30 02:55:58.724348 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:58.724329    4461 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-szgzh\""
Oct 30 02:55:58.724542 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:58.724443    4461 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Oct 30 02:55:58.757119 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:58.757074    4461 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Oct 30 02:55:58.810570 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:58.810538    4461 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/61b66fcf-4ed1-40d0-acb5-d0490b21fd6b-node-exporter-accelerators-collector-config\") pod \"node-exporter-n769x\" (UID: \"61b66fcf-4ed1-40d0-acb5-d0490b21fd6b\") " pod="openshift-monitoring/node-exporter-n769x"
Oct 30 02:55:58.810570 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:58.810577    4461 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/61b66fcf-4ed1-40d0-acb5-d0490b21fd6b-node-exporter-textfile\") pod \"node-exporter-n769x\" (UID: \"61b66fcf-4ed1-40d0-acb5-d0490b21fd6b\") " pod="openshift-monitoring/node-exporter-n769x"
Oct 30 02:55:58.810792 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:58.810603    4461 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/61b66fcf-4ed1-40d0-acb5-d0490b21fd6b-sys\") pod \"node-exporter-n769x\" (UID: \"61b66fcf-4ed1-40d0-acb5-d0490b21fd6b\") " pod="openshift-monitoring/node-exporter-n769x"
Oct 30 02:55:58.810792 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:58.810704    4461 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/61b66fcf-4ed1-40d0-acb5-d0490b21fd6b-root\") pod \"node-exporter-n769x\" (UID: \"61b66fcf-4ed1-40d0-acb5-d0490b21fd6b\") " pod="openshift-monitoring/node-exporter-n769x"
Oct 30 02:55:58.810792 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:58.810721    4461 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fs88h\" (UniqueName: \"kubernetes.io/projected/61b66fcf-4ed1-40d0-acb5-d0490b21fd6b-kube-api-access-fs88h\") pod \"node-exporter-n769x\" (UID: \"61b66fcf-4ed1-40d0-acb5-d0490b21fd6b\") " pod="openshift-monitoring/node-exporter-n769x"
Oct 30 02:55:58.810792 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:58.810750    4461 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/61b66fcf-4ed1-40d0-acb5-d0490b21fd6b-node-exporter-tls\") pod \"node-exporter-n769x\" (UID: \"61b66fcf-4ed1-40d0-acb5-d0490b21fd6b\") " pod="openshift-monitoring/node-exporter-n769x"
Oct 30 02:55:58.810792 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:58.810778    4461 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/61b66fcf-4ed1-40d0-acb5-d0490b21fd6b-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-n769x\" (UID: \"61b66fcf-4ed1-40d0-acb5-d0490b21fd6b\") " pod="openshift-monitoring/node-exporter-n769x"
Oct 30 02:55:58.810998 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:58.810800    4461 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/61b66fcf-4ed1-40d0-acb5-d0490b21fd6b-node-exporter-wtmp\") pod \"node-exporter-n769x\" (UID: \"61b66fcf-4ed1-40d0-acb5-d0490b21fd6b\") " pod="openshift-monitoring/node-exporter-n769x"
Oct 30 02:55:58.810998 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:58.810845    4461 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/61b66fcf-4ed1-40d0-acb5-d0490b21fd6b-metrics-client-ca\") pod \"node-exporter-n769x\" (UID: \"61b66fcf-4ed1-40d0-acb5-d0490b21fd6b\") " pod="openshift-monitoring/node-exporter-n769x"
Oct 30 02:55:58.825739 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:58.825712    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 02:55:58.911996 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:58.911959    4461 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/61b66fcf-4ed1-40d0-acb5-d0490b21fd6b-node-exporter-tls\") pod \"node-exporter-n769x\" (UID: \"61b66fcf-4ed1-40d0-acb5-d0490b21fd6b\") " pod="openshift-monitoring/node-exporter-n769x"
Oct 30 02:55:58.912127 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:58.912003    4461 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/61b66fcf-4ed1-40d0-acb5-d0490b21fd6b-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-n769x\" (UID: \"61b66fcf-4ed1-40d0-acb5-d0490b21fd6b\") " pod="openshift-monitoring/node-exporter-n769x"
Oct 30 02:55:58.912166 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:58.912141    4461 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/61b66fcf-4ed1-40d0-acb5-d0490b21fd6b-node-exporter-wtmp\") pod \"node-exporter-n769x\" (UID: \"61b66fcf-4ed1-40d0-acb5-d0490b21fd6b\") " pod="openshift-monitoring/node-exporter-n769x"
Oct 30 02:55:58.912200 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:58.912176    4461 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/61b66fcf-4ed1-40d0-acb5-d0490b21fd6b-metrics-client-ca\") pod \"node-exporter-n769x\" (UID: \"61b66fcf-4ed1-40d0-acb5-d0490b21fd6b\") " pod="openshift-monitoring/node-exporter-n769x"
Oct 30 02:55:58.912237 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:58.912221    4461 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/61b66fcf-4ed1-40d0-acb5-d0490b21fd6b-node-exporter-accelerators-collector-config\") pod \"node-exporter-n769x\" (UID: \"61b66fcf-4ed1-40d0-acb5-d0490b21fd6b\") " pod="openshift-monitoring/node-exporter-n769x"
Oct 30 02:55:58.912272 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:58.912256    4461 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/61b66fcf-4ed1-40d0-acb5-d0490b21fd6b-node-exporter-textfile\") pod \"node-exporter-n769x\" (UID: \"61b66fcf-4ed1-40d0-acb5-d0490b21fd6b\") " pod="openshift-monitoring/node-exporter-n769x"
Oct 30 02:55:58.912316 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:58.912284    4461 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/61b66fcf-4ed1-40d0-acb5-d0490b21fd6b-sys\") pod \"node-exporter-n769x\" (UID: \"61b66fcf-4ed1-40d0-acb5-d0490b21fd6b\") " pod="openshift-monitoring/node-exporter-n769x"
Oct 30 02:55:58.912367 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:58.912318    4461 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/61b66fcf-4ed1-40d0-acb5-d0490b21fd6b-root\") pod \"node-exporter-n769x\" (UID: \"61b66fcf-4ed1-40d0-acb5-d0490b21fd6b\") " pod="openshift-monitoring/node-exporter-n769x"
Oct 30 02:55:58.912367 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:58.912341    4461 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fs88h\" (UniqueName: \"kubernetes.io/projected/61b66fcf-4ed1-40d0-acb5-d0490b21fd6b-kube-api-access-fs88h\") pod \"node-exporter-n769x\" (UID: \"61b66fcf-4ed1-40d0-acb5-d0490b21fd6b\") " pod="openshift-monitoring/node-exporter-n769x"
Oct 30 02:55:58.912367 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:58.912333    4461 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/61b66fcf-4ed1-40d0-acb5-d0490b21fd6b-node-exporter-wtmp\") pod \"node-exporter-n769x\" (UID: \"61b66fcf-4ed1-40d0-acb5-d0490b21fd6b\") " pod="openshift-monitoring/node-exporter-n769x"
Oct 30 02:55:58.912510 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:58.912385    4461 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/61b66fcf-4ed1-40d0-acb5-d0490b21fd6b-sys\") pod \"node-exporter-n769x\" (UID: \"61b66fcf-4ed1-40d0-acb5-d0490b21fd6b\") " pod="openshift-monitoring/node-exporter-n769x"
Oct 30 02:55:58.912510 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:58.912408    4461 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/61b66fcf-4ed1-40d0-acb5-d0490b21fd6b-root\") pod \"node-exporter-n769x\" (UID: \"61b66fcf-4ed1-40d0-acb5-d0490b21fd6b\") " pod="openshift-monitoring/node-exporter-n769x"
Oct 30 02:55:58.912685 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:58.912665    4461 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/61b66fcf-4ed1-40d0-acb5-d0490b21fd6b-node-exporter-textfile\") pod \"node-exporter-n769x\" (UID: \"61b66fcf-4ed1-40d0-acb5-d0490b21fd6b\") " pod="openshift-monitoring/node-exporter-n769x"
Oct 30 02:55:58.912815 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:58.912797    4461 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/61b66fcf-4ed1-40d0-acb5-d0490b21fd6b-metrics-client-ca\") pod \"node-exporter-n769x\" (UID: \"61b66fcf-4ed1-40d0-acb5-d0490b21fd6b\") " pod="openshift-monitoring/node-exporter-n769x"
Oct 30 02:55:58.912880 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:58.912843    4461 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/61b66fcf-4ed1-40d0-acb5-d0490b21fd6b-node-exporter-accelerators-collector-config\") pod \"node-exporter-n769x\" (UID: \"61b66fcf-4ed1-40d0-acb5-d0490b21fd6b\") " pod="openshift-monitoring/node-exporter-n769x"
Oct 30 02:55:58.914469 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:58.914453    4461 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/61b66fcf-4ed1-40d0-acb5-d0490b21fd6b-node-exporter-tls\") pod \"node-exporter-n769x\" (UID: \"61b66fcf-4ed1-40d0-acb5-d0490b21fd6b\") " pod="openshift-monitoring/node-exporter-n769x"
Oct 30 02:55:58.914565 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:58.914549    4461 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/61b66fcf-4ed1-40d0-acb5-d0490b21fd6b-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-n769x\" (UID: \"61b66fcf-4ed1-40d0-acb5-d0490b21fd6b\") " pod="openshift-monitoring/node-exporter-n769x"
Oct 30 02:55:58.938685 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:58.938621    4461 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fs88h\" (UniqueName: \"kubernetes.io/projected/61b66fcf-4ed1-40d0-acb5-d0490b21fd6b-kube-api-access-fs88h\") pod \"node-exporter-n769x\" (UID: \"61b66fcf-4ed1-40d0-acb5-d0490b21fd6b\") " pod="openshift-monitoring/node-exporter-n769x"
Oct 30 02:55:59.002817 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:59.002775    4461 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-n769x"
Oct 30 02:55:59.014244 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:55:59.014216    4461 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod61b66fcf_4ed1_40d0_acb5_d0490b21fd6b.slice/crio-cb7c23356f05f0ff7a7ad880e1eff28ed4a7908a58bad5d3595701468b32993e WatchSource:0}: Error finding container cb7c23356f05f0ff7a7ad880e1eff28ed4a7908a58bad5d3595701468b32993e: Status 404 returned error can't find the container with id cb7c23356f05f0ff7a7ad880e1eff28ed4a7908a58bad5d3595701468b32993e
Oct 30 02:55:59.030036 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:59.030009    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 02:55:59.233711 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:59.233677    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 02:55:59.254354 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:59.254322    4461 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-n769x" event={"ID":"61b66fcf-4ed1-40d0-acb5-d0490b21fd6b","Type":"ContainerStarted","Data":"cb7c23356f05f0ff7a7ad880e1eff28ed4a7908a58bad5d3595701468b32993e"}
Oct 30 02:55:59.408321 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:59.408290    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 02:55:59.932310 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:55:59.932283    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 02:56:00.098545 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:56:00.098511    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 02:56:00.259002 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:56:00.258972    4461 generic.go:356] "Generic (PLEG): container finished" podID="61b66fcf-4ed1-40d0-acb5-d0490b21fd6b" containerID="c355935cd20434580598d0f6266b5e92225f56a35264baedb8a5bd8c33f19bef" exitCode=0
Oct 30 02:56:00.259192 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:56:00.259014    4461 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-n769x" event={"ID":"61b66fcf-4ed1-40d0-acb5-d0490b21fd6b","Type":"ContainerDied","Data":"c355935cd20434580598d0f6266b5e92225f56a35264baedb8a5bd8c33f19bef"}
Oct 30 02:56:00.269265 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:56:00.269232    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 02:56:01.264444 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:56:01.264406    4461 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-n769x" event={"ID":"61b66fcf-4ed1-40d0-acb5-d0490b21fd6b","Type":"ContainerStarted","Data":"bfa1cd9df40095cc0b6abccb20972a55776838c45f73dbe3457e0b684445c092"}
Oct 30 02:56:01.264852 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:56:01.264458    4461 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-n769x" event={"ID":"61b66fcf-4ed1-40d0-acb5-d0490b21fd6b","Type":"ContainerStarted","Data":"d9bec3f8567aa50e4df2c6fe1fc87e57ee55ddf8b825a47dc7fc20b3c44082c3"}
Oct 30 02:56:01.333788 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:56:01.333741    4461 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-n769x" podStartSLOduration=2.414977707 podStartE2EDuration="3.33372764s" totalImagesPullingTime="852.50147ms" totalInitContainerRuntime="66.248463ms" isStatefulPod=true podCreationTimestamp="2026-10-30 02:55:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:56:01.321436512 +0000 UTC m=+29.371241699" watchObservedRunningTime="2026-10-30 02:56:01.33372764 +0000 UTC m=+29.383532815"
Oct 30 02:56:01.967382 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:56:01.967346    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 02:56:02.164597 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:56:02.164565    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 02:56:02.230640 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:56:02.230555    4461 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-wjll7"
Oct 30 02:56:02.330792 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:56:02.330760    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 02:56:03.117570 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:56:03.117537    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 02:56:03.145411 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:56:03.145377    4461 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-7d8666798c-cnjhs"]
Oct 30 02:56:03.170983 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:56:03.170961    4461 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-7d8666798c-cnjhs"
Oct 30 02:56:03.174843 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:56:03.174818    4461 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-7d8666798c-cnjhs"]
Oct 30 02:56:03.198822 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:56:03.198786    4461 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\""
Oct 30 02:56:03.198980 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:56:03.198854    4461 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-8kh9a08ce1jd\""
Oct 30 02:56:03.198980 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:56:03.198919    4461 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-v267c\""
Oct 30 02:56:03.198980 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:56:03.198955    4461 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\""
Oct 30 02:56:03.199085 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:56:03.198991    4461 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Oct 30 02:56:03.199085 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:56:03.198997    4461 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\""
Oct 30 02:56:03.246070 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:56:03.246039    4461 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9e87e1b3-bbad-44a8-9684-83aed2d040d5-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-7d8666798c-cnjhs\" (UID: \"9e87e1b3-bbad-44a8-9684-83aed2d040d5\") " pod="openshift-monitoring/metrics-server-7d8666798c-cnjhs"
Oct 30 02:56:03.246246 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:56:03.246075    4461 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-57fsg\" (UniqueName: \"kubernetes.io/projected/9e87e1b3-bbad-44a8-9684-83aed2d040d5-kube-api-access-57fsg\") pod \"metrics-server-7d8666798c-cnjhs\" (UID: \"9e87e1b3-bbad-44a8-9684-83aed2d040d5\") " pod="openshift-monitoring/metrics-server-7d8666798c-cnjhs"
Oct 30 02:56:03.246246 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:56:03.246111    4461 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/9e87e1b3-bbad-44a8-9684-83aed2d040d5-secret-metrics-server-tls\") pod \"metrics-server-7d8666798c-cnjhs\" (UID: \"9e87e1b3-bbad-44a8-9684-83aed2d040d5\") " pod="openshift-monitoring/metrics-server-7d8666798c-cnjhs"
Oct 30 02:56:03.246246 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:56:03.246197    4461 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/9e87e1b3-bbad-44a8-9684-83aed2d040d5-audit-log\") pod \"metrics-server-7d8666798c-cnjhs\" (UID: \"9e87e1b3-bbad-44a8-9684-83aed2d040d5\") " pod="openshift-monitoring/metrics-server-7d8666798c-cnjhs"
Oct 30 02:56:03.246246 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:56:03.246215    4461 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/9e87e1b3-bbad-44a8-9684-83aed2d040d5-client-ca-bundle\") pod \"metrics-server-7d8666798c-cnjhs\" (UID: \"9e87e1b3-bbad-44a8-9684-83aed2d040d5\") " pod="openshift-monitoring/metrics-server-7d8666798c-cnjhs"
Oct 30 02:56:03.246246 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:56:03.246235    4461 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/9e87e1b3-bbad-44a8-9684-83aed2d040d5-secret-metrics-server-client-certs\") pod \"metrics-server-7d8666798c-cnjhs\" (UID: \"9e87e1b3-bbad-44a8-9684-83aed2d040d5\") " pod="openshift-monitoring/metrics-server-7d8666798c-cnjhs"
Oct 30 02:56:03.246397 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:56:03.246263    4461 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/9e87e1b3-bbad-44a8-9684-83aed2d040d5-metrics-server-audit-profiles\") pod \"metrics-server-7d8666798c-cnjhs\" (UID: \"9e87e1b3-bbad-44a8-9684-83aed2d040d5\") " pod="openshift-monitoring/metrics-server-7d8666798c-cnjhs"
Oct 30 02:56:03.285602 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:56:03.285571    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 02:56:03.346684 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:56:03.346636    4461 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9e87e1b3-bbad-44a8-9684-83aed2d040d5-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-7d8666798c-cnjhs\" (UID: \"9e87e1b3-bbad-44a8-9684-83aed2d040d5\") " pod="openshift-monitoring/metrics-server-7d8666798c-cnjhs"
Oct 30 02:56:03.347230 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:56:03.346689    4461 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-57fsg\" (UniqueName: \"kubernetes.io/projected/9e87e1b3-bbad-44a8-9684-83aed2d040d5-kube-api-access-57fsg\") pod \"metrics-server-7d8666798c-cnjhs\" (UID: \"9e87e1b3-bbad-44a8-9684-83aed2d040d5\") " pod="openshift-monitoring/metrics-server-7d8666798c-cnjhs"
Oct 30 02:56:03.347230 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:56:03.346720    4461 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/9e87e1b3-bbad-44a8-9684-83aed2d040d5-secret-metrics-server-tls\") pod \"metrics-server-7d8666798c-cnjhs\" (UID: \"9e87e1b3-bbad-44a8-9684-83aed2d040d5\") " pod="openshift-monitoring/metrics-server-7d8666798c-cnjhs"
Oct 30 02:56:03.347230 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:56:03.346762    4461 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/9e87e1b3-bbad-44a8-9684-83aed2d040d5-audit-log\") pod \"metrics-server-7d8666798c-cnjhs\" (UID: \"9e87e1b3-bbad-44a8-9684-83aed2d040d5\") " pod="openshift-monitoring/metrics-server-7d8666798c-cnjhs"
Oct 30 02:56:03.347230 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:56:03.346784    4461 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/9e87e1b3-bbad-44a8-9684-83aed2d040d5-client-ca-bundle\") pod \"metrics-server-7d8666798c-cnjhs\" (UID: \"9e87e1b3-bbad-44a8-9684-83aed2d040d5\") " pod="openshift-monitoring/metrics-server-7d8666798c-cnjhs"
Oct 30 02:56:03.347230 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:56:03.346811    4461 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/9e87e1b3-bbad-44a8-9684-83aed2d040d5-secret-metrics-server-client-certs\") pod \"metrics-server-7d8666798c-cnjhs\" (UID: \"9e87e1b3-bbad-44a8-9684-83aed2d040d5\") " pod="openshift-monitoring/metrics-server-7d8666798c-cnjhs"
Oct 30 02:56:03.347230 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:56:03.346852    4461 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/9e87e1b3-bbad-44a8-9684-83aed2d040d5-metrics-server-audit-profiles\") pod \"metrics-server-7d8666798c-cnjhs\" (UID: \"9e87e1b3-bbad-44a8-9684-83aed2d040d5\") " pod="openshift-monitoring/metrics-server-7d8666798c-cnjhs"
Oct 30 02:56:03.347485 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:56:03.347293    4461 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/9e87e1b3-bbad-44a8-9684-83aed2d040d5-audit-log\") pod \"metrics-server-7d8666798c-cnjhs\" (UID: \"9e87e1b3-bbad-44a8-9684-83aed2d040d5\") " pod="openshift-monitoring/metrics-server-7d8666798c-cnjhs"
Oct 30 02:56:03.347582 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:56:03.347558    4461 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9e87e1b3-bbad-44a8-9684-83aed2d040d5-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-7d8666798c-cnjhs\" (UID: \"9e87e1b3-bbad-44a8-9684-83aed2d040d5\") " pod="openshift-monitoring/metrics-server-7d8666798c-cnjhs"
Oct 30 02:56:03.347805 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:56:03.347790    4461 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/9e87e1b3-bbad-44a8-9684-83aed2d040d5-metrics-server-audit-profiles\") pod \"metrics-server-7d8666798c-cnjhs\" (UID: \"9e87e1b3-bbad-44a8-9684-83aed2d040d5\") " pod="openshift-monitoring/metrics-server-7d8666798c-cnjhs"
Oct 30 02:56:03.349805 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:56:03.349778    4461 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/9e87e1b3-bbad-44a8-9684-83aed2d040d5-secret-metrics-server-client-certs\") pod \"metrics-server-7d8666798c-cnjhs\" (UID: \"9e87e1b3-bbad-44a8-9684-83aed2d040d5\") " pod="openshift-monitoring/metrics-server-7d8666798c-cnjhs"
Oct 30 02:56:03.349805 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:56:03.349794    4461 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/9e87e1b3-bbad-44a8-9684-83aed2d040d5-secret-metrics-server-tls\") pod \"metrics-server-7d8666798c-cnjhs\" (UID: \"9e87e1b3-bbad-44a8-9684-83aed2d040d5\") " pod="openshift-monitoring/metrics-server-7d8666798c-cnjhs"
Oct 30 02:56:03.349937 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:56:03.349409    4461 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/9e87e1b3-bbad-44a8-9684-83aed2d040d5-client-ca-bundle\") pod \"metrics-server-7d8666798c-cnjhs\" (UID: \"9e87e1b3-bbad-44a8-9684-83aed2d040d5\") " pod="openshift-monitoring/metrics-server-7d8666798c-cnjhs"
Oct 30 02:56:03.370255 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:56:03.370193    4461 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-57fsg\" (UniqueName: \"kubernetes.io/projected/9e87e1b3-bbad-44a8-9684-83aed2d040d5-kube-api-access-57fsg\") pod \"metrics-server-7d8666798c-cnjhs\" (UID: \"9e87e1b3-bbad-44a8-9684-83aed2d040d5\") " pod="openshift-monitoring/metrics-server-7d8666798c-cnjhs"
Oct 30 02:56:03.492897 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:56:03.492867    4461 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-st7wb"]
Oct 30 02:56:03.493193 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:56:03.493161    4461 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-7d8666798c-cnjhs"
Oct 30 02:56:03.511537 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:56:03.511505    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 02:56:03.523067 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:56:03.523040    4461 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-st7wb"]
Oct 30 02:56:03.523206 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:56:03.523074    4461 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-77f4f76594-st7wb"
Oct 30 02:56:03.538134 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:56:03.538112    4461 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\""
Oct 30 02:56:03.538245 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:56:03.538172    4461 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-8fhr6\""
Oct 30 02:56:03.649472 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:56:03.649393    4461 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/924e9f2a-f91e-43b5-ad1c-ea75d8aaa765-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-st7wb\" (UID: \"924e9f2a-f91e-43b5-ad1c-ea75d8aaa765\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-st7wb"
Oct 30 02:56:03.649472 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:56:03.649425    4461 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t7qgj\" (UniqueName: \"kubernetes.io/projected/924e9f2a-f91e-43b5-ad1c-ea75d8aaa765-kube-api-access-t7qgj\") pod \"monitoring-plugin-77f4f76594-st7wb\" (UID: \"924e9f2a-f91e-43b5-ad1c-ea75d8aaa765\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-st7wb"
Oct 30 02:56:03.682369 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:56:03.682339    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 02:56:03.750167 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:56:03.750131    4461 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/924e9f2a-f91e-43b5-ad1c-ea75d8aaa765-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-st7wb\" (UID: \"924e9f2a-f91e-43b5-ad1c-ea75d8aaa765\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-st7wb"
Oct 30 02:56:03.750167 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:56:03.750168    4461 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t7qgj\" (UniqueName: \"kubernetes.io/projected/924e9f2a-f91e-43b5-ad1c-ea75d8aaa765-kube-api-access-t7qgj\") pod \"monitoring-plugin-77f4f76594-st7wb\" (UID: \"924e9f2a-f91e-43b5-ad1c-ea75d8aaa765\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-st7wb"
Oct 30 02:56:03.752535 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:56:03.752512    4461 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/924e9f2a-f91e-43b5-ad1c-ea75d8aaa765-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-st7wb\" (UID: \"924e9f2a-f91e-43b5-ad1c-ea75d8aaa765\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-st7wb"
Oct 30 02:56:03.773547 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:56:03.773524    4461 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t7qgj\" (UniqueName: \"kubernetes.io/projected/924e9f2a-f91e-43b5-ad1c-ea75d8aaa765-kube-api-access-t7qgj\") pod \"monitoring-plugin-77f4f76594-st7wb\" (UID: \"924e9f2a-f91e-43b5-ad1c-ea75d8aaa765\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-st7wb"
Oct 30 02:56:03.830083 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:56:03.830051    4461 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-7d8666798c-cnjhs"]
Oct 30 02:56:03.833537 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:56:03.833506    4461 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9e87e1b3_bbad_44a8_9684_83aed2d040d5.slice/crio-880b288f27e1a839a28923c364188443cded9d6f4bb252c16e152d81f233fe23 WatchSource:0}: Error finding container 880b288f27e1a839a28923c364188443cded9d6f4bb252c16e152d81f233fe23: Status 404 returned error can't find the container with id 880b288f27e1a839a28923c364188443cded9d6f4bb252c16e152d81f233fe23
Oct 30 02:56:03.845154 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:56:03.845127    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 02:56:03.867318 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:56:03.867294    4461 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-77f4f76594-st7wb"
Oct 30 02:56:04.189032 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:56:04.188961    4461 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-st7wb"]
Oct 30 02:56:04.192142 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:56:04.192112    4461 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod924e9f2a_f91e_43b5_ad1c_ea75d8aaa765.slice/crio-de8c935f50862861dd7cb2831c0c8d62973ad290a9b3b3c049740452ef031001 WatchSource:0}: Error finding container de8c935f50862861dd7cb2831c0c8d62973ad290a9b3b3c049740452ef031001: Status 404 returned error can't find the container with id de8c935f50862861dd7cb2831c0c8d62973ad290a9b3b3c049740452ef031001
Oct 30 02:56:04.275092 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:56:04.275060    4461 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-7d8666798c-cnjhs" event={"ID":"9e87e1b3-bbad-44a8-9684-83aed2d040d5","Type":"ContainerStarted","Data":"880b288f27e1a839a28923c364188443cded9d6f4bb252c16e152d81f233fe23"}
Oct 30 02:56:04.276513 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:56:04.276484    4461 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-77f4f76594-st7wb" event={"ID":"924e9f2a-f91e-43b5-ad1c-ea75d8aaa765","Type":"ContainerStarted","Data":"de8c935f50862861dd7cb2831c0c8d62973ad290a9b3b3c049740452ef031001"}
Oct 30 02:56:05.606020 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:56:05.605987    4461 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Oct 30 02:56:05.637120 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:56:05.637078    4461 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Oct 30 02:56:05.637283 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:56:05.637223    4461 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 02:56:05.655325 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:56:05.655297    4461 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Oct 30 02:56:05.655325 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:56:05.655297    4461 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Oct 30 02:56:05.655596 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:56:05.655575    4461 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Oct 30 02:56:05.655691 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:56:05.655678    4461 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Oct 30 02:56:05.655825 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:56:05.655811    4461 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-ktdw7\""
Oct 30 02:56:05.655890 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:56:05.655878    4461 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Oct 30 02:56:05.655942 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:56:05.655921    4461 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Oct 30 02:56:05.656016 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:56:05.656005    4461 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Oct 30 02:56:05.656070 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:56:05.656051    4461 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-8lokipe3p784l\""
Oct 30 02:56:05.674360 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:56:05.674330    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 02:56:05.755727 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:56:05.755702    4461 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Oct 30 02:56:05.755851 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:56:05.755717    4461 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Oct 30 02:56:05.755908 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:56:05.755728    4461 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Oct 30 02:56:05.756009 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:56:05.755990    4461 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Oct 30 02:56:05.756139 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:56:05.756082    4461 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Oct 30 02:56:05.765724 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:56:05.765697    4461 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/4c1f3a6e-55b5-49a6-a08b-e0a6ef968f92-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"4c1f3a6e-55b5-49a6-a08b-e0a6ef968f92\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 02:56:05.765819 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:56:05.765776    4461 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4c1f3a6e-55b5-49a6-a08b-e0a6ef968f92-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"4c1f3a6e-55b5-49a6-a08b-e0a6ef968f92\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 02:56:05.765819 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:56:05.765805    4461 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/4c1f3a6e-55b5-49a6-a08b-e0a6ef968f92-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"4c1f3a6e-55b5-49a6-a08b-e0a6ef968f92\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 02:56:05.765923 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:56:05.765898    4461 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/4c1f3a6e-55b5-49a6-a08b-e0a6ef968f92-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"4c1f3a6e-55b5-49a6-a08b-e0a6ef968f92\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 02:56:05.765971 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:56:05.765941    4461 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4c1f3a6e-55b5-49a6-a08b-e0a6ef968f92-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"4c1f3a6e-55b5-49a6-a08b-e0a6ef968f92\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 02:56:05.766035 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:56:05.766018    4461 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/4c1f3a6e-55b5-49a6-a08b-e0a6ef968f92-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"4c1f3a6e-55b5-49a6-a08b-e0a6ef968f92\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 02:56:05.766089 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:56:05.766074    4461 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/4c1f3a6e-55b5-49a6-a08b-e0a6ef968f92-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"4c1f3a6e-55b5-49a6-a08b-e0a6ef968f92\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 02:56:05.766192 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:56:05.766148    4461 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4c1f3a6e-55b5-49a6-a08b-e0a6ef968f92-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"4c1f3a6e-55b5-49a6-a08b-e0a6ef968f92\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 02:56:05.766192 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:56:05.766179    4461 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/4c1f3a6e-55b5-49a6-a08b-e0a6ef968f92-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"4c1f3a6e-55b5-49a6-a08b-e0a6ef968f92\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 02:56:05.766263 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:56:05.766237    4461 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/4c1f3a6e-55b5-49a6-a08b-e0a6ef968f92-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"4c1f3a6e-55b5-49a6-a08b-e0a6ef968f92\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 02:56:05.766300 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:56:05.766261    4461 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/4c1f3a6e-55b5-49a6-a08b-e0a6ef968f92-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"4c1f3a6e-55b5-49a6-a08b-e0a6ef968f92\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 02:56:05.766300 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:56:05.766288    4461 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/4c1f3a6e-55b5-49a6-a08b-e0a6ef968f92-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"4c1f3a6e-55b5-49a6-a08b-e0a6ef968f92\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 02:56:05.766409 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:56:05.766344    4461 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/4c1f3a6e-55b5-49a6-a08b-e0a6ef968f92-config\") pod \"prometheus-k8s-0\" (UID: \"4c1f3a6e-55b5-49a6-a08b-e0a6ef968f92\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 02:56:05.766459 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:56:05.766422    4461 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/4c1f3a6e-55b5-49a6-a08b-e0a6ef968f92-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"4c1f3a6e-55b5-49a6-a08b-e0a6ef968f92\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 02:56:05.766508 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:56:05.766474    4461 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/4c1f3a6e-55b5-49a6-a08b-e0a6ef968f92-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"4c1f3a6e-55b5-49a6-a08b-e0a6ef968f92\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 02:56:05.766508 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:56:05.766501    4461 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/4c1f3a6e-55b5-49a6-a08b-e0a6ef968f92-web-config\") pod \"prometheus-k8s-0\" (UID: \"4c1f3a6e-55b5-49a6-a08b-e0a6ef968f92\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 02:56:05.766605 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:56:05.766533    4461 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/4c1f3a6e-55b5-49a6-a08b-e0a6ef968f92-config-out\") pod \"prometheus-k8s-0\" (UID: \"4c1f3a6e-55b5-49a6-a08b-e0a6ef968f92\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 02:56:05.766605 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:56:05.766571    4461 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4c1f3a6e-55b5-49a6-a08b-e0a6ef968f92-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"4c1f3a6e-55b5-49a6-a08b-e0a6ef968f92\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 02:56:05.766707 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:56:05.766603    4461 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k794n\" (UniqueName: \"kubernetes.io/projected/4c1f3a6e-55b5-49a6-a08b-e0a6ef968f92-kube-api-access-k794n\") pod \"prometheus-k8s-0\" (UID: \"4c1f3a6e-55b5-49a6-a08b-e0a6ef968f92\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 02:56:05.766707 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:56:05.766637    4461 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/4c1f3a6e-55b5-49a6-a08b-e0a6ef968f92-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"4c1f3a6e-55b5-49a6-a08b-e0a6ef968f92\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 02:56:05.769309 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:56:05.769292    4461 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Oct 30 02:56:05.769381 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:56:05.769331    4461 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Oct 30 02:56:05.867495 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:56:05.867422    4461 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/4c1f3a6e-55b5-49a6-a08b-e0a6ef968f92-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"4c1f3a6e-55b5-49a6-a08b-e0a6ef968f92\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 02:56:05.867495 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:56:05.867452    4461 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4c1f3a6e-55b5-49a6-a08b-e0a6ef968f92-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"4c1f3a6e-55b5-49a6-a08b-e0a6ef968f92\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 02:56:05.867495 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:56:05.867470    4461 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/4c1f3a6e-55b5-49a6-a08b-e0a6ef968f92-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"4c1f3a6e-55b5-49a6-a08b-e0a6ef968f92\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 02:56:05.867495 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:56:05.867494    4461 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/4c1f3a6e-55b5-49a6-a08b-e0a6ef968f92-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"4c1f3a6e-55b5-49a6-a08b-e0a6ef968f92\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 02:56:05.867872 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:56:05.867511    4461 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4c1f3a6e-55b5-49a6-a08b-e0a6ef968f92-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"4c1f3a6e-55b5-49a6-a08b-e0a6ef968f92\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 02:56:05.867872 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:56:05.867535    4461 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/4c1f3a6e-55b5-49a6-a08b-e0a6ef968f92-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"4c1f3a6e-55b5-49a6-a08b-e0a6ef968f92\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 02:56:05.867872 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:56:05.867558    4461 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/4c1f3a6e-55b5-49a6-a08b-e0a6ef968f92-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"4c1f3a6e-55b5-49a6-a08b-e0a6ef968f92\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 02:56:05.867872 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:56:05.867588    4461 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4c1f3a6e-55b5-49a6-a08b-e0a6ef968f92-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"4c1f3a6e-55b5-49a6-a08b-e0a6ef968f92\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 02:56:05.867872 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:56:05.867620    4461 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/4c1f3a6e-55b5-49a6-a08b-e0a6ef968f92-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"4c1f3a6e-55b5-49a6-a08b-e0a6ef968f92\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 02:56:05.867872 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:56:05.867656    4461 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/4c1f3a6e-55b5-49a6-a08b-e0a6ef968f92-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"4c1f3a6e-55b5-49a6-a08b-e0a6ef968f92\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 02:56:05.867872 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:56:05.867682    4461 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/4c1f3a6e-55b5-49a6-a08b-e0a6ef968f92-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"4c1f3a6e-55b5-49a6-a08b-e0a6ef968f92\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 02:56:05.867872 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:56:05.867703    4461 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/4c1f3a6e-55b5-49a6-a08b-e0a6ef968f92-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"4c1f3a6e-55b5-49a6-a08b-e0a6ef968f92\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 02:56:05.867872 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:56:05.867735    4461 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/4c1f3a6e-55b5-49a6-a08b-e0a6ef968f92-config\") pod \"prometheus-k8s-0\" (UID: \"4c1f3a6e-55b5-49a6-a08b-e0a6ef968f92\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 02:56:05.867872 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:56:05.867758    4461 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/4c1f3a6e-55b5-49a6-a08b-e0a6ef968f92-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"4c1f3a6e-55b5-49a6-a08b-e0a6ef968f92\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 02:56:05.867872 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:56:05.867787    4461 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/4c1f3a6e-55b5-49a6-a08b-e0a6ef968f92-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"4c1f3a6e-55b5-49a6-a08b-e0a6ef968f92\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 02:56:05.867872 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:56:05.867811    4461 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/4c1f3a6e-55b5-49a6-a08b-e0a6ef968f92-web-config\") pod \"prometheus-k8s-0\" (UID: \"4c1f3a6e-55b5-49a6-a08b-e0a6ef968f92\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 02:56:05.867872 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:56:05.867842    4461 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/4c1f3a6e-55b5-49a6-a08b-e0a6ef968f92-config-out\") pod \"prometheus-k8s-0\" (UID: \"4c1f3a6e-55b5-49a6-a08b-e0a6ef968f92\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 02:56:05.867872 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:56:05.867873    4461 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4c1f3a6e-55b5-49a6-a08b-e0a6ef968f92-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"4c1f3a6e-55b5-49a6-a08b-e0a6ef968f92\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 02:56:05.868549 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:56:05.867905    4461 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k794n\" (UniqueName: \"kubernetes.io/projected/4c1f3a6e-55b5-49a6-a08b-e0a6ef968f92-kube-api-access-k794n\") pod \"prometheus-k8s-0\" (UID: \"4c1f3a6e-55b5-49a6-a08b-e0a6ef968f92\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 02:56:05.868549 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:56:05.867939    4461 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/4c1f3a6e-55b5-49a6-a08b-e0a6ef968f92-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"4c1f3a6e-55b5-49a6-a08b-e0a6ef968f92\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 02:56:05.868549 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:56:05.868340    4461 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/4c1f3a6e-55b5-49a6-a08b-e0a6ef968f92-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"4c1f3a6e-55b5-49a6-a08b-e0a6ef968f92\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 02:56:05.868549 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:56:05.868426    4461 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4c1f3a6e-55b5-49a6-a08b-e0a6ef968f92-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"4c1f3a6e-55b5-49a6-a08b-e0a6ef968f92\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 02:56:05.868756 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:56:05.868546    4461 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4c1f3a6e-55b5-49a6-a08b-e0a6ef968f92-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"4c1f3a6e-55b5-49a6-a08b-e0a6ef968f92\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 02:56:05.868921 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:56:05.868893    4461 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/4c1f3a6e-55b5-49a6-a08b-e0a6ef968f92-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"4c1f3a6e-55b5-49a6-a08b-e0a6ef968f92\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 02:56:05.869148 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:56:05.869124    4461 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4c1f3a6e-55b5-49a6-a08b-e0a6ef968f92-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"4c1f3a6e-55b5-49a6-a08b-e0a6ef968f92\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 02:56:05.871077 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:56:05.871049    4461 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/4c1f3a6e-55b5-49a6-a08b-e0a6ef968f92-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"4c1f3a6e-55b5-49a6-a08b-e0a6ef968f92\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 02:56:05.871571 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:56:05.871517    4461 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/4c1f3a6e-55b5-49a6-a08b-e0a6ef968f92-config\") pod \"prometheus-k8s-0\" (UID: \"4c1f3a6e-55b5-49a6-a08b-e0a6ef968f92\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 02:56:05.872014 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:56:05.871987    4461 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/4c1f3a6e-55b5-49a6-a08b-e0a6ef968f92-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"4c1f3a6e-55b5-49a6-a08b-e0a6ef968f92\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 02:56:05.872211 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:56:05.872189    4461 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/4c1f3a6e-55b5-49a6-a08b-e0a6ef968f92-web-config\") pod \"prometheus-k8s-0\" (UID: \"4c1f3a6e-55b5-49a6-a08b-e0a6ef968f92\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 02:56:05.872288 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:56:05.872224    4461 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4c1f3a6e-55b5-49a6-a08b-e0a6ef968f92-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"4c1f3a6e-55b5-49a6-a08b-e0a6ef968f92\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 02:56:05.873125 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:56:05.873056    4461 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/4c1f3a6e-55b5-49a6-a08b-e0a6ef968f92-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"4c1f3a6e-55b5-49a6-a08b-e0a6ef968f92\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 02:56:05.873541 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:56:05.873516    4461 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/4c1f3a6e-55b5-49a6-a08b-e0a6ef968f92-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"4c1f3a6e-55b5-49a6-a08b-e0a6ef968f92\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 02:56:05.873623 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:56:05.873609    4461 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/4c1f3a6e-55b5-49a6-a08b-e0a6ef968f92-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"4c1f3a6e-55b5-49a6-a08b-e0a6ef968f92\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 02:56:05.873905 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:56:05.873888    4461 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/4c1f3a6e-55b5-49a6-a08b-e0a6ef968f92-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"4c1f3a6e-55b5-49a6-a08b-e0a6ef968f92\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 02:56:05.874115 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:56:05.874077    4461 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/4c1f3a6e-55b5-49a6-a08b-e0a6ef968f92-config-out\") pod \"prometheus-k8s-0\" (UID: \"4c1f3a6e-55b5-49a6-a08b-e0a6ef968f92\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 02:56:05.874292 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:56:05.874276    4461 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/4c1f3a6e-55b5-49a6-a08b-e0a6ef968f92-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"4c1f3a6e-55b5-49a6-a08b-e0a6ef968f92\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 02:56:05.874444 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:56:05.874424    4461 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/4c1f3a6e-55b5-49a6-a08b-e0a6ef968f92-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"4c1f3a6e-55b5-49a6-a08b-e0a6ef968f92\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 02:56:05.874497 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:56:05.874454    4461 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/4c1f3a6e-55b5-49a6-a08b-e0a6ef968f92-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"4c1f3a6e-55b5-49a6-a08b-e0a6ef968f92\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 02:56:05.875176 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:56:05.875156    4461 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/4c1f3a6e-55b5-49a6-a08b-e0a6ef968f92-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"4c1f3a6e-55b5-49a6-a08b-e0a6ef968f92\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 02:56:05.895549 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:56:05.895520    4461 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k794n\" (UniqueName: \"kubernetes.io/projected/4c1f3a6e-55b5-49a6-a08b-e0a6ef968f92-kube-api-access-k794n\") pod \"prometheus-k8s-0\" (UID: \"4c1f3a6e-55b5-49a6-a08b-e0a6ef968f92\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 02:56:05.997809 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:56:05.997772    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 02:56:06.083359 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:56:06.083310    4461 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 02:56:06.200049 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:56:06.200019    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 02:56:06.289175 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:56:06.289136    4461 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-7d8666798c-cnjhs" event={"ID":"9e87e1b3-bbad-44a8-9684-83aed2d040d5","Type":"ContainerStarted","Data":"15234ab6c83ce78bb53355e29121547db6139f6f1f3193924e06cbdb8039708c"}
Oct 30 02:56:06.290674 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:56:06.290650    4461 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-77f4f76594-st7wb" event={"ID":"924e9f2a-f91e-43b5-ad1c-ea75d8aaa765","Type":"ContainerStarted","Data":"6a4c2dac4ca8f2d9ac9017c38cede7b49d9120e4b44b86599d691ed2bed514ee"}
Oct 30 02:56:06.290866 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:56:06.290848    4461 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-77f4f76594-st7wb"
Oct 30 02:56:06.295611 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:56:06.295579    4461 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-77f4f76594-st7wb"
Oct 30 02:56:06.332716 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:56:06.332670    4461 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-7d8666798c-cnjhs" podStartSLOduration=1.446770582 podStartE2EDuration="3.332658187s" totalImagesPullingTime="1.885887605s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:56:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:56:06.332057738 +0000 UTC m=+34.381862942" watchObservedRunningTime="2026-10-30 02:56:06.332658187 +0000 UTC m=+34.382463366"
Oct 30 02:56:06.372473 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:56:06.372397    4461 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-77f4f76594-st7wb" podStartSLOduration=1.473341337 podStartE2EDuration="3.372387575s" totalImagesPullingTime="1.899046238s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:56:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:56:06.371925165 +0000 UTC m=+34.421730341" watchObservedRunningTime="2026-10-30 02:56:06.372387575 +0000 UTC m=+34.422192744"
Oct 30 02:56:06.437848 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:56:06.437829    4461 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Oct 30 02:56:06.440353 ip-10-0-143-156 kubenswrapper[4461]: W1030 02:56:06.440324    4461 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4c1f3a6e_55b5_49a6_a08b_e0a6ef968f92.slice/crio-7ad2e0b55f2efba4cf1c6e9c22b4a057333759d53cdc6b9f7f508f3348b839a5 WatchSource:0}: Error finding container 7ad2e0b55f2efba4cf1c6e9c22b4a057333759d53cdc6b9f7f508f3348b839a5: Status 404 returned error can't find the container with id 7ad2e0b55f2efba4cf1c6e9c22b4a057333759d53cdc6b9f7f508f3348b839a5
Oct 30 02:56:07.295656 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:56:07.295615    4461 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"4c1f3a6e-55b5-49a6-a08b-e0a6ef968f92","Type":"ContainerStarted","Data":"7ad2e0b55f2efba4cf1c6e9c22b4a057333759d53cdc6b9f7f508f3348b839a5"}
Oct 30 02:56:07.329465 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:56:07.329434    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 02:56:07.494367 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:56:07.494341    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 02:56:08.300684 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:56:08.300653    4461 generic.go:356] "Generic (PLEG): container finished" podID="4c1f3a6e-55b5-49a6-a08b-e0a6ef968f92" containerID="1405902b2ac02df8827320d7c99de9720ffb9139130796104ca52c317defb55e" exitCode=0
Oct 30 02:56:08.301084 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:56:08.300744    4461 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"4c1f3a6e-55b5-49a6-a08b-e0a6ef968f92","Type":"ContainerDied","Data":"1405902b2ac02df8827320d7c99de9720ffb9139130796104ca52c317defb55e"}
Oct 30 02:56:09.637165 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:56:09.637131    4461 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-7rmdj"
Oct 30 02:56:11.313285 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:56:11.313254    4461 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"4c1f3a6e-55b5-49a6-a08b-e0a6ef968f92","Type":"ContainerStarted","Data":"0562553763f223274cd2c7f0b1443f96a1448b3562cd9badbeef5b26874ac48c"}
Oct 30 02:56:11.943220 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:56:11.943187    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 02:56:12.116497 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:56:12.116461    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 02:56:12.318602 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:56:12.318566    4461 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"4c1f3a6e-55b5-49a6-a08b-e0a6ef968f92","Type":"ContainerStarted","Data":"8991c3c095e62bcaaba42b6f2206c1068c8bf0cd399e4ab78d66fd96856f9a1b"}
Oct 30 02:56:13.118864 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:56:13.118823    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 02:56:13.233935 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:56:13.233902    4461 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-7fbbd597bb-twtnh"
Oct 30 02:56:13.295307 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:56:13.295272    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 02:56:13.521284 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:56:13.521223    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 02:56:13.693149 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:56:13.693049    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 02:56:14.329538 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:56:14.329509    4461 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"4c1f3a6e-55b5-49a6-a08b-e0a6ef968f92","Type":"ContainerStarted","Data":"197a8e7cbd60429b401c8fcd33d7c0fc405fedf277e300e9920ceae84ce253bb"}
Oct 30 02:56:14.329731 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:56:14.329570    4461 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"4c1f3a6e-55b5-49a6-a08b-e0a6ef968f92","Type":"ContainerStarted","Data":"eaebcf3d35aff92a6e8f433f3210b74a2f6af9a370aec55ee3ec8a9edc815d56"}
Oct 30 02:56:14.329731 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:56:14.329596    4461 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"4c1f3a6e-55b5-49a6-a08b-e0a6ef968f92","Type":"ContainerStarted","Data":"4ba57a60620682c924f73732b0c33c86c9e2fd829b1d02af85962be1d71d1a35"}
Oct 30 02:56:14.329731 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:56:14.329621    4461 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"4c1f3a6e-55b5-49a6-a08b-e0a6ef968f92","Type":"ContainerStarted","Data":"4a7dfd5cddfe28978a985b3748a2086295130a2e4b26f8f5d85b8817060236a1"}
Oct 30 02:56:14.412089 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:56:14.411996    4461 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=3.54271855 podStartE2EDuration="9.411983369s" totalImagesPullingTime="5.850465837s" totalInitContainerRuntime="18.798982ms" isStatefulPod=false podCreationTimestamp="2026-10-30 02:56:05 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:56:14.399270124 +0000 UTC m=+42.449075339" watchObservedRunningTime="2026-10-30 02:56:14.411983369 +0000 UTC m=+42.461788545"
Oct 30 02:56:15.289984 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:56:15.289950    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 02:56:15.458812 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:56:15.458783    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 02:56:16.084363 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:56:16.084321    4461 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 02:56:18.403425 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:56:18.403390    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 02:56:18.580619 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:56:18.580584    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 02:56:20.997838 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:56:20.997809    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 02:56:21.199391 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:56:21.199360    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 02:56:21.395308 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:56:21.395235    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 02:56:23.493753 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:56:23.493722    4461 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-7d8666798c-cnjhs"
Oct 30 02:56:23.494264 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:56:23.493822    4461 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-7d8666798c-cnjhs"
Oct 30 02:56:24.503896 ip-10-0-143-156 kubenswrapper[4461]: E1030 02:56:24.503863    4461 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2k85b" containerName="extractor"
Oct 30 02:56:28.399056 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:56:28.399016    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 02:56:28.566688 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:56:28.566655    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 02:56:31.754740 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:56:31.754706    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 02:56:31.928461 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:56:31.928427    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 02:56:36.024916 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:56:36.024884    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 02:56:36.200776 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:56:36.200741    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 02:56:43.500302 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:56:43.500265    4461 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-7d8666798c-cnjhs"
Oct 30 02:56:43.504191 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:56:43.504169    4461 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-7d8666798c-cnjhs"
Oct 30 02:56:44.733901 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:56:44.733868    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 02:56:45.162298 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:56:45.162214    4461 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 02:56:45.172675 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:56:45.172649    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 02:56:45.180213 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:56:45.180192    4461 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 02:56:45.455211 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:56:45.455140    4461 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 02:56:50.987259 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:56:50.987225    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 02:56:51.170347 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:56:51.170310    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 02:56:54.503030 ip-10-0-143-156 kubenswrapper[4461]: E1030 02:56:54.502990    4461 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2k85b" containerName="extractor"
Oct 30 02:56:58.987745 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:56:58.987685    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 02:56:59.155930 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:56:59.155898    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 02:57:07.817845 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:57:07.817809    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 02:57:07.987131 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:57:07.987083    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 02:57:08.158829 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:57:08.158753    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 02:57:14.886860 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:57:14.886820    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 02:57:15.051757 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:57:15.051714    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 02:57:18.884726 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:57:18.884688    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 02:57:19.055113 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:57:19.055071    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 02:57:19.293009 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:57:19.292971    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 02:57:19.463052 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:57:19.463020    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 02:57:21.367541 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:57:21.367510    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 02:57:24.503458 ip-10-0-143-156 kubenswrapper[4461]: E1030 02:57:24.503419    4461 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2k85b" containerName="extractor"
Oct 30 02:57:27.593558 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:57:27.593525    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 02:57:27.771976 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:57:27.771941    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 02:57:36.010841 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:57:36.010796    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 02:57:36.181490 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:57:36.181451    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 02:57:54.503769 ip-10-0-143-156 kubenswrapper[4461]: E1030 02:57:54.503681    4461 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2k85b" containerName="extractor"
Oct 30 02:57:59.405475 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:57:59.405427    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 02:58:20.844041 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:58:20.844011    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 02:58:21.482282 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:58:21.482241    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 02:58:24.110932 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:58:24.110894    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 02:58:24.278315 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:58:24.278281    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 02:58:24.503292 ip-10-0-143-156 kubenswrapper[4461]: E1030 02:58:24.503257    4461 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2k85b" containerName="extractor"
Oct 30 02:58:54.503516 ip-10-0-143-156 kubenswrapper[4461]: E1030 02:58:54.503471    4461 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2k85b" containerName="extractor"
Oct 30 02:58:56.971518 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:58:56.971477    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 02:59:01.902683 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:59:01.902646    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 02:59:02.075063 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:59:02.075029    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 02:59:21.614043 ip-10-0-143-156 kubenswrapper[4461]: I1030 02:59:21.613950    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 02:59:24.503023 ip-10-0-143-156 kubenswrapper[4461]: E1030 02:59:24.502987    4461 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2k85b" containerName="extractor"
Oct 30 02:59:54.503328 ip-10-0-143-156 kubenswrapper[4461]: E1030 02:59:54.503286    4461 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2k85b" containerName="extractor"
Oct 30 03:00:16.354400 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:00:16.354365    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:00:21.655026 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:00:21.654985    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:00:24.503539 ip-10-0-143-156 kubenswrapper[4461]: E1030 03:00:24.503505    4461 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2k85b" containerName="extractor"
Oct 30 03:00:36.286718 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:00:36.286673    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:00:36.463808 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:00:36.463775    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:00:54.502987 ip-10-0-143-156 kubenswrapper[4461]: E1030 03:00:54.502908    4461 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2k85b" containerName="extractor"
Oct 30 03:01:22.130790 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:01:22.130745    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:01:24.503176 ip-10-0-143-156 kubenswrapper[4461]: E1030 03:01:24.503133    4461 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2k85b" containerName="extractor"
Oct 30 03:01:34.895191 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:01:34.895155    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:01:35.213331 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:01:35.213244    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:01:37.839614 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:01:37.839580    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:01:38.008756 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:01:38.008726    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:01:54.503823 ip-10-0-143-156 kubenswrapper[4461]: E1030 03:01:54.503783    4461 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2k85b" containerName="extractor"
Oct 30 03:02:21.966487 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:02:21.966399    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:02:24.503900 ip-10-0-143-156 kubenswrapper[4461]: E1030 03:02:24.503856    4461 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2k85b" containerName="extractor"
Oct 30 03:02:54.503570 ip-10-0-143-156 kubenswrapper[4461]: E1030 03:02:54.503534    4461 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2k85b" containerName="extractor"
Oct 30 03:03:22.088089 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:03:22.088030    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:03:24.503687 ip-10-0-143-156 kubenswrapper[4461]: E1030 03:03:24.503650    4461 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2k85b" containerName="extractor"
Oct 30 03:03:54.502923 ip-10-0-143-156 kubenswrapper[4461]: E1030 03:03:54.502840    4461 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2k85b" containerName="extractor"
Oct 30 03:04:22.232060 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:04:22.232020    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:04:24.502806 ip-10-0-143-156 kubenswrapper[4461]: E1030 03:04:24.502772    4461 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2k85b" containerName="extractor"
Oct 30 03:04:46.716108 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:04:46.716058    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:04:54.503510 ip-10-0-143-156 kubenswrapper[4461]: E1030 03:04:54.503411    4461 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2k85b" containerName="extractor"
Oct 30 03:05:06.849282 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:05:06.849248    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:05:22.348657 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:05:22.348581    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:05:24.503590 ip-10-0-143-156 kubenswrapper[4461]: E1030 03:05:24.503549    4461 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2k85b" containerName="extractor"
Oct 30 03:05:47.417649 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:05:47.417611    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:05:54.503315 ip-10-0-143-156 kubenswrapper[4461]: E1030 03:05:54.503273    4461 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2k85b" containerName="extractor"
Oct 30 03:06:22.454215 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:06:22.454176    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:06:24.503130 ip-10-0-143-156 kubenswrapper[4461]: E1030 03:06:24.503080    4461 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2k85b" containerName="extractor"
Oct 30 03:06:54.502972 ip-10-0-143-156 kubenswrapper[4461]: E1030 03:06:54.502893    4461 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2k85b" containerName="extractor"
Oct 30 03:07:22.576999 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:07:22.576962    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:07:24.504185 ip-10-0-143-156 kubenswrapper[4461]: E1030 03:07:24.504151    4461 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2k85b" containerName="extractor"
Oct 30 03:07:50.736708 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:07:50.736674    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:07:54.503021 ip-10-0-143-156 kubenswrapper[4461]: E1030 03:07:54.502985    4461 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2k85b" containerName="extractor"
Oct 30 03:08:22.712573 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:08:22.712500    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:08:24.503205 ip-10-0-143-156 kubenswrapper[4461]: E1030 03:08:24.503171    4461 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2k85b" containerName="extractor"
Oct 30 03:08:54.503866 ip-10-0-143-156 kubenswrapper[4461]: E1030 03:08:54.503831    4461 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2k85b" containerName="extractor"
Oct 30 03:09:22.908588 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:09:22.908555    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:09:24.503441 ip-10-0-143-156 kubenswrapper[4461]: E1030 03:09:24.503399    4461 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2k85b" containerName="extractor"
Oct 30 03:09:42.991956 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:09:42.991921    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:09:43.157434 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:09:43.157399    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:09:54.504170 ip-10-0-143-156 kubenswrapper[4461]: E1030 03:09:54.504063    4461 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2k85b" containerName="extractor"
Oct 30 03:10:06.014199 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:10:06.014165    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:10:06.215061 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:10:06.215021    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:10:12.388544 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:10:12.388503    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:10:13.053374 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:10:13.053341    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:10:13.221995 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:10:13.221961    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:10:13.581642 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:10:13.581600    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:10:13.749852 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:10:13.749804    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:10:13.922553 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:10:13.922452    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:10:14.116883 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:10:14.116834    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:10:14.871904 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:10:14.871869    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:10:15.147220 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:10:15.147134    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:10:15.385146 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:10:15.385091    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:10:15.547883 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:10:15.547839    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:10:22.909403 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:10:22.909364    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:10:24.503385 ip-10-0-143-156 kubenswrapper[4461]: E1030 03:10:24.503351    4461 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2k85b" containerName="extractor"
Oct 30 03:10:54.502879 ip-10-0-143-156 kubenswrapper[4461]: E1030 03:10:54.502834    4461 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2k85b" containerName="extractor"
Oct 30 03:11:07.129848 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:11:07.129810    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:11:09.894659 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:11:09.894626    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:11:19.658071 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:11:19.657955    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:11:23.122969 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:11:23.122929    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:11:23.316870 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:11:23.316829    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:11:24.502583 ip-10-0-143-156 kubenswrapper[4461]: E1030 03:11:24.502552    4461 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2k85b" containerName="extractor"
Oct 30 03:11:24.877077 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:11:24.876997    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:11:25.042002 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:11:25.041972    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:11:26.560426 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:11:26.560395    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:11:26.743808 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:11:26.743770    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:11:28.722820 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:11:28.722784    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:11:29.099145 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:11:29.099045    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:11:29.470444 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:11:29.470388    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:11:29.640363 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:11:29.640330    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:11:29.808018 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:11:29.807941    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:11:31.240575 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:11:31.240544    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:11:31.405980 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:11:31.405949    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:11:32.860570 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:11:32.860523    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:11:33.026297 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:11:33.026263    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:11:37.315881 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:11:37.315843    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:11:39.410041 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:11:39.410014    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:11:39.579402 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:11:39.579366    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:11:45.182548 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:11:45.182506    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:11:45.354472 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:11:45.354436    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:11:54.504011 ip-10-0-143-156 kubenswrapper[4461]: E1030 03:11:54.503973    4461 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2k85b" containerName="extractor"
Oct 30 03:11:59.281260 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:11:59.281230    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:12:04.504641 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:12:04.504596    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:12:04.677369 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:12:04.677338    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:12:05.803433 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:12:05.803401    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:12:08.433844 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:12:08.433809    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:12:08.600465 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:12:08.600428    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:12:12.591916 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:12:12.591882    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:12:12.791372 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:12:12.791252    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:12:16.645681 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:12:16.645641    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:12:17.565433 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:12:17.565403    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:12:17.581307 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:12:17.581277    4461 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-6784dd78d9-drnrz"]
Oct 30 03:12:17.584534 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:12:17.584514    4461 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-6784dd78d9-drnrz"
Oct 30 03:12:17.601659 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:12:17.601638    4461 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-mbvvd\""
Oct 30 03:12:17.614189 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:12:17.614168    4461 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\""
Oct 30 03:12:17.614489 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:12:17.614471    4461 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\""
Oct 30 03:12:17.614564 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:12:17.614512    4461 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\""
Oct 30 03:12:17.615496 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:12:17.615476    4461 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-6784dd78d9-drnrz"]
Oct 30 03:12:17.615882 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:12:17.615863    4461 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\""
Oct 30 03:12:17.655843 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:12:17.655809    4461 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gbz6q\" (UniqueName: \"kubernetes.io/projected/f8b2e6c0-006d-4839-a0d3-5a6469794df7-kube-api-access-gbz6q\") pod \"lws-controller-manager-6784dd78d9-drnrz\" (UID: \"f8b2e6c0-006d-4839-a0d3-5a6469794df7\") " pod="openshift-lws-operator/lws-controller-manager-6784dd78d9-drnrz"
Oct 30 03:12:17.656214 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:12:17.655855    4461 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/f8b2e6c0-006d-4839-a0d3-5a6469794df7-metrics-cert\") pod \"lws-controller-manager-6784dd78d9-drnrz\" (UID: \"f8b2e6c0-006d-4839-a0d3-5a6469794df7\") " pod="openshift-lws-operator/lws-controller-manager-6784dd78d9-drnrz"
Oct 30 03:12:17.656214 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:12:17.655927    4461 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f8b2e6c0-006d-4839-a0d3-5a6469794df7-cert\") pod \"lws-controller-manager-6784dd78d9-drnrz\" (UID: \"f8b2e6c0-006d-4839-a0d3-5a6469794df7\") " pod="openshift-lws-operator/lws-controller-manager-6784dd78d9-drnrz"
Oct 30 03:12:17.656214 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:12:17.655953    4461 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/f8b2e6c0-006d-4839-a0d3-5a6469794df7-manager-config\") pod \"lws-controller-manager-6784dd78d9-drnrz\" (UID: \"f8b2e6c0-006d-4839-a0d3-5a6469794df7\") " pod="openshift-lws-operator/lws-controller-manager-6784dd78d9-drnrz"
Oct 30 03:12:17.703621 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:12:17.703596    4461 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\""
Oct 30 03:12:17.730869 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:12:17.730844    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:12:17.757030 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:12:17.757001    4461 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/f8b2e6c0-006d-4839-a0d3-5a6469794df7-manager-config\") pod \"lws-controller-manager-6784dd78d9-drnrz\" (UID: \"f8b2e6c0-006d-4839-a0d3-5a6469794df7\") " pod="openshift-lws-operator/lws-controller-manager-6784dd78d9-drnrz"
Oct 30 03:12:17.757177 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:12:17.757052    4461 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gbz6q\" (UniqueName: \"kubernetes.io/projected/f8b2e6c0-006d-4839-a0d3-5a6469794df7-kube-api-access-gbz6q\") pod \"lws-controller-manager-6784dd78d9-drnrz\" (UID: \"f8b2e6c0-006d-4839-a0d3-5a6469794df7\") " pod="openshift-lws-operator/lws-controller-manager-6784dd78d9-drnrz"
Oct 30 03:12:17.757177 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:12:17.757080    4461 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/f8b2e6c0-006d-4839-a0d3-5a6469794df7-metrics-cert\") pod \"lws-controller-manager-6784dd78d9-drnrz\" (UID: \"f8b2e6c0-006d-4839-a0d3-5a6469794df7\") " pod="openshift-lws-operator/lws-controller-manager-6784dd78d9-drnrz"
Oct 30 03:12:17.757177 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:12:17.757125    4461 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f8b2e6c0-006d-4839-a0d3-5a6469794df7-cert\") pod \"lws-controller-manager-6784dd78d9-drnrz\" (UID: \"f8b2e6c0-006d-4839-a0d3-5a6469794df7\") " pod="openshift-lws-operator/lws-controller-manager-6784dd78d9-drnrz"
Oct 30 03:12:17.757754 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:12:17.757731    4461 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/f8b2e6c0-006d-4839-a0d3-5a6469794df7-manager-config\") pod \"lws-controller-manager-6784dd78d9-drnrz\" (UID: \"f8b2e6c0-006d-4839-a0d3-5a6469794df7\") " pod="openshift-lws-operator/lws-controller-manager-6784dd78d9-drnrz"
Oct 30 03:12:17.759615 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:12:17.759587    4461 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/f8b2e6c0-006d-4839-a0d3-5a6469794df7-metrics-cert\") pod \"lws-controller-manager-6784dd78d9-drnrz\" (UID: \"f8b2e6c0-006d-4839-a0d3-5a6469794df7\") " pod="openshift-lws-operator/lws-controller-manager-6784dd78d9-drnrz"
Oct 30 03:12:17.759708 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:12:17.759643    4461 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f8b2e6c0-006d-4839-a0d3-5a6469794df7-cert\") pod \"lws-controller-manager-6784dd78d9-drnrz\" (UID: \"f8b2e6c0-006d-4839-a0d3-5a6469794df7\") " pod="openshift-lws-operator/lws-controller-manager-6784dd78d9-drnrz"
Oct 30 03:12:17.781611 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:12:17.781578    4461 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gbz6q\" (UniqueName: \"kubernetes.io/projected/f8b2e6c0-006d-4839-a0d3-5a6469794df7-kube-api-access-gbz6q\") pod \"lws-controller-manager-6784dd78d9-drnrz\" (UID: \"f8b2e6c0-006d-4839-a0d3-5a6469794df7\") " pod="openshift-lws-operator/lws-controller-manager-6784dd78d9-drnrz"
Oct 30 03:12:17.895615 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:12:17.895523    4461 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-6784dd78d9-drnrz"
Oct 30 03:12:18.193425 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:12:18.193344    4461 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-6784dd78d9-drnrz"]
Oct 30 03:12:18.196471 ip-10-0-143-156 kubenswrapper[4461]: W1030 03:12:18.196443    4461 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf8b2e6c0_006d_4839_a0d3_5a6469794df7.slice/crio-47e195c2b136ab8f91f9282c0d3ced5c0afa462de56b77eb86d0a4fdc7af0d07 WatchSource:0}: Error finding container 47e195c2b136ab8f91f9282c0d3ced5c0afa462de56b77eb86d0a4fdc7af0d07: Status 404 returned error can't find the container with id 47e195c2b136ab8f91f9282c0d3ced5c0afa462de56b77eb86d0a4fdc7af0d07
Oct 30 03:12:18.198304 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:12:18.198290    4461 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 03:12:18.440389 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:12:18.440355    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:12:18.479134 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:12:18.479089    4461 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-6784dd78d9-drnrz" event={"ID":"f8b2e6c0-006d-4839-a0d3-5a6469794df7","Type":"ContainerStarted","Data":"47e195c2b136ab8f91f9282c0d3ced5c0afa462de56b77eb86d0a4fdc7af0d07"}
Oct 30 03:12:18.631157 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:12:18.631123    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:12:21.497569 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:12:21.497535    4461 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-6784dd78d9-drnrz" event={"ID":"f8b2e6c0-006d-4839-a0d3-5a6469794df7","Type":"ContainerStarted","Data":"c39d84fee3fba6ef9107a743d4f2f2d50ab4e765e02750449298cc1cceb7b3ef"}
Oct 30 03:12:21.497968 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:12:21.497687    4461 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-6784dd78d9-drnrz"
Oct 30 03:12:21.539859 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:12:21.539813    4461 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-6784dd78d9-drnrz" podStartSLOduration=2.294335419 podStartE2EDuration="4.539801061s" totalImagesPullingTime="2.245465642s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:12:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:12:21.539585339 +0000 UTC m=+1009.589390539" watchObservedRunningTime="2026-10-30 03:12:21.539801061 +0000 UTC m=+1009.589606238"
Oct 30 03:12:23.152020 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:12:23.151989    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:12:24.503315 ip-10-0-143-156 kubenswrapper[4461]: E1030 03:12:24.503281    4461 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2k85b" containerName="extractor"
Oct 30 03:12:29.396714 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:12:29.396680    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:12:32.022776 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:12:32.022727    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:12:32.386112 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:12:32.386000    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:12:32.503212 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:12:32.503179    4461 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-6784dd78d9-drnrz"
Oct 30 03:12:32.793318 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:12:32.793280    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:12:33.364837 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:12:33.364805    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:12:34.279167 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:12:34.279117    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:12:34.559713 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:12:34.559600    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:12:35.997149 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:12:35.997116    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:12:36.380809 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:12:36.380714    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:12:38.098346 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:12:38.098310    4461 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-66498fdf6c-gvm55"]
Oct 30 03:12:38.102020 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:12:38.101995    4461 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-66498fdf6c-gvm55"
Oct 30 03:12:38.119913 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:12:38.119891    4461 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-4jsdh\""
Oct 30 03:12:38.132713 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:12:38.132688    4461 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Oct 30 03:12:38.132713 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:12:38.132700    4461 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Oct 30 03:12:38.133670 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:12:38.133651    4461 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-66498fdf6c-gvm55"]
Oct 30 03:12:38.226416 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:12:38.226384    4461 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k4sfm\" (UniqueName: \"kubernetes.io/projected/94fdc4cf-9363-4925-b3df-01c4487615e6-kube-api-access-k4sfm\") pod \"authorino-operator-66498fdf6c-gvm55\" (UID: \"94fdc4cf-9363-4925-b3df-01c4487615e6\") " pod="kuadrant-system/authorino-operator-66498fdf6c-gvm55"
Oct 30 03:12:38.327197 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:12:38.327165    4461 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k4sfm\" (UniqueName: \"kubernetes.io/projected/94fdc4cf-9363-4925-b3df-01c4487615e6-kube-api-access-k4sfm\") pod \"authorino-operator-66498fdf6c-gvm55\" (UID: \"94fdc4cf-9363-4925-b3df-01c4487615e6\") " pod="kuadrant-system/authorino-operator-66498fdf6c-gvm55"
Oct 30 03:12:38.352226 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:12:38.352170    4461 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k4sfm\" (UniqueName: \"kubernetes.io/projected/94fdc4cf-9363-4925-b3df-01c4487615e6-kube-api-access-k4sfm\") pod \"authorino-operator-66498fdf6c-gvm55\" (UID: \"94fdc4cf-9363-4925-b3df-01c4487615e6\") " pod="kuadrant-system/authorino-operator-66498fdf6c-gvm55"
Oct 30 03:12:38.413800 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:12:38.413769    4461 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-66498fdf6c-gvm55"
Oct 30 03:12:38.647935 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:12:38.647854    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:12:38.718160 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:12:38.718128    4461 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-66498fdf6c-gvm55"]
Oct 30 03:12:38.720896 ip-10-0-143-156 kubenswrapper[4461]: W1030 03:12:38.720862    4461 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod94fdc4cf_9363_4925_b3df_01c4487615e6.slice/crio-c93eee78a325bceab12cc8acec9e37014d88e4195b19875fd50ba7ab8f169add WatchSource:0}: Error finding container c93eee78a325bceab12cc8acec9e37014d88e4195b19875fd50ba7ab8f169add: Status 404 returned error can't find the container with id c93eee78a325bceab12cc8acec9e37014d88e4195b19875fd50ba7ab8f169add
Oct 30 03:12:38.820405 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:12:38.820369    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:12:39.561236 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:12:39.561198    4461 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-66498fdf6c-gvm55" event={"ID":"94fdc4cf-9363-4925-b3df-01c4487615e6","Type":"ContainerStarted","Data":"c93eee78a325bceab12cc8acec9e37014d88e4195b19875fd50ba7ab8f169add"}
Oct 30 03:12:40.039392 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:12:40.039345    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:12:41.569866 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:12:41.569778    4461 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-66498fdf6c-gvm55" event={"ID":"94fdc4cf-9363-4925-b3df-01c4487615e6","Type":"ContainerStarted","Data":"55e021a3bd6129030b9b08edf08e1673bc51f00a90c2fc5c7477c69e388b0d67"}
Oct 30 03:12:41.570240 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:12:41.569963    4461 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-66498fdf6c-gvm55"
Oct 30 03:12:41.614726 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:12:41.614674    4461 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-66498fdf6c-gvm55" podStartSLOduration=1.091753599 podStartE2EDuration="3.614660385s" totalImagesPullingTime="2.522906786s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:12:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:12:41.613532975 +0000 UTC m=+1029.663338166" watchObservedRunningTime="2026-10-30 03:12:41.614660385 +0000 UTC m=+1029.664465560"
Oct 30 03:12:42.014368 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:12:42.014315    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:12:42.226414 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:12:42.226362    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:12:45.989528 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:12:45.989454    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:12:46.172941 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:12:46.172908    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:12:46.403340 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:12:46.403261    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:12:46.587264 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:12:46.587217    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:12:46.816284 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:12:46.816252    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:12:47.082227 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:12:47.082112    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:12:47.225323 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:12:47.225288    4461 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-44jp4"]
Oct 30 03:12:47.234667 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:12:47.234644    4461 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-44jp4"
Oct 30 03:12:47.260738 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:12:47.260708    4461 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-44jp4"]
Oct 30 03:12:47.262258 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:12:47.262236    4461 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\""
Oct 30 03:12:47.267049 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:12:47.267027    4461 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\""
Oct 30 03:12:47.267303 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:12:47.267286    4461 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-54rq2\""
Oct 30 03:12:47.294813 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:12:47.294786    4461 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/49dc33fe-1441-4d7c-822e-4472497f6685-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-44jp4\" (UID: \"49dc33fe-1441-4d7c-822e-4472497f6685\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-44jp4"
Oct 30 03:12:47.294941 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:12:47.294819    4461 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7gvcr\" (UniqueName: \"kubernetes.io/projected/49dc33fe-1441-4d7c-822e-4472497f6685-kube-api-access-7gvcr\") pod \"kuadrant-console-plugin-66c4fb575d-44jp4\" (UID: \"49dc33fe-1441-4d7c-822e-4472497f6685\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-44jp4"
Oct 30 03:12:47.294941 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:12:47.294844    4461 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/49dc33fe-1441-4d7c-822e-4472497f6685-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-44jp4\" (UID: \"49dc33fe-1441-4d7c-822e-4472497f6685\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-44jp4"
Oct 30 03:12:47.339259 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:12:47.339189    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:12:47.396014 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:12:47.395978    4461 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/49dc33fe-1441-4d7c-822e-4472497f6685-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-44jp4\" (UID: \"49dc33fe-1441-4d7c-822e-4472497f6685\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-44jp4"
Oct 30 03:12:47.396014 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:12:47.396013    4461 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7gvcr\" (UniqueName: \"kubernetes.io/projected/49dc33fe-1441-4d7c-822e-4472497f6685-kube-api-access-7gvcr\") pod \"kuadrant-console-plugin-66c4fb575d-44jp4\" (UID: \"49dc33fe-1441-4d7c-822e-4472497f6685\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-44jp4"
Oct 30 03:12:47.396258 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:12:47.396033    4461 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/49dc33fe-1441-4d7c-822e-4472497f6685-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-44jp4\" (UID: \"49dc33fe-1441-4d7c-822e-4472497f6685\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-44jp4"
Oct 30 03:12:47.396737 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:12:47.396715    4461 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/49dc33fe-1441-4d7c-822e-4472497f6685-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-44jp4\" (UID: \"49dc33fe-1441-4d7c-822e-4472497f6685\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-44jp4"
Oct 30 03:12:47.398413 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:12:47.398393    4461 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/49dc33fe-1441-4d7c-822e-4472497f6685-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-44jp4\" (UID: \"49dc33fe-1441-4d7c-822e-4472497f6685\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-44jp4"
Oct 30 03:12:47.448107 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:12:47.448067    4461 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7gvcr\" (UniqueName: \"kubernetes.io/projected/49dc33fe-1441-4d7c-822e-4472497f6685-kube-api-access-7gvcr\") pod \"kuadrant-console-plugin-66c4fb575d-44jp4\" (UID: \"49dc33fe-1441-4d7c-822e-4472497f6685\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-44jp4"
Oct 30 03:12:47.554207 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:12:47.554170    4461 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-44jp4"
Oct 30 03:12:47.578845 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:12:47.578816    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:12:47.850478 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:12:47.850447    4461 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-44jp4"]
Oct 30 03:12:47.853842 ip-10-0-143-156 kubenswrapper[4461]: W1030 03:12:47.853816    4461 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod49dc33fe_1441_4d7c_822e_4472497f6685.slice/crio-76305306e1274d0018c700a82464b187107b0424f3a4ae289b742b5eb7d77dcc WatchSource:0}: Error finding container 76305306e1274d0018c700a82464b187107b0424f3a4ae289b742b5eb7d77dcc: Status 404 returned error can't find the container with id 76305306e1274d0018c700a82464b187107b0424f3a4ae289b742b5eb7d77dcc
Oct 30 03:12:48.595779 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:12:48.595747    4461 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-44jp4" event={"ID":"49dc33fe-1441-4d7c-822e-4472497f6685","Type":"ContainerStarted","Data":"76305306e1274d0018c700a82464b187107b0424f3a4ae289b742b5eb7d77dcc"}
Oct 30 03:12:52.575247 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:12:52.575216    4461 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-66498fdf6c-gvm55"
Oct 30 03:12:53.616512 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:12:53.616477    4461 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-44jp4" event={"ID":"49dc33fe-1441-4d7c-822e-4472497f6685","Type":"ContainerStarted","Data":"7e6796e64e5c46d553fee83026e8b2fcbab5cfff76d4fd88a36b6dae66371fd9"}
Oct 30 03:12:53.657459 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:12:53.657415    4461 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-44jp4" podStartSLOduration=1.873431508 podStartE2EDuration="6.657401838s" totalImagesPullingTime="4.78397033s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:12:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:12:53.657332184 +0000 UTC m=+1041.707137361" watchObservedRunningTime="2026-10-30 03:12:53.657401838 +0000 UTC m=+1041.707207014"
Oct 30 03:12:54.466013 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:12:54.465965    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:12:54.503250 ip-10-0-143-156 kubenswrapper[4461]: E1030 03:12:54.503217    4461 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2k85b" containerName="extractor"
Oct 30 03:12:54.634678 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:12:54.634646    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:12:56.970786 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:12:56.970742    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:12:57.168836 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:12:57.168801    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:13:01.284113 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:13:01.284051    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:13:01.450914 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:13:01.450876    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:13:04.877949 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:13:04.877918    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:13:05.048186 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:13:05.048156    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:13:05.281711 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:13:05.281680    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:13:05.445539 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:13:05.445504    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:13:05.608502 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:13:05.608420    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:13:13.890722 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:13:13.890680    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:13:14.062032 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:13:14.061991    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:13:16.332685 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:13:16.332651    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:13:16.513768 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:13:16.513726    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:13:23.270678 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:13:23.270640    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:13:24.504216 ip-10-0-143-156 kubenswrapper[4461]: E1030 03:13:24.504182    4461 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2k85b" containerName="extractor"
Oct 30 03:13:27.733220 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:13:27.733186    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:13:27.909239 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:13:27.909202    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:13:28.067280 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:13:28.067195    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:13:28.423243 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:13:28.423171    4461 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gkz9m"]
Oct 30 03:13:28.443494 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:13:28.443465    4461 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-7c9b87d574-gkz9m"
Oct 30 03:13:28.445216 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:13:28.445189    4461 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gkz9m"]
Oct 30 03:13:28.458512 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:13:28.458490    4461 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
Oct 30 03:13:28.501619 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:13:28.501591    4461 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gkz9m"]
Oct 30 03:13:28.505946 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:13:28.505920    4461 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/2f5b6e59-9823-4958-846b-7b9b481114ef-config-file\") pod \"limitador-limitador-7c9b87d574-gkz9m\" (UID: \"2f5b6e59-9823-4958-846b-7b9b481114ef\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-gkz9m"
Oct 30 03:13:28.506060 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:13:28.505961    4461 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nhkp4\" (UniqueName: \"kubernetes.io/projected/2f5b6e59-9823-4958-846b-7b9b481114ef-kube-api-access-nhkp4\") pod \"limitador-limitador-7c9b87d574-gkz9m\" (UID: \"2f5b6e59-9823-4958-846b-7b9b481114ef\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-gkz9m"
Oct 30 03:13:28.606742 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:13:28.606712    4461 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/2f5b6e59-9823-4958-846b-7b9b481114ef-config-file\") pod \"limitador-limitador-7c9b87d574-gkz9m\" (UID: \"2f5b6e59-9823-4958-846b-7b9b481114ef\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-gkz9m"
Oct 30 03:13:28.606922 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:13:28.606761    4461 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nhkp4\" (UniqueName: \"kubernetes.io/projected/2f5b6e59-9823-4958-846b-7b9b481114ef-kube-api-access-nhkp4\") pod \"limitador-limitador-7c9b87d574-gkz9m\" (UID: \"2f5b6e59-9823-4958-846b-7b9b481114ef\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-gkz9m"
Oct 30 03:13:28.607350 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:13:28.607329    4461 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/2f5b6e59-9823-4958-846b-7b9b481114ef-config-file\") pod \"limitador-limitador-7c9b87d574-gkz9m\" (UID: \"2f5b6e59-9823-4958-846b-7b9b481114ef\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-gkz9m"
Oct 30 03:13:28.630708 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:13:28.630688    4461 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nhkp4\" (UniqueName: \"kubernetes.io/projected/2f5b6e59-9823-4958-846b-7b9b481114ef-kube-api-access-nhkp4\") pod \"limitador-limitador-7c9b87d574-gkz9m\" (UID: \"2f5b6e59-9823-4958-846b-7b9b481114ef\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-gkz9m"
Oct 30 03:13:28.691276 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:13:28.691204    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:13:28.756341 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:13:28.756310    4461 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-7c9b87d574-gkz9m"
Oct 30 03:13:28.864369 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:13:28.864339    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:13:29.052764 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:13:29.052733    4461 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gkz9m"]
Oct 30 03:13:29.055997 ip-10-0-143-156 kubenswrapper[4461]: W1030 03:13:29.055972    4461 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f5b6e59_9823_4958_846b_7b9b481114ef.slice/crio-571de0f653f9fd4a72ed107a9d8e26f1db8a28cbf6b3c3cea737b45002e61419 WatchSource:0}: Error finding container 571de0f653f9fd4a72ed107a9d8e26f1db8a28cbf6b3c3cea737b45002e61419: Status 404 returned error can't find the container with id 571de0f653f9fd4a72ed107a9d8e26f1db8a28cbf6b3c3cea737b45002e61419
Oct 30 03:13:29.735369 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:13:29.735334    4461 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-7c9b87d574-gkz9m" event={"ID":"2f5b6e59-9823-4958-846b-7b9b481114ef","Type":"ContainerStarted","Data":"571de0f653f9fd4a72ed107a9d8e26f1db8a28cbf6b3c3cea737b45002e61419"}
Oct 30 03:13:30.741026 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:13:30.740994    4461 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-7c9b87d574-gkz9m" event={"ID":"2f5b6e59-9823-4958-846b-7b9b481114ef","Type":"ContainerStarted","Data":"d280fd25aca71a4cfa3b3549c9015d8bf5fb586d6a0ed8d0fc3dc9f556cb8d58"}
Oct 30 03:13:30.741408 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:13:30.741057    4461 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-7c9b87d574-gkz9m"
Oct 30 03:13:30.784651 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:13:30.784595    4461 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-7c9b87d574-gkz9m" podStartSLOduration=1.677830901 podStartE2EDuration="2.784581882s" totalImagesPullingTime="1.106750981s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:13:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:13:30.784146941 +0000 UTC m=+1078.833952118" watchObservedRunningTime="2026-10-30 03:13:30.784581882 +0000 UTC m=+1078.834387058"
Oct 30 03:13:41.746708 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:13:41.746675    4461 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-7c9b87d574-gkz9m"
Oct 30 03:13:43.574986 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:13:43.574954    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:13:43.743201 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:13:43.743171    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:13:47.383106 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:13:47.383071    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:13:48.724460 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:13:48.724425    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:13:48.903435 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:13:48.903403    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:13:54.503132 ip-10-0-143-156 kubenswrapper[4461]: E1030 03:13:54.503081    4461 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2k85b" containerName="extractor"
Oct 30 03:13:59.426089 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:13:59.426047    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:13:59.605011 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:13:59.604975    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:14:23.446785 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:14:23.446710    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:14:24.503191 ip-10-0-143-156 kubenswrapper[4461]: E1030 03:14:24.503155    4461 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2k85b" containerName="extractor"
Oct 30 03:14:34.399531 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:14:34.399500    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:14:34.603423 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:14:34.603390    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:14:34.782930 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:14:34.782897    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:14:35.893140 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:14:35.893087    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:14:36.064924 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:14:36.064884    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:14:36.238152 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:14:36.238121    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:14:54.503302 ip-10-0-143-156 kubenswrapper[4461]: E1030 03:14:54.503267    4461 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2k85b" containerName="extractor"
Oct 30 03:15:23.531343 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:15:23.531309    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:15:24.503970 ip-10-0-143-156 kubenswrapper[4461]: E1030 03:15:24.503929    4461 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2k85b" containerName="extractor"
Oct 30 03:15:52.724596 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:15:52.724494    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:15:54.503186 ip-10-0-143-156 kubenswrapper[4461]: E1030 03:15:54.503148    4461 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2k85b" containerName="extractor"
Oct 30 03:15:57.821539 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:15:57.821505    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:15:58.170321 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:15:58.170248    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:15:58.346393 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:15:58.346359    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:16:01.887496 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:16:01.887465    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:16:02.062879 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:16:02.062847    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:16:15.732957 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:16:15.732921    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:16:23.567547 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:16:23.567513    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:16:24.503334 ip-10-0-143-156 kubenswrapper[4461]: E1030 03:16:24.503302    4461 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2k85b" containerName="extractor"
Oct 30 03:16:35.761914 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:16:35.761882    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:16:54.503631 ip-10-0-143-156 kubenswrapper[4461]: E1030 03:16:54.503591    4461 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2k85b" containerName="extractor"
Oct 30 03:17:20.844260 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:17:20.844180    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:17:21.026772 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:17:21.026738    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:17:23.758209 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:17:23.758171    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:17:24.502978 ip-10-0-143-156 kubenswrapper[4461]: E1030 03:17:24.502940    4461 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2k85b" containerName="extractor"
Oct 30 03:17:26.126173 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:17:26.126125    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:17:28.570696 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:17:28.570663    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:17:28.742799 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:17:28.742760    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:17:32.837382 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:17:32.837331    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:17:33.018345 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:17:33.018307    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:17:54.504353 ip-10-0-143-156 kubenswrapper[4461]: E1030 03:17:54.504315    4461 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2k85b" containerName="extractor"
Oct 30 03:17:57.696574 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:17:57.696533    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:17:57.979520 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:17:57.979465    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:18:14.097908 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:18:14.097864    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:18:14.603319 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:18:14.603276    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:18:21.426923 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:18:21.426894    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:18:21.688171 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:18:21.688061    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:18:23.870649 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:18:23.870617    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:18:24.504349 ip-10-0-143-156 kubenswrapper[4461]: E1030 03:18:24.504313    4461 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2k85b" containerName="extractor"
Oct 30 03:18:25.416733 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:18:25.416700    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:18:28.854557 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:18:28.854524    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:18:29.091271 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:18:29.091240    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:18:30.574269 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:18:30.574235    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:18:31.517475 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:18:31.517442    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:18:34.890347 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:18:34.890306    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:18:35.206926 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:18:35.206848    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:18:35.371294 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:18:35.371258    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:18:39.881335 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:18:39.881298    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:18:40.051988 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:18:40.051950    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:18:46.183535 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:18:46.183444    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:18:46.352595 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:18:46.352563    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:18:49.788614 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:18:49.788578    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:18:50.141995 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:18:50.141908    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:18:51.823391 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:18:51.823360    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:18:52.016066 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:18:52.016021    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:18:54.503899 ip-10-0-143-156 kubenswrapper[4461]: E1030 03:18:54.503861    4461 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2k85b" containerName="extractor"
Oct 30 03:18:56.027986 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:18:56.027950    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:18:58.660901 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:18:58.660865    4461 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-fhwng"]
Oct 30 03:18:58.664032 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:18:58.664012    4461 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-fhwng"
Oct 30 03:18:58.680576 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:18:58.680555    4461 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Oct 30 03:18:58.693001 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:18:58.692974    4461 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Oct 30 03:18:58.693157 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:18:58.693141    4461 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Oct 30 03:18:58.693234 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:18:58.693191    4461 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"openshift-ai-inference-openshift-default-dockercfg-pzw79\""
Oct 30 03:18:58.694820 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:18:58.694793    4461 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-fhwng"]
Oct 30 03:18:58.704719 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:18:58.704688    4461 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9428d98d-2f15-4c68-81a6-59066080b401-istio-token\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-fhwng\" (UID: \"9428d98d-2f15-4c68-81a6-59066080b401\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-fhwng"
Oct 30 03:18:58.704829 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:18:58.704742    4461 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9428d98d-2f15-4c68-81a6-59066080b401-workload-certs\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-fhwng\" (UID: \"9428d98d-2f15-4c68-81a6-59066080b401\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-fhwng"
Oct 30 03:18:58.704898 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:18:58.704835    4461 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9428d98d-2f15-4c68-81a6-59066080b401-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-fhwng\" (UID: \"9428d98d-2f15-4c68-81a6-59066080b401\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-fhwng"
Oct 30 03:18:58.704898 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:18:58.704880    4461 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9428d98d-2f15-4c68-81a6-59066080b401-credential-socket\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-fhwng\" (UID: \"9428d98d-2f15-4c68-81a6-59066080b401\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-fhwng"
Oct 30 03:18:58.704974 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:18:58.704953    4461 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9428d98d-2f15-4c68-81a6-59066080b401-istio-envoy\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-fhwng\" (UID: \"9428d98d-2f15-4c68-81a6-59066080b401\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-fhwng"
Oct 30 03:18:58.705013 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:18:58.704999    4461 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fng8q\" (UniqueName: \"kubernetes.io/projected/9428d98d-2f15-4c68-81a6-59066080b401-kube-api-access-fng8q\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-fhwng\" (UID: \"9428d98d-2f15-4c68-81a6-59066080b401\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-fhwng"
Oct 30 03:18:58.705069 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:18:58.705044    4461 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9428d98d-2f15-4c68-81a6-59066080b401-istio-podinfo\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-fhwng\" (UID: \"9428d98d-2f15-4c68-81a6-59066080b401\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-fhwng"
Oct 30 03:18:58.705153 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:18:58.705074    4461 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9428d98d-2f15-4c68-81a6-59066080b401-workload-socket\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-fhwng\" (UID: \"9428d98d-2f15-4c68-81a6-59066080b401\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-fhwng"
Oct 30 03:18:58.705153 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:18:58.705110    4461 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9428d98d-2f15-4c68-81a6-59066080b401-istio-data\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-fhwng\" (UID: \"9428d98d-2f15-4c68-81a6-59066080b401\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-fhwng"
Oct 30 03:18:58.805550 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:18:58.805508    4461 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9428d98d-2f15-4c68-81a6-59066080b401-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-fhwng\" (UID: \"9428d98d-2f15-4c68-81a6-59066080b401\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-fhwng"
Oct 30 03:18:58.805550 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:18:58.805562    4461 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9428d98d-2f15-4c68-81a6-59066080b401-credential-socket\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-fhwng\" (UID: \"9428d98d-2f15-4c68-81a6-59066080b401\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-fhwng"
Oct 30 03:18:58.805772 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:18:58.805674    4461 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9428d98d-2f15-4c68-81a6-59066080b401-istio-envoy\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-fhwng\" (UID: \"9428d98d-2f15-4c68-81a6-59066080b401\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-fhwng"
Oct 30 03:18:58.805772 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:18:58.805711    4461 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fng8q\" (UniqueName: \"kubernetes.io/projected/9428d98d-2f15-4c68-81a6-59066080b401-kube-api-access-fng8q\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-fhwng\" (UID: \"9428d98d-2f15-4c68-81a6-59066080b401\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-fhwng"
Oct 30 03:18:58.805772 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:18:58.805760    4461 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9428d98d-2f15-4c68-81a6-59066080b401-istio-podinfo\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-fhwng\" (UID: \"9428d98d-2f15-4c68-81a6-59066080b401\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-fhwng"
Oct 30 03:18:58.805916 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:18:58.805794    4461 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9428d98d-2f15-4c68-81a6-59066080b401-workload-socket\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-fhwng\" (UID: \"9428d98d-2f15-4c68-81a6-59066080b401\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-fhwng"
Oct 30 03:18:58.805916 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:18:58.805826    4461 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9428d98d-2f15-4c68-81a6-59066080b401-istio-data\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-fhwng\" (UID: \"9428d98d-2f15-4c68-81a6-59066080b401\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-fhwng"
Oct 30 03:18:58.805916 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:18:58.805890    4461 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9428d98d-2f15-4c68-81a6-59066080b401-istio-token\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-fhwng\" (UID: \"9428d98d-2f15-4c68-81a6-59066080b401\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-fhwng"
Oct 30 03:18:58.806062 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:18:58.805930    4461 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9428d98d-2f15-4c68-81a6-59066080b401-workload-certs\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-fhwng\" (UID: \"9428d98d-2f15-4c68-81a6-59066080b401\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-fhwng"
Oct 30 03:18:58.806146 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:18:58.806065    4461 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9428d98d-2f15-4c68-81a6-59066080b401-credential-socket\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-fhwng\" (UID: \"9428d98d-2f15-4c68-81a6-59066080b401\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-fhwng"
Oct 30 03:18:58.806291 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:18:58.806266    4461 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9428d98d-2f15-4c68-81a6-59066080b401-workload-socket\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-fhwng\" (UID: \"9428d98d-2f15-4c68-81a6-59066080b401\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-fhwng"
Oct 30 03:18:58.806336 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:18:58.806303    4461 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9428d98d-2f15-4c68-81a6-59066080b401-istio-data\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-fhwng\" (UID: \"9428d98d-2f15-4c68-81a6-59066080b401\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-fhwng"
Oct 30 03:18:58.806576 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:18:58.806552    4461 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9428d98d-2f15-4c68-81a6-59066080b401-workload-certs\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-fhwng\" (UID: \"9428d98d-2f15-4c68-81a6-59066080b401\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-fhwng"
Oct 30 03:18:58.806690 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:18:58.806643    4461 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9428d98d-2f15-4c68-81a6-59066080b401-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-fhwng\" (UID: \"9428d98d-2f15-4c68-81a6-59066080b401\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-fhwng"
Oct 30 03:18:58.807878 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:18:58.807852    4461 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9428d98d-2f15-4c68-81a6-59066080b401-istio-envoy\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-fhwng\" (UID: \"9428d98d-2f15-4c68-81a6-59066080b401\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-fhwng"
Oct 30 03:18:58.808262 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:18:58.808242    4461 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9428d98d-2f15-4c68-81a6-59066080b401-istio-podinfo\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-fhwng\" (UID: \"9428d98d-2f15-4c68-81a6-59066080b401\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-fhwng"
Oct 30 03:18:58.830219 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:18:58.830193    4461 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9428d98d-2f15-4c68-81a6-59066080b401-istio-token\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-fhwng\" (UID: \"9428d98d-2f15-4c68-81a6-59066080b401\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-fhwng"
Oct 30 03:18:58.830328 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:18:58.830276    4461 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fng8q\" (UniqueName: \"kubernetes.io/projected/9428d98d-2f15-4c68-81a6-59066080b401-kube-api-access-fng8q\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-fhwng\" (UID: \"9428d98d-2f15-4c68-81a6-59066080b401\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-fhwng"
Oct 30 03:18:58.977738 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:18:58.977707    4461 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-fhwng"
Oct 30 03:18:59.295602 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:18:59.295511    4461 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-fhwng"]
Oct 30 03:18:59.299656 ip-10-0-143-156 kubenswrapper[4461]: W1030 03:18:59.299630    4461 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9428d98d_2f15_4c68_81a6_59066080b401.slice/crio-2afec69a672b00a4e68772033cd1430a4e9c193f69b247f38482d8ab414262b9 WatchSource:0}: Error finding container 2afec69a672b00a4e68772033cd1430a4e9c193f69b247f38482d8ab414262b9: Status 404 returned error can't find the container with id 2afec69a672b00a4e68772033cd1430a4e9c193f69b247f38482d8ab414262b9
Oct 30 03:18:59.301485 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:18:59.301471    4461 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 03:18:59.866466 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:18:59.866424    4461 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-fhwng" event={"ID":"9428d98d-2f15-4c68-81a6-59066080b401","Type":"ContainerStarted","Data":"2afec69a672b00a4e68772033cd1430a4e9c193f69b247f38482d8ab414262b9"}
Oct 30 03:19:01.796311 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:19:01.796277    4461 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 03:19:01.796551 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:19:01.796372    4461 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 03:19:01.796551 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:19:01.796405    4461 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 03:19:01.876424 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:19:01.876389    4461 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-fhwng" event={"ID":"9428d98d-2f15-4c68-81a6-59066080b401","Type":"ContainerStarted","Data":"f1e6e45c30abc4aac701f0518ca60bb2891777d2b7ce9ff7fed73c5203d17e9b"}
Oct 30 03:19:01.926651 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:19:01.926602    4461 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-fhwng" podStartSLOduration=1.432877636 podStartE2EDuration="3.926588494s" totalImagesPullingTime="2.493710858s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:18:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:19:01.925785381 +0000 UTC m=+1409.975590559" watchObservedRunningTime="2026-10-30 03:19:01.926588494 +0000 UTC m=+1409.976393670"
Oct 30 03:19:01.978728 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:19:01.978692    4461 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-fhwng"
Oct 30 03:19:02.983449 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:19:02.983411    4461 patch_prober.go:29] interesting pod/openshift-ai-inference-openshift-default-5b4f8cc59b-fhwng container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.20:15021/healthz/ready\": dial tcp 10.133.0.20:15021: connect: connection refused" start-of-body=
Oct 30 03:19:02.984026 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:19:02.983980    4461 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-fhwng" podUID="9428d98d-2f15-4c68-81a6-59066080b401" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.20:15021/healthz/ready\": dial tcp 10.133.0.20:15021: connect: connection refused"
Oct 30 03:19:03.981583 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:19:03.981543    4461 patch_prober.go:29] interesting pod/openshift-ai-inference-openshift-default-5b4f8cc59b-fhwng container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.20:15021/healthz/ready\": dial tcp 10.133.0.20:15021: connect: connection refused" start-of-body=
Oct 30 03:19:03.981891 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:19:03.981865    4461 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-fhwng" podUID="9428d98d-2f15-4c68-81a6-59066080b401" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.20:15021/healthz/ready\": dial tcp 10.133.0.20:15021: connect: connection refused"
Oct 30 03:19:04.982255 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:19:04.982223    4461 patch_prober.go:29] interesting pod/openshift-ai-inference-openshift-default-5b4f8cc59b-fhwng container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.20:15021/healthz/ready\": dial tcp 10.133.0.20:15021: connect: connection refused" start-of-body=
Oct 30 03:19:04.982829 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:19:04.982798    4461 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-fhwng" podUID="9428d98d-2f15-4c68-81a6-59066080b401" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.20:15021/healthz/ready\": dial tcp 10.133.0.20:15021: connect: connection refused"
Oct 30 03:19:05.982589 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:19:05.982549    4461 patch_prober.go:29] interesting pod/openshift-ai-inference-openshift-default-5b4f8cc59b-fhwng container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.20:15021/healthz/ready\": dial tcp 10.133.0.20:15021: connect: connection refused" start-of-body=
Oct 30 03:19:05.983201 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:19:05.983171    4461 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-fhwng" podUID="9428d98d-2f15-4c68-81a6-59066080b401" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.20:15021/healthz/ready\": dial tcp 10.133.0.20:15021: connect: connection refused"
Oct 30 03:19:06.985142 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:19:06.981505    4461 patch_prober.go:29] interesting pod/openshift-ai-inference-openshift-default-5b4f8cc59b-fhwng container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.20:15021/healthz/ready\": dial tcp 10.133.0.20:15021: connect: connection refused" start-of-body=
Oct 30 03:19:06.985142 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:19:06.981566    4461 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-fhwng" podUID="9428d98d-2f15-4c68-81a6-59066080b401" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.20:15021/healthz/ready\": dial tcp 10.133.0.20:15021: connect: connection refused"
Oct 30 03:19:07.982137 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:19:07.982089    4461 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-fhwng"
Oct 30 03:19:07.982368 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:19:07.982347    4461 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-fhwng"
Oct 30 03:19:07.983110 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:19:07.983079    4461 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-fhwng"
Oct 30 03:19:13.713941 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:19:13.713904    4461 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-7s92s"]
Oct 30 03:19:13.717439 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:19:13.717422    4461 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-7s92s"
Oct 30 03:19:13.740469 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:19:13.740439    4461 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-7s92s"]
Oct 30 03:19:13.743557 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:19:13.743530    4461 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/74841243-fe62-45e3-ac11-6480ca9913c5-istio-podinfo\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-7s92s\" (UID: \"74841243-fe62-45e3-ac11-6480ca9913c5\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-7s92s"
Oct 30 03:19:13.743706 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:19:13.743617    4461 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/74841243-fe62-45e3-ac11-6480ca9913c5-workload-certs\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-7s92s\" (UID: \"74841243-fe62-45e3-ac11-6480ca9913c5\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-7s92s"
Oct 30 03:19:13.743774 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:19:13.743699    4461 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/74841243-fe62-45e3-ac11-6480ca9913c5-credential-socket\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-7s92s\" (UID: \"74841243-fe62-45e3-ac11-6480ca9913c5\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-7s92s"
Oct 30 03:19:13.743774 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:19:13.743742    4461 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/74841243-fe62-45e3-ac11-6480ca9913c5-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-7s92s\" (UID: \"74841243-fe62-45e3-ac11-6480ca9913c5\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-7s92s"
Oct 30 03:19:13.743888 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:19:13.743835    4461 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/74841243-fe62-45e3-ac11-6480ca9913c5-workload-socket\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-7s92s\" (UID: \"74841243-fe62-45e3-ac11-6480ca9913c5\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-7s92s"
Oct 30 03:19:13.743888 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:19:13.743863    4461 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/74841243-fe62-45e3-ac11-6480ca9913c5-istio-data\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-7s92s\" (UID: \"74841243-fe62-45e3-ac11-6480ca9913c5\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-7s92s"
Oct 30 03:19:13.743990 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:19:13.743889    4461 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mtkvl\" (UniqueName: \"kubernetes.io/projected/74841243-fe62-45e3-ac11-6480ca9913c5-kube-api-access-mtkvl\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-7s92s\" (UID: \"74841243-fe62-45e3-ac11-6480ca9913c5\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-7s92s"
Oct 30 03:19:13.743990 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:19:13.743925    4461 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/74841243-fe62-45e3-ac11-6480ca9913c5-istio-envoy\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-7s92s\" (UID: \"74841243-fe62-45e3-ac11-6480ca9913c5\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-7s92s"
Oct 30 03:19:13.744064 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:19:13.743994    4461 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/74841243-fe62-45e3-ac11-6480ca9913c5-istio-token\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-7s92s\" (UID: \"74841243-fe62-45e3-ac11-6480ca9913c5\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-7s92s"
Oct 30 03:19:13.752071 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:19:13.752044    4461 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-lp2z8"]
Oct 30 03:19:13.755415 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:19:13.755390    4461 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-lp2z8"
Oct 30 03:19:13.780499 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:19:13.780467    4461 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-lp2z8"]
Oct 30 03:19:13.844396 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:19:13.844368    4461 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/74841243-fe62-45e3-ac11-6480ca9913c5-workload-certs\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-7s92s\" (UID: \"74841243-fe62-45e3-ac11-6480ca9913c5\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-7s92s"
Oct 30 03:19:13.844396 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:19:13.844407    4461 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/74841243-fe62-45e3-ac11-6480ca9913c5-credential-socket\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-7s92s\" (UID: \"74841243-fe62-45e3-ac11-6480ca9913c5\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-7s92s"
Oct 30 03:19:13.844628 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:19:13.844438    4461 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/74841243-fe62-45e3-ac11-6480ca9913c5-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-7s92s\" (UID: \"74841243-fe62-45e3-ac11-6480ca9913c5\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-7s92s"
Oct 30 03:19:13.844628 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:19:13.844474    4461 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/74841243-fe62-45e3-ac11-6480ca9913c5-workload-socket\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-7s92s\" (UID: \"74841243-fe62-45e3-ac11-6480ca9913c5\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-7s92s"
Oct 30 03:19:13.844628 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:19:13.844496    4461 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/74841243-fe62-45e3-ac11-6480ca9913c5-istio-data\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-7s92s\" (UID: \"74841243-fe62-45e3-ac11-6480ca9913c5\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-7s92s"
Oct 30 03:19:13.844628 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:19:13.844521    4461 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mtkvl\" (UniqueName: \"kubernetes.io/projected/74841243-fe62-45e3-ac11-6480ca9913c5-kube-api-access-mtkvl\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-7s92s\" (UID: \"74841243-fe62-45e3-ac11-6480ca9913c5\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-7s92s"
Oct 30 03:19:13.844628 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:19:13.844552    4461 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/74841243-fe62-45e3-ac11-6480ca9913c5-istio-envoy\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-7s92s\" (UID: \"74841243-fe62-45e3-ac11-6480ca9913c5\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-7s92s"
Oct 30 03:19:13.844628 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:19:13.844591    4461 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7056833-a12d-45d3-bf13-101db3dc1d24-workload-socket\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-lp2z8\" (UID: \"f7056833-a12d-45d3-bf13-101db3dc1d24\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-lp2z8"
Oct 30 03:19:13.844628 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:19:13.844622    4461 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7056833-a12d-45d3-bf13-101db3dc1d24-credential-socket\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-lp2z8\" (UID: \"f7056833-a12d-45d3-bf13-101db3dc1d24\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-lp2z8"
Oct 30 03:19:13.845020 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:19:13.844775    4461 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f7056833-a12d-45d3-bf13-101db3dc1d24-istio-data\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-lp2z8\" (UID: \"f7056833-a12d-45d3-bf13-101db3dc1d24\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-lp2z8"
Oct 30 03:19:13.845020 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:19:13.844829    4461 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/74841243-fe62-45e3-ac11-6480ca9913c5-istio-token\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-7s92s\" (UID: \"74841243-fe62-45e3-ac11-6480ca9913c5\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-7s92s"
Oct 30 03:19:13.845020 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:19:13.844841    4461 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/74841243-fe62-45e3-ac11-6480ca9913c5-workload-certs\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-7s92s\" (UID: \"74841243-fe62-45e3-ac11-6480ca9913c5\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-7s92s"
Oct 30 03:19:13.845020 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:19:13.844873    4461 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/74841243-fe62-45e3-ac11-6480ca9913c5-istio-podinfo\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-7s92s\" (UID: \"74841243-fe62-45e3-ac11-6480ca9913c5\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-7s92s"
Oct 30 03:19:13.845020 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:19:13.844898    4461 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f7056833-a12d-45d3-bf13-101db3dc1d24-istio-token\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-lp2z8\" (UID: \"f7056833-a12d-45d3-bf13-101db3dc1d24\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-lp2z8"
Oct 30 03:19:13.845020 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:19:13.844928    4461 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f7056833-a12d-45d3-bf13-101db3dc1d24-istio-envoy\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-lp2z8\" (UID: \"f7056833-a12d-45d3-bf13-101db3dc1d24\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-lp2z8"
Oct 30 03:19:13.845020 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:19:13.844942    4461 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/74841243-fe62-45e3-ac11-6480ca9913c5-credential-socket\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-7s92s\" (UID: \"74841243-fe62-45e3-ac11-6480ca9913c5\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-7s92s"
Oct 30 03:19:13.845020 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:19:13.844954    4461 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f7056833-a12d-45d3-bf13-101db3dc1d24-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-lp2z8\" (UID: \"f7056833-a12d-45d3-bf13-101db3dc1d24\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-lp2z8"
Oct 30 03:19:13.845020 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:19:13.844966    4461 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/74841243-fe62-45e3-ac11-6480ca9913c5-istio-data\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-7s92s\" (UID: \"74841243-fe62-45e3-ac11-6480ca9913c5\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-7s92s"
Oct 30 03:19:13.845407 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:19:13.845066    4461 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/74841243-fe62-45e3-ac11-6480ca9913c5-workload-socket\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-7s92s\" (UID: \"74841243-fe62-45e3-ac11-6480ca9913c5\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-7s92s"
Oct 30 03:19:13.845407 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:19:13.845141    4461 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f7056833-a12d-45d3-bf13-101db3dc1d24-istio-podinfo\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-lp2z8\" (UID: \"f7056833-a12d-45d3-bf13-101db3dc1d24\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-lp2z8"
Oct 30 03:19:13.845407 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:19:13.845224    4461 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f7056833-a12d-45d3-bf13-101db3dc1d24-workload-certs\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-lp2z8\" (UID: \"f7056833-a12d-45d3-bf13-101db3dc1d24\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-lp2z8"
Oct 30 03:19:13.845407 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:19:13.845281    4461 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v9cdr\" (UniqueName: \"kubernetes.io/projected/f7056833-a12d-45d3-bf13-101db3dc1d24-kube-api-access-v9cdr\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-lp2z8\" (UID: \"f7056833-a12d-45d3-bf13-101db3dc1d24\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-lp2z8"
Oct 30 03:19:13.845526 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:19:13.845447    4461 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/74841243-fe62-45e3-ac11-6480ca9913c5-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-7s92s\" (UID: \"74841243-fe62-45e3-ac11-6480ca9913c5\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-7s92s"
Oct 30 03:19:13.847209 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:19:13.847184    4461 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/74841243-fe62-45e3-ac11-6480ca9913c5-istio-envoy\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-7s92s\" (UID: \"74841243-fe62-45e3-ac11-6480ca9913c5\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-7s92s"
Oct 30 03:19:13.847370 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:19:13.847354    4461 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/74841243-fe62-45e3-ac11-6480ca9913c5-istio-podinfo\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-7s92s\" (UID: \"74841243-fe62-45e3-ac11-6480ca9913c5\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-7s92s"
Oct 30 03:19:13.868427 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:19:13.868394    4461 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/74841243-fe62-45e3-ac11-6480ca9913c5-istio-token\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-7s92s\" (UID: \"74841243-fe62-45e3-ac11-6480ca9913c5\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-7s92s"
Oct 30 03:19:13.868532 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:19:13.868514    4461 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mtkvl\" (UniqueName: \"kubernetes.io/projected/74841243-fe62-45e3-ac11-6480ca9913c5-kube-api-access-mtkvl\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-7s92s\" (UID: \"74841243-fe62-45e3-ac11-6480ca9913c5\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-7s92s"
Oct 30 03:19:13.946001 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:19:13.945968    4461 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7056833-a12d-45d3-bf13-101db3dc1d24-workload-socket\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-lp2z8\" (UID: \"f7056833-a12d-45d3-bf13-101db3dc1d24\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-lp2z8"
Oct 30 03:19:13.946001 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:19:13.946001    4461 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7056833-a12d-45d3-bf13-101db3dc1d24-credential-socket\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-lp2z8\" (UID: \"f7056833-a12d-45d3-bf13-101db3dc1d24\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-lp2z8"
Oct 30 03:19:13.946282 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:19:13.946017    4461 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f7056833-a12d-45d3-bf13-101db3dc1d24-istio-data\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-lp2z8\" (UID: \"f7056833-a12d-45d3-bf13-101db3dc1d24\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-lp2z8"
Oct 30 03:19:13.946282 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:19:13.946045    4461 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f7056833-a12d-45d3-bf13-101db3dc1d24-istio-token\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-lp2z8\" (UID: \"f7056833-a12d-45d3-bf13-101db3dc1d24\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-lp2z8"
Oct 30 03:19:13.946282 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:19:13.946067    4461 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f7056833-a12d-45d3-bf13-101db3dc1d24-istio-envoy\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-lp2z8\" (UID: \"f7056833-a12d-45d3-bf13-101db3dc1d24\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-lp2z8"
Oct 30 03:19:13.946282 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:19:13.946083    4461 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f7056833-a12d-45d3-bf13-101db3dc1d24-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-lp2z8\" (UID: \"f7056833-a12d-45d3-bf13-101db3dc1d24\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-lp2z8"
Oct 30 03:19:13.946282 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:19:13.946135    4461 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f7056833-a12d-45d3-bf13-101db3dc1d24-istio-podinfo\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-lp2z8\" (UID: \"f7056833-a12d-45d3-bf13-101db3dc1d24\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-lp2z8"
Oct 30 03:19:13.946282 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:19:13.946162    4461 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f7056833-a12d-45d3-bf13-101db3dc1d24-workload-certs\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-lp2z8\" (UID: \"f7056833-a12d-45d3-bf13-101db3dc1d24\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-lp2z8"
Oct 30 03:19:13.946282 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:19:13.946185    4461 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v9cdr\" (UniqueName: \"kubernetes.io/projected/f7056833-a12d-45d3-bf13-101db3dc1d24-kube-api-access-v9cdr\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-lp2z8\" (UID: \"f7056833-a12d-45d3-bf13-101db3dc1d24\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-lp2z8"
Oct 30 03:19:13.946603 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:19:13.946447    4461 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f7056833-a12d-45d3-bf13-101db3dc1d24-istio-data\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-lp2z8\" (UID: \"f7056833-a12d-45d3-bf13-101db3dc1d24\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-lp2z8"
Oct 30 03:19:13.946603 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:19:13.946509    4461 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7056833-a12d-45d3-bf13-101db3dc1d24-workload-socket\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-lp2z8\" (UID: \"f7056833-a12d-45d3-bf13-101db3dc1d24\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-lp2z8"
Oct 30 03:19:13.946732 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:19:13.946709    4461 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7056833-a12d-45d3-bf13-101db3dc1d24-credential-socket\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-lp2z8\" (UID: \"f7056833-a12d-45d3-bf13-101db3dc1d24\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-lp2z8"
Oct 30 03:19:13.946732 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:19:13.946723    4461 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f7056833-a12d-45d3-bf13-101db3dc1d24-workload-certs\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-lp2z8\" (UID: \"f7056833-a12d-45d3-bf13-101db3dc1d24\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-lp2z8"
Oct 30 03:19:13.946827 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:19:13.946791    4461 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f7056833-a12d-45d3-bf13-101db3dc1d24-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-lp2z8\" (UID: \"f7056833-a12d-45d3-bf13-101db3dc1d24\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-lp2z8"
Oct 30 03:19:13.948510 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:19:13.948480    4461 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f7056833-a12d-45d3-bf13-101db3dc1d24-istio-envoy\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-lp2z8\" (UID: \"f7056833-a12d-45d3-bf13-101db3dc1d24\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-lp2z8"
Oct 30 03:19:13.948738 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:19:13.948721    4461 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f7056833-a12d-45d3-bf13-101db3dc1d24-istio-podinfo\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-lp2z8\" (UID: \"f7056833-a12d-45d3-bf13-101db3dc1d24\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-lp2z8"
Oct 30 03:19:13.970233 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:19:13.970165    4461 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f7056833-a12d-45d3-bf13-101db3dc1d24-istio-token\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-lp2z8\" (UID: \"f7056833-a12d-45d3-bf13-101db3dc1d24\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-lp2z8"
Oct 30 03:19:13.970233 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:19:13.970225    4461 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v9cdr\" (UniqueName: \"kubernetes.io/projected/f7056833-a12d-45d3-bf13-101db3dc1d24-kube-api-access-v9cdr\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-lp2z8\" (UID: \"f7056833-a12d-45d3-bf13-101db3dc1d24\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-lp2z8"
Oct 30 03:19:14.030894 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:19:14.030855    4461 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-7s92s"
Oct 30 03:19:14.068707 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:19:14.068655    4461 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-lp2z8"
Oct 30 03:19:14.347320 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:19:14.347296    4461 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-7s92s"]
Oct 30 03:19:14.349837 ip-10-0-143-156 kubenswrapper[4461]: W1030 03:19:14.349810    4461 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod74841243_fe62_45e3_ac11_6480ca9913c5.slice/crio-e69d72bc6a944e4cdf3a62d5ed29bab011e9c0c0d0236ace00f791b92465803e WatchSource:0}: Error finding container e69d72bc6a944e4cdf3a62d5ed29bab011e9c0c0d0236ace00f791b92465803e: Status 404 returned error can't find the container with id e69d72bc6a944e4cdf3a62d5ed29bab011e9c0c0d0236ace00f791b92465803e
Oct 30 03:19:14.351956 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:19:14.351918    4461 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 03:19:14.352059 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:19:14.351998    4461 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 03:19:14.352059 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:19:14.352042    4461 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 03:19:14.392429 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:19:14.392404    4461 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-lp2z8"]
Oct 30 03:19:14.394644 ip-10-0-143-156 kubenswrapper[4461]: W1030 03:19:14.394615    4461 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf7056833_a12d_45d3_bf13_101db3dc1d24.slice/crio-96d6d8ce97fd8db44529901bc82aaab1b1d64a587c794f54eef061c7c2e99bda WatchSource:0}: Error finding container 96d6d8ce97fd8db44529901bc82aaab1b1d64a587c794f54eef061c7c2e99bda: Status 404 returned error can't find the container with id 96d6d8ce97fd8db44529901bc82aaab1b1d64a587c794f54eef061c7c2e99bda
Oct 30 03:19:14.396884 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:19:14.396855    4461 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 03:19:14.396935 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:19:14.396917    4461 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 03:19:14.396972 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:19:14.396951    4461 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 03:19:14.932863 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:19:14.932826    4461 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-7s92s" event={"ID":"74841243-fe62-45e3-ac11-6480ca9913c5","Type":"ContainerStarted","Data":"d9108f9e40204e67f93220a50ed63e4e57a20b7abe2fda7f2c060f90aab129d0"}
Oct 30 03:19:14.933353 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:19:14.932882    4461 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-7s92s" event={"ID":"74841243-fe62-45e3-ac11-6480ca9913c5","Type":"ContainerStarted","Data":"e69d72bc6a944e4cdf3a62d5ed29bab011e9c0c0d0236ace00f791b92465803e"}
Oct 30 03:19:14.934433 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:19:14.934406    4461 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-lp2z8" event={"ID":"f7056833-a12d-45d3-bf13-101db3dc1d24","Type":"ContainerStarted","Data":"8848879ad8abe8e8da97b7205cab605ce5fb4fce4a2de558d3e4fac58e68c2d4"}
Oct 30 03:19:14.934541 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:19:14.934454    4461 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-lp2z8" event={"ID":"f7056833-a12d-45d3-bf13-101db3dc1d24","Type":"ContainerStarted","Data":"96d6d8ce97fd8db44529901bc82aaab1b1d64a587c794f54eef061c7c2e99bda"}
Oct 30 03:19:15.011807 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:19:15.011745    4461 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-7s92s" podStartSLOduration=2.011731684 podStartE2EDuration="2.011731684s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:19:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:19:15.010606636 +0000 UTC m=+1423.060411824" watchObservedRunningTime="2026-10-30 03:19:15.011731684 +0000 UTC m=+1423.061536860"
Oct 30 03:19:15.031252 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:19:15.031218    4461 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-7s92s"
Oct 30 03:19:15.032757 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:19:15.032720    4461 patch_prober.go:29] interesting pod/openshift-ai-inference-openshift-default-5b4f8cc59b-7s92s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.21:15021/healthz/ready\": dial tcp 10.133.0.21:15021: connect: connection refused" start-of-body=
Oct 30 03:19:15.032860 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:19:15.032773    4461 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-7s92s" podUID="74841243-fe62-45e3-ac11-6480ca9913c5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.21:15021/healthz/ready\": dial tcp 10.133.0.21:15021: connect: connection refused"
Oct 30 03:19:15.060750 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:19:15.060705    4461 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-lp2z8" podStartSLOduration=2.060692313 podStartE2EDuration="2.060692313s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:19:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:19:15.060003094 +0000 UTC m=+1423.109808295" watchObservedRunningTime="2026-10-30 03:19:15.060692313 +0000 UTC m=+1423.110497489"
Oct 30 03:19:15.071232 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:19:15.071154    4461 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-lp2z8"
Oct 30 03:19:15.071664 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:19:15.071633    4461 patch_prober.go:29] interesting pod/openshift-ai-inference-openshift-default-5b4f8cc59b-lp2z8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.22:15021/healthz/ready\": dial tcp 10.133.0.22:15021: connect: connection refused" start-of-body=
Oct 30 03:19:15.071857 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:19:15.071830    4461 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-lp2z8" podUID="f7056833-a12d-45d3-bf13-101db3dc1d24" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.22:15021/healthz/ready\": dial tcp 10.133.0.22:15021: connect: connection refused"
Oct 30 03:19:16.032045 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:19:16.032009    4461 patch_prober.go:29] interesting pod/openshift-ai-inference-openshift-default-5b4f8cc59b-7s92s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.21:15021/healthz/ready\": dial tcp 10.133.0.21:15021: connect: connection refused" start-of-body=
Oct 30 03:19:16.032461 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:19:16.032060    4461 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-7s92s" podUID="74841243-fe62-45e3-ac11-6480ca9913c5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.21:15021/healthz/ready\": dial tcp 10.133.0.21:15021: connect: connection refused"
Oct 30 03:19:16.084121 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:19:16.081207    4461 patch_prober.go:29] interesting pod/openshift-ai-inference-openshift-default-5b4f8cc59b-lp2z8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.22:15021/healthz/ready\": dial tcp 10.133.0.22:15021: connect: connection refused" start-of-body=
Oct 30 03:19:16.084121 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:19:16.081271    4461 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-lp2z8" podUID="f7056833-a12d-45d3-bf13-101db3dc1d24" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.22:15021/healthz/ready\": dial tcp 10.133.0.22:15021: connect: connection refused"
Oct 30 03:19:17.031927 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:19:17.031890    4461 patch_prober.go:29] interesting pod/openshift-ai-inference-openshift-default-5b4f8cc59b-7s92s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.21:15021/healthz/ready\": dial tcp 10.133.0.21:15021: connect: connection refused" start-of-body=
Oct 30 03:19:17.032151 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:19:17.031946    4461 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-7s92s" podUID="74841243-fe62-45e3-ac11-6480ca9913c5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.21:15021/healthz/ready\": dial tcp 10.133.0.21:15021: connect: connection refused"
Oct 30 03:19:17.073121 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:19:17.069478    4461 patch_prober.go:29] interesting pod/openshift-ai-inference-openshift-default-5b4f8cc59b-lp2z8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.22:15021/healthz/ready\": dial tcp 10.133.0.22:15021: connect: connection refused" start-of-body=
Oct 30 03:19:17.073121 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:19:17.069537    4461 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-lp2z8" podUID="f7056833-a12d-45d3-bf13-101db3dc1d24" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.22:15021/healthz/ready\": dial tcp 10.133.0.22:15021: connect: connection refused"
Oct 30 03:19:18.031334 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:19:18.031298    4461 patch_prober.go:29] interesting pod/openshift-ai-inference-openshift-default-5b4f8cc59b-7s92s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.21:15021/healthz/ready\": dial tcp 10.133.0.21:15021: connect: connection refused" start-of-body=
Oct 30 03:19:18.031518 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:19:18.031357    4461 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-7s92s" podUID="74841243-fe62-45e3-ac11-6480ca9913c5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.21:15021/healthz/ready\": dial tcp 10.133.0.21:15021: connect: connection refused"
Oct 30 03:19:18.069524 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:19:18.069486    4461 patch_prober.go:29] interesting pod/openshift-ai-inference-openshift-default-5b4f8cc59b-lp2z8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.22:15021/healthz/ready\": dial tcp 10.133.0.22:15021: connect: connection refused" start-of-body=
Oct 30 03:19:18.070089 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:19:18.070059    4461 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-lp2z8" podUID="f7056833-a12d-45d3-bf13-101db3dc1d24" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.22:15021/healthz/ready\": dial tcp 10.133.0.22:15021: connect: connection refused"
Oct 30 03:19:19.031762 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:19:19.031720    4461 patch_prober.go:29] interesting pod/openshift-ai-inference-openshift-default-5b4f8cc59b-7s92s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.21:15021/healthz/ready\": dial tcp 10.133.0.21:15021: connect: connection refused" start-of-body=
Oct 30 03:19:19.031978 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:19:19.031778    4461 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-7s92s" podUID="74841243-fe62-45e3-ac11-6480ca9913c5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.21:15021/healthz/ready\": dial tcp 10.133.0.21:15021: connect: connection refused"
Oct 30 03:19:19.069527 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:19:19.069490    4461 patch_prober.go:29] interesting pod/openshift-ai-inference-openshift-default-5b4f8cc59b-lp2z8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.22:15021/healthz/ready\": dial tcp 10.133.0.22:15021: connect: connection refused" start-of-body=
Oct 30 03:19:19.070069 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:19:19.070038    4461 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-lp2z8" podUID="f7056833-a12d-45d3-bf13-101db3dc1d24" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.22:15021/healthz/ready\": dial tcp 10.133.0.22:15021: connect: connection refused"
Oct 30 03:19:20.031926 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:19:20.031887    4461 patch_prober.go:29] interesting pod/openshift-ai-inference-openshift-default-5b4f8cc59b-7s92s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.21:15021/healthz/ready\": dial tcp 10.133.0.21:15021: connect: connection refused" start-of-body=
Oct 30 03:19:20.032083 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:19:20.031946    4461 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-7s92s" podUID="74841243-fe62-45e3-ac11-6480ca9913c5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.21:15021/healthz/ready\": dial tcp 10.133.0.21:15021: connect: connection refused"
Oct 30 03:19:20.078982 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:19:20.069589    4461 patch_prober.go:29] interesting pod/openshift-ai-inference-openshift-default-5b4f8cc59b-lp2z8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.22:15021/healthz/ready\": dial tcp 10.133.0.22:15021: connect: connection refused" start-of-body=
Oct 30 03:19:20.078982 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:19:20.069649    4461 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-lp2z8" podUID="f7056833-a12d-45d3-bf13-101db3dc1d24" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.22:15021/healthz/ready\": dial tcp 10.133.0.22:15021: connect: connection refused"
Oct 30 03:19:21.035016 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:19:21.034972    4461 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-7s92s"
Oct 30 03:19:21.035360 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:19:21.035337    4461 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-7s92s"
Oct 30 03:19:21.035960 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:19:21.035941    4461 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-7s92s"
Oct 30 03:19:21.073053 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:19:21.073026    4461 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-lp2z8"
Oct 30 03:19:21.073318 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:19:21.073297    4461 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-lp2z8"
Oct 30 03:19:21.073865 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:19:21.073846    4461 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-lp2z8"
Oct 30 03:19:23.993358 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:19:23.993318    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:19:24.504255 ip-10-0-143-156 kubenswrapper[4461]: E1030 03:19:24.504222    4461 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2k85b" containerName="extractor"
Oct 30 03:19:42.989823 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:19:42.989790    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:19:43.159877 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:19:43.159841    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:19:43.335296 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:19:43.335218    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:19:43.500046 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:19:43.500016    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:19:47.449588 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:19:47.449555    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:19:52.586762 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:19:52.586707    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:19:54.039237 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:19:54.039205    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:19:54.206910 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:19:54.206877    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:19:54.503727 ip-10-0-143-156 kubenswrapper[4461]: E1030 03:19:54.503690    4461 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2k85b" containerName="extractor"
Oct 30 03:20:17.549427 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:20:17.549314    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:20:17.717970 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:20:17.717923    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:20:24.116327 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:20:24.116283    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:20:24.503833 ip-10-0-143-156 kubenswrapper[4461]: E1030 03:20:24.503793    4461 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2k85b" containerName="extractor"
Oct 30 03:20:54.503959 ip-10-0-143-156 kubenswrapper[4461]: E1030 03:20:54.503913    4461 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2k85b" containerName="extractor"
Oct 30 03:21:07.687145 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:21:07.687110    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:21:07.879808 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:21:07.879771    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:21:12.018129 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:21:12.015928    4461 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-767f9954b4-dhhvt"]
Oct 30 03:21:12.019715 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:21:12.019685    4461 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-767f9954b4-dhhvt"
Oct 30 03:21:12.039510 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:21:12.039485    4461 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-15d6126c\"/\"kube-root-ca.crt\""
Oct 30 03:21:12.052023 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:21:12.051994    4461 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-15d6126c\"/\"openshift-service-ca.crt\""
Oct 30 03:21:12.052169 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:21:12.052082    4461 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-15d6126c\"/\"scheduler-configmap-ref-test-kserve-self-signed-certs\""
Oct 30 03:21:12.052169 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:21:12.052135    4461 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-15d6126c\"/\"default-dockercfg-wg8tb\""
Oct 30 03:21:12.053481 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:21:12.053458    4461 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-767f9954b4-dhhvt"]
Oct 30 03:21:12.120777 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:21:12.120733    4461 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e962f31c-ce65-4f08-b172-f166789eb0a0-dshm\") pod \"scheduler-configmap-ref-test-kserve-767f9954b4-dhhvt\" (UID: \"e962f31c-ce65-4f08-b172-f166789eb0a0\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-767f9954b4-dhhvt"
Oct 30 03:21:12.120777 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:21:12.120780    4461 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c4vt2\" (UniqueName: \"kubernetes.io/projected/e962f31c-ce65-4f08-b172-f166789eb0a0-kube-api-access-c4vt2\") pod \"scheduler-configmap-ref-test-kserve-767f9954b4-dhhvt\" (UID: \"e962f31c-ce65-4f08-b172-f166789eb0a0\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-767f9954b4-dhhvt"
Oct 30 03:21:12.121006 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:21:12.120808    4461 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e962f31c-ce65-4f08-b172-f166789eb0a0-tmp-dir\") pod \"scheduler-configmap-ref-test-kserve-767f9954b4-dhhvt\" (UID: \"e962f31c-ce65-4f08-b172-f166789eb0a0\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-767f9954b4-dhhvt"
Oct 30 03:21:12.121006 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:21:12.120870    4461 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e962f31c-ce65-4f08-b172-f166789eb0a0-tls-certs\") pod \"scheduler-configmap-ref-test-kserve-767f9954b4-dhhvt\" (UID: \"e962f31c-ce65-4f08-b172-f166789eb0a0\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-767f9954b4-dhhvt"
Oct 30 03:21:12.121006 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:21:12.120895    4461 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e962f31c-ce65-4f08-b172-f166789eb0a0-model-cache\") pod \"scheduler-configmap-ref-test-kserve-767f9954b4-dhhvt\" (UID: \"e962f31c-ce65-4f08-b172-f166789eb0a0\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-767f9954b4-dhhvt"
Oct 30 03:21:12.121006 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:21:12.120913    4461 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e962f31c-ce65-4f08-b172-f166789eb0a0-home\") pod \"scheduler-configmap-ref-test-kserve-767f9954b4-dhhvt\" (UID: \"e962f31c-ce65-4f08-b172-f166789eb0a0\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-767f9954b4-dhhvt"
Oct 30 03:21:12.133877 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:21:12.133846    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:21:12.222382 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:21:12.222333    4461 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e962f31c-ce65-4f08-b172-f166789eb0a0-tmp-dir\") pod \"scheduler-configmap-ref-test-kserve-767f9954b4-dhhvt\" (UID: \"e962f31c-ce65-4f08-b172-f166789eb0a0\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-767f9954b4-dhhvt"
Oct 30 03:21:12.222382 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:21:12.222388    4461 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e962f31c-ce65-4f08-b172-f166789eb0a0-tls-certs\") pod \"scheduler-configmap-ref-test-kserve-767f9954b4-dhhvt\" (UID: \"e962f31c-ce65-4f08-b172-f166789eb0a0\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-767f9954b4-dhhvt"
Oct 30 03:21:12.222620 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:21:12.222416    4461 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e962f31c-ce65-4f08-b172-f166789eb0a0-model-cache\") pod \"scheduler-configmap-ref-test-kserve-767f9954b4-dhhvt\" (UID: \"e962f31c-ce65-4f08-b172-f166789eb0a0\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-767f9954b4-dhhvt"
Oct 30 03:21:12.222620 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:21:12.222433    4461 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e962f31c-ce65-4f08-b172-f166789eb0a0-home\") pod \"scheduler-configmap-ref-test-kserve-767f9954b4-dhhvt\" (UID: \"e962f31c-ce65-4f08-b172-f166789eb0a0\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-767f9954b4-dhhvt"
Oct 30 03:21:12.222620 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:21:12.222515    4461 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e962f31c-ce65-4f08-b172-f166789eb0a0-dshm\") pod \"scheduler-configmap-ref-test-kserve-767f9954b4-dhhvt\" (UID: \"e962f31c-ce65-4f08-b172-f166789eb0a0\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-767f9954b4-dhhvt"
Oct 30 03:21:12.222620 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:21:12.222534    4461 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c4vt2\" (UniqueName: \"kubernetes.io/projected/e962f31c-ce65-4f08-b172-f166789eb0a0-kube-api-access-c4vt2\") pod \"scheduler-configmap-ref-test-kserve-767f9954b4-dhhvt\" (UID: \"e962f31c-ce65-4f08-b172-f166789eb0a0\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-767f9954b4-dhhvt"
Oct 30 03:21:12.222843 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:21:12.222800    4461 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e962f31c-ce65-4f08-b172-f166789eb0a0-tmp-dir\") pod \"scheduler-configmap-ref-test-kserve-767f9954b4-dhhvt\" (UID: \"e962f31c-ce65-4f08-b172-f166789eb0a0\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-767f9954b4-dhhvt"
Oct 30 03:21:12.222843 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:21:12.222811    4461 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e962f31c-ce65-4f08-b172-f166789eb0a0-model-cache\") pod \"scheduler-configmap-ref-test-kserve-767f9954b4-dhhvt\" (UID: \"e962f31c-ce65-4f08-b172-f166789eb0a0\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-767f9954b4-dhhvt"
Oct 30 03:21:12.222939 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:21:12.222871    4461 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e962f31c-ce65-4f08-b172-f166789eb0a0-home\") pod \"scheduler-configmap-ref-test-kserve-767f9954b4-dhhvt\" (UID: \"e962f31c-ce65-4f08-b172-f166789eb0a0\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-767f9954b4-dhhvt"
Oct 30 03:21:12.224845 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:21:12.224824    4461 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e962f31c-ce65-4f08-b172-f166789eb0a0-dshm\") pod \"scheduler-configmap-ref-test-kserve-767f9954b4-dhhvt\" (UID: \"e962f31c-ce65-4f08-b172-f166789eb0a0\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-767f9954b4-dhhvt"
Oct 30 03:21:12.224932 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:21:12.224913    4461 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e962f31c-ce65-4f08-b172-f166789eb0a0-tls-certs\") pod \"scheduler-configmap-ref-test-kserve-767f9954b4-dhhvt\" (UID: \"e962f31c-ce65-4f08-b172-f166789eb0a0\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-767f9954b4-dhhvt"
Oct 30 03:21:12.246434 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:21:12.246411    4461 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c4vt2\" (UniqueName: \"kubernetes.io/projected/e962f31c-ce65-4f08-b172-f166789eb0a0-kube-api-access-c4vt2\") pod \"scheduler-configmap-ref-test-kserve-767f9954b4-dhhvt\" (UID: \"e962f31c-ce65-4f08-b172-f166789eb0a0\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-767f9954b4-dhhvt"
Oct 30 03:21:12.333449 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:21:12.333367    4461 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-767f9954b4-dhhvt"
Oct 30 03:21:12.437345 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:21:12.437283    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:21:12.634063 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:21:12.632574    4461 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-767f9954b4-dhhvt"]
Oct 30 03:21:12.637358 ip-10-0-143-156 kubenswrapper[4461]: W1030 03:21:12.637328    4461 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode962f31c_ce65_4f08_b172_f166789eb0a0.slice/crio-f493ffd36e4c81803af7ea48505adc22dbe01f5a0150e9e7f701d181a6b29cf9 WatchSource:0}: Error finding container f493ffd36e4c81803af7ea48505adc22dbe01f5a0150e9e7f701d181a6b29cf9: Status 404 returned error can't find the container with id f493ffd36e4c81803af7ea48505adc22dbe01f5a0150e9e7f701d181a6b29cf9
Oct 30 03:21:13.394867 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:21:13.394837    4461 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-767f9954b4-dhhvt" event={"ID":"e962f31c-ce65-4f08-b172-f166789eb0a0","Type":"ContainerStarted","Data":"f493ffd36e4c81803af7ea48505adc22dbe01f5a0150e9e7f701d181a6b29cf9"}
Oct 30 03:21:14.400761 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:21:14.400730    4461 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-767f9954b4-dhhvt" event={"ID":"e962f31c-ce65-4f08-b172-f166789eb0a0","Type":"ContainerStarted","Data":"5dee061f771236e3d8eead0741c84f16b41c549221155cb452f0b947c059740d"}
Oct 30 03:21:14.444318 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:21:14.444266    4461 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-767f9954b4-dhhvt" podStartSLOduration=1.828654749 podStartE2EDuration="3.444251954s" totalImagesPullingTime="1.615597205s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:21:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:21:14.442431967 +0000 UTC m=+1542.492237142" watchObservedRunningTime="2026-10-30 03:21:14.444251954 +0000 UTC m=+1542.494057129"
Oct 30 03:21:14.586886 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:21:14.586805    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:21:15.079826 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:21:15.079791    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:21:15.334010 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:21:15.333926    4461 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-767f9954b4-dhhvt"
Oct 30 03:21:16.814584 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:21:16.814552    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:21:16.987329 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:21:16.987291    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:21:21.128232 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:21:21.128196    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:21:22.334178 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:21:22.334132    4461 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-767f9954b4-dhhvt"
Oct 30 03:21:22.346712 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:21:22.346688    4461 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-767f9954b4-dhhvt"
Oct 30 03:21:22.442421 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:21:22.442389    4461 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-767f9954b4-dhhvt"
Oct 30 03:21:22.801322 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:21:22.801280    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:21:24.215947 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:21:24.215912    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:21:24.503581 ip-10-0-143-156 kubenswrapper[4461]: E1030 03:21:24.503488    4461 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2k85b" containerName="extractor"
Oct 30 03:21:46.840458 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:21:46.840360    4461 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-767f9954b4-dhhvt"]
Oct 30 03:21:46.870348 ip-10-0-143-156 kubenswrapper[4461]: E1030 03:21:46.869283    4461 log.go:32] "ExecSync cmd from runtime service failed" err="rpc error: code = Unknown desc = command error: , stdout: , stderr: , exit code -1" containerID="5dee061f771236e3d8eead0741c84f16b41c549221155cb452f0b947c059740d" cmd=["/bin/sleep","15"]
Oct 30 03:21:46.870348 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:21:46.869361    4461 handlers.go:85] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-767f9954b4-dhhvt" message="" err="rpc error: code = Unknown desc = command error: , stdout: , stderr: , exit code -1"
Oct 30 03:21:46.870348 ip-10-0-143-156 kubenswrapper[4461]: E1030 03:21:46.869382    4461 kuberuntime_container.go:780] "PreStop hook failed" err="rpc error: code = Unknown desc = command error: , stdout: , stderr: , exit code -1" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-767f9954b4-dhhvt" podUID="e962f31c-ce65-4f08-b172-f166789eb0a0" containerName="main" containerID="cri-o://5dee061f771236e3d8eead0741c84f16b41c549221155cb452f0b947c059740d"
Oct 30 03:21:46.870348 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:21:46.869425    4461 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-767f9954b4-dhhvt" podUID="e962f31c-ce65-4f08-b172-f166789eb0a0" containerName="main" containerID="cri-o://5dee061f771236e3d8eead0741c84f16b41c549221155cb452f0b947c059740d" gracePeriod=60
Oct 30 03:21:46.886977 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:21:46.886894    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:21:47.159067 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:21:47.159041    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:21:47.220626 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:21:47.220602    4461 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-767f9954b4-dhhvt"
Oct 30 03:21:47.339742 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:21:47.339706    4461 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e962f31c-ce65-4f08-b172-f166789eb0a0-dshm\" (UniqueName: \"kubernetes.io/empty-dir/e962f31c-ce65-4f08-b172-f166789eb0a0-dshm\") pod \"e962f31c-ce65-4f08-b172-f166789eb0a0\" (UID: \"e962f31c-ce65-4f08-b172-f166789eb0a0\") "
Oct 30 03:21:47.339928 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:21:47.339754    4461 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e962f31c-ce65-4f08-b172-f166789eb0a0-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e962f31c-ce65-4f08-b172-f166789eb0a0-model-cache\") pod \"e962f31c-ce65-4f08-b172-f166789eb0a0\" (UID: \"e962f31c-ce65-4f08-b172-f166789eb0a0\") "
Oct 30 03:21:47.339928 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:21:47.339789    4461 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e962f31c-ce65-4f08-b172-f166789eb0a0-home\" (UniqueName: \"kubernetes.io/empty-dir/e962f31c-ce65-4f08-b172-f166789eb0a0-home\") pod \"e962f31c-ce65-4f08-b172-f166789eb0a0\" (UID: \"e962f31c-ce65-4f08-b172-f166789eb0a0\") "
Oct 30 03:21:47.339928 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:21:47.339823    4461 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e962f31c-ce65-4f08-b172-f166789eb0a0-kube-api-access-c4vt2\" (UniqueName: \"kubernetes.io/projected/e962f31c-ce65-4f08-b172-f166789eb0a0-kube-api-access-c4vt2\") pod \"e962f31c-ce65-4f08-b172-f166789eb0a0\" (UID: \"e962f31c-ce65-4f08-b172-f166789eb0a0\") "
Oct 30 03:21:47.339928 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:21:47.339904    4461 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e962f31c-ce65-4f08-b172-f166789eb0a0-tls-certs\" (UniqueName: \"kubernetes.io/secret/e962f31c-ce65-4f08-b172-f166789eb0a0-tls-certs\") pod \"e962f31c-ce65-4f08-b172-f166789eb0a0\" (UID: \"e962f31c-ce65-4f08-b172-f166789eb0a0\") "
Oct 30 03:21:47.339928 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:21:47.339924    4461 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e962f31c-ce65-4f08-b172-f166789eb0a0-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e962f31c-ce65-4f08-b172-f166789eb0a0-tmp-dir\") pod \"e962f31c-ce65-4f08-b172-f166789eb0a0\" (UID: \"e962f31c-ce65-4f08-b172-f166789eb0a0\") "
Oct 30 03:21:47.340225 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:21:47.340033    4461 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e962f31c-ce65-4f08-b172-f166789eb0a0-model-cache" pod "e962f31c-ce65-4f08-b172-f166789eb0a0" (UID: "e962f31c-ce65-4f08-b172-f166789eb0a0"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:21:47.340225 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:21:47.340065    4461 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e962f31c-ce65-4f08-b172-f166789eb0a0-home" pod "e962f31c-ce65-4f08-b172-f166789eb0a0" (UID: "e962f31c-ce65-4f08-b172-f166789eb0a0"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:21:47.340225 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:21:47.340214    4461 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e962f31c-ce65-4f08-b172-f166789eb0a0-model-cache\") on node \"ip-10-0-143-156.ec2.internal\" DevicePath \"\""
Oct 30 03:21:47.340349 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:21:47.340235    4461 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e962f31c-ce65-4f08-b172-f166789eb0a0-home\") on node \"ip-10-0-143-156.ec2.internal\" DevicePath \"\""
Oct 30 03:21:47.340349 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:21:47.340257    4461 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e962f31c-ce65-4f08-b172-f166789eb0a0-tmp-dir" pod "e962f31c-ce65-4f08-b172-f166789eb0a0" (UID: "e962f31c-ce65-4f08-b172-f166789eb0a0"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:21:47.342107 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:21:47.342076    4461 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e962f31c-ce65-4f08-b172-f166789eb0a0-dshm" pod "e962f31c-ce65-4f08-b172-f166789eb0a0" (UID: "e962f31c-ce65-4f08-b172-f166789eb0a0"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:21:47.342210 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:21:47.342182    4461 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e962f31c-ce65-4f08-b172-f166789eb0a0-kube-api-access-c4vt2" pod "e962f31c-ce65-4f08-b172-f166789eb0a0" (UID: "e962f31c-ce65-4f08-b172-f166789eb0a0"). InnerVolumeSpecName "kube-api-access-c4vt2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:21:47.342297 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:21:47.342238    4461 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e962f31c-ce65-4f08-b172-f166789eb0a0-tls-certs" pod "e962f31c-ce65-4f08-b172-f166789eb0a0" (UID: "e962f31c-ce65-4f08-b172-f166789eb0a0"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:21:47.441609 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:21:47.441530    4461 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e962f31c-ce65-4f08-b172-f166789eb0a0-tls-certs\") on node \"ip-10-0-143-156.ec2.internal\" DevicePath \"\""
Oct 30 03:21:47.441609 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:21:47.441556    4461 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e962f31c-ce65-4f08-b172-f166789eb0a0-tmp-dir\") on node \"ip-10-0-143-156.ec2.internal\" DevicePath \"\""
Oct 30 03:21:47.441609 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:21:47.441566    4461 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e962f31c-ce65-4f08-b172-f166789eb0a0-dshm\") on node \"ip-10-0-143-156.ec2.internal\" DevicePath \"\""
Oct 30 03:21:47.441609 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:21:47.441575    4461 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c4vt2\" (UniqueName: \"kubernetes.io/projected/e962f31c-ce65-4f08-b172-f166789eb0a0-kube-api-access-c4vt2\") on node \"ip-10-0-143-156.ec2.internal\" DevicePath \"\""
Oct 30 03:21:47.530271 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:21:47.530230    4461 generic.go:356] "Generic (PLEG): container finished" podID="e962f31c-ce65-4f08-b172-f166789eb0a0" containerID="5dee061f771236e3d8eead0741c84f16b41c549221155cb452f0b947c059740d" exitCode=0
Oct 30 03:21:47.530432 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:21:47.530279    4461 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-767f9954b4-dhhvt"
Oct 30 03:21:47.530432 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:21:47.530313    4461 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-767f9954b4-dhhvt" event={"ID":"e962f31c-ce65-4f08-b172-f166789eb0a0","Type":"ContainerDied","Data":"5dee061f771236e3d8eead0741c84f16b41c549221155cb452f0b947c059740d"}
Oct 30 03:21:47.530432 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:21:47.530369    4461 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-767f9954b4-dhhvt" event={"ID":"e962f31c-ce65-4f08-b172-f166789eb0a0","Type":"ContainerDied","Data":"f493ffd36e4c81803af7ea48505adc22dbe01f5a0150e9e7f701d181a6b29cf9"}
Oct 30 03:21:47.530432 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:21:47.530391    4461 scope.go:122] "RemoveContainer" containerID="5dee061f771236e3d8eead0741c84f16b41c549221155cb452f0b947c059740d"
Oct 30 03:21:47.540184 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:21:47.540168    4461 scope.go:122] "RemoveContainer" containerID="5dee061f771236e3d8eead0741c84f16b41c549221155cb452f0b947c059740d"
Oct 30 03:21:47.540450 ip-10-0-143-156 kubenswrapper[4461]: E1030 03:21:47.540431    4461 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5dee061f771236e3d8eead0741c84f16b41c549221155cb452f0b947c059740d\": container with ID starting with 5dee061f771236e3d8eead0741c84f16b41c549221155cb452f0b947c059740d not found: ID does not exist" containerID="5dee061f771236e3d8eead0741c84f16b41c549221155cb452f0b947c059740d"
Oct 30 03:21:47.540518 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:21:47.540463    4461 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5dee061f771236e3d8eead0741c84f16b41c549221155cb452f0b947c059740d"} err="failed to get container status \"5dee061f771236e3d8eead0741c84f16b41c549221155cb452f0b947c059740d\": rpc error: code = NotFound desc = could not find container \"5dee061f771236e3d8eead0741c84f16b41c549221155cb452f0b947c059740d\": container with ID starting with 5dee061f771236e3d8eead0741c84f16b41c549221155cb452f0b947c059740d not found: ID does not exist"
Oct 30 03:21:47.583876 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:21:47.583841    4461 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-767f9954b4-dhhvt"]
Oct 30 03:21:47.588687 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:21:47.588660    4461 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-767f9954b4-dhhvt"]
Oct 30 03:21:48.114601 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:21:48.114569    4461 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e962f31c-ce65-4f08-b172-f166789eb0a0" path="/var/lib/kubelet/pods/e962f31c-ce65-4f08-b172-f166789eb0a0/volumes"
Oct 30 03:21:52.696962 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:21:52.696931    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:21:54.503040 ip-10-0-143-156 kubenswrapper[4461]: E1030 03:21:54.503000    4461 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2k85b" containerName="extractor"
Oct 30 03:22:06.893273 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:22:06.893234    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:22:07.077429 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:22:07.077396    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:22:11.137773 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:22:11.137737    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:22:11.435700 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:22:11.435622    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:22:13.969707 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:22:13.969672    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:22:14.210469 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:22:14.210433    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:22:20.542604 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:22:20.542564    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:22:20.705606 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:22:20.705568    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:22:24.340066 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:22:24.340024    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:22:24.503520 ip-10-0-143-156 kubenswrapper[4461]: E1030 03:22:24.503474    4461 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2k85b" containerName="extractor"
Oct 30 03:22:36.676180 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:22:36.676146    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:22:36.939451 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:22:36.939361    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:22:39.291431 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:22:39.291396    4461 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-74cd6f7448-spgqf"]
Oct 30 03:22:39.291809 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:22:39.291706    4461 cpu_manager.go:414] "containerMap: removing container" podUID="e962f31c-ce65-4f08-b172-f166789eb0a0" containerName="main"
Oct 30 03:22:39.291809 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:22:39.291717    4461 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e962f31c-ce65-4f08-b172-f166789eb0a0" containerName="main"
Oct 30 03:22:39.291809 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:22:39.291782    4461 memory_manager.go:365] "RemoveStaleState removing state" podUID="e962f31c-ce65-4f08-b172-f166789eb0a0" containerName="main"
Oct 30 03:22:39.295519 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:22:39.295500    4461 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-74cd6f7448-spgqf"
Oct 30 03:22:39.327826 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:22:39.327663    4461 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-2541cd65\"/\"openshift-service-ca.crt\""
Oct 30 03:22:39.327826 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:22:39.327794    4461 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-2541cd65\"/\"kube-root-ca.crt\""
Oct 30 03:22:39.328131 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:22:39.328116    4461 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-2541cd65\"/\"scheduler-ha-replicas-test-kserve-self-signed-certs\""
Oct 30 03:22:39.328302 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:22:39.328289    4461 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-2541cd65\"/\"default-dockercfg-7zh75\""
Oct 30 03:22:39.329198 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:22:39.329178    4461 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-74cd6f7448-spgqf"]
Oct 30 03:22:39.384906 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:22:39.384872    4461 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6e43795c-d576-47f3-94e5-192856898684-tmp-dir\") pod \"scheduler-ha-replicas-test-kserve-74cd6f7448-spgqf\" (UID: \"6e43795c-d576-47f3-94e5-192856898684\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-74cd6f7448-spgqf"
Oct 30 03:22:39.385066 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:22:39.384922    4461 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6e43795c-d576-47f3-94e5-192856898684-model-cache\") pod \"scheduler-ha-replicas-test-kserve-74cd6f7448-spgqf\" (UID: \"6e43795c-d576-47f3-94e5-192856898684\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-74cd6f7448-spgqf"
Oct 30 03:22:39.385066 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:22:39.385040    4461 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6e43795c-d576-47f3-94e5-192856898684-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-74cd6f7448-spgqf\" (UID: \"6e43795c-d576-47f3-94e5-192856898684\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-74cd6f7448-spgqf"
Oct 30 03:22:39.385183 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:22:39.385067    4461 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6e43795c-d576-47f3-94e5-192856898684-home\") pod \"scheduler-ha-replicas-test-kserve-74cd6f7448-spgqf\" (UID: \"6e43795c-d576-47f3-94e5-192856898684\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-74cd6f7448-spgqf"
Oct 30 03:22:39.385183 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:22:39.385138    4461 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qbdc9\" (UniqueName: \"kubernetes.io/projected/6e43795c-d576-47f3-94e5-192856898684-kube-api-access-qbdc9\") pod \"scheduler-ha-replicas-test-kserve-74cd6f7448-spgqf\" (UID: \"6e43795c-d576-47f3-94e5-192856898684\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-74cd6f7448-spgqf"
Oct 30 03:22:39.385246 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:22:39.385224    4461 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6e43795c-d576-47f3-94e5-192856898684-dshm\") pod \"scheduler-ha-replicas-test-kserve-74cd6f7448-spgqf\" (UID: \"6e43795c-d576-47f3-94e5-192856898684\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-74cd6f7448-spgqf"
Oct 30 03:22:39.404699 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:22:39.404664    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:22:39.486586 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:22:39.486547    4461 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6e43795c-d576-47f3-94e5-192856898684-dshm\") pod \"scheduler-ha-replicas-test-kserve-74cd6f7448-spgqf\" (UID: \"6e43795c-d576-47f3-94e5-192856898684\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-74cd6f7448-spgqf"
Oct 30 03:22:39.486787 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:22:39.486606    4461 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6e43795c-d576-47f3-94e5-192856898684-tmp-dir\") pod \"scheduler-ha-replicas-test-kserve-74cd6f7448-spgqf\" (UID: \"6e43795c-d576-47f3-94e5-192856898684\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-74cd6f7448-spgqf"
Oct 30 03:22:39.486787 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:22:39.486638    4461 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6e43795c-d576-47f3-94e5-192856898684-model-cache\") pod \"scheduler-ha-replicas-test-kserve-74cd6f7448-spgqf\" (UID: \"6e43795c-d576-47f3-94e5-192856898684\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-74cd6f7448-spgqf"
Oct 30 03:22:39.486787 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:22:39.486659    4461 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6e43795c-d576-47f3-94e5-192856898684-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-74cd6f7448-spgqf\" (UID: \"6e43795c-d576-47f3-94e5-192856898684\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-74cd6f7448-spgqf"
Oct 30 03:22:39.486787 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:22:39.486675    4461 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6e43795c-d576-47f3-94e5-192856898684-home\") pod \"scheduler-ha-replicas-test-kserve-74cd6f7448-spgqf\" (UID: \"6e43795c-d576-47f3-94e5-192856898684\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-74cd6f7448-spgqf"
Oct 30 03:22:39.486787 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:22:39.486692    4461 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qbdc9\" (UniqueName: \"kubernetes.io/projected/6e43795c-d576-47f3-94e5-192856898684-kube-api-access-qbdc9\") pod \"scheduler-ha-replicas-test-kserve-74cd6f7448-spgqf\" (UID: \"6e43795c-d576-47f3-94e5-192856898684\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-74cd6f7448-spgqf"
Oct 30 03:22:39.487177 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:22:39.487152    4461 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6e43795c-d576-47f3-94e5-192856898684-home\") pod \"scheduler-ha-replicas-test-kserve-74cd6f7448-spgqf\" (UID: \"6e43795c-d576-47f3-94e5-192856898684\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-74cd6f7448-spgqf"
Oct 30 03:22:39.487177 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:22:39.487168    4461 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6e43795c-d576-47f3-94e5-192856898684-model-cache\") pod \"scheduler-ha-replicas-test-kserve-74cd6f7448-spgqf\" (UID: \"6e43795c-d576-47f3-94e5-192856898684\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-74cd6f7448-spgqf"
Oct 30 03:22:39.487301 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:22:39.487151    4461 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6e43795c-d576-47f3-94e5-192856898684-tmp-dir\") pod \"scheduler-ha-replicas-test-kserve-74cd6f7448-spgqf\" (UID: \"6e43795c-d576-47f3-94e5-192856898684\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-74cd6f7448-spgqf"
Oct 30 03:22:39.488964 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:22:39.488948    4461 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6e43795c-d576-47f3-94e5-192856898684-dshm\") pod \"scheduler-ha-replicas-test-kserve-74cd6f7448-spgqf\" (UID: \"6e43795c-d576-47f3-94e5-192856898684\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-74cd6f7448-spgqf"
Oct 30 03:22:39.489057 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:22:39.489042    4461 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6e43795c-d576-47f3-94e5-192856898684-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-74cd6f7448-spgqf\" (UID: \"6e43795c-d576-47f3-94e5-192856898684\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-74cd6f7448-spgqf"
Oct 30 03:22:39.511692 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:22:39.511664    4461 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qbdc9\" (UniqueName: \"kubernetes.io/projected/6e43795c-d576-47f3-94e5-192856898684-kube-api-access-qbdc9\") pod \"scheduler-ha-replicas-test-kserve-74cd6f7448-spgqf\" (UID: \"6e43795c-d576-47f3-94e5-192856898684\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-74cd6f7448-spgqf"
Oct 30 03:22:39.543213 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:22:39.543146    4461 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-84865b6lfs4x"]
Oct 30 03:22:39.546542 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:22:39.546524    4461 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-84865b6lfs4x"
Oct 30 03:22:39.567667 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:22:39.567638    4461 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-84865b6lfs4x"]
Oct 30 03:22:39.587550 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:22:39.587524    4461 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3e283094-9a35-4d83-8dd7-d17db3cb6e2b-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-84865b6lfs4x\" (UID: \"3e283094-9a35-4d83-8dd7-d17db3cb6e2b\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-84865b6lfs4x"
Oct 30 03:22:39.587700 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:22:39.587586    4461 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nxp6w\" (UniqueName: \"kubernetes.io/projected/3e283094-9a35-4d83-8dd7-d17db3cb6e2b-kube-api-access-nxp6w\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-84865b6lfs4x\" (UID: \"3e283094-9a35-4d83-8dd7-d17db3cb6e2b\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-84865b6lfs4x"
Oct 30 03:22:39.600795 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:22:39.600764    4461 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-2541cd65\"/\"scheduler-ha-replicas-test-epp-sa-dockercfg-zdvmn\""
Oct 30 03:22:39.607948 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:22:39.607923    4461 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-74cd6f7448-spgqf"
Oct 30 03:22:39.640072 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:22:39.640033    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:22:39.688653 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:22:39.688623    4461 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3e283094-9a35-4d83-8dd7-d17db3cb6e2b-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-84865b6lfs4x\" (UID: \"3e283094-9a35-4d83-8dd7-d17db3cb6e2b\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-84865b6lfs4x"
Oct 30 03:22:39.688797 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:22:39.688673    4461 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nxp6w\" (UniqueName: \"kubernetes.io/projected/3e283094-9a35-4d83-8dd7-d17db3cb6e2b-kube-api-access-nxp6w\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-84865b6lfs4x\" (UID: \"3e283094-9a35-4d83-8dd7-d17db3cb6e2b\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-84865b6lfs4x"
Oct 30 03:22:39.690950 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:22:39.690930    4461 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3e283094-9a35-4d83-8dd7-d17db3cb6e2b-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-84865b6lfs4x\" (UID: \"3e283094-9a35-4d83-8dd7-d17db3cb6e2b\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-84865b6lfs4x"
Oct 30 03:22:39.715981 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:22:39.715950    4461 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nxp6w\" (UniqueName: \"kubernetes.io/projected/3e283094-9a35-4d83-8dd7-d17db3cb6e2b-kube-api-access-nxp6w\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-84865b6lfs4x\" (UID: \"3e283094-9a35-4d83-8dd7-d17db3cb6e2b\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-84865b6lfs4x"
Oct 30 03:22:39.861635 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:22:39.861561    4461 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-84865b6lfs4x"
Oct 30 03:22:39.905443 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:22:39.905412    4461 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-74cd6f7448-spgqf"]
Oct 30 03:22:39.909884 ip-10-0-143-156 kubenswrapper[4461]: W1030 03:22:39.909651    4461 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6e43795c_d576_47f3_94e5_192856898684.slice/crio-a53deba11d329644d62368da5deadbd4ae8191e1da7d8aeba47a7a733e9e63a6 WatchSource:0}: Error finding container a53deba11d329644d62368da5deadbd4ae8191e1da7d8aeba47a7a733e9e63a6: Status 404 returned error can't find the container with id a53deba11d329644d62368da5deadbd4ae8191e1da7d8aeba47a7a733e9e63a6
Oct 30 03:22:40.161232 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:22:40.161158    4461 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-84865b6lfs4x"]
Oct 30 03:22:40.163667 ip-10-0-143-156 kubenswrapper[4461]: W1030 03:22:40.163644    4461 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3e283094_9a35_4d83_8dd7_d17db3cb6e2b.slice/crio-444d18112aa241e93916ab8ac7bc473e41e672ff21f86afc1b54b68cd0aebaf5 WatchSource:0}: Error finding container 444d18112aa241e93916ab8ac7bc473e41e672ff21f86afc1b54b68cd0aebaf5: Status 404 returned error can't find the container with id 444d18112aa241e93916ab8ac7bc473e41e672ff21f86afc1b54b68cd0aebaf5
Oct 30 03:22:40.744384 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:22:40.744343    4461 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-74cd6f7448-spgqf" event={"ID":"6e43795c-d576-47f3-94e5-192856898684","Type":"ContainerStarted","Data":"c04f80d668eba7388146e91acb2acf477d874502ec121e6f06eac44821becb1a"}
Oct 30 03:22:40.744835 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:22:40.744408    4461 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-74cd6f7448-spgqf" event={"ID":"6e43795c-d576-47f3-94e5-192856898684","Type":"ContainerStarted","Data":"a53deba11d329644d62368da5deadbd4ae8191e1da7d8aeba47a7a733e9e63a6"}
Oct 30 03:22:40.745656 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:22:40.745626    4461 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-84865b6lfs4x" event={"ID":"3e283094-9a35-4d83-8dd7-d17db3cb6e2b","Type":"ContainerStarted","Data":"444d18112aa241e93916ab8ac7bc473e41e672ff21f86afc1b54b68cd0aebaf5"}
Oct 30 03:22:40.788942 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:22:40.788886    4461 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-74cd6f7448-spgqf" podStartSLOduration=1.788870745 podStartE2EDuration="1.788870745s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:22:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:22:40.787905952 +0000 UTC m=+1628.837711128" watchObservedRunningTime="2026-10-30 03:22:40.788870745 +0000 UTC m=+1628.838675921"
Oct 30 03:22:41.105026 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:22:41.104948    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:22:41.608412 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:22:41.608377    4461 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-74cd6f7448-spgqf"
Oct 30 03:22:41.751637 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:22:41.751604    4461 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-84865b6lfs4x" event={"ID":"3e283094-9a35-4d83-8dd7-d17db3cb6e2b","Type":"ContainerStarted","Data":"f901bbc451f3e83feb9c713e33c44e38c942d1ae864d4d3f5ffab122ab848a5b"}
Oct 30 03:22:41.752073 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:22:41.751798    4461 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-84865b6lfs4x"
Oct 30 03:22:41.795896 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:22:41.795844    4461 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-84865b6lfs4x" podStartSLOduration=1.27947404 podStartE2EDuration="2.795828968s" totalImagesPullingTime="1.516354928s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:22:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:22:41.794531177 +0000 UTC m=+1629.844336349" watchObservedRunningTime="2026-10-30 03:22:41.795828968 +0000 UTC m=+1629.845634173"
Oct 30 03:22:43.534937 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:22:43.534900    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:22:44.268398 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:22:44.268361    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:22:49.608238 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:22:49.608198    4461 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-74cd6f7448-spgqf"
Oct 30 03:22:49.620780 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:22:49.620753    4461 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-74cd6f7448-spgqf"
Oct 30 03:22:49.799294 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:22:49.799268    4461 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-74cd6f7448-spgqf"
Oct 30 03:22:49.906066 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:22:49.905963    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:22:50.101401 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:22:50.101341    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:22:51.840709 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:22:51.840672    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:22:52.011310 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:22:52.011269    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:22:54.503342 ip-10-0-143-156 kubenswrapper[4461]: E1030 03:22:54.503299    4461 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2k85b" containerName="extractor"
Oct 30 03:23:12.757407 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:23:12.757366    4461 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-84865b6lfs4x" podUID="3e283094-9a35-4d83-8dd7-d17db3cb6e2b" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 03:23:14.397176 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:23:14.397076    4461 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-84865b6lfs4x"]
Oct 30 03:23:14.398896 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:23:14.398815    4461 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-74cd6f7448-spgqf"]
Oct 30 03:23:14.399038 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:23:14.398906    4461 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-84865b6lfs4x" podUID="3e283094-9a35-4d83-8dd7-d17db3cb6e2b" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 03:23:14.408318 ip-10-0-143-156 kubenswrapper[4461]: E1030 03:23:14.408280    4461 log.go:32] "ExecSync cmd from runtime service failed" err="rpc error: code = Unknown desc = command error: , stdout: , stderr: , exit code -1" containerID="c04f80d668eba7388146e91acb2acf477d874502ec121e6f06eac44821becb1a" cmd=["/bin/sleep","15"]
Oct 30 03:23:14.408454 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:23:14.408332    4461 handlers.go:85] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-74cd6f7448-spgqf" message="" err="rpc error: code = Unknown desc = command error: , stdout: , stderr: , exit code -1"
Oct 30 03:23:14.408454 ip-10-0-143-156 kubenswrapper[4461]: E1030 03:23:14.408345    4461 kuberuntime_container.go:780] "PreStop hook failed" err="rpc error: code = Unknown desc = command error: , stdout: , stderr: , exit code -1" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-74cd6f7448-spgqf" podUID="6e43795c-d576-47f3-94e5-192856898684" containerName="main" containerID="cri-o://c04f80d668eba7388146e91acb2acf477d874502ec121e6f06eac44821becb1a"
Oct 30 03:23:14.408454 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:23:14.408381    4461 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-74cd6f7448-spgqf" podUID="6e43795c-d576-47f3-94e5-192856898684" containerName="main" containerID="cri-o://c04f80d668eba7388146e91acb2acf477d874502ec121e6f06eac44821becb1a" gracePeriod=60
Oct 30 03:23:14.462356 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:23:14.462320    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:23:14.654378 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:23:14.654314    4461 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-74cd6f7448-spgqf"
Oct 30 03:23:14.659748 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:23:14.659726    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:23:14.707688 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:23:14.707656    4461 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6e43795c-d576-47f3-94e5-192856898684-tls-certs\" (UniqueName: \"kubernetes.io/secret/6e43795c-d576-47f3-94e5-192856898684-tls-certs\") pod \"6e43795c-d576-47f3-94e5-192856898684\" (UID: \"6e43795c-d576-47f3-94e5-192856898684\") "
Oct 30 03:23:14.707860 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:23:14.707700    4461 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6e43795c-d576-47f3-94e5-192856898684-home\" (UniqueName: \"kubernetes.io/empty-dir/6e43795c-d576-47f3-94e5-192856898684-home\") pod \"6e43795c-d576-47f3-94e5-192856898684\" (UID: \"6e43795c-d576-47f3-94e5-192856898684\") "
Oct 30 03:23:14.707860 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:23:14.707736    4461 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6e43795c-d576-47f3-94e5-192856898684-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6e43795c-d576-47f3-94e5-192856898684-model-cache\") pod \"6e43795c-d576-47f3-94e5-192856898684\" (UID: \"6e43795c-d576-47f3-94e5-192856898684\") "
Oct 30 03:23:14.707860 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:23:14.707802    4461 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6e43795c-d576-47f3-94e5-192856898684-kube-api-access-qbdc9\" (UniqueName: \"kubernetes.io/projected/6e43795c-d576-47f3-94e5-192856898684-kube-api-access-qbdc9\") pod \"6e43795c-d576-47f3-94e5-192856898684\" (UID: \"6e43795c-d576-47f3-94e5-192856898684\") "
Oct 30 03:23:14.707860 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:23:14.707831    4461 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6e43795c-d576-47f3-94e5-192856898684-dshm\" (UniqueName: \"kubernetes.io/empty-dir/6e43795c-d576-47f3-94e5-192856898684-dshm\") pod \"6e43795c-d576-47f3-94e5-192856898684\" (UID: \"6e43795c-d576-47f3-94e5-192856898684\") "
Oct 30 03:23:14.707860 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:23:14.707857    4461 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6e43795c-d576-47f3-94e5-192856898684-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6e43795c-d576-47f3-94e5-192856898684-tmp-dir\") pod \"6e43795c-d576-47f3-94e5-192856898684\" (UID: \"6e43795c-d576-47f3-94e5-192856898684\") "
Oct 30 03:23:14.708126 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:23:14.708012    4461 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6e43795c-d576-47f3-94e5-192856898684-home" pod "6e43795c-d576-47f3-94e5-192856898684" (UID: "6e43795c-d576-47f3-94e5-192856898684"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:23:14.708196 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:23:14.708121    4461 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6e43795c-d576-47f3-94e5-192856898684-model-cache" pod "6e43795c-d576-47f3-94e5-192856898684" (UID: "6e43795c-d576-47f3-94e5-192856898684"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:23:14.708196 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:23:14.708162    4461 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6e43795c-d576-47f3-94e5-192856898684-home\") on node \"ip-10-0-143-156.ec2.internal\" DevicePath \"\""
Oct 30 03:23:14.708288 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:23:14.708228    4461 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6e43795c-d576-47f3-94e5-192856898684-tmp-dir" pod "6e43795c-d576-47f3-94e5-192856898684" (UID: "6e43795c-d576-47f3-94e5-192856898684"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:23:14.710071 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:23:14.710042    4461 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6e43795c-d576-47f3-94e5-192856898684-dshm" pod "6e43795c-d576-47f3-94e5-192856898684" (UID: "6e43795c-d576-47f3-94e5-192856898684"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:23:14.710238 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:23:14.710071    4461 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6e43795c-d576-47f3-94e5-192856898684-tls-certs" pod "6e43795c-d576-47f3-94e5-192856898684" (UID: "6e43795c-d576-47f3-94e5-192856898684"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:23:14.710238 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:23:14.710078    4461 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6e43795c-d576-47f3-94e5-192856898684-kube-api-access-qbdc9" pod "6e43795c-d576-47f3-94e5-192856898684" (UID: "6e43795c-d576-47f3-94e5-192856898684"). InnerVolumeSpecName "kube-api-access-qbdc9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:23:14.809241 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:23:14.809210    4461 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6e43795c-d576-47f3-94e5-192856898684-tmp-dir\") on node \"ip-10-0-143-156.ec2.internal\" DevicePath \"\""
Oct 30 03:23:14.809241 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:23:14.809236    4461 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6e43795c-d576-47f3-94e5-192856898684-tls-certs\") on node \"ip-10-0-143-156.ec2.internal\" DevicePath \"\""
Oct 30 03:23:14.809241 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:23:14.809245    4461 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6e43795c-d576-47f3-94e5-192856898684-model-cache\") on node \"ip-10-0-143-156.ec2.internal\" DevicePath \"\""
Oct 30 03:23:14.809458 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:23:14.809253    4461 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qbdc9\" (UniqueName: \"kubernetes.io/projected/6e43795c-d576-47f3-94e5-192856898684-kube-api-access-qbdc9\") on node \"ip-10-0-143-156.ec2.internal\" DevicePath \"\""
Oct 30 03:23:14.809458 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:23:14.809262    4461 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6e43795c-d576-47f3-94e5-192856898684-dshm\") on node \"ip-10-0-143-156.ec2.internal\" DevicePath \"\""
Oct 30 03:23:14.890021 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:23:14.889991    4461 generic.go:356] "Generic (PLEG): container finished" podID="6e43795c-d576-47f3-94e5-192856898684" containerID="c04f80d668eba7388146e91acb2acf477d874502ec121e6f06eac44821becb1a" exitCode=0
Oct 30 03:23:14.890203 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:23:14.890038    4461 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-74cd6f7448-spgqf"
Oct 30 03:23:14.890203 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:23:14.890053    4461 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-74cd6f7448-spgqf" event={"ID":"6e43795c-d576-47f3-94e5-192856898684","Type":"ContainerDied","Data":"c04f80d668eba7388146e91acb2acf477d874502ec121e6f06eac44821becb1a"}
Oct 30 03:23:14.890203 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:23:14.890108    4461 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-74cd6f7448-spgqf" event={"ID":"6e43795c-d576-47f3-94e5-192856898684","Type":"ContainerDied","Data":"a53deba11d329644d62368da5deadbd4ae8191e1da7d8aeba47a7a733e9e63a6"}
Oct 30 03:23:14.890203 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:23:14.890130    4461 scope.go:122] "RemoveContainer" containerID="c04f80d668eba7388146e91acb2acf477d874502ec121e6f06eac44821becb1a"
Oct 30 03:23:14.899169 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:23:14.899147    4461 scope.go:122] "RemoveContainer" containerID="c04f80d668eba7388146e91acb2acf477d874502ec121e6f06eac44821becb1a"
Oct 30 03:23:14.899431 ip-10-0-143-156 kubenswrapper[4461]: E1030 03:23:14.899414    4461 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c04f80d668eba7388146e91acb2acf477d874502ec121e6f06eac44821becb1a\": container with ID starting with c04f80d668eba7388146e91acb2acf477d874502ec121e6f06eac44821becb1a not found: ID does not exist" containerID="c04f80d668eba7388146e91acb2acf477d874502ec121e6f06eac44821becb1a"
Oct 30 03:23:14.899497 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:23:14.899445    4461 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c04f80d668eba7388146e91acb2acf477d874502ec121e6f06eac44821becb1a"} err="failed to get container status \"c04f80d668eba7388146e91acb2acf477d874502ec121e6f06eac44821becb1a\": rpc error: code = NotFound desc = could not find container \"c04f80d668eba7388146e91acb2acf477d874502ec121e6f06eac44821becb1a\": container with ID starting with c04f80d668eba7388146e91acb2acf477d874502ec121e6f06eac44821becb1a not found: ID does not exist"
Oct 30 03:23:14.942408 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:23:14.942346    4461 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-74cd6f7448-spgqf"]
Oct 30 03:23:14.948285 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:23:14.948264    4461 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-74cd6f7448-spgqf"]
Oct 30 03:23:16.115596 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:23:16.115556    4461 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6e43795c-d576-47f3-94e5-192856898684" path="/var/lib/kubelet/pods/6e43795c-d576-47f3-94e5-192856898684/volumes"
Oct 30 03:23:24.398802 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:23:24.398753    4461 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-84865b6lfs4x" podUID="3e283094-9a35-4d83-8dd7-d17db3cb6e2b" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 03:23:24.465833 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:23:24.465787    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:23:24.504171 ip-10-0-143-156 kubenswrapper[4461]: E1030 03:23:24.504138    4461 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2k85b" containerName="extractor"
Oct 30 03:23:29.417775 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:23:29.417733    4461 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-84865b6lfs4x" podUID="3e283094-9a35-4d83-8dd7-d17db3cb6e2b" containerName="main" containerID="cri-o://f901bbc451f3e83feb9c713e33c44e38c942d1ae864d4d3f5ffab122ab848a5b" gracePeriod=45
Oct 30 03:23:34.399623 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:23:34.399585    4461 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-84865b6lfs4x" podUID="3e283094-9a35-4d83-8dd7-d17db3cb6e2b" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 03:23:41.990411 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:23:41.990372    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:23:42.167006 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:23:42.166961    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:23:44.399040 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:23:44.399005    4461 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-84865b6lfs4x" podUID="3e283094-9a35-4d83-8dd7-d17db3cb6e2b" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 03:23:46.063425 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:23:46.063379    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:23:46.367345 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:23:46.367211    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:23:48.761235 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:23:48.761206    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:23:49.343195 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:23:49.343143    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:23:54.399607 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:23:54.399569    4461 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-84865b6lfs4x" podUID="3e283094-9a35-4d83-8dd7-d17db3cb6e2b" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 03:23:54.503442 ip-10-0-143-156 kubenswrapper[4461]: E1030 03:23:54.503407    4461 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2k85b" containerName="extractor"
Oct 30 03:23:55.233292 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:23:55.233253    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:23:55.415884 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:23:55.415814    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:23:59.538580 ip-10-0-143-156 kubenswrapper[4461]: E1030 03:23:59.538539    4461 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3e283094_9a35_4d83_8dd7_d17db3cb6e2b.slice/crio-conmon-f901bbc451f3e83feb9c713e33c44e38c942d1ae864d4d3f5ffab122ab848a5b.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3e283094_9a35_4d83_8dd7_d17db3cb6e2b.slice/crio-f901bbc451f3e83feb9c713e33c44e38c942d1ae864d4d3f5ffab122ab848a5b.scope\": RecentStats: unable to find data in memory cache]"
Oct 30 03:23:59.539063 ip-10-0-143-156 kubenswrapper[4461]: E1030 03:23:59.538558    4461 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3e283094_9a35_4d83_8dd7_d17db3cb6e2b.slice/crio-conmon-f901bbc451f3e83feb9c713e33c44e38c942d1ae864d4d3f5ffab122ab848a5b.scope\": RecentStats: unable to find data in memory cache]"
Oct 30 03:23:59.539205 ip-10-0-143-156 kubenswrapper[4461]: E1030 03:23:59.538609    4461 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3e283094_9a35_4d83_8dd7_d17db3cb6e2b.slice/crio-conmon-f901bbc451f3e83feb9c713e33c44e38c942d1ae864d4d3f5ffab122ab848a5b.scope\": RecentStats: unable to find data in memory cache]"
Oct 30 03:23:59.540689 ip-10-0-143-156 kubenswrapper[4461]: E1030 03:23:59.538814    4461 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3e283094_9a35_4d83_8dd7_d17db3cb6e2b.slice/crio-f901bbc451f3e83feb9c713e33c44e38c942d1ae864d4d3f5ffab122ab848a5b.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3e283094_9a35_4d83_8dd7_d17db3cb6e2b.slice/crio-conmon-f901bbc451f3e83feb9c713e33c44e38c942d1ae864d4d3f5ffab122ab848a5b.scope\": RecentStats: unable to find data in memory cache]"
Oct 30 03:23:59.678465 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:23:59.678399    4461 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-84865b6lfs4x"
Oct 30 03:23:59.699580 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:23:59.699535    4461 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3e283094-9a35-4d83-8dd7-d17db3cb6e2b-kube-api-access-nxp6w\" (UniqueName: \"kubernetes.io/projected/3e283094-9a35-4d83-8dd7-d17db3cb6e2b-kube-api-access-nxp6w\") pod \"3e283094-9a35-4d83-8dd7-d17db3cb6e2b\" (UID: \"3e283094-9a35-4d83-8dd7-d17db3cb6e2b\") "
Oct 30 03:23:59.699580 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:23:59.699585    4461 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3e283094-9a35-4d83-8dd7-d17db3cb6e2b-tls-certs\" (UniqueName: \"kubernetes.io/secret/3e283094-9a35-4d83-8dd7-d17db3cb6e2b-tls-certs\") pod \"3e283094-9a35-4d83-8dd7-d17db3cb6e2b\" (UID: \"3e283094-9a35-4d83-8dd7-d17db3cb6e2b\") "
Oct 30 03:23:59.701593 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:23:59.701561    4461 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3e283094-9a35-4d83-8dd7-d17db3cb6e2b-tls-certs" pod "3e283094-9a35-4d83-8dd7-d17db3cb6e2b" (UID: "3e283094-9a35-4d83-8dd7-d17db3cb6e2b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:23:59.701593 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:23:59.701575    4461 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3e283094-9a35-4d83-8dd7-d17db3cb6e2b-kube-api-access-nxp6w" pod "3e283094-9a35-4d83-8dd7-d17db3cb6e2b" (UID: "3e283094-9a35-4d83-8dd7-d17db3cb6e2b"). InnerVolumeSpecName "kube-api-access-nxp6w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:23:59.800272 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:23:59.800238    4461 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nxp6w\" (UniqueName: \"kubernetes.io/projected/3e283094-9a35-4d83-8dd7-d17db3cb6e2b-kube-api-access-nxp6w\") on node \"ip-10-0-143-156.ec2.internal\" DevicePath \"\""
Oct 30 03:23:59.800272 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:23:59.800267    4461 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3e283094-9a35-4d83-8dd7-d17db3cb6e2b-tls-certs\") on node \"ip-10-0-143-156.ec2.internal\" DevicePath \"\""
Oct 30 03:24:00.071627 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:24:00.071596    4461 generic.go:356] "Generic (PLEG): container finished" podID="3e283094-9a35-4d83-8dd7-d17db3cb6e2b" containerID="f901bbc451f3e83feb9c713e33c44e38c942d1ae864d4d3f5ffab122ab848a5b" exitCode=0
Oct 30 03:24:00.071867 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:24:00.071638    4461 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-84865b6lfs4x"
Oct 30 03:24:00.071867 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:24:00.071653    4461 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-84865b6lfs4x" event={"ID":"3e283094-9a35-4d83-8dd7-d17db3cb6e2b","Type":"ContainerDied","Data":"f901bbc451f3e83feb9c713e33c44e38c942d1ae864d4d3f5ffab122ab848a5b"}
Oct 30 03:24:00.071867 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:24:00.071728    4461 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-84865b6lfs4x" event={"ID":"3e283094-9a35-4d83-8dd7-d17db3cb6e2b","Type":"ContainerDied","Data":"444d18112aa241e93916ab8ac7bc473e41e672ff21f86afc1b54b68cd0aebaf5"}
Oct 30 03:24:00.071867 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:24:00.071751    4461 scope.go:122] "RemoveContainer" containerID="f901bbc451f3e83feb9c713e33c44e38c942d1ae864d4d3f5ffab122ab848a5b"
Oct 30 03:24:00.080241 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:24:00.080221    4461 scope.go:122] "RemoveContainer" containerID="f901bbc451f3e83feb9c713e33c44e38c942d1ae864d4d3f5ffab122ab848a5b"
Oct 30 03:24:00.080559 ip-10-0-143-156 kubenswrapper[4461]: E1030 03:24:00.080536    4461 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f901bbc451f3e83feb9c713e33c44e38c942d1ae864d4d3f5ffab122ab848a5b\": container with ID starting with f901bbc451f3e83feb9c713e33c44e38c942d1ae864d4d3f5ffab122ab848a5b not found: ID does not exist" containerID="f901bbc451f3e83feb9c713e33c44e38c942d1ae864d4d3f5ffab122ab848a5b"
Oct 30 03:24:00.080648 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:24:00.080572    4461 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f901bbc451f3e83feb9c713e33c44e38c942d1ae864d4d3f5ffab122ab848a5b"} err="failed to get container status \"f901bbc451f3e83feb9c713e33c44e38c942d1ae864d4d3f5ffab122ab848a5b\": rpc error: code = NotFound desc = could not find container \"f901bbc451f3e83feb9c713e33c44e38c942d1ae864d4d3f5ffab122ab848a5b\": container with ID starting with f901bbc451f3e83feb9c713e33c44e38c942d1ae864d4d3f5ffab122ab848a5b not found: ID does not exist"
Oct 30 03:24:00.123541 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:24:00.123510    4461 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-84865b6lfs4x"]
Oct 30 03:24:00.136202 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:24:00.136171    4461 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-84865b6lfs4x"]
Oct 30 03:24:02.115051 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:24:02.115017    4461 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3e283094-9a35-4d83-8dd7-d17db3cb6e2b" path="/var/lib/kubelet/pods/3e283094-9a35-4d83-8dd7-d17db3cb6e2b/volumes"
Oct 30 03:24:04.714893 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:24:04.714851    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:24:04.933008 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:24:04.932969    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:24:09.218306 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:24:09.218272    4461 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-6667fc56b8-gr2f9"]
Oct 30 03:24:09.219684 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:24:09.219627    4461 cpu_manager.go:414] "containerMap: removing container" podUID="6e43795c-d576-47f3-94e5-192856898684" containerName="main"
Oct 30 03:24:09.219873 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:24:09.219857    4461 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6e43795c-d576-47f3-94e5-192856898684" containerName="main"
Oct 30 03:24:09.220014 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:24:09.220003    4461 cpu_manager.go:414] "containerMap: removing container" podUID="3e283094-9a35-4d83-8dd7-d17db3cb6e2b" containerName="main"
Oct 30 03:24:09.220134 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:24:09.220122    4461 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3e283094-9a35-4d83-8dd7-d17db3cb6e2b" containerName="main"
Oct 30 03:24:09.220470 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:24:09.220458    4461 memory_manager.go:365] "RemoveStaleState removing state" podUID="6e43795c-d576-47f3-94e5-192856898684" containerName="main"
Oct 30 03:24:09.220785 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:24:09.220771    4461 memory_manager.go:365] "RemoveStaleState removing state" podUID="3e283094-9a35-4d83-8dd7-d17db3cb6e2b" containerName="main"
Oct 30 03:24:09.228146 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:24:09.228111    4461 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-6667fc56b8-gr2f9"
Oct 30 03:24:09.241694 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:24:09.241668    4461 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-6667fc56b8-gr2f9"]
Oct 30 03:24:09.252230 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:24:09.252197    4461 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-35377f35\"/\"openshift-service-ca.crt\""
Oct 30 03:24:09.252230 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:24:09.252215    4461 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-35377f35\"/\"kube-root-ca.crt\""
Oct 30 03:24:09.252505 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:24:09.252255    4461 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-35377f35\"/\"scheduler-custom-template-test-kserve-self-signed-certs\""
Oct 30 03:24:09.255559 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:24:09.255519    4461 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-35377f35\"/\"default-dockercfg-sxsgc\""
Oct 30 03:24:09.284856 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:24:09.284822    4461 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/70bd8cb1-e22a-4a38-8688-2566a80e2cdb-dshm\") pod \"scheduler-custom-template-test-kserve-6667fc56b8-gr2f9\" (UID: \"70bd8cb1-e22a-4a38-8688-2566a80e2cdb\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-6667fc56b8-gr2f9"
Oct 30 03:24:09.284856 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:24:09.284861    4461 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/70bd8cb1-e22a-4a38-8688-2566a80e2cdb-tls-certs\") pod \"scheduler-custom-template-test-kserve-6667fc56b8-gr2f9\" (UID: \"70bd8cb1-e22a-4a38-8688-2566a80e2cdb\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-6667fc56b8-gr2f9"
Oct 30 03:24:09.285054 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:24:09.284985    4461 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/70bd8cb1-e22a-4a38-8688-2566a80e2cdb-home\") pod \"scheduler-custom-template-test-kserve-6667fc56b8-gr2f9\" (UID: \"70bd8cb1-e22a-4a38-8688-2566a80e2cdb\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-6667fc56b8-gr2f9"
Oct 30 03:24:09.285054 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:24:09.285011    4461 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/70bd8cb1-e22a-4a38-8688-2566a80e2cdb-model-cache\") pod \"scheduler-custom-template-test-kserve-6667fc56b8-gr2f9\" (UID: \"70bd8cb1-e22a-4a38-8688-2566a80e2cdb\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-6667fc56b8-gr2f9"
Oct 30 03:24:09.285054 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:24:09.285043    4461 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/70bd8cb1-e22a-4a38-8688-2566a80e2cdb-tmp-dir\") pod \"scheduler-custom-template-test-kserve-6667fc56b8-gr2f9\" (UID: \"70bd8cb1-e22a-4a38-8688-2566a80e2cdb\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-6667fc56b8-gr2f9"
Oct 30 03:24:09.285183 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:24:09.285091    4461 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7vcwc\" (UniqueName: \"kubernetes.io/projected/70bd8cb1-e22a-4a38-8688-2566a80e2cdb-kube-api-access-7vcwc\") pod \"scheduler-custom-template-test-kserve-6667fc56b8-gr2f9\" (UID: \"70bd8cb1-e22a-4a38-8688-2566a80e2cdb\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-6667fc56b8-gr2f9"
Oct 30 03:24:09.345959 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:24:09.345925    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:24:09.385984 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:24:09.385930    4461 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/70bd8cb1-e22a-4a38-8688-2566a80e2cdb-dshm\") pod \"scheduler-custom-template-test-kserve-6667fc56b8-gr2f9\" (UID: \"70bd8cb1-e22a-4a38-8688-2566a80e2cdb\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-6667fc56b8-gr2f9"
Oct 30 03:24:09.385984 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:24:09.385989    4461 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/70bd8cb1-e22a-4a38-8688-2566a80e2cdb-tls-certs\") pod \"scheduler-custom-template-test-kserve-6667fc56b8-gr2f9\" (UID: \"70bd8cb1-e22a-4a38-8688-2566a80e2cdb\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-6667fc56b8-gr2f9"
Oct 30 03:24:09.386241 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:24:09.386034    4461 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/70bd8cb1-e22a-4a38-8688-2566a80e2cdb-home\") pod \"scheduler-custom-template-test-kserve-6667fc56b8-gr2f9\" (UID: \"70bd8cb1-e22a-4a38-8688-2566a80e2cdb\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-6667fc56b8-gr2f9"
Oct 30 03:24:09.386241 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:24:09.386051    4461 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/70bd8cb1-e22a-4a38-8688-2566a80e2cdb-model-cache\") pod \"scheduler-custom-template-test-kserve-6667fc56b8-gr2f9\" (UID: \"70bd8cb1-e22a-4a38-8688-2566a80e2cdb\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-6667fc56b8-gr2f9"
Oct 30 03:24:09.386241 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:24:09.386072    4461 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/70bd8cb1-e22a-4a38-8688-2566a80e2cdb-tmp-dir\") pod \"scheduler-custom-template-test-kserve-6667fc56b8-gr2f9\" (UID: \"70bd8cb1-e22a-4a38-8688-2566a80e2cdb\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-6667fc56b8-gr2f9"
Oct 30 03:24:09.386241 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:24:09.386088    4461 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7vcwc\" (UniqueName: \"kubernetes.io/projected/70bd8cb1-e22a-4a38-8688-2566a80e2cdb-kube-api-access-7vcwc\") pod \"scheduler-custom-template-test-kserve-6667fc56b8-gr2f9\" (UID: \"70bd8cb1-e22a-4a38-8688-2566a80e2cdb\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-6667fc56b8-gr2f9"
Oct 30 03:24:09.386480 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:24:09.386453    4461 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/70bd8cb1-e22a-4a38-8688-2566a80e2cdb-model-cache\") pod \"scheduler-custom-template-test-kserve-6667fc56b8-gr2f9\" (UID: \"70bd8cb1-e22a-4a38-8688-2566a80e2cdb\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-6667fc56b8-gr2f9"
Oct 30 03:24:09.386597 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:24:09.386488    4461 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/70bd8cb1-e22a-4a38-8688-2566a80e2cdb-home\") pod \"scheduler-custom-template-test-kserve-6667fc56b8-gr2f9\" (UID: \"70bd8cb1-e22a-4a38-8688-2566a80e2cdb\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-6667fc56b8-gr2f9"
Oct 30 03:24:09.386597 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:24:09.386554    4461 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/70bd8cb1-e22a-4a38-8688-2566a80e2cdb-tmp-dir\") pod \"scheduler-custom-template-test-kserve-6667fc56b8-gr2f9\" (UID: \"70bd8cb1-e22a-4a38-8688-2566a80e2cdb\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-6667fc56b8-gr2f9"
Oct 30 03:24:09.388378 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:24:09.388359    4461 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/70bd8cb1-e22a-4a38-8688-2566a80e2cdb-dshm\") pod \"scheduler-custom-template-test-kserve-6667fc56b8-gr2f9\" (UID: \"70bd8cb1-e22a-4a38-8688-2566a80e2cdb\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-6667fc56b8-gr2f9"
Oct 30 03:24:09.388541 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:24:09.388526    4461 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/70bd8cb1-e22a-4a38-8688-2566a80e2cdb-tls-certs\") pod \"scheduler-custom-template-test-kserve-6667fc56b8-gr2f9\" (UID: \"70bd8cb1-e22a-4a38-8688-2566a80e2cdb\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-6667fc56b8-gr2f9"
Oct 30 03:24:09.410245 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:24:09.410214    4461 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7vcwc\" (UniqueName: \"kubernetes.io/projected/70bd8cb1-e22a-4a38-8688-2566a80e2cdb-kube-api-access-7vcwc\") pod \"scheduler-custom-template-test-kserve-6667fc56b8-gr2f9\" (UID: \"70bd8cb1-e22a-4a38-8688-2566a80e2cdb\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-6667fc56b8-gr2f9"
Oct 30 03:24:09.541989 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:24:09.541896    4461 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-6667fc56b8-gr2f9"
Oct 30 03:24:09.689403 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:24:09.689365    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:24:09.845339 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:24:09.845262    4461 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-6667fc56b8-gr2f9"]
Oct 30 03:24:09.848829 ip-10-0-143-156 kubenswrapper[4461]: W1030 03:24:09.848801    4461 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod70bd8cb1_e22a_4a38_8688_2566a80e2cdb.slice/crio-c88b9d6550232183e4192b7e3c41127695f78990ae4cff0f8f136b3ab599435f WatchSource:0}: Error finding container c88b9d6550232183e4192b7e3c41127695f78990ae4cff0f8f136b3ab599435f: Status 404 returned error can't find the container with id c88b9d6550232183e4192b7e3c41127695f78990ae4cff0f8f136b3ab599435f
Oct 30 03:24:10.117555 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:24:10.117459    4461 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-6667fc56b8-gr2f9" event={"ID":"70bd8cb1-e22a-4a38-8688-2566a80e2cdb","Type":"ContainerStarted","Data":"980d13f3193e61c639ef3bbd5a71b6b899710546c04ccfed6b8d5d453f071c44"}
Oct 30 03:24:10.117555 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:24:10.117519    4461 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-6667fc56b8-gr2f9" event={"ID":"70bd8cb1-e22a-4a38-8688-2566a80e2cdb","Type":"ContainerStarted","Data":"c88b9d6550232183e4192b7e3c41127695f78990ae4cff0f8f136b3ab599435f"}
Oct 30 03:24:10.160413 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:24:10.160360    4461 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-6667fc56b8-gr2f9" podStartSLOduration=1.160344991 podStartE2EDuration="1.160344991s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:24:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:24:10.159243388 +0000 UTC m=+1718.209048572" watchObservedRunningTime="2026-10-30 03:24:10.160344991 +0000 UTC m=+1718.210150167"
Oct 30 03:24:10.542444 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:24:10.542391    4461 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-6667fc56b8-gr2f9"
Oct 30 03:24:11.492109 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:24:11.492074    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:24:14.185550 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:24:14.185518    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:24:17.322796 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:24:17.322758    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:24:17.487375 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:24:17.487334    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:24:19.542089 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:24:19.542057    4461 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-6667fc56b8-gr2f9"
Oct 30 03:24:19.554554 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:24:19.554530    4461 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-6667fc56b8-gr2f9"
Oct 30 03:24:20.168141 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:24:20.168085    4461 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-6667fc56b8-gr2f9"
Oct 30 03:24:24.504109 ip-10-0-143-156 kubenswrapper[4461]: E1030 03:24:24.504064    4461 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2k85b" containerName="extractor"
Oct 30 03:24:24.592530 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:24:24.592492    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:24:25.936985 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:24:25.936949    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:24:26.104121 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:24:26.104067    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:24:42.753618 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:24:42.753560    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:24:43.717689 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:24:43.717629    4461 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-6667fc56b8-gr2f9"]
Oct 30 03:24:43.753921 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:24:43.753887    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:24:43.763227 ip-10-0-143-156 kubenswrapper[4461]: E1030 03:24:43.763191    4461 log.go:32] "ExecSync cmd from runtime service failed" err="rpc error: code = Unknown desc = command error: , stdout: , stderr: , exit code -1" containerID="980d13f3193e61c639ef3bbd5a71b6b899710546c04ccfed6b8d5d453f071c44" cmd=["/bin/sleep","15"]
Oct 30 03:24:43.763400 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:24:43.763246    4461 handlers.go:85] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-6667fc56b8-gr2f9" message="" err="rpc error: code = Unknown desc = command error: , stdout: , stderr: , exit code -1"
Oct 30 03:24:43.763400 ip-10-0-143-156 kubenswrapper[4461]: E1030 03:24:43.763262    4461 kuberuntime_container.go:780] "PreStop hook failed" err="rpc error: code = Unknown desc = command error: , stdout: , stderr: , exit code -1" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-6667fc56b8-gr2f9" podUID="70bd8cb1-e22a-4a38-8688-2566a80e2cdb" containerName="main" containerID="cri-o://980d13f3193e61c639ef3bbd5a71b6b899710546c04ccfed6b8d5d453f071c44"
Oct 30 03:24:43.763400 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:24:43.763302    4461 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-6667fc56b8-gr2f9" podUID="70bd8cb1-e22a-4a38-8688-2566a80e2cdb" containerName="main" containerID="cri-o://980d13f3193e61c639ef3bbd5a71b6b899710546c04ccfed6b8d5d453f071c44" gracePeriod=60
Oct 30 03:24:43.925410 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:24:43.925374    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:24:44.106913 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:24:44.106890    4461 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-6667fc56b8-gr2f9"
Oct 30 03:24:44.192829 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:24:44.192790    4461 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/70bd8cb1-e22a-4a38-8688-2566a80e2cdb-home\" (UniqueName: \"kubernetes.io/empty-dir/70bd8cb1-e22a-4a38-8688-2566a80e2cdb-home\") pod \"70bd8cb1-e22a-4a38-8688-2566a80e2cdb\" (UID: \"70bd8cb1-e22a-4a38-8688-2566a80e2cdb\") "
Oct 30 03:24:44.193029 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:24:44.192845    4461 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/70bd8cb1-e22a-4a38-8688-2566a80e2cdb-kube-api-access-7vcwc\" (UniqueName: \"kubernetes.io/projected/70bd8cb1-e22a-4a38-8688-2566a80e2cdb-kube-api-access-7vcwc\") pod \"70bd8cb1-e22a-4a38-8688-2566a80e2cdb\" (UID: \"70bd8cb1-e22a-4a38-8688-2566a80e2cdb\") "
Oct 30 03:24:44.193029 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:24:44.192880    4461 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/70bd8cb1-e22a-4a38-8688-2566a80e2cdb-dshm\" (UniqueName: \"kubernetes.io/empty-dir/70bd8cb1-e22a-4a38-8688-2566a80e2cdb-dshm\") pod \"70bd8cb1-e22a-4a38-8688-2566a80e2cdb\" (UID: \"70bd8cb1-e22a-4a38-8688-2566a80e2cdb\") "
Oct 30 03:24:44.193029 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:24:44.192900    4461 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/70bd8cb1-e22a-4a38-8688-2566a80e2cdb-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/70bd8cb1-e22a-4a38-8688-2566a80e2cdb-tmp-dir\") pod \"70bd8cb1-e22a-4a38-8688-2566a80e2cdb\" (UID: \"70bd8cb1-e22a-4a38-8688-2566a80e2cdb\") "
Oct 30 03:24:44.193029 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:24:44.192920    4461 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/70bd8cb1-e22a-4a38-8688-2566a80e2cdb-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/70bd8cb1-e22a-4a38-8688-2566a80e2cdb-model-cache\") pod \"70bd8cb1-e22a-4a38-8688-2566a80e2cdb\" (UID: \"70bd8cb1-e22a-4a38-8688-2566a80e2cdb\") "
Oct 30 03:24:44.193029 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:24:44.193001    4461 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/70bd8cb1-e22a-4a38-8688-2566a80e2cdb-tls-certs\" (UniqueName: \"kubernetes.io/secret/70bd8cb1-e22a-4a38-8688-2566a80e2cdb-tls-certs\") pod \"70bd8cb1-e22a-4a38-8688-2566a80e2cdb\" (UID: \"70bd8cb1-e22a-4a38-8688-2566a80e2cdb\") "
Oct 30 03:24:44.193329 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:24:44.193138    4461 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/70bd8cb1-e22a-4a38-8688-2566a80e2cdb-home" pod "70bd8cb1-e22a-4a38-8688-2566a80e2cdb" (UID: "70bd8cb1-e22a-4a38-8688-2566a80e2cdb"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:24:44.193329 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:24:44.193217    4461 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/70bd8cb1-e22a-4a38-8688-2566a80e2cdb-tmp-dir" pod "70bd8cb1-e22a-4a38-8688-2566a80e2cdb" (UID: "70bd8cb1-e22a-4a38-8688-2566a80e2cdb"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:24:44.193329 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:24:44.193279    4461 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/70bd8cb1-e22a-4a38-8688-2566a80e2cdb-model-cache" pod "70bd8cb1-e22a-4a38-8688-2566a80e2cdb" (UID: "70bd8cb1-e22a-4a38-8688-2566a80e2cdb"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:24:44.193499 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:24:44.193477    4461 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/70bd8cb1-e22a-4a38-8688-2566a80e2cdb-home\") on node \"ip-10-0-143-156.ec2.internal\" DevicePath \"\""
Oct 30 03:24:44.193559 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:24:44.193512    4461 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/70bd8cb1-e22a-4a38-8688-2566a80e2cdb-tmp-dir\") on node \"ip-10-0-143-156.ec2.internal\" DevicePath \"\""
Oct 30 03:24:44.193559 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:24:44.193529    4461 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/70bd8cb1-e22a-4a38-8688-2566a80e2cdb-model-cache\") on node \"ip-10-0-143-156.ec2.internal\" DevicePath \"\""
Oct 30 03:24:44.195192 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:24:44.195167    4461 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/70bd8cb1-e22a-4a38-8688-2566a80e2cdb-dshm" pod "70bd8cb1-e22a-4a38-8688-2566a80e2cdb" (UID: "70bd8cb1-e22a-4a38-8688-2566a80e2cdb"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:24:44.195397 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:24:44.195374    4461 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/70bd8cb1-e22a-4a38-8688-2566a80e2cdb-kube-api-access-7vcwc" pod "70bd8cb1-e22a-4a38-8688-2566a80e2cdb" (UID: "70bd8cb1-e22a-4a38-8688-2566a80e2cdb"). InnerVolumeSpecName "kube-api-access-7vcwc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:24:44.195539 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:24:44.195512    4461 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/70bd8cb1-e22a-4a38-8688-2566a80e2cdb-tls-certs" pod "70bd8cb1-e22a-4a38-8688-2566a80e2cdb" (UID: "70bd8cb1-e22a-4a38-8688-2566a80e2cdb"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:24:44.248153 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:24:44.248055    4461 generic.go:356] "Generic (PLEG): container finished" podID="70bd8cb1-e22a-4a38-8688-2566a80e2cdb" containerID="980d13f3193e61c639ef3bbd5a71b6b899710546c04ccfed6b8d5d453f071c44" exitCode=0
Oct 30 03:24:44.248153 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:24:44.248127    4461 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-6667fc56b8-gr2f9"
Oct 30 03:24:44.248153 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:24:44.248140    4461 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-6667fc56b8-gr2f9" event={"ID":"70bd8cb1-e22a-4a38-8688-2566a80e2cdb","Type":"ContainerDied","Data":"980d13f3193e61c639ef3bbd5a71b6b899710546c04ccfed6b8d5d453f071c44"}
Oct 30 03:24:44.248376 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:24:44.248198    4461 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-6667fc56b8-gr2f9" event={"ID":"70bd8cb1-e22a-4a38-8688-2566a80e2cdb","Type":"ContainerDied","Data":"c88b9d6550232183e4192b7e3c41127695f78990ae4cff0f8f136b3ab599435f"}
Oct 30 03:24:44.248376 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:24:44.248219    4461 scope.go:122] "RemoveContainer" containerID="980d13f3193e61c639ef3bbd5a71b6b899710546c04ccfed6b8d5d453f071c44"
Oct 30 03:24:44.256939 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:24:44.256917    4461 scope.go:122] "RemoveContainer" containerID="980d13f3193e61c639ef3bbd5a71b6b899710546c04ccfed6b8d5d453f071c44"
Oct 30 03:24:44.257298 ip-10-0-143-156 kubenswrapper[4461]: E1030 03:24:44.257276    4461 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"980d13f3193e61c639ef3bbd5a71b6b899710546c04ccfed6b8d5d453f071c44\": container with ID starting with 980d13f3193e61c639ef3bbd5a71b6b899710546c04ccfed6b8d5d453f071c44 not found: ID does not exist" containerID="980d13f3193e61c639ef3bbd5a71b6b899710546c04ccfed6b8d5d453f071c44"
Oct 30 03:24:44.257344 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:24:44.257312    4461 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"980d13f3193e61c639ef3bbd5a71b6b899710546c04ccfed6b8d5d453f071c44"} err="failed to get container status \"980d13f3193e61c639ef3bbd5a71b6b899710546c04ccfed6b8d5d453f071c44\": rpc error: code = NotFound desc = could not find container \"980d13f3193e61c639ef3bbd5a71b6b899710546c04ccfed6b8d5d453f071c44\": container with ID starting with 980d13f3193e61c639ef3bbd5a71b6b899710546c04ccfed6b8d5d453f071c44 not found: ID does not exist"
Oct 30 03:24:44.294117 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:24:44.294073    4461 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/70bd8cb1-e22a-4a38-8688-2566a80e2cdb-tls-certs\") on node \"ip-10-0-143-156.ec2.internal\" DevicePath \"\""
Oct 30 03:24:44.294117 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:24:44.294116    4461 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7vcwc\" (UniqueName: \"kubernetes.io/projected/70bd8cb1-e22a-4a38-8688-2566a80e2cdb-kube-api-access-7vcwc\") on node \"ip-10-0-143-156.ec2.internal\" DevicePath \"\""
Oct 30 03:24:44.294294 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:24:44.294128    4461 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/70bd8cb1-e22a-4a38-8688-2566a80e2cdb-dshm\") on node \"ip-10-0-143-156.ec2.internal\" DevicePath \"\""
Oct 30 03:24:44.299021 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:24:44.298997    4461 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-6667fc56b8-gr2f9"]
Oct 30 03:24:44.303485 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:24:44.303459    4461 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-6667fc56b8-gr2f9"]
Oct 30 03:24:46.114824 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:24:46.114787    4461 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="70bd8cb1-e22a-4a38-8688-2566a80e2cdb" path="/var/lib/kubelet/pods/70bd8cb1-e22a-4a38-8688-2566a80e2cdb/volumes"
Oct 30 03:24:54.504410 ip-10-0-143-156 kubenswrapper[4461]: E1030 03:24:54.504368    4461 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2k85b" containerName="extractor"
Oct 30 03:25:16.030273 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:25:16.030232    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:25:16.245522 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:25:16.245484    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:25:16.608014 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:25:16.607979    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:25:16.789013 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:25:16.788916    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:25:20.644296 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:25:20.644256    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:25:20.863311 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:25:20.863227    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:25:21.534437 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:25:21.534399    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:25:21.704414 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:25:21.704371    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:25:22.160227 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:25:22.160193    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:25:22.344157 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:25:22.344060    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:25:24.261553 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:25:24.261517    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:25:24.504182 ip-10-0-143-156 kubenswrapper[4461]: E1030 03:25:24.504149    4461 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2k85b" containerName="extractor"
Oct 30 03:25:24.704345 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:25:24.704268    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:25:25.455409 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:25:25.455376    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:25:26.552300 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:25:26.552267    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:25:26.717748 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:25:26.717713    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:25:27.841322 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:25:27.841286    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:25:29.499436 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:25:29.499399    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:25:29.666912 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:25:29.666875    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:25:30.777821 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:25:30.777784    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:25:30.953137 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:25:30.953085    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:25:33.069486 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:25:33.069447    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:25:33.677976 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:25:33.677937    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:25:33.854370 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:25:33.854335    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:25:34.416608 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:25:34.416580    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:25:34.719985 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:25:34.719955    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:25:37.114722 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:25:37.114687    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:25:37.354000 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:25:37.353965    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:25:40.377123 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:25:40.375141    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:25:40.962885 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:25:40.962831    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:25:41.872134 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:25:41.870915    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:25:47.017501 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:25:47.017469    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:25:47.348172 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:25:47.348064    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:25:47.513573 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:25:47.513528    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:25:48.124403 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:25:48.124373    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:25:49.564292 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:25:49.564255    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:25:54.508784 ip-10-0-143-156 kubenswrapper[4461]: E1030 03:25:54.508742    4461 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2k85b" containerName="extractor"
Oct 30 03:25:54.603984 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:25:54.603953    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:25:55.872796 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:25:55.872757    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:25:56.036639 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:25:56.036603    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:25:56.201007 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:25:56.200927    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:25:59.858346 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:25:59.858311    4461 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-qf5vt"]
Oct 30 03:25:59.858814 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:25:59.858642    4461 cpu_manager.go:414] "containerMap: removing container" podUID="70bd8cb1-e22a-4a38-8688-2566a80e2cdb" containerName="main"
Oct 30 03:25:59.858814 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:25:59.858653    4461 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="70bd8cb1-e22a-4a38-8688-2566a80e2cdb" containerName="main"
Oct 30 03:25:59.858814 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:25:59.858718    4461 memory_manager.go:365] "RemoveStaleState removing state" podUID="70bd8cb1-e22a-4a38-8688-2566a80e2cdb" containerName="main"
Oct 30 03:25:59.861665 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:25:59.861641    4461 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-qf5vt"
Oct 30 03:25:59.884730 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:25:59.884705    4461 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-qf5vt"]
Oct 30 03:25:59.930838 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:25:59.930806    4461 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/68c83adb-d031-4a39-93b6-925bed75adc4-workload-socket\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-qf5vt\" (UID: \"68c83adb-d031-4a39-93b6-925bed75adc4\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-qf5vt"
Oct 30 03:25:59.930838 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:25:59.930838    4461 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/68c83adb-d031-4a39-93b6-925bed75adc4-credential-socket\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-qf5vt\" (UID: \"68c83adb-d031-4a39-93b6-925bed75adc4\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-qf5vt"
Oct 30 03:25:59.931058 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:25:59.930857    4461 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/68c83adb-d031-4a39-93b6-925bed75adc4-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-qf5vt\" (UID: \"68c83adb-d031-4a39-93b6-925bed75adc4\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-qf5vt"
Oct 30 03:25:59.931058 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:25:59.930948    4461 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/68c83adb-d031-4a39-93b6-925bed75adc4-workload-certs\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-qf5vt\" (UID: \"68c83adb-d031-4a39-93b6-925bed75adc4\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-qf5vt"
Oct 30 03:25:59.931058 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:25:59.930973    4461 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/68c83adb-d031-4a39-93b6-925bed75adc4-istio-envoy\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-qf5vt\" (UID: \"68c83adb-d031-4a39-93b6-925bed75adc4\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-qf5vt"
Oct 30 03:25:59.931058 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:25:59.931034    4461 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/68c83adb-d031-4a39-93b6-925bed75adc4-istio-token\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-qf5vt\" (UID: \"68c83adb-d031-4a39-93b6-925bed75adc4\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-qf5vt"
Oct 30 03:25:59.931058 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:25:59.931056    4461 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/68c83adb-d031-4a39-93b6-925bed75adc4-istio-data\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-qf5vt\" (UID: \"68c83adb-d031-4a39-93b6-925bed75adc4\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-qf5vt"
Oct 30 03:25:59.931275 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:25:59.931086    4461 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f484p\" (UniqueName: \"kubernetes.io/projected/68c83adb-d031-4a39-93b6-925bed75adc4-kube-api-access-f484p\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-qf5vt\" (UID: \"68c83adb-d031-4a39-93b6-925bed75adc4\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-qf5vt"
Oct 30 03:25:59.931275 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:25:59.931185    4461 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/68c83adb-d031-4a39-93b6-925bed75adc4-istio-podinfo\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-qf5vt\" (UID: \"68c83adb-d031-4a39-93b6-925bed75adc4\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-qf5vt"
Oct 30 03:26:00.031529 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:26:00.031502    4461 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f484p\" (UniqueName: \"kubernetes.io/projected/68c83adb-d031-4a39-93b6-925bed75adc4-kube-api-access-f484p\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-qf5vt\" (UID: \"68c83adb-d031-4a39-93b6-925bed75adc4\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-qf5vt"
Oct 30 03:26:00.031665 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:26:00.031537    4461 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/68c83adb-d031-4a39-93b6-925bed75adc4-istio-podinfo\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-qf5vt\" (UID: \"68c83adb-d031-4a39-93b6-925bed75adc4\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-qf5vt"
Oct 30 03:26:00.031738 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:26:00.031666    4461 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/68c83adb-d031-4a39-93b6-925bed75adc4-workload-socket\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-qf5vt\" (UID: \"68c83adb-d031-4a39-93b6-925bed75adc4\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-qf5vt"
Oct 30 03:26:00.031738 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:26:00.031710    4461 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/68c83adb-d031-4a39-93b6-925bed75adc4-credential-socket\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-qf5vt\" (UID: \"68c83adb-d031-4a39-93b6-925bed75adc4\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-qf5vt"
Oct 30 03:26:00.031844 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:26:00.031738    4461 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/68c83adb-d031-4a39-93b6-925bed75adc4-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-qf5vt\" (UID: \"68c83adb-d031-4a39-93b6-925bed75adc4\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-qf5vt"
Oct 30 03:26:00.031844 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:26:00.031791    4461 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/68c83adb-d031-4a39-93b6-925bed75adc4-workload-certs\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-qf5vt\" (UID: \"68c83adb-d031-4a39-93b6-925bed75adc4\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-qf5vt"
Oct 30 03:26:00.031952 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:26:00.031921    4461 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/68c83adb-d031-4a39-93b6-925bed75adc4-istio-envoy\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-qf5vt\" (UID: \"68c83adb-d031-4a39-93b6-925bed75adc4\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-qf5vt"
Oct 30 03:26:00.032015 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:26:00.031962    4461 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/68c83adb-d031-4a39-93b6-925bed75adc4-istio-token\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-qf5vt\" (UID: \"68c83adb-d031-4a39-93b6-925bed75adc4\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-qf5vt"
Oct 30 03:26:00.032015 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:26:00.031989    4461 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/68c83adb-d031-4a39-93b6-925bed75adc4-istio-data\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-qf5vt\" (UID: \"68c83adb-d031-4a39-93b6-925bed75adc4\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-qf5vt"
Oct 30 03:26:00.032140 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:26:00.032114    4461 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/68c83adb-d031-4a39-93b6-925bed75adc4-workload-socket\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-qf5vt\" (UID: \"68c83adb-d031-4a39-93b6-925bed75adc4\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-qf5vt"
Oct 30 03:26:00.032202 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:26:00.032158    4461 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/68c83adb-d031-4a39-93b6-925bed75adc4-workload-certs\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-qf5vt\" (UID: \"68c83adb-d031-4a39-93b6-925bed75adc4\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-qf5vt"
Oct 30 03:26:00.032414 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:26:00.032393    4461 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/68c83adb-d031-4a39-93b6-925bed75adc4-credential-socket\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-qf5vt\" (UID: \"68c83adb-d031-4a39-93b6-925bed75adc4\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-qf5vt"
Oct 30 03:26:00.032474 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:26:00.032453    4461 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/68c83adb-d031-4a39-93b6-925bed75adc4-istio-data\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-qf5vt\" (UID: \"68c83adb-d031-4a39-93b6-925bed75adc4\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-qf5vt"
Oct 30 03:26:00.032743 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:26:00.032722    4461 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/68c83adb-d031-4a39-93b6-925bed75adc4-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-qf5vt\" (UID: \"68c83adb-d031-4a39-93b6-925bed75adc4\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-qf5vt"
Oct 30 03:26:00.034122 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:26:00.034083    4461 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/68c83adb-d031-4a39-93b6-925bed75adc4-istio-envoy\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-qf5vt\" (UID: \"68c83adb-d031-4a39-93b6-925bed75adc4\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-qf5vt"
Oct 30 03:26:00.034208 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:26:00.034164    4461 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/68c83adb-d031-4a39-93b6-925bed75adc4-istio-podinfo\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-qf5vt\" (UID: \"68c83adb-d031-4a39-93b6-925bed75adc4\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-qf5vt"
Oct 30 03:26:00.063425 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:26:00.063386    4461 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f484p\" (UniqueName: \"kubernetes.io/projected/68c83adb-d031-4a39-93b6-925bed75adc4-kube-api-access-f484p\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-qf5vt\" (UID: \"68c83adb-d031-4a39-93b6-925bed75adc4\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-qf5vt"
Oct 30 03:26:00.063566 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:26:00.063547    4461 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/68c83adb-d031-4a39-93b6-925bed75adc4-istio-token\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-qf5vt\" (UID: \"68c83adb-d031-4a39-93b6-925bed75adc4\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-qf5vt"
Oct 30 03:26:00.173906 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:26:00.173822    4461 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-qf5vt"
Oct 30 03:26:00.491533 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:26:00.491509    4461 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-qf5vt"]
Oct 30 03:26:00.493181 ip-10-0-143-156 kubenswrapper[4461]: W1030 03:26:00.493152    4461 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod68c83adb_d031_4a39_93b6_925bed75adc4.slice/crio-24d83a76202f9cd59f2cf77f93f16fbd6be7be704cc1182bef35fe65e3390e7c WatchSource:0}: Error finding container 24d83a76202f9cd59f2cf77f93f16fbd6be7be704cc1182bef35fe65e3390e7c: Status 404 returned error can't find the container with id 24d83a76202f9cd59f2cf77f93f16fbd6be7be704cc1182bef35fe65e3390e7c
Oct 30 03:26:00.495260 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:26:00.495227    4461 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 03:26:00.495348 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:26:00.495292    4461 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 03:26:00.495348 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:26:00.495324    4461 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 03:26:00.615799 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:26:00.615759    4461 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-qf5vt" event={"ID":"68c83adb-d031-4a39-93b6-925bed75adc4","Type":"ContainerStarted","Data":"ef0f9a0ab729567ac2f5bac6fb6ebde1165a67a035f1b83d3ec20abc5d87080a"}
Oct 30 03:26:00.615959 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:26:00.615809    4461 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-qf5vt" event={"ID":"68c83adb-d031-4a39-93b6-925bed75adc4","Type":"ContainerStarted","Data":"24d83a76202f9cd59f2cf77f93f16fbd6be7be704cc1182bef35fe65e3390e7c"}
Oct 30 03:26:00.662596 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:26:00.662547    4461 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-qf5vt" podStartSLOduration=1.662533858 podStartE2EDuration="1.662533858s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:25:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:26:00.66217509 +0000 UTC m=+1828.711980296" watchObservedRunningTime="2026-10-30 03:26:00.662533858 +0000 UTC m=+1828.712339034"
Oct 30 03:26:01.184033 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:26:01.183992    4461 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-qf5vt"
Oct 30 03:26:01.184767 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:26:01.184740    4461 patch_prober.go:29] interesting pod/openshift-ai-inference-openshift-default-5b4f8cc59b-qf5vt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.27:15021/healthz/ready\": dial tcp 10.133.0.27:15021: connect: connection refused" start-of-body=
Oct 30 03:26:01.184926 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:26:01.184900    4461 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-qf5vt" podUID="68c83adb-d031-4a39-93b6-925bed75adc4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.27:15021/healthz/ready\": dial tcp 10.133.0.27:15021: connect: connection refused"
Oct 30 03:26:02.180558 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:26:02.180521    4461 patch_prober.go:29] interesting pod/openshift-ai-inference-openshift-default-5b4f8cc59b-qf5vt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.27:15021/healthz/ready\": dial tcp 10.133.0.27:15021: connect: connection refused" start-of-body=
Oct 30 03:26:02.180845 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:26:02.180812    4461 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-qf5vt" podUID="68c83adb-d031-4a39-93b6-925bed75adc4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.27:15021/healthz/ready\": dial tcp 10.133.0.27:15021: connect: connection refused"
Oct 30 03:26:03.179592 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:26:03.179544    4461 patch_prober.go:29] interesting pod/openshift-ai-inference-openshift-default-5b4f8cc59b-qf5vt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.27:15021/healthz/ready\": dial tcp 10.133.0.27:15021: connect: connection refused" start-of-body=
Oct 30 03:26:03.180231 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:26:03.180201    4461 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-qf5vt" podUID="68c83adb-d031-4a39-93b6-925bed75adc4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.27:15021/healthz/ready\": dial tcp 10.133.0.27:15021: connect: connection refused"
Oct 30 03:26:04.182119 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:26:04.181568    4461 patch_prober.go:29] interesting pod/openshift-ai-inference-openshift-default-5b4f8cc59b-qf5vt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.27:15021/healthz/ready\": dial tcp 10.133.0.27:15021: connect: connection refused" start-of-body=
Oct 30 03:26:04.182119 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:26:04.181650    4461 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-qf5vt" podUID="68c83adb-d031-4a39-93b6-925bed75adc4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.27:15021/healthz/ready\": dial tcp 10.133.0.27:15021: connect: connection refused"
Oct 30 03:26:05.179602 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:26:05.179550    4461 patch_prober.go:29] interesting pod/openshift-ai-inference-openshift-default-5b4f8cc59b-qf5vt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.27:15021/healthz/ready\": dial tcp 10.133.0.27:15021: connect: connection refused" start-of-body=
Oct 30 03:26:05.179894 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:26:05.179863    4461 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-qf5vt" podUID="68c83adb-d031-4a39-93b6-925bed75adc4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.27:15021/healthz/ready\": dial tcp 10.133.0.27:15021: connect: connection refused"
Oct 30 03:26:06.182121 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:26:06.180553    4461 patch_prober.go:29] interesting pod/openshift-ai-inference-openshift-default-5b4f8cc59b-qf5vt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.27:15021/healthz/ready\": dial tcp 10.133.0.27:15021: connect: connection refused" start-of-body=
Oct 30 03:26:06.182121 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:26:06.180617    4461 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-qf5vt" podUID="68c83adb-d031-4a39-93b6-925bed75adc4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.27:15021/healthz/ready\": dial tcp 10.133.0.27:15021: connect: connection refused"
Oct 30 03:26:07.178757 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:26:07.178720    4461 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-qf5vt"
Oct 30 03:26:07.179113 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:26:07.179078    4461 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-qf5vt"
Oct 30 03:26:07.179820 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:26:07.179803    4461 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-qf5vt"
Oct 30 03:26:24.502850 ip-10-0-143-156 kubenswrapper[4461]: E1030 03:26:24.502773    4461 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2k85b" containerName="extractor"
Oct 30 03:26:24.849283 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:26:24.849205    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:26:29.730741 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:26:29.730708    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:26:34.837799 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:26:34.837767    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:26:54.504022 ip-10-0-143-156 kubenswrapper[4461]: E1030 03:26:54.503985    4461 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2k85b" containerName="extractor"
Oct 30 03:27:24.504392 ip-10-0-143-156 kubenswrapper[4461]: E1030 03:27:24.504353    4461 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2k85b" containerName="extractor"
Oct 30 03:27:34.965828 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:27:34.965791    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:27:44.956615 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:27:44.956543    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:27:45.716300 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:27:45.716256    4461 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-8468d8f4d9-5q7xd"]
Oct 30 03:27:45.720343 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:27:45.720321    4461 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-8468d8f4d9-5q7xd"
Oct 30 03:27:45.748638 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:27:45.748614    4461 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-9c386aa9\"/\"scheduler-v06-pd-migration-test-kserve-dockercfg-4tq4s\""
Oct 30 03:27:45.748638 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:27:45.748613    4461 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-9c386aa9\"/\"default-dockercfg-gsg7j\""
Oct 30 03:27:45.748817 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:27:45.748672    4461 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-9c386aa9\"/\"scheduler-v06-pd-migration-test-kserve-self-signed-certs\""
Oct 30 03:27:45.748817 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:27:45.748672    4461 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-9c386aa9\"/\"openshift-service-ca.crt\""
Oct 30 03:27:45.748817 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:27:45.748723    4461 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-9c386aa9\"/\"kube-root-ca.crt\""
Oct 30 03:27:45.759560 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:27:45.759532    4461 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/18bcde21-fc86-49f7-957f-463d8549c37a-home\") pod \"scheduler-v06-pd-migration-test-kserve-8468d8f4d9-5q7xd\" (UID: \"18bcde21-fc86-49f7-957f-463d8549c37a\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-8468d8f4d9-5q7xd"
Oct 30 03:27:45.759693 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:27:45.759573    4461 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/18bcde21-fc86-49f7-957f-463d8549c37a-model-cache\") pod \"scheduler-v06-pd-migration-test-kserve-8468d8f4d9-5q7xd\" (UID: \"18bcde21-fc86-49f7-957f-463d8549c37a\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-8468d8f4d9-5q7xd"
Oct 30 03:27:45.759693 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:27:45.759607    4461 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-8468d8f4d9-5q7xd"]
Oct 30 03:27:45.759765 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:27:45.759725    4461 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/18bcde21-fc86-49f7-957f-463d8549c37a-tmp-dir\") pod \"scheduler-v06-pd-migration-test-kserve-8468d8f4d9-5q7xd\" (UID: \"18bcde21-fc86-49f7-957f-463d8549c37a\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-8468d8f4d9-5q7xd"
Oct 30 03:27:45.759800 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:27:45.759772    4461 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/18bcde21-fc86-49f7-957f-463d8549c37a-tls-certs\") pod \"scheduler-v06-pd-migration-test-kserve-8468d8f4d9-5q7xd\" (UID: \"18bcde21-fc86-49f7-957f-463d8549c37a\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-8468d8f4d9-5q7xd"
Oct 30 03:27:45.759800 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:27:45.759793    4461 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9v7jl\" (UniqueName: \"kubernetes.io/projected/18bcde21-fc86-49f7-957f-463d8549c37a-kube-api-access-9v7jl\") pod \"scheduler-v06-pd-migration-test-kserve-8468d8f4d9-5q7xd\" (UID: \"18bcde21-fc86-49f7-957f-463d8549c37a\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-8468d8f4d9-5q7xd"
Oct 30 03:27:45.759875 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:27:45.759851    4461 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/18bcde21-fc86-49f7-957f-463d8549c37a-dshm\") pod \"scheduler-v06-pd-migration-test-kserve-8468d8f4d9-5q7xd\" (UID: \"18bcde21-fc86-49f7-957f-463d8549c37a\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-8468d8f4d9-5q7xd"
Oct 30 03:27:45.840262 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:27:45.840225    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:27:45.860359 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:27:45.860332    4461 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/18bcde21-fc86-49f7-957f-463d8549c37a-home\") pod \"scheduler-v06-pd-migration-test-kserve-8468d8f4d9-5q7xd\" (UID: \"18bcde21-fc86-49f7-957f-463d8549c37a\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-8468d8f4d9-5q7xd"
Oct 30 03:27:45.860492 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:27:45.860364    4461 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/18bcde21-fc86-49f7-957f-463d8549c37a-model-cache\") pod \"scheduler-v06-pd-migration-test-kserve-8468d8f4d9-5q7xd\" (UID: \"18bcde21-fc86-49f7-957f-463d8549c37a\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-8468d8f4d9-5q7xd"
Oct 30 03:27:45.860492 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:27:45.860413    4461 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/18bcde21-fc86-49f7-957f-463d8549c37a-tmp-dir\") pod \"scheduler-v06-pd-migration-test-kserve-8468d8f4d9-5q7xd\" (UID: \"18bcde21-fc86-49f7-957f-463d8549c37a\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-8468d8f4d9-5q7xd"
Oct 30 03:27:45.860492 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:27:45.860441    4461 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/18bcde21-fc86-49f7-957f-463d8549c37a-tls-certs\") pod \"scheduler-v06-pd-migration-test-kserve-8468d8f4d9-5q7xd\" (UID: \"18bcde21-fc86-49f7-957f-463d8549c37a\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-8468d8f4d9-5q7xd"
Oct 30 03:27:45.860492 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:27:45.860466    4461 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9v7jl\" (UniqueName: \"kubernetes.io/projected/18bcde21-fc86-49f7-957f-463d8549c37a-kube-api-access-9v7jl\") pod \"scheduler-v06-pd-migration-test-kserve-8468d8f4d9-5q7xd\" (UID: \"18bcde21-fc86-49f7-957f-463d8549c37a\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-8468d8f4d9-5q7xd"
Oct 30 03:27:45.860700 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:27:45.860535    4461 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/18bcde21-fc86-49f7-957f-463d8549c37a-dshm\") pod \"scheduler-v06-pd-migration-test-kserve-8468d8f4d9-5q7xd\" (UID: \"18bcde21-fc86-49f7-957f-463d8549c37a\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-8468d8f4d9-5q7xd"
Oct 30 03:27:45.860820 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:27:45.860801    4461 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/18bcde21-fc86-49f7-957f-463d8549c37a-home\") pod \"scheduler-v06-pd-migration-test-kserve-8468d8f4d9-5q7xd\" (UID: \"18bcde21-fc86-49f7-957f-463d8549c37a\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-8468d8f4d9-5q7xd"
Oct 30 03:27:45.860906 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:27:45.860887    4461 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/18bcde21-fc86-49f7-957f-463d8549c37a-model-cache\") pod \"scheduler-v06-pd-migration-test-kserve-8468d8f4d9-5q7xd\" (UID: \"18bcde21-fc86-49f7-957f-463d8549c37a\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-8468d8f4d9-5q7xd"
Oct 30 03:27:45.860962 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:27:45.860917    4461 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/18bcde21-fc86-49f7-957f-463d8549c37a-tmp-dir\") pod \"scheduler-v06-pd-migration-test-kserve-8468d8f4d9-5q7xd\" (UID: \"18bcde21-fc86-49f7-957f-463d8549c37a\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-8468d8f4d9-5q7xd"
Oct 30 03:27:45.862823 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:27:45.862799    4461 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/18bcde21-fc86-49f7-957f-463d8549c37a-dshm\") pod \"scheduler-v06-pd-migration-test-kserve-8468d8f4d9-5q7xd\" (UID: \"18bcde21-fc86-49f7-957f-463d8549c37a\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-8468d8f4d9-5q7xd"
Oct 30 03:27:45.863002 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:27:45.862986    4461 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/18bcde21-fc86-49f7-957f-463d8549c37a-tls-certs\") pod \"scheduler-v06-pd-migration-test-kserve-8468d8f4d9-5q7xd\" (UID: \"18bcde21-fc86-49f7-957f-463d8549c37a\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-8468d8f4d9-5q7xd"
Oct 30 03:27:45.884171 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:27:45.884143    4461 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9v7jl\" (UniqueName: \"kubernetes.io/projected/18bcde21-fc86-49f7-957f-463d8549c37a-kube-api-access-9v7jl\") pod \"scheduler-v06-pd-migration-test-kserve-8468d8f4d9-5q7xd\" (UID: \"18bcde21-fc86-49f7-957f-463d8549c37a\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-8468d8f4d9-5q7xd"
Oct 30 03:27:46.034800 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:27:46.034716    4461 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-8468d8f4d9-5q7xd"
Oct 30 03:27:46.217312 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:27:46.217277    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:27:46.357901 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:27:46.357823    4461 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-8468d8f4d9-5q7xd"]
Oct 30 03:27:46.360736 ip-10-0-143-156 kubenswrapper[4461]: W1030 03:27:46.360710    4461 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod18bcde21_fc86_49f7_957f_463d8549c37a.slice/crio-d887851712ab7ab40f23007d4cc7d0af23c8daea0cf34e99fd2f5a43fdc69918 WatchSource:0}: Error finding container d887851712ab7ab40f23007d4cc7d0af23c8daea0cf34e99fd2f5a43fdc69918: Status 404 returned error can't find the container with id d887851712ab7ab40f23007d4cc7d0af23c8daea0cf34e99fd2f5a43fdc69918
Oct 30 03:27:46.362663 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:27:46.362645    4461 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 03:27:47.028053 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:27:47.028011    4461 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-8468d8f4d9-5q7xd" event={"ID":"18bcde21-fc86-49f7-957f-463d8549c37a","Type":"ContainerStarted","Data":"d887851712ab7ab40f23007d4cc7d0af23c8daea0cf34e99fd2f5a43fdc69918"}
Oct 30 03:27:49.056313 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:27:49.056275    4461 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-8468d8f4d9-5q7xd" event={"ID":"18bcde21-fc86-49f7-957f-463d8549c37a","Type":"ContainerStarted","Data":"e3f7ef758f04181c8cd928f249259816fbdf23322402763a8eb07cf8a5027bb7"}
Oct 30 03:27:49.056704 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:27:49.056350    4461 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-8468d8f4d9-5q7xd"
Oct 30 03:27:50.061751 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:27:50.061715    4461 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-8468d8f4d9-5q7xd" event={"ID":"18bcde21-fc86-49f7-957f-463d8549c37a","Type":"ContainerStarted","Data":"b3505dd7e35d79dacaba5f3235fc7991809718f3e4b1781fef5212da8a12212d"}
Oct 30 03:27:50.109223 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:27:50.109166    4461 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-8468d8f4d9-5q7xd" podStartSLOduration=3.583170431 podStartE2EDuration="5.109151853s" totalImagesPullingTime="1.525981422s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:27:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:27:50.108199613 +0000 UTC m=+1938.158004790" watchObservedRunningTime="2026-10-30 03:27:50.109151853 +0000 UTC m=+1938.158957028"
Oct 30 03:27:51.035342 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:27:51.035306    4461 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-8468d8f4d9-5q7xd"
Oct 30 03:27:53.907686 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:27:53.907656    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:27:54.504384 ip-10-0-143-156 kubenswrapper[4461]: E1030 03:27:54.504344    4461 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2k85b" containerName="extractor"
Oct 30 03:27:56.035085 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:27:56.034998    4461 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-8468d8f4d9-5q7xd"
Oct 30 03:27:56.047500 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:27:56.047473    4461 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-8468d8f4d9-5q7xd"
Oct 30 03:27:56.095339 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:27:56.095306    4461 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-8468d8f4d9-5q7xd"
Oct 30 03:28:07.101229 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:28:07.101193    4461 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-8468d8f4d9-5q7xd"
Oct 30 03:28:14.167756 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:28:14.167719    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:28:14.404763 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:28:14.404726    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:28:24.503422 ip-10-0-143-156 kubenswrapper[4461]: E1030 03:28:24.503382    4461 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2k85b" containerName="extractor"
Oct 30 03:28:34.859864 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:28:34.859825    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:28:45.071884 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:28:45.071846    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:28:54.504020 ip-10-0-143-156 kubenswrapper[4461]: E1030 03:28:54.503984    4461 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2k85b" containerName="extractor"
Oct 30 03:28:59.588743 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:28:59.588705    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:29:03.404781 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:29:03.404747    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:29:03.572766 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:29:03.572732    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:29:06.799237 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:29:06.799203    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:29:07.096743 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:29:07.096658    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:29:10.901298 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:29:10.901262    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:29:11.589496 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:29:11.589467    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:29:14.950354 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:29:14.950272    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:29:17.111557 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:29:17.111527    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:29:17.310539 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:29:17.310505    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:29:23.225114 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:29:23.225070    4461 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-8468d8f4d9-5q7xd"]
Oct 30 03:29:23.278498 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:29:23.278469    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:29:23.442381 ip-10-0-143-156 kubenswrapper[4461]: E1030 03:29:23.442342    4461 log.go:32] "ExecSync cmd from runtime service failed" err="rpc error: code = Unknown desc = command error: , stdout: , stderr: , exit code -1" containerID="b3505dd7e35d79dacaba5f3235fc7991809718f3e4b1781fef5212da8a12212d" cmd=["/bin/sleep","15"]
Oct 30 03:29:23.442534 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:29:23.442395    4461 handlers.go:85] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-8468d8f4d9-5q7xd" message="" err="rpc error: code = Unknown desc = command error: , stdout: , stderr: , exit code -1"
Oct 30 03:29:23.442534 ip-10-0-143-156 kubenswrapper[4461]: E1030 03:29:23.442408    4461 kuberuntime_container.go:780] "PreStop hook failed" err="rpc error: code = Unknown desc = command error: , stdout: , stderr: , exit code -1" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-8468d8f4d9-5q7xd" podUID="18bcde21-fc86-49f7-957f-463d8549c37a" containerName="main" containerID="cri-o://b3505dd7e35d79dacaba5f3235fc7991809718f3e4b1781fef5212da8a12212d"
Oct 30 03:29:23.442534 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:29:23.442475    4461 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-8468d8f4d9-5q7xd" podUID="18bcde21-fc86-49f7-957f-463d8549c37a" containerName="main" containerID="cri-o://b3505dd7e35d79dacaba5f3235fc7991809718f3e4b1781fef5212da8a12212d" gracePeriod=60
Oct 30 03:29:23.443482 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:29:23.443454    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:29:23.654309 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:29:23.654276    4461 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-8468d8f4d9-5q7xd" podUID="18bcde21-fc86-49f7-957f-463d8549c37a" containerName="llm-d-routing-sidecar" containerID="cri-o://e3f7ef758f04181c8cd928f249259816fbdf23322402763a8eb07cf8a5027bb7" gracePeriod=60
Oct 30 03:29:23.893228 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:29:23.893203    4461 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-8468d8f4d9-5q7xd"
Oct 30 03:29:23.952453 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:29:23.952420    4461 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/18bcde21-fc86-49f7-957f-463d8549c37a-tls-certs\" (UniqueName: \"kubernetes.io/secret/18bcde21-fc86-49f7-957f-463d8549c37a-tls-certs\") pod \"18bcde21-fc86-49f7-957f-463d8549c37a\" (UID: \"18bcde21-fc86-49f7-957f-463d8549c37a\") "
Oct 30 03:29:23.952620 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:29:23.952499    4461 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/18bcde21-fc86-49f7-957f-463d8549c37a-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/18bcde21-fc86-49f7-957f-463d8549c37a-model-cache\") pod \"18bcde21-fc86-49f7-957f-463d8549c37a\" (UID: \"18bcde21-fc86-49f7-957f-463d8549c37a\") "
Oct 30 03:29:23.952620 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:29:23.952598    4461 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/18bcde21-fc86-49f7-957f-463d8549c37a-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/18bcde21-fc86-49f7-957f-463d8549c37a-tmp-dir\") pod \"18bcde21-fc86-49f7-957f-463d8549c37a\" (UID: \"18bcde21-fc86-49f7-957f-463d8549c37a\") "
Oct 30 03:29:23.952724 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:29:23.952638    4461 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/18bcde21-fc86-49f7-957f-463d8549c37a-dshm\" (UniqueName: \"kubernetes.io/empty-dir/18bcde21-fc86-49f7-957f-463d8549c37a-dshm\") pod \"18bcde21-fc86-49f7-957f-463d8549c37a\" (UID: \"18bcde21-fc86-49f7-957f-463d8549c37a\") "
Oct 30 03:29:23.952724 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:29:23.952672    4461 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/18bcde21-fc86-49f7-957f-463d8549c37a-kube-api-access-9v7jl\" (UniqueName: \"kubernetes.io/projected/18bcde21-fc86-49f7-957f-463d8549c37a-kube-api-access-9v7jl\") pod \"18bcde21-fc86-49f7-957f-463d8549c37a\" (UID: \"18bcde21-fc86-49f7-957f-463d8549c37a\") "
Oct 30 03:29:23.952724 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:29:23.952723    4461 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/18bcde21-fc86-49f7-957f-463d8549c37a-home\" (UniqueName: \"kubernetes.io/empty-dir/18bcde21-fc86-49f7-957f-463d8549c37a-home\") pod \"18bcde21-fc86-49f7-957f-463d8549c37a\" (UID: \"18bcde21-fc86-49f7-957f-463d8549c37a\") "
Oct 30 03:29:23.952878 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:29:23.952734    4461 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/18bcde21-fc86-49f7-957f-463d8549c37a-model-cache" pod "18bcde21-fc86-49f7-957f-463d8549c37a" (UID: "18bcde21-fc86-49f7-957f-463d8549c37a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:29:23.953074 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:29:23.953043    4461 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/18bcde21-fc86-49f7-957f-463d8549c37a-model-cache\") on node \"ip-10-0-143-156.ec2.internal\" DevicePath \"\""
Oct 30 03:29:23.953231 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:29:23.953116    4461 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/18bcde21-fc86-49f7-957f-463d8549c37a-home" pod "18bcde21-fc86-49f7-957f-463d8549c37a" (UID: "18bcde21-fc86-49f7-957f-463d8549c37a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:29:23.953231 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:29:23.953185    4461 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/18bcde21-fc86-49f7-957f-463d8549c37a-tmp-dir" pod "18bcde21-fc86-49f7-957f-463d8549c37a" (UID: "18bcde21-fc86-49f7-957f-463d8549c37a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:29:23.954768 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:29:23.954746    4461 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/18bcde21-fc86-49f7-957f-463d8549c37a-dshm" pod "18bcde21-fc86-49f7-957f-463d8549c37a" (UID: "18bcde21-fc86-49f7-957f-463d8549c37a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:29:23.954905 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:29:23.954888    4461 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/18bcde21-fc86-49f7-957f-463d8549c37a-tls-certs" pod "18bcde21-fc86-49f7-957f-463d8549c37a" (UID: "18bcde21-fc86-49f7-957f-463d8549c37a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:29:23.954969 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:29:23.954931    4461 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/18bcde21-fc86-49f7-957f-463d8549c37a-kube-api-access-9v7jl" pod "18bcde21-fc86-49f7-957f-463d8549c37a" (UID: "18bcde21-fc86-49f7-957f-463d8549c37a"). InnerVolumeSpecName "kube-api-access-9v7jl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:29:24.054153 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:29:24.054113    4461 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/18bcde21-fc86-49f7-957f-463d8549c37a-home\") on node \"ip-10-0-143-156.ec2.internal\" DevicePath \"\""
Oct 30 03:29:24.054153 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:29:24.054147    4461 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/18bcde21-fc86-49f7-957f-463d8549c37a-tls-certs\") on node \"ip-10-0-143-156.ec2.internal\" DevicePath \"\""
Oct 30 03:29:24.054153 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:29:24.054157    4461 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/18bcde21-fc86-49f7-957f-463d8549c37a-tmp-dir\") on node \"ip-10-0-143-156.ec2.internal\" DevicePath \"\""
Oct 30 03:29:24.054153 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:29:24.054165    4461 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/18bcde21-fc86-49f7-957f-463d8549c37a-dshm\") on node \"ip-10-0-143-156.ec2.internal\" DevicePath \"\""
Oct 30 03:29:24.054497 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:29:24.054174    4461 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9v7jl\" (UniqueName: \"kubernetes.io/projected/18bcde21-fc86-49f7-957f-463d8549c37a-kube-api-access-9v7jl\") on node \"ip-10-0-143-156.ec2.internal\" DevicePath \"\""
Oct 30 03:29:24.439247 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:29:24.439165    4461 generic.go:356] "Generic (PLEG): container finished" podID="18bcde21-fc86-49f7-957f-463d8549c37a" containerID="b3505dd7e35d79dacaba5f3235fc7991809718f3e4b1781fef5212da8a12212d" exitCode=0
Oct 30 03:29:24.439247 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:29:24.439191    4461 generic.go:356] "Generic (PLEG): container finished" podID="18bcde21-fc86-49f7-957f-463d8549c37a" containerID="e3f7ef758f04181c8cd928f249259816fbdf23322402763a8eb07cf8a5027bb7" exitCode=0
Oct 30 03:29:24.439247 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:29:24.439220    4461 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-8468d8f4d9-5q7xd"
Oct 30 03:29:24.439737 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:29:24.439250    4461 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-8468d8f4d9-5q7xd" event={"ID":"18bcde21-fc86-49f7-957f-463d8549c37a","Type":"ContainerDied","Data":"b3505dd7e35d79dacaba5f3235fc7991809718f3e4b1781fef5212da8a12212d"}
Oct 30 03:29:24.439737 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:29:24.439311    4461 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-8468d8f4d9-5q7xd" event={"ID":"18bcde21-fc86-49f7-957f-463d8549c37a","Type":"ContainerDied","Data":"e3f7ef758f04181c8cd928f249259816fbdf23322402763a8eb07cf8a5027bb7"}
Oct 30 03:29:24.439737 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:29:24.439340    4461 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-8468d8f4d9-5q7xd" event={"ID":"18bcde21-fc86-49f7-957f-463d8549c37a","Type":"ContainerDied","Data":"d887851712ab7ab40f23007d4cc7d0af23c8daea0cf34e99fd2f5a43fdc69918"}
Oct 30 03:29:24.439737 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:29:24.439368    4461 scope.go:122] "RemoveContainer" containerID="b3505dd7e35d79dacaba5f3235fc7991809718f3e4b1781fef5212da8a12212d"
Oct 30 03:29:24.447861 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:29:24.447643    4461 scope.go:122] "RemoveContainer" containerID="e3f7ef758f04181c8cd928f249259816fbdf23322402763a8eb07cf8a5027bb7"
Oct 30 03:29:24.454824 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:29:24.454808    4461 scope.go:122] "RemoveContainer" containerID="b3505dd7e35d79dacaba5f3235fc7991809718f3e4b1781fef5212da8a12212d"
Oct 30 03:29:24.455117 ip-10-0-143-156 kubenswrapper[4461]: E1030 03:29:24.455084    4461 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b3505dd7e35d79dacaba5f3235fc7991809718f3e4b1781fef5212da8a12212d\": container with ID starting with b3505dd7e35d79dacaba5f3235fc7991809718f3e4b1781fef5212da8a12212d not found: ID does not exist" containerID="b3505dd7e35d79dacaba5f3235fc7991809718f3e4b1781fef5212da8a12212d"
Oct 30 03:29:24.455219 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:29:24.455124    4461 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b3505dd7e35d79dacaba5f3235fc7991809718f3e4b1781fef5212da8a12212d"} err="failed to get container status \"b3505dd7e35d79dacaba5f3235fc7991809718f3e4b1781fef5212da8a12212d\": rpc error: code = NotFound desc = could not find container \"b3505dd7e35d79dacaba5f3235fc7991809718f3e4b1781fef5212da8a12212d\": container with ID starting with b3505dd7e35d79dacaba5f3235fc7991809718f3e4b1781fef5212da8a12212d not found: ID does not exist"
Oct 30 03:29:24.455219 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:29:24.455150    4461 scope.go:122] "RemoveContainer" containerID="e3f7ef758f04181c8cd928f249259816fbdf23322402763a8eb07cf8a5027bb7"
Oct 30 03:29:24.455385 ip-10-0-143-156 kubenswrapper[4461]: E1030 03:29:24.455365    4461 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e3f7ef758f04181c8cd928f249259816fbdf23322402763a8eb07cf8a5027bb7\": container with ID starting with e3f7ef758f04181c8cd928f249259816fbdf23322402763a8eb07cf8a5027bb7 not found: ID does not exist" containerID="e3f7ef758f04181c8cd928f249259816fbdf23322402763a8eb07cf8a5027bb7"
Oct 30 03:29:24.455466 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:29:24.455396    4461 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e3f7ef758f04181c8cd928f249259816fbdf23322402763a8eb07cf8a5027bb7"} err="failed to get container status \"e3f7ef758f04181c8cd928f249259816fbdf23322402763a8eb07cf8a5027bb7\": rpc error: code = NotFound desc = could not find container \"e3f7ef758f04181c8cd928f249259816fbdf23322402763a8eb07cf8a5027bb7\": container with ID starting with e3f7ef758f04181c8cd928f249259816fbdf23322402763a8eb07cf8a5027bb7 not found: ID does not exist"
Oct 30 03:29:24.455466 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:29:24.455418    4461 scope.go:122] "RemoveContainer" containerID="b3505dd7e35d79dacaba5f3235fc7991809718f3e4b1781fef5212da8a12212d"
Oct 30 03:29:24.455724 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:29:24.455696    4461 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b3505dd7e35d79dacaba5f3235fc7991809718f3e4b1781fef5212da8a12212d"} err="failed to get container status \"b3505dd7e35d79dacaba5f3235fc7991809718f3e4b1781fef5212da8a12212d\": rpc error: code = NotFound desc = could not find container \"b3505dd7e35d79dacaba5f3235fc7991809718f3e4b1781fef5212da8a12212d\": container with ID starting with b3505dd7e35d79dacaba5f3235fc7991809718f3e4b1781fef5212da8a12212d not found: ID does not exist"
Oct 30 03:29:24.455770 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:29:24.455724    4461 scope.go:122] "RemoveContainer" containerID="e3f7ef758f04181c8cd928f249259816fbdf23322402763a8eb07cf8a5027bb7"
Oct 30 03:29:24.455951 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:29:24.455933    4461 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e3f7ef758f04181c8cd928f249259816fbdf23322402763a8eb07cf8a5027bb7"} err="failed to get container status \"e3f7ef758f04181c8cd928f249259816fbdf23322402763a8eb07cf8a5027bb7\": rpc error: code = NotFound desc = could not find container \"e3f7ef758f04181c8cd928f249259816fbdf23322402763a8eb07cf8a5027bb7\": container with ID starting with e3f7ef758f04181c8cd928f249259816fbdf23322402763a8eb07cf8a5027bb7 not found: ID does not exist"
Oct 30 03:29:24.491346 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:29:24.491318    4461 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-8468d8f4d9-5q7xd"]
Oct 30 03:29:24.503893 ip-10-0-143-156 kubenswrapper[4461]: E1030 03:29:24.503863    4461 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2k85b" containerName="extractor"
Oct 30 03:29:24.504138 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:29:24.504119    4461 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-8468d8f4d9-5q7xd"]
Oct 30 03:29:26.115325 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:29:26.115295    4461 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="18bcde21-fc86-49f7-957f-463d8549c37a" path="/var/lib/kubelet/pods/18bcde21-fc86-49f7-957f-463d8549c37a/volumes"
Oct 30 03:29:44.978185 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:29:44.978145    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:29:45.215410 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:29:45.215373    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:29:54.503883 ip-10-0-143-156 kubenswrapper[4461]: E1030 03:29:54.503844    4461 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2k85b" containerName="extractor"
Oct 30 03:30:24.504354 ip-10-0-143-156 kubenswrapper[4461]: E1030 03:30:24.504318    4461 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2k85b" containerName="extractor"
Oct 30 03:30:45.102644 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:30:45.102544    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:30:45.292610 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:30:45.292576    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:30:54.503618 ip-10-0-143-156 kubenswrapper[4461]: E1030 03:30:54.503575    4461 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2k85b" containerName="extractor"
Oct 30 03:30:55.097880 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:30:55.097843    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:31:09.339402 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:31:09.339369    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:31:09.562894 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:31:09.562861    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:31:09.939074 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:31:09.939038    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:31:10.119033 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:31:10.118998    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:31:15.704933 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:31:15.704889    4461 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-fhwng"]
Oct 30 03:31:15.705404 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:31:15.705136    4461 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-fhwng" podUID="9428d98d-2f15-4c68-81a6-59066080b401" containerName="istio-proxy" containerID="cri-o://f1e6e45c30abc4aac701f0518ca60bb2891777d2b7ce9ff7fed73c5203d17e9b" gracePeriod=30
Oct 30 03:31:15.717656 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:31:15.717632    4461 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-lp2z8"]
Oct 30 03:31:15.717847 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:31:15.717825    4461 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-lp2z8" podUID="f7056833-a12d-45d3-bf13-101db3dc1d24" containerName="istio-proxy" containerID="cri-o://8848879ad8abe8e8da97b7205cab605ce5fb4fce4a2de558d3e4fac58e68c2d4" gracePeriod=30
Oct 30 03:31:15.718579 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:31:15.718557    4461 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-qf5vt"]
Oct 30 03:31:15.718740 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:31:15.718722    4461 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-qf5vt" podUID="68c83adb-d031-4a39-93b6-925bed75adc4" containerName="istio-proxy" containerID="cri-o://ef0f9a0ab729567ac2f5bac6fb6ebde1165a67a035f1b83d3ec20abc5d87080a" gracePeriod=30
Oct 30 03:31:15.730285 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:31:15.730260    4461 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-7s92s"]
Oct 30 03:31:15.730437 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:31:15.730420    4461 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-7s92s" podUID="74841243-fe62-45e3-ac11-6480ca9913c5" containerName="istio-proxy" containerID="cri-o://d9108f9e40204e67f93220a50ed63e4e57a20b7abe2fda7f2c060f90aab129d0" gracePeriod=30
Oct 30 03:31:20.893374 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:31:20.893276    4461 generic.go:356] "Generic (PLEG): container finished" podID="74841243-fe62-45e3-ac11-6480ca9913c5" containerID="d9108f9e40204e67f93220a50ed63e4e57a20b7abe2fda7f2c060f90aab129d0" exitCode=0
Oct 30 03:31:20.893374 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:31:20.893349    4461 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-7s92s" event={"ID":"74841243-fe62-45e3-ac11-6480ca9913c5","Type":"ContainerDied","Data":"d9108f9e40204e67f93220a50ed63e4e57a20b7abe2fda7f2c060f90aab129d0"}
Oct 30 03:31:20.895680 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:31:20.895512    4461 generic.go:356] "Generic (PLEG): container finished" podID="f7056833-a12d-45d3-bf13-101db3dc1d24" containerID="8848879ad8abe8e8da97b7205cab605ce5fb4fce4a2de558d3e4fac58e68c2d4" exitCode=0
Oct 30 03:31:20.895680 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:31:20.895617    4461 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-lp2z8" event={"ID":"f7056833-a12d-45d3-bf13-101db3dc1d24","Type":"ContainerDied","Data":"8848879ad8abe8e8da97b7205cab605ce5fb4fce4a2de558d3e4fac58e68c2d4"}
Oct 30 03:31:20.897476 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:31:20.897354    4461 generic.go:356] "Generic (PLEG): container finished" podID="9428d98d-2f15-4c68-81a6-59066080b401" containerID="f1e6e45c30abc4aac701f0518ca60bb2891777d2b7ce9ff7fed73c5203d17e9b" exitCode=0
Oct 30 03:31:20.897476 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:31:20.897430    4461 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-fhwng" event={"ID":"9428d98d-2f15-4c68-81a6-59066080b401","Type":"ContainerDied","Data":"f1e6e45c30abc4aac701f0518ca60bb2891777d2b7ce9ff7fed73c5203d17e9b"}
Oct 30 03:31:20.901169 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:31:20.900376    4461 generic.go:356] "Generic (PLEG): container finished" podID="68c83adb-d031-4a39-93b6-925bed75adc4" containerID="ef0f9a0ab729567ac2f5bac6fb6ebde1165a67a035f1b83d3ec20abc5d87080a" exitCode=0
Oct 30 03:31:20.901169 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:31:20.900413    4461 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-qf5vt" event={"ID":"68c83adb-d031-4a39-93b6-925bed75adc4","Type":"ContainerDied","Data":"ef0f9a0ab729567ac2f5bac6fb6ebde1165a67a035f1b83d3ec20abc5d87080a"}
Oct 30 03:31:20.990308 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:31:20.990282    4461 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-qf5vt"
Oct 30 03:31:21.035772 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:31:21.035749    4461 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-7s92s"
Oct 30 03:31:21.057863 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:31:21.057836    4461 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/68c83adb-d031-4a39-93b6-925bed75adc4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/68c83adb-d031-4a39-93b6-925bed75adc4-istio-data\") pod \"68c83adb-d031-4a39-93b6-925bed75adc4\" (UID: \"68c83adb-d031-4a39-93b6-925bed75adc4\") "
Oct 30 03:31:21.057992 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:31:21.057884    4461 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/68c83adb-d031-4a39-93b6-925bed75adc4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/68c83adb-d031-4a39-93b6-925bed75adc4-istio-podinfo\") pod \"68c83adb-d031-4a39-93b6-925bed75adc4\" (UID: \"68c83adb-d031-4a39-93b6-925bed75adc4\") "
Oct 30 03:31:21.057992 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:31:21.057945    4461 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/68c83adb-d031-4a39-93b6-925bed75adc4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/68c83adb-d031-4a39-93b6-925bed75adc4-workload-socket\") pod \"68c83adb-d031-4a39-93b6-925bed75adc4\" (UID: \"68c83adb-d031-4a39-93b6-925bed75adc4\") "
Oct 30 03:31:21.057992 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:31:21.057978    4461 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/68c83adb-d031-4a39-93b6-925bed75adc4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/68c83adb-d031-4a39-93b6-925bed75adc4-istiod-ca-cert\") pod \"68c83adb-d031-4a39-93b6-925bed75adc4\" (UID: \"68c83adb-d031-4a39-93b6-925bed75adc4\") "
Oct 30 03:31:21.058193 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:31:21.058139    4461 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/68c83adb-d031-4a39-93b6-925bed75adc4-istio-data" pod "68c83adb-d031-4a39-93b6-925bed75adc4" (UID: "68c83adb-d031-4a39-93b6-925bed75adc4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:31:21.058250 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:31:21.058198    4461 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/68c83adb-d031-4a39-93b6-925bed75adc4-kube-api-access-f484p\" (UniqueName: \"kubernetes.io/projected/68c83adb-d031-4a39-93b6-925bed75adc4-kube-api-access-f484p\") pod \"68c83adb-d031-4a39-93b6-925bed75adc4\" (UID: \"68c83adb-d031-4a39-93b6-925bed75adc4\") "
Oct 30 03:31:21.058322 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:31:21.058303    4461 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/68c83adb-d031-4a39-93b6-925bed75adc4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/68c83adb-d031-4a39-93b6-925bed75adc4-istio-envoy\") pod \"68c83adb-d031-4a39-93b6-925bed75adc4\" (UID: \"68c83adb-d031-4a39-93b6-925bed75adc4\") "
Oct 30 03:31:21.058539 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:31:21.058501    4461 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/68c83adb-d031-4a39-93b6-925bed75adc4-istiod-ca-cert" pod "68c83adb-d031-4a39-93b6-925bed75adc4" (UID: "68c83adb-d031-4a39-93b6-925bed75adc4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:31:21.058724 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:31:21.058702    4461 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/68c83adb-d031-4a39-93b6-925bed75adc4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/68c83adb-d031-4a39-93b6-925bed75adc4-credential-socket\") pod \"68c83adb-d031-4a39-93b6-925bed75adc4\" (UID: \"68c83adb-d031-4a39-93b6-925bed75adc4\") "
Oct 30 03:31:21.058811 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:31:21.058783    4461 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/68c83adb-d031-4a39-93b6-925bed75adc4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/68c83adb-d031-4a39-93b6-925bed75adc4-workload-certs\") pod \"68c83adb-d031-4a39-93b6-925bed75adc4\" (UID: \"68c83adb-d031-4a39-93b6-925bed75adc4\") "
Oct 30 03:31:21.058866 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:31:21.058807    4461 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/68c83adb-d031-4a39-93b6-925bed75adc4-istio-token\" (UniqueName: \"kubernetes.io/projected/68c83adb-d031-4a39-93b6-925bed75adc4-istio-token\") pod \"68c83adb-d031-4a39-93b6-925bed75adc4\" (UID: \"68c83adb-d031-4a39-93b6-925bed75adc4\") "
Oct 30 03:31:21.058923 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:31:21.058878    4461 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/68c83adb-d031-4a39-93b6-925bed75adc4-credential-socket" pod "68c83adb-d031-4a39-93b6-925bed75adc4" (UID: "68c83adb-d031-4a39-93b6-925bed75adc4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:31:21.059366 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:31:21.058949    4461 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/68c83adb-d031-4a39-93b6-925bed75adc4-workload-socket" pod "68c83adb-d031-4a39-93b6-925bed75adc4" (UID: "68c83adb-d031-4a39-93b6-925bed75adc4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:31:21.059366 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:31:21.059028    4461 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/68c83adb-d031-4a39-93b6-925bed75adc4-workload-certs" pod "68c83adb-d031-4a39-93b6-925bed75adc4" (UID: "68c83adb-d031-4a39-93b6-925bed75adc4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:31:21.059366 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:31:21.059134    4461 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/68c83adb-d031-4a39-93b6-925bed75adc4-istio-data\") on node \"ip-10-0-143-156.ec2.internal\" DevicePath \"\""
Oct 30 03:31:21.059366 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:31:21.059303    4461 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/68c83adb-d031-4a39-93b6-925bed75adc4-istiod-ca-cert\") on node \"ip-10-0-143-156.ec2.internal\" DevicePath \"\""
Oct 30 03:31:21.059366 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:31:21.059323    4461 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/68c83adb-d031-4a39-93b6-925bed75adc4-credential-socket\") on node \"ip-10-0-143-156.ec2.internal\" DevicePath \"\""
Oct 30 03:31:21.061174 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:31:21.061144    4461 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/68c83adb-d031-4a39-93b6-925bed75adc4-kube-api-access-f484p" pod "68c83adb-d031-4a39-93b6-925bed75adc4" (UID: "68c83adb-d031-4a39-93b6-925bed75adc4"). InnerVolumeSpecName "kube-api-access-f484p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:31:21.061283 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:31:21.061267    4461 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/68c83adb-d031-4a39-93b6-925bed75adc4-istio-podinfo" pod "68c83adb-d031-4a39-93b6-925bed75adc4" (UID: "68c83adb-d031-4a39-93b6-925bed75adc4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 03:31:21.061358 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:31:21.061278    4461 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/68c83adb-d031-4a39-93b6-925bed75adc4-istio-envoy" pod "68c83adb-d031-4a39-93b6-925bed75adc4" (UID: "68c83adb-d031-4a39-93b6-925bed75adc4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:31:21.061358 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:31:21.061345    4461 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/68c83adb-d031-4a39-93b6-925bed75adc4-istio-token" pod "68c83adb-d031-4a39-93b6-925bed75adc4" (UID: "68c83adb-d031-4a39-93b6-925bed75adc4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:31:21.065363 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:31:21.065341    4461 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-fhwng"
Oct 30 03:31:21.067415 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:31:21.067397    4461 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-lp2z8"
Oct 30 03:31:21.159940 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:31:21.159908    4461 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9428d98d-2f15-4c68-81a6-59066080b401-istio-token\" (UniqueName: \"kubernetes.io/projected/9428d98d-2f15-4c68-81a6-59066080b401-istio-token\") pod \"9428d98d-2f15-4c68-81a6-59066080b401\" (UID: \"9428d98d-2f15-4c68-81a6-59066080b401\") "
Oct 30 03:31:21.159940 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:31:21.159942    4461 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/74841243-fe62-45e3-ac11-6480ca9913c5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/74841243-fe62-45e3-ac11-6480ca9913c5-workload-socket\") pod \"74841243-fe62-45e3-ac11-6480ca9913c5\" (UID: \"74841243-fe62-45e3-ac11-6480ca9913c5\") "
Oct 30 03:31:21.160217 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:31:21.159967    4461 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9428d98d-2f15-4c68-81a6-59066080b401-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9428d98d-2f15-4c68-81a6-59066080b401-istio-envoy\") pod \"9428d98d-2f15-4c68-81a6-59066080b401\" (UID: \"9428d98d-2f15-4c68-81a6-59066080b401\") "
Oct 30 03:31:21.160217 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:31:21.160014    4461 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f7056833-a12d-45d3-bf13-101db3dc1d24-istio-token\" (UniqueName: \"kubernetes.io/projected/f7056833-a12d-45d3-bf13-101db3dc1d24-istio-token\") pod \"f7056833-a12d-45d3-bf13-101db3dc1d24\" (UID: \"f7056833-a12d-45d3-bf13-101db3dc1d24\") "
Oct 30 03:31:21.160217 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:31:21.160031    4461 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f7056833-a12d-45d3-bf13-101db3dc1d24-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f7056833-a12d-45d3-bf13-101db3dc1d24-istio-envoy\") pod \"f7056833-a12d-45d3-bf13-101db3dc1d24\" (UID: \"f7056833-a12d-45d3-bf13-101db3dc1d24\") "
Oct 30 03:31:21.160217 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:31:21.160055    4461 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f7056833-a12d-45d3-bf13-101db3dc1d24-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f7056833-a12d-45d3-bf13-101db3dc1d24-istio-data\") pod \"f7056833-a12d-45d3-bf13-101db3dc1d24\" (UID: \"f7056833-a12d-45d3-bf13-101db3dc1d24\") "
Oct 30 03:31:21.160217 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:31:21.160077    4461 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9428d98d-2f15-4c68-81a6-59066080b401-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9428d98d-2f15-4c68-81a6-59066080b401-istio-data\") pod \"9428d98d-2f15-4c68-81a6-59066080b401\" (UID: \"9428d98d-2f15-4c68-81a6-59066080b401\") "
Oct 30 03:31:21.160217 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:31:21.160112    4461 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/74841243-fe62-45e3-ac11-6480ca9913c5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/74841243-fe62-45e3-ac11-6480ca9913c5-istiod-ca-cert\") pod \"74841243-fe62-45e3-ac11-6480ca9913c5\" (UID: \"74841243-fe62-45e3-ac11-6480ca9913c5\") "
Oct 30 03:31:21.160217 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:31:21.160150    4461 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f7056833-a12d-45d3-bf13-101db3dc1d24-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f7056833-a12d-45d3-bf13-101db3dc1d24-istiod-ca-cert\") pod \"f7056833-a12d-45d3-bf13-101db3dc1d24\" (UID: \"f7056833-a12d-45d3-bf13-101db3dc1d24\") "
Oct 30 03:31:21.160217 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:31:21.160181    4461 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/74841243-fe62-45e3-ac11-6480ca9913c5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/74841243-fe62-45e3-ac11-6480ca9913c5-workload-certs\") pod \"74841243-fe62-45e3-ac11-6480ca9913c5\" (UID: \"74841243-fe62-45e3-ac11-6480ca9913c5\") "
Oct 30 03:31:21.160668 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:31:21.160225    4461 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/74841243-fe62-45e3-ac11-6480ca9913c5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/74841243-fe62-45e3-ac11-6480ca9913c5-istio-data\") pod \"74841243-fe62-45e3-ac11-6480ca9913c5\" (UID: \"74841243-fe62-45e3-ac11-6480ca9913c5\") "
Oct 30 03:31:21.160668 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:31:21.160256    4461 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/74841243-fe62-45e3-ac11-6480ca9913c5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/74841243-fe62-45e3-ac11-6480ca9913c5-credential-socket\") pod \"74841243-fe62-45e3-ac11-6480ca9913c5\" (UID: \"74841243-fe62-45e3-ac11-6480ca9913c5\") "
Oct 30 03:31:21.160668 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:31:21.160274    4461 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/74841243-fe62-45e3-ac11-6480ca9913c5-workload-socket" pod "74841243-fe62-45e3-ac11-6480ca9913c5" (UID: "74841243-fe62-45e3-ac11-6480ca9913c5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:31:21.160668 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:31:21.160340    4461 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9428d98d-2f15-4c68-81a6-59066080b401-kube-api-access-fng8q\" (UniqueName: \"kubernetes.io/projected/9428d98d-2f15-4c68-81a6-59066080b401-kube-api-access-fng8q\") pod \"9428d98d-2f15-4c68-81a6-59066080b401\" (UID: \"9428d98d-2f15-4c68-81a6-59066080b401\") "
Oct 30 03:31:21.160668 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:31:21.160372    4461 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f7056833-a12d-45d3-bf13-101db3dc1d24-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7056833-a12d-45d3-bf13-101db3dc1d24-credential-socket\") pod \"f7056833-a12d-45d3-bf13-101db3dc1d24\" (UID: \"f7056833-a12d-45d3-bf13-101db3dc1d24\") "
Oct 30 03:31:21.160668 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:31:21.160400    4461 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f7056833-a12d-45d3-bf13-101db3dc1d24-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7056833-a12d-45d3-bf13-101db3dc1d24-workload-socket\") pod \"f7056833-a12d-45d3-bf13-101db3dc1d24\" (UID: \"f7056833-a12d-45d3-bf13-101db3dc1d24\") "
Oct 30 03:31:21.160668 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:31:21.160427    4461 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f7056833-a12d-45d3-bf13-101db3dc1d24-kube-api-access-v9cdr\" (UniqueName: \"kubernetes.io/projected/f7056833-a12d-45d3-bf13-101db3dc1d24-kube-api-access-v9cdr\") pod \"f7056833-a12d-45d3-bf13-101db3dc1d24\" (UID: \"f7056833-a12d-45d3-bf13-101db3dc1d24\") "
Oct 30 03:31:21.160668 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:31:21.160467    4461 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9428d98d-2f15-4c68-81a6-59066080b401-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9428d98d-2f15-4c68-81a6-59066080b401-istio-podinfo\") pod \"9428d98d-2f15-4c68-81a6-59066080b401\" (UID: \"9428d98d-2f15-4c68-81a6-59066080b401\") "
Oct 30 03:31:21.160668 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:31:21.160515    4461 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9428d98d-2f15-4c68-81a6-59066080b401-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9428d98d-2f15-4c68-81a6-59066080b401-workload-certs\") pod \"9428d98d-2f15-4c68-81a6-59066080b401\" (UID: \"9428d98d-2f15-4c68-81a6-59066080b401\") "
Oct 30 03:31:21.160668 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:31:21.160546    4461 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/74841243-fe62-45e3-ac11-6480ca9913c5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/74841243-fe62-45e3-ac11-6480ca9913c5-istio-podinfo\") pod \"74841243-fe62-45e3-ac11-6480ca9913c5\" (UID: \"74841243-fe62-45e3-ac11-6480ca9913c5\") "
Oct 30 03:31:21.160668 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:31:21.160611    4461 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9428d98d-2f15-4c68-81a6-59066080b401-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9428d98d-2f15-4c68-81a6-59066080b401-istiod-ca-cert\") pod \"9428d98d-2f15-4c68-81a6-59066080b401\" (UID: \"9428d98d-2f15-4c68-81a6-59066080b401\") "
Oct 30 03:31:21.160668 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:31:21.160617    4461 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f7056833-a12d-45d3-bf13-101db3dc1d24-istio-data" pod "f7056833-a12d-45d3-bf13-101db3dc1d24" (UID: "f7056833-a12d-45d3-bf13-101db3dc1d24"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:31:21.160668 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:31:21.160641    4461 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/74841243-fe62-45e3-ac11-6480ca9913c5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/74841243-fe62-45e3-ac11-6480ca9913c5-istio-envoy\") pod \"74841243-fe62-45e3-ac11-6480ca9913c5\" (UID: \"74841243-fe62-45e3-ac11-6480ca9913c5\") "
Oct 30 03:31:21.160668 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:31:21.160668    4461 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9428d98d-2f15-4c68-81a6-59066080b401-istio-data" pod "9428d98d-2f15-4c68-81a6-59066080b401" (UID: "9428d98d-2f15-4c68-81a6-59066080b401"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:31:21.161753 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:31:21.160672    4461 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9428d98d-2f15-4c68-81a6-59066080b401-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9428d98d-2f15-4c68-81a6-59066080b401-credential-socket\") pod \"9428d98d-2f15-4c68-81a6-59066080b401\" (UID: \"9428d98d-2f15-4c68-81a6-59066080b401\") "
Oct 30 03:31:21.161753 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:31:21.160699    4461 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/74841243-fe62-45e3-ac11-6480ca9913c5-istiod-ca-cert" pod "74841243-fe62-45e3-ac11-6480ca9913c5" (UID: "74841243-fe62-45e3-ac11-6480ca9913c5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:31:21.161753 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:31:21.160723    4461 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/74841243-fe62-45e3-ac11-6480ca9913c5-kube-api-access-mtkvl\" (UniqueName: \"kubernetes.io/projected/74841243-fe62-45e3-ac11-6480ca9913c5-kube-api-access-mtkvl\") pod \"74841243-fe62-45e3-ac11-6480ca9913c5\" (UID: \"74841243-fe62-45e3-ac11-6480ca9913c5\") "
Oct 30 03:31:21.161753 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:31:21.160805    4461 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f7056833-a12d-45d3-bf13-101db3dc1d24-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f7056833-a12d-45d3-bf13-101db3dc1d24-workload-certs\") pod \"f7056833-a12d-45d3-bf13-101db3dc1d24\" (UID: \"f7056833-a12d-45d3-bf13-101db3dc1d24\") "
Oct 30 03:31:21.161753 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:31:21.160836    4461 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f7056833-a12d-45d3-bf13-101db3dc1d24-credential-socket" pod "f7056833-a12d-45d3-bf13-101db3dc1d24" (UID: "f7056833-a12d-45d3-bf13-101db3dc1d24"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:31:21.161753 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:31:21.160868    4461 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9428d98d-2f15-4c68-81a6-59066080b401-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9428d98d-2f15-4c68-81a6-59066080b401-workload-socket\") pod \"9428d98d-2f15-4c68-81a6-59066080b401\" (UID: \"9428d98d-2f15-4c68-81a6-59066080b401\") "
Oct 30 03:31:21.161753 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:31:21.160896    4461 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/74841243-fe62-45e3-ac11-6480ca9913c5-istio-token\" (UniqueName: \"kubernetes.io/projected/74841243-fe62-45e3-ac11-6480ca9913c5-istio-token\") pod \"74841243-fe62-45e3-ac11-6480ca9913c5\" (UID: \"74841243-fe62-45e3-ac11-6480ca9913c5\") "
Oct 30 03:31:21.161753 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:31:21.160913    4461 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9428d98d-2f15-4c68-81a6-59066080b401-credential-socket" pod "9428d98d-2f15-4c68-81a6-59066080b401" (UID: "9428d98d-2f15-4c68-81a6-59066080b401"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:31:21.161753 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:31:21.160927    4461 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f7056833-a12d-45d3-bf13-101db3dc1d24-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f7056833-a12d-45d3-bf13-101db3dc1d24-istio-podinfo\") pod \"f7056833-a12d-45d3-bf13-101db3dc1d24\" (UID: \"f7056833-a12d-45d3-bf13-101db3dc1d24\") "
Oct 30 03:31:21.161753 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:31:21.160950    4461 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9428d98d-2f15-4c68-81a6-59066080b401-workload-certs" pod "9428d98d-2f15-4c68-81a6-59066080b401" (UID: "9428d98d-2f15-4c68-81a6-59066080b401"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:31:21.161753 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:31:21.161231    4461 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/74841243-fe62-45e3-ac11-6480ca9913c5-workload-socket\") on node \"ip-10-0-143-156.ec2.internal\" DevicePath \"\""
Oct 30 03:31:21.161753 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:31:21.161257    4461 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f7056833-a12d-45d3-bf13-101db3dc1d24-istio-data\") on node \"ip-10-0-143-156.ec2.internal\" DevicePath \"\""
Oct 30 03:31:21.161753 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:31:21.161270    4461 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9428d98d-2f15-4c68-81a6-59066080b401-istio-data\") on node \"ip-10-0-143-156.ec2.internal\" DevicePath \"\""
Oct 30 03:31:21.161753 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:31:21.161283    4461 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/74841243-fe62-45e3-ac11-6480ca9913c5-istiod-ca-cert\") on node \"ip-10-0-143-156.ec2.internal\" DevicePath \"\""
Oct 30 03:31:21.161753 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:31:21.161295    4461 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/68c83adb-d031-4a39-93b6-925bed75adc4-workload-certs\") on node \"ip-10-0-143-156.ec2.internal\" DevicePath \"\""
Oct 30 03:31:21.161753 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:31:21.161307    4461 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/68c83adb-d031-4a39-93b6-925bed75adc4-istio-token\") on node \"ip-10-0-143-156.ec2.internal\" DevicePath \"\""
Oct 30 03:31:21.161753 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:31:21.161319    4461 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/68c83adb-d031-4a39-93b6-925bed75adc4-istio-podinfo\") on node \"ip-10-0-143-156.ec2.internal\" DevicePath \"\""
Oct 30 03:31:21.161753 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:31:21.161297    4461 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f7056833-a12d-45d3-bf13-101db3dc1d24-workload-certs" pod "f7056833-a12d-45d3-bf13-101db3dc1d24" (UID: "f7056833-a12d-45d3-bf13-101db3dc1d24"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:31:21.161753 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:31:21.161332    4461 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7056833-a12d-45d3-bf13-101db3dc1d24-credential-socket\") on node \"ip-10-0-143-156.ec2.internal\" DevicePath \"\""
Oct 30 03:31:21.161753 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:31:21.161346    4461 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/68c83adb-d031-4a39-93b6-925bed75adc4-workload-socket\") on node \"ip-10-0-143-156.ec2.internal\" DevicePath \"\""
Oct 30 03:31:21.161753 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:31:21.161363    4461 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f484p\" (UniqueName: \"kubernetes.io/projected/68c83adb-d031-4a39-93b6-925bed75adc4-kube-api-access-f484p\") on node \"ip-10-0-143-156.ec2.internal\" DevicePath \"\""
Oct 30 03:31:21.161753 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:31:21.161378    4461 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9428d98d-2f15-4c68-81a6-59066080b401-workload-certs\") on node \"ip-10-0-143-156.ec2.internal\" DevicePath \"\""
Oct 30 03:31:21.161753 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:31:21.161391    4461 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9428d98d-2f15-4c68-81a6-59066080b401-credential-socket\") on node \"ip-10-0-143-156.ec2.internal\" DevicePath \"\""
Oct 30 03:31:21.161753 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:31:21.161403    4461 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/68c83adb-d031-4a39-93b6-925bed75adc4-istio-envoy\") on node \"ip-10-0-143-156.ec2.internal\" DevicePath \"\""
Oct 30 03:31:21.161753 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:31:21.161361    4461 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/74841243-fe62-45e3-ac11-6480ca9913c5-workload-certs" pod "74841243-fe62-45e3-ac11-6480ca9913c5" (UID: "74841243-fe62-45e3-ac11-6480ca9913c5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:31:21.161753 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:31:21.161552    4461 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/74841243-fe62-45e3-ac11-6480ca9913c5-istio-data" pod "74841243-fe62-45e3-ac11-6480ca9913c5" (UID: "74841243-fe62-45e3-ac11-6480ca9913c5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:31:21.161753 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:31:21.161603    4461 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9428d98d-2f15-4c68-81a6-59066080b401-workload-socket" pod "9428d98d-2f15-4c68-81a6-59066080b401" (UID: "9428d98d-2f15-4c68-81a6-59066080b401"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:31:21.161753 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:31:21.161762    4461 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/74841243-fe62-45e3-ac11-6480ca9913c5-credential-socket" pod "74841243-fe62-45e3-ac11-6480ca9913c5" (UID: "74841243-fe62-45e3-ac11-6480ca9913c5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:31:21.163295 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:31:21.162197    4461 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f7056833-a12d-45d3-bf13-101db3dc1d24-istiod-ca-cert" pod "f7056833-a12d-45d3-bf13-101db3dc1d24" (UID: "f7056833-a12d-45d3-bf13-101db3dc1d24"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:31:21.163295 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:31:21.162477    4461 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f7056833-a12d-45d3-bf13-101db3dc1d24-workload-socket" pod "f7056833-a12d-45d3-bf13-101db3dc1d24" (UID: "f7056833-a12d-45d3-bf13-101db3dc1d24"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:31:21.163295 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:31:21.162707    4461 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9428d98d-2f15-4c68-81a6-59066080b401-istiod-ca-cert" pod "9428d98d-2f15-4c68-81a6-59066080b401" (UID: "9428d98d-2f15-4c68-81a6-59066080b401"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:31:21.164705 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:31:21.164587    4461 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/74841243-fe62-45e3-ac11-6480ca9913c5-istio-token" pod "74841243-fe62-45e3-ac11-6480ca9913c5" (UID: "74841243-fe62-45e3-ac11-6480ca9913c5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:31:21.164705 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:31:21.164651    4461 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9428d98d-2f15-4c68-81a6-59066080b401-istio-podinfo" pod "9428d98d-2f15-4c68-81a6-59066080b401" (UID: "9428d98d-2f15-4c68-81a6-59066080b401"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 03:31:21.164875 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:31:21.164708    4461 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/74841243-fe62-45e3-ac11-6480ca9913c5-kube-api-access-mtkvl" pod "74841243-fe62-45e3-ac11-6480ca9913c5" (UID: "74841243-fe62-45e3-ac11-6480ca9913c5"). InnerVolumeSpecName "kube-api-access-mtkvl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:31:21.165408 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:31:21.165385    4461 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/74841243-fe62-45e3-ac11-6480ca9913c5-istio-podinfo" pod "74841243-fe62-45e3-ac11-6480ca9913c5" (UID: "74841243-fe62-45e3-ac11-6480ca9913c5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 03:31:21.165521 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:31:21.165455    4461 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f7056833-a12d-45d3-bf13-101db3dc1d24-istio-token" pod "f7056833-a12d-45d3-bf13-101db3dc1d24" (UID: "f7056833-a12d-45d3-bf13-101db3dc1d24"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:31:21.165583 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:31:21.165558    4461 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f7056833-a12d-45d3-bf13-101db3dc1d24-istio-envoy" pod "f7056833-a12d-45d3-bf13-101db3dc1d24" (UID: "f7056833-a12d-45d3-bf13-101db3dc1d24"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:31:21.165628 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:31:21.165601    4461 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9428d98d-2f15-4c68-81a6-59066080b401-istio-envoy" pod "9428d98d-2f15-4c68-81a6-59066080b401" (UID: "9428d98d-2f15-4c68-81a6-59066080b401"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:31:21.165731 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:31:21.165711    4461 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f7056833-a12d-45d3-bf13-101db3dc1d24-kube-api-access-v9cdr" pod "f7056833-a12d-45d3-bf13-101db3dc1d24" (UID: "f7056833-a12d-45d3-bf13-101db3dc1d24"). InnerVolumeSpecName "kube-api-access-v9cdr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:31:21.166150 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:31:21.166086    4461 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9428d98d-2f15-4c68-81a6-59066080b401-istio-token" pod "9428d98d-2f15-4c68-81a6-59066080b401" (UID: "9428d98d-2f15-4c68-81a6-59066080b401"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:31:21.166279 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:31:21.166254    4461 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f7056833-a12d-45d3-bf13-101db3dc1d24-istio-podinfo" pod "f7056833-a12d-45d3-bf13-101db3dc1d24" (UID: "f7056833-a12d-45d3-bf13-101db3dc1d24"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 03:31:21.166565 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:31:21.166546    4461 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9428d98d-2f15-4c68-81a6-59066080b401-kube-api-access-fng8q" pod "9428d98d-2f15-4c68-81a6-59066080b401" (UID: "9428d98d-2f15-4c68-81a6-59066080b401"). InnerVolumeSpecName "kube-api-access-fng8q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:31:21.166933 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:31:21.166916    4461 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/74841243-fe62-45e3-ac11-6480ca9913c5-istio-envoy" pod "74841243-fe62-45e3-ac11-6480ca9913c5" (UID: "74841243-fe62-45e3-ac11-6480ca9913c5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:31:21.262087 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:31:21.262008    4461 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/74841243-fe62-45e3-ac11-6480ca9913c5-istio-data\") on node \"ip-10-0-143-156.ec2.internal\" DevicePath \"\""
Oct 30 03:31:21.262087 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:31:21.262049    4461 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/74841243-fe62-45e3-ac11-6480ca9913c5-credential-socket\") on node \"ip-10-0-143-156.ec2.internal\" DevicePath \"\""
Oct 30 03:31:21.262087 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:31:21.262065    4461 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fng8q\" (UniqueName: \"kubernetes.io/projected/9428d98d-2f15-4c68-81a6-59066080b401-kube-api-access-fng8q\") on node \"ip-10-0-143-156.ec2.internal\" DevicePath \"\""
Oct 30 03:31:21.262087 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:31:21.262081    4461 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7056833-a12d-45d3-bf13-101db3dc1d24-workload-socket\") on node \"ip-10-0-143-156.ec2.internal\" DevicePath \"\""
Oct 30 03:31:21.262344 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:31:21.262122    4461 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v9cdr\" (UniqueName: \"kubernetes.io/projected/f7056833-a12d-45d3-bf13-101db3dc1d24-kube-api-access-v9cdr\") on node \"ip-10-0-143-156.ec2.internal\" DevicePath \"\""
Oct 30 03:31:21.262344 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:31:21.262137    4461 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9428d98d-2f15-4c68-81a6-59066080b401-istio-podinfo\") on node \"ip-10-0-143-156.ec2.internal\" DevicePath \"\""
Oct 30 03:31:21.262344 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:31:21.262151    4461 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/74841243-fe62-45e3-ac11-6480ca9913c5-istio-podinfo\") on node \"ip-10-0-143-156.ec2.internal\" DevicePath \"\""
Oct 30 03:31:21.262344 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:31:21.262164    4461 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9428d98d-2f15-4c68-81a6-59066080b401-istiod-ca-cert\") on node \"ip-10-0-143-156.ec2.internal\" DevicePath \"\""
Oct 30 03:31:21.262344 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:31:21.262177    4461 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/74841243-fe62-45e3-ac11-6480ca9913c5-istio-envoy\") on node \"ip-10-0-143-156.ec2.internal\" DevicePath \"\""
Oct 30 03:31:21.262344 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:31:21.262191    4461 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mtkvl\" (UniqueName: \"kubernetes.io/projected/74841243-fe62-45e3-ac11-6480ca9913c5-kube-api-access-mtkvl\") on node \"ip-10-0-143-156.ec2.internal\" DevicePath \"\""
Oct 30 03:31:21.262344 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:31:21.262204    4461 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f7056833-a12d-45d3-bf13-101db3dc1d24-workload-certs\") on node \"ip-10-0-143-156.ec2.internal\" DevicePath \"\""
Oct 30 03:31:21.262344 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:31:21.262218    4461 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9428d98d-2f15-4c68-81a6-59066080b401-workload-socket\") on node \"ip-10-0-143-156.ec2.internal\" DevicePath \"\""
Oct 30 03:31:21.262344 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:31:21.262231    4461 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/74841243-fe62-45e3-ac11-6480ca9913c5-istio-token\") on node \"ip-10-0-143-156.ec2.internal\" DevicePath \"\""
Oct 30 03:31:21.262344 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:31:21.262244    4461 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f7056833-a12d-45d3-bf13-101db3dc1d24-istio-podinfo\") on node \"ip-10-0-143-156.ec2.internal\" DevicePath \"\""
Oct 30 03:31:21.262344 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:31:21.262257    4461 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9428d98d-2f15-4c68-81a6-59066080b401-istio-token\") on node \"ip-10-0-143-156.ec2.internal\" DevicePath \"\""
Oct 30 03:31:21.262344 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:31:21.262271    4461 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9428d98d-2f15-4c68-81a6-59066080b401-istio-envoy\") on node \"ip-10-0-143-156.ec2.internal\" DevicePath \"\""
Oct 30 03:31:21.262344 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:31:21.262285    4461 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f7056833-a12d-45d3-bf13-101db3dc1d24-istio-token\") on node \"ip-10-0-143-156.ec2.internal\" DevicePath \"\""
Oct 30 03:31:21.262344 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:31:21.262297    4461 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f7056833-a12d-45d3-bf13-101db3dc1d24-istio-envoy\") on node \"ip-10-0-143-156.ec2.internal\" DevicePath \"\""
Oct 30 03:31:21.262344 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:31:21.262309    4461 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f7056833-a12d-45d3-bf13-101db3dc1d24-istiod-ca-cert\") on node \"ip-10-0-143-156.ec2.internal\" DevicePath \"\""
Oct 30 03:31:21.262344 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:31:21.262322    4461 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/74841243-fe62-45e3-ac11-6480ca9913c5-workload-certs\") on node \"ip-10-0-143-156.ec2.internal\" DevicePath \"\""
Oct 30 03:31:21.905552 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:31:21.905518    4461 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-lp2z8"
Oct 30 03:31:21.905552 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:31:21.905527    4461 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-lp2z8" event={"ID":"f7056833-a12d-45d3-bf13-101db3dc1d24","Type":"ContainerDied","Data":"96d6d8ce97fd8db44529901bc82aaab1b1d64a587c794f54eef061c7c2e99bda"}
Oct 30 03:31:21.906065 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:31:21.905594    4461 scope.go:122] "RemoveContainer" containerID="8848879ad8abe8e8da97b7205cab605ce5fb4fce4a2de558d3e4fac58e68c2d4"
Oct 30 03:31:21.906962 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:31:21.906939    4461 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-fhwng"
Oct 30 03:31:21.906962 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:31:21.906951    4461 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-fhwng" event={"ID":"9428d98d-2f15-4c68-81a6-59066080b401","Type":"ContainerDied","Data":"2afec69a672b00a4e68772033cd1430a4e9c193f69b247f38482d8ab414262b9"}
Oct 30 03:31:21.908336 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:31:21.908314    4461 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-qf5vt"
Oct 30 03:31:21.908609 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:31:21.908334    4461 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-qf5vt" event={"ID":"68c83adb-d031-4a39-93b6-925bed75adc4","Type":"ContainerDied","Data":"24d83a76202f9cd59f2cf77f93f16fbd6be7be704cc1182bef35fe65e3390e7c"}
Oct 30 03:31:21.909745 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:31:21.909723    4461 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-7s92s" event={"ID":"74841243-fe62-45e3-ac11-6480ca9913c5","Type":"ContainerDied","Data":"e69d72bc6a944e4cdf3a62d5ed29bab011e9c0c0d0236ace00f791b92465803e"}
Oct 30 03:31:21.909840 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:31:21.909751    4461 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-7s92s"
Oct 30 03:31:21.920358 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:31:21.920339    4461 scope.go:122] "RemoveContainer" containerID="f1e6e45c30abc4aac701f0518ca60bb2891777d2b7ce9ff7fed73c5203d17e9b"
Oct 30 03:31:21.928078 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:31:21.928061    4461 scope.go:122] "RemoveContainer" containerID="ef0f9a0ab729567ac2f5bac6fb6ebde1165a67a035f1b83d3ec20abc5d87080a"
Oct 30 03:31:21.935104 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:31:21.935078    4461 scope.go:122] "RemoveContainer" containerID="d9108f9e40204e67f93220a50ed63e4e57a20b7abe2fda7f2c060f90aab129d0"
Oct 30 03:31:21.973911 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:31:21.973836    4461 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-lp2z8"]
Oct 30 03:31:21.986506 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:31:21.986482    4461 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-lp2z8"]
Oct 30 03:31:22.044899 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:31:22.044870    4461 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-fhwng"]
Oct 30 03:31:22.049744 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:31:22.049717    4461 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-fhwng"]
Oct 30 03:31:22.085048 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:31:22.085020    4461 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-qf5vt"]
Oct 30 03:31:22.091225 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:31:22.091201    4461 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-qf5vt"]
Oct 30 03:31:22.114657 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:31:22.114627    4461 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="68c83adb-d031-4a39-93b6-925bed75adc4" path="/var/lib/kubelet/pods/68c83adb-d031-4a39-93b6-925bed75adc4/volumes"
Oct 30 03:31:22.115035 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:31:22.115023    4461 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9428d98d-2f15-4c68-81a6-59066080b401" path="/var/lib/kubelet/pods/9428d98d-2f15-4c68-81a6-59066080b401/volumes"
Oct 30 03:31:22.115411 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:31:22.115399    4461 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f7056833-a12d-45d3-bf13-101db3dc1d24" path="/var/lib/kubelet/pods/f7056833-a12d-45d3-bf13-101db3dc1d24/volumes"
Oct 30 03:31:22.125767 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:31:22.125746    4461 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-7s92s"]
Oct 30 03:31:22.130674 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:31:22.130650    4461 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-7s92s"]
Oct 30 03:31:24.114734 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:31:24.114698    4461 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="74841243-fe62-45e3-ac11-6480ca9913c5" path="/var/lib/kubelet/pods/74841243-fe62-45e3-ac11-6480ca9913c5/volumes"
Oct 30 03:31:24.503678 ip-10-0-143-156 kubenswrapper[4461]: E1030 03:31:24.503640    4461 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2k85b" containerName="extractor"
Oct 30 03:31:33.635408 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:31:33.635373    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:31:35.605702 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:31:35.605669    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:31:40.265519 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:31:40.265483    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:31:42.604939 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:31:42.604909    4461 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-bc5ffbf5b-nvgcr"]
Oct 30 03:31:42.605361 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:31:42.605234    4461 cpu_manager.go:414] "containerMap: removing container" podUID="68c83adb-d031-4a39-93b6-925bed75adc4" containerName="istio-proxy"
Oct 30 03:31:42.605361 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:31:42.605247    4461 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="68c83adb-d031-4a39-93b6-925bed75adc4" containerName="istio-proxy"
Oct 30 03:31:42.605361 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:31:42.605262    4461 cpu_manager.go:414] "containerMap: removing container" podUID="74841243-fe62-45e3-ac11-6480ca9913c5" containerName="istio-proxy"
Oct 30 03:31:42.605361 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:31:42.605268    4461 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="74841243-fe62-45e3-ac11-6480ca9913c5" containerName="istio-proxy"
Oct 30 03:31:42.605361 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:31:42.605279    4461 cpu_manager.go:414] "containerMap: removing container" podUID="f7056833-a12d-45d3-bf13-101db3dc1d24" containerName="istio-proxy"
Oct 30 03:31:42.605361 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:31:42.605284    4461 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f7056833-a12d-45d3-bf13-101db3dc1d24" containerName="istio-proxy"
Oct 30 03:31:42.605361 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:31:42.605302    4461 cpu_manager.go:414] "containerMap: removing container" podUID="18bcde21-fc86-49f7-957f-463d8549c37a" containerName="llm-d-routing-sidecar"
Oct 30 03:31:42.605361 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:31:42.605310    4461 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="18bcde21-fc86-49f7-957f-463d8549c37a" containerName="llm-d-routing-sidecar"
Oct 30 03:31:42.605361 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:31:42.605324    4461 cpu_manager.go:414] "containerMap: removing container" podUID="18bcde21-fc86-49f7-957f-463d8549c37a" containerName="main"
Oct 30 03:31:42.605361 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:31:42.605329    4461 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="18bcde21-fc86-49f7-957f-463d8549c37a" containerName="main"
Oct 30 03:31:42.605361 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:31:42.605337    4461 cpu_manager.go:414] "containerMap: removing container" podUID="9428d98d-2f15-4c68-81a6-59066080b401" containerName="istio-proxy"
Oct 30 03:31:42.605361 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:31:42.605341    4461 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9428d98d-2f15-4c68-81a6-59066080b401" containerName="istio-proxy"
Oct 30 03:31:42.605707 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:31:42.605389    4461 memory_manager.go:365] "RemoveStaleState removing state" podUID="9428d98d-2f15-4c68-81a6-59066080b401" containerName="istio-proxy"
Oct 30 03:31:42.605707 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:31:42.605398    4461 memory_manager.go:365] "RemoveStaleState removing state" podUID="68c83adb-d031-4a39-93b6-925bed75adc4" containerName="istio-proxy"
Oct 30 03:31:42.605707 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:31:42.605403    4461 memory_manager.go:365] "RemoveStaleState removing state" podUID="74841243-fe62-45e3-ac11-6480ca9913c5" containerName="istio-proxy"
Oct 30 03:31:42.605707 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:31:42.605409    4461 memory_manager.go:365] "RemoveStaleState removing state" podUID="f7056833-a12d-45d3-bf13-101db3dc1d24" containerName="istio-proxy"
Oct 30 03:31:42.605707 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:31:42.605416    4461 memory_manager.go:365] "RemoveStaleState removing state" podUID="18bcde21-fc86-49f7-957f-463d8549c37a" containerName="llm-d-routing-sidecar"
Oct 30 03:31:42.605707 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:31:42.605422    4461 memory_manager.go:365] "RemoveStaleState removing state" podUID="18bcde21-fc86-49f7-957f-463d8549c37a" containerName="main"
Oct 30 03:31:42.610056 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:31:42.610027    4461 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-bc5ffbf5b-nvgcr"
Oct 30 03:31:42.631924 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:31:42.631891    4461 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8b4448d3-ed82-4c9d-9d33-6f4c3f60b2da-dshm\") pod \"scheduler-v06-threshold-migration-test-kserve-bc5ffbf5b-nvgcr\" (UID: \"8b4448d3-ed82-4c9d-9d33-6f4c3f60b2da\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-bc5ffbf5b-nvgcr"
Oct 30 03:31:42.632049 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:31:42.631956    4461 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8b4448d3-ed82-4c9d-9d33-6f4c3f60b2da-model-cache\") pod \"scheduler-v06-threshold-migration-test-kserve-bc5ffbf5b-nvgcr\" (UID: \"8b4448d3-ed82-4c9d-9d33-6f4c3f60b2da\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-bc5ffbf5b-nvgcr"
Oct 30 03:31:42.632139 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:31:42.632116    4461 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8b4448d3-ed82-4c9d-9d33-6f4c3f60b2da-tmp-dir\") pod \"scheduler-v06-threshold-migration-test-kserve-bc5ffbf5b-nvgcr\" (UID: \"8b4448d3-ed82-4c9d-9d33-6f4c3f60b2da\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-bc5ffbf5b-nvgcr"
Oct 30 03:31:42.632197 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:31:42.632173    4461 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mf48d\" (UniqueName: \"kubernetes.io/projected/8b4448d3-ed82-4c9d-9d33-6f4c3f60b2da-kube-api-access-mf48d\") pod \"scheduler-v06-threshold-migration-test-kserve-bc5ffbf5b-nvgcr\" (UID: \"8b4448d3-ed82-4c9d-9d33-6f4c3f60b2da\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-bc5ffbf5b-nvgcr"
Oct 30 03:31:42.632248 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:31:42.632206    4461 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8b4448d3-ed82-4c9d-9d33-6f4c3f60b2da-tls-certs\") pod \"scheduler-v06-threshold-migration-test-kserve-bc5ffbf5b-nvgcr\" (UID: \"8b4448d3-ed82-4c9d-9d33-6f4c3f60b2da\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-bc5ffbf5b-nvgcr"
Oct 30 03:31:42.632248 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:31:42.632232    4461 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8b4448d3-ed82-4c9d-9d33-6f4c3f60b2da-home\") pod \"scheduler-v06-threshold-migration-test-kserve-bc5ffbf5b-nvgcr\" (UID: \"8b4448d3-ed82-4c9d-9d33-6f4c3f60b2da\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-bc5ffbf5b-nvgcr"
Oct 30 03:31:42.632360 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:31:42.632342    4461 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-bc5ffbf5b-nvgcr"]
Oct 30 03:31:42.643719 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:31:42.643693    4461 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-40b85f95\"/\"kube-root-ca.crt\""
Oct 30 03:31:42.643924 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:31:42.643904    4461 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-40b85f95\"/\"scheduler-v06-threshold-migration-test-kserve-self-signed-certs\""
Oct 30 03:31:42.644008 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:31:42.643905    4461 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-40b85f95\"/\"scheduler-v06-threshold-migration-test-kserve-dockercfg-h67n2\""
Oct 30 03:31:42.644008 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:31:42.643936    4461 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-40b85f95\"/\"default-dockercfg-rzfmv\""
Oct 30 03:31:42.644008 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:31:42.643951    4461 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-40b85f95\"/\"openshift-service-ca.crt\""
Oct 30 03:31:42.712737 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:31:42.712694    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:31:42.732931 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:31:42.732900    4461 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mf48d\" (UniqueName: \"kubernetes.io/projected/8b4448d3-ed82-4c9d-9d33-6f4c3f60b2da-kube-api-access-mf48d\") pod \"scheduler-v06-threshold-migration-test-kserve-bc5ffbf5b-nvgcr\" (UID: \"8b4448d3-ed82-4c9d-9d33-6f4c3f60b2da\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-bc5ffbf5b-nvgcr"
Oct 30 03:31:42.733066 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:31:42.732938    4461 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8b4448d3-ed82-4c9d-9d33-6f4c3f60b2da-tls-certs\") pod \"scheduler-v06-threshold-migration-test-kserve-bc5ffbf5b-nvgcr\" (UID: \"8b4448d3-ed82-4c9d-9d33-6f4c3f60b2da\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-bc5ffbf5b-nvgcr"
Oct 30 03:31:42.733066 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:31:42.732961    4461 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8b4448d3-ed82-4c9d-9d33-6f4c3f60b2da-home\") pod \"scheduler-v06-threshold-migration-test-kserve-bc5ffbf5b-nvgcr\" (UID: \"8b4448d3-ed82-4c9d-9d33-6f4c3f60b2da\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-bc5ffbf5b-nvgcr"
Oct 30 03:31:42.733066 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:31:42.732991    4461 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8b4448d3-ed82-4c9d-9d33-6f4c3f60b2da-dshm\") pod \"scheduler-v06-threshold-migration-test-kserve-bc5ffbf5b-nvgcr\" (UID: \"8b4448d3-ed82-4c9d-9d33-6f4c3f60b2da\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-bc5ffbf5b-nvgcr"
Oct 30 03:31:42.733066 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:31:42.733045    4461 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8b4448d3-ed82-4c9d-9d33-6f4c3f60b2da-model-cache\") pod \"scheduler-v06-threshold-migration-test-kserve-bc5ffbf5b-nvgcr\" (UID: \"8b4448d3-ed82-4c9d-9d33-6f4c3f60b2da\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-bc5ffbf5b-nvgcr"
Oct 30 03:31:42.733301 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:31:42.733112    4461 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8b4448d3-ed82-4c9d-9d33-6f4c3f60b2da-tmp-dir\") pod \"scheduler-v06-threshold-migration-test-kserve-bc5ffbf5b-nvgcr\" (UID: \"8b4448d3-ed82-4c9d-9d33-6f4c3f60b2da\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-bc5ffbf5b-nvgcr"
Oct 30 03:31:42.733452 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:31:42.733431    4461 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8b4448d3-ed82-4c9d-9d33-6f4c3f60b2da-home\") pod \"scheduler-v06-threshold-migration-test-kserve-bc5ffbf5b-nvgcr\" (UID: \"8b4448d3-ed82-4c9d-9d33-6f4c3f60b2da\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-bc5ffbf5b-nvgcr"
Oct 30 03:31:42.733520 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:31:42.733476    4461 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8b4448d3-ed82-4c9d-9d33-6f4c3f60b2da-model-cache\") pod \"scheduler-v06-threshold-migration-test-kserve-bc5ffbf5b-nvgcr\" (UID: \"8b4448d3-ed82-4c9d-9d33-6f4c3f60b2da\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-bc5ffbf5b-nvgcr"
Oct 30 03:31:42.733520 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:31:42.733494    4461 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8b4448d3-ed82-4c9d-9d33-6f4c3f60b2da-tmp-dir\") pod \"scheduler-v06-threshold-migration-test-kserve-bc5ffbf5b-nvgcr\" (UID: \"8b4448d3-ed82-4c9d-9d33-6f4c3f60b2da\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-bc5ffbf5b-nvgcr"
Oct 30 03:31:42.735268 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:31:42.735246    4461 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8b4448d3-ed82-4c9d-9d33-6f4c3f60b2da-dshm\") pod \"scheduler-v06-threshold-migration-test-kserve-bc5ffbf5b-nvgcr\" (UID: \"8b4448d3-ed82-4c9d-9d33-6f4c3f60b2da\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-bc5ffbf5b-nvgcr"
Oct 30 03:31:42.735436 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:31:42.735421    4461 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8b4448d3-ed82-4c9d-9d33-6f4c3f60b2da-tls-certs\") pod \"scheduler-v06-threshold-migration-test-kserve-bc5ffbf5b-nvgcr\" (UID: \"8b4448d3-ed82-4c9d-9d33-6f4c3f60b2da\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-bc5ffbf5b-nvgcr"
Oct 30 03:31:42.756486 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:31:42.756462    4461 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mf48d\" (UniqueName: \"kubernetes.io/projected/8b4448d3-ed82-4c9d-9d33-6f4c3f60b2da-kube-api-access-mf48d\") pod \"scheduler-v06-threshold-migration-test-kserve-bc5ffbf5b-nvgcr\" (UID: \"8b4448d3-ed82-4c9d-9d33-6f4c3f60b2da\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-bc5ffbf5b-nvgcr"
Oct 30 03:31:42.900191 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:31:42.900069    4461 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedp57qq"]
Oct 30 03:31:42.903424 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:31:42.903407    4461 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedp57qq"
Oct 30 03:31:42.919661 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:31:42.919636    4461 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-40b85f95\"/\"scheduler-v06-threshold-migration-test-epp-sa-dockercfg-m9lpc\""
Oct 30 03:31:42.923689 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:31:42.923662    4461 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-bc5ffbf5b-nvgcr"
Oct 30 03:31:42.924065 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:31:42.924039    4461 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedp57qq"]
Oct 30 03:31:42.934825 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:31:42.934795    4461 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6c81b022-5c85-4c62-8c0a-f3d4d86db0b8-tls-certs\") pod \"scheduler-v06-threshold-migration-test-kserve-router-schedp57qq\" (UID: \"6c81b022-5c85-4c62-8c0a-f3d4d86db0b8\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedp57qq"
Oct 30 03:31:42.934946 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:31:42.934868    4461 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wms9g\" (UniqueName: \"kubernetes.io/projected/6c81b022-5c85-4c62-8c0a-f3d4d86db0b8-kube-api-access-wms9g\") pod \"scheduler-v06-threshold-migration-test-kserve-router-schedp57qq\" (UID: \"6c81b022-5c85-4c62-8c0a-f3d4d86db0b8\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedp57qq"
Oct 30 03:31:43.014615 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:31:43.014584    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:31:43.035825 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:31:43.035794    4461 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wms9g\" (UniqueName: \"kubernetes.io/projected/6c81b022-5c85-4c62-8c0a-f3d4d86db0b8-kube-api-access-wms9g\") pod \"scheduler-v06-threshold-migration-test-kserve-router-schedp57qq\" (UID: \"6c81b022-5c85-4c62-8c0a-f3d4d86db0b8\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedp57qq"
Oct 30 03:31:43.035966 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:31:43.035861    4461 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6c81b022-5c85-4c62-8c0a-f3d4d86db0b8-tls-certs\") pod \"scheduler-v06-threshold-migration-test-kserve-router-schedp57qq\" (UID: \"6c81b022-5c85-4c62-8c0a-f3d4d86db0b8\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedp57qq"
Oct 30 03:31:43.038139 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:31:43.038121    4461 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6c81b022-5c85-4c62-8c0a-f3d4d86db0b8-tls-certs\") pod \"scheduler-v06-threshold-migration-test-kserve-router-schedp57qq\" (UID: \"6c81b022-5c85-4c62-8c0a-f3d4d86db0b8\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedp57qq"
Oct 30 03:31:43.059483 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:31:43.059455    4461 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wms9g\" (UniqueName: \"kubernetes.io/projected/6c81b022-5c85-4c62-8c0a-f3d4d86db0b8-kube-api-access-wms9g\") pod \"scheduler-v06-threshold-migration-test-kserve-router-schedp57qq\" (UID: \"6c81b022-5c85-4c62-8c0a-f3d4d86db0b8\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedp57qq"
Oct 30 03:31:43.215898 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:31:43.215864    4461 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedp57qq"
Oct 30 03:31:43.243190 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:31:43.243058    4461 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-bc5ffbf5b-nvgcr"]
Oct 30 03:31:43.245782 ip-10-0-143-156 kubenswrapper[4461]: W1030 03:31:43.245754    4461 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b4448d3_ed82_4c9d_9d33_6f4c3f60b2da.slice/crio-69b4062758ba627ca08660dd2ec04e9369262d5ab15641567bda4b2295e0643d WatchSource:0}: Error finding container 69b4062758ba627ca08660dd2ec04e9369262d5ab15641567bda4b2295e0643d: Status 404 returned error can't find the container with id 69b4062758ba627ca08660dd2ec04e9369262d5ab15641567bda4b2295e0643d
Oct 30 03:31:43.513078 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:31:43.513006    4461 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedp57qq"]
Oct 30 03:31:43.516579 ip-10-0-143-156 kubenswrapper[4461]: W1030 03:31:43.516549    4461 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6c81b022_5c85_4c62_8c0a_f3d4d86db0b8.slice/crio-9cfadde3e1227508fbcd67b4e3fb005ea5c921f005ba7d046fdea6f5ce475723 WatchSource:0}: Error finding container 9cfadde3e1227508fbcd67b4e3fb005ea5c921f005ba7d046fdea6f5ce475723: Status 404 returned error can't find the container with id 9cfadde3e1227508fbcd67b4e3fb005ea5c921f005ba7d046fdea6f5ce475723
Oct 30 03:31:43.659680 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:31:43.659645    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:31:44.009278 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:31:44.009235    4461 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedp57qq" event={"ID":"6c81b022-5c85-4c62-8c0a-f3d4d86db0b8","Type":"ContainerStarted","Data":"518e0fc52121f9c63abc447f94adf888648f93d39e3fc3ac1627236e8b610b42"}
Oct 30 03:31:44.009479 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:31:44.009408    4461 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedp57qq"
Oct 30 03:31:44.009479 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:31:44.009441    4461 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedp57qq" event={"ID":"6c81b022-5c85-4c62-8c0a-f3d4d86db0b8","Type":"ContainerStarted","Data":"9cfadde3e1227508fbcd67b4e3fb005ea5c921f005ba7d046fdea6f5ce475723"}
Oct 30 03:31:44.010722 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:31:44.010701    4461 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-bc5ffbf5b-nvgcr" event={"ID":"8b4448d3-ed82-4c9d-9d33-6f4c3f60b2da","Type":"ContainerStarted","Data":"876145dd46a988cbab18acf578a7c21be721ce48df6fbbf0acdab68999c72f18"}
Oct 30 03:31:44.010827 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:31:44.010737    4461 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-bc5ffbf5b-nvgcr" event={"ID":"8b4448d3-ed82-4c9d-9d33-6f4c3f60b2da","Type":"ContainerStarted","Data":"69b4062758ba627ca08660dd2ec04e9369262d5ab15641567bda4b2295e0643d"}
Oct 30 03:31:44.010891 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:31:44.010840    4461 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-bc5ffbf5b-nvgcr"
Oct 30 03:31:44.055773 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:31:44.055718    4461 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedp57qq" podStartSLOduration=2.055700694 podStartE2EDuration="2.055700694s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:31:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:31:44.054478571 +0000 UTC m=+2172.104283762" watchObservedRunningTime="2026-10-30 03:31:44.055700694 +0000 UTC m=+2172.105505871"
Oct 30 03:31:45.016588 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:31:45.016550    4461 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-bc5ffbf5b-nvgcr" event={"ID":"8b4448d3-ed82-4c9d-9d33-6f4c3f60b2da","Type":"ContainerStarted","Data":"fc4b37072f4c3e36ea5c2e56dc6dd8e4ddc0c2b3075b218688777ab576782123"}
Oct 30 03:31:45.074091 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:31:45.074043    4461 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-bc5ffbf5b-nvgcr" podStartSLOduration=3.074031253 podStartE2EDuration="3.074031253s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:31:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:31:45.061359917 +0000 UTC m=+2173.111165092" watchObservedRunningTime="2026-10-30 03:31:45.074031253 +0000 UTC m=+2173.123836428"
Oct 30 03:31:45.258288 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:31:45.258256    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:31:45.924371 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:31:45.924327    4461 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-bc5ffbf5b-nvgcr"
Oct 30 03:31:46.356558 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:31:46.356527    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:31:46.519236 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:31:46.519201    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:31:48.362056 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:31:48.362024    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:31:52.923945 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:31:52.923910    4461 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-bc5ffbf5b-nvgcr"
Oct 30 03:31:52.936229 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:31:52.936205    4461 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-bc5ffbf5b-nvgcr"
Oct 30 03:31:53.058339 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:31:53.058312    4461 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-bc5ffbf5b-nvgcr"
Oct 30 03:31:53.059567 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:31:53.059542    4461 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-bc5ffbf5b-nvgcr"
Oct 30 03:31:54.503879 ip-10-0-143-156 kubenswrapper[4461]: E1030 03:31:54.503842    4461 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2k85b" containerName="extractor"
Oct 30 03:31:55.222260 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:31:55.222209    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:31:59.096650 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:31:59.096615    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:31:59.361040 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:31:59.360954    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:32:01.955956 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:32:01.955920    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:32:02.268365 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:32:02.268331    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:32:05.553108 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:32:05.553064    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:32:10.122505 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:32:10.122469    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:32:11.592856 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:32:11.592825    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:32:11.763287 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:32:11.763253    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:32:15.018925 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:32:15.018848    4461 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedp57qq"
Oct 30 03:32:18.036940 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:32:18.036905    4461 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedp57qq"]
Oct 30 03:32:18.043550 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:32:18.043518    4461 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-bc5ffbf5b-nvgcr"]
Oct 30 03:32:18.053001 ip-10-0-143-156 kubenswrapper[4461]: E1030 03:32:18.052967    4461 log.go:32] "ExecSync cmd from runtime service failed" err="rpc error: code = Unknown desc = command error: , stdout: , stderr: , exit code -1" containerID="fc4b37072f4c3e36ea5c2e56dc6dd8e4ddc0c2b3075b218688777ab576782123" cmd=["/bin/sleep","15"]
Oct 30 03:32:18.053153 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:32:18.053024    4461 handlers.go:85] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-bc5ffbf5b-nvgcr" message="" err="rpc error: code = Unknown desc = command error: , stdout: , stderr: , exit code -1"
Oct 30 03:32:18.053153 ip-10-0-143-156 kubenswrapper[4461]: E1030 03:32:18.053042    4461 kuberuntime_container.go:780] "PreStop hook failed" err="rpc error: code = Unknown desc = command error: , stdout: , stderr: , exit code -1" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-bc5ffbf5b-nvgcr" podUID="8b4448d3-ed82-4c9d-9d33-6f4c3f60b2da" containerName="main" containerID="cri-o://fc4b37072f4c3e36ea5c2e56dc6dd8e4ddc0c2b3075b218688777ab576782123"
Oct 30 03:32:18.053153 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:32:18.053123    4461 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-bc5ffbf5b-nvgcr" podUID="8b4448d3-ed82-4c9d-9d33-6f4c3f60b2da" containerName="main" containerID="cri-o://fc4b37072f4c3e36ea5c2e56dc6dd8e4ddc0c2b3075b218688777ab576782123" gracePeriod=60
Oct 30 03:32:18.087840 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:32:18.087800    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:32:18.257825 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:32:18.257789    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:32:18.271153 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:32:18.271120    4461 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-bc5ffbf5b-nvgcr" podUID="8b4448d3-ed82-4c9d-9d33-6f4c3f60b2da" containerName="llm-d-routing-sidecar" containerID="cri-o://876145dd46a988cbab18acf578a7c21be721ce48df6fbbf0acdab68999c72f18" gracePeriod=60
Oct 30 03:32:18.514822 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:32:18.514790    4461 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-bc5ffbf5b-nvgcr"
Oct 30 03:32:18.631773 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:32:18.631738    4461 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8b4448d3-ed82-4c9d-9d33-6f4c3f60b2da-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8b4448d3-ed82-4c9d-9d33-6f4c3f60b2da-model-cache\") pod \"8b4448d3-ed82-4c9d-9d33-6f4c3f60b2da\" (UID: \"8b4448d3-ed82-4c9d-9d33-6f4c3f60b2da\") "
Oct 30 03:32:18.631941 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:32:18.631817    4461 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8b4448d3-ed82-4c9d-9d33-6f4c3f60b2da-dshm\" (UniqueName: \"kubernetes.io/empty-dir/8b4448d3-ed82-4c9d-9d33-6f4c3f60b2da-dshm\") pod \"8b4448d3-ed82-4c9d-9d33-6f4c3f60b2da\" (UID: \"8b4448d3-ed82-4c9d-9d33-6f4c3f60b2da\") "
Oct 30 03:32:18.631941 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:32:18.631877    4461 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8b4448d3-ed82-4c9d-9d33-6f4c3f60b2da-kube-api-access-mf48d\" (UniqueName: \"kubernetes.io/projected/8b4448d3-ed82-4c9d-9d33-6f4c3f60b2da-kube-api-access-mf48d\") pod \"8b4448d3-ed82-4c9d-9d33-6f4c3f60b2da\" (UID: \"8b4448d3-ed82-4c9d-9d33-6f4c3f60b2da\") "
Oct 30 03:32:18.631941 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:32:18.631909    4461 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8b4448d3-ed82-4c9d-9d33-6f4c3f60b2da-tls-certs\" (UniqueName: \"kubernetes.io/secret/8b4448d3-ed82-4c9d-9d33-6f4c3f60b2da-tls-certs\") pod \"8b4448d3-ed82-4c9d-9d33-6f4c3f60b2da\" (UID: \"8b4448d3-ed82-4c9d-9d33-6f4c3f60b2da\") "
Oct 30 03:32:18.632064 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:32:18.631940    4461 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8b4448d3-ed82-4c9d-9d33-6f4c3f60b2da-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8b4448d3-ed82-4c9d-9d33-6f4c3f60b2da-tmp-dir\") pod \"8b4448d3-ed82-4c9d-9d33-6f4c3f60b2da\" (UID: \"8b4448d3-ed82-4c9d-9d33-6f4c3f60b2da\") "
Oct 30 03:32:18.632064 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:32:18.632000    4461 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8b4448d3-ed82-4c9d-9d33-6f4c3f60b2da-home\" (UniqueName: \"kubernetes.io/empty-dir/8b4448d3-ed82-4c9d-9d33-6f4c3f60b2da-home\") pod \"8b4448d3-ed82-4c9d-9d33-6f4c3f60b2da\" (UID: \"8b4448d3-ed82-4c9d-9d33-6f4c3f60b2da\") "
Oct 30 03:32:18.632180 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:32:18.632072    4461 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b4448d3-ed82-4c9d-9d33-6f4c3f60b2da-model-cache" pod "8b4448d3-ed82-4c9d-9d33-6f4c3f60b2da" (UID: "8b4448d3-ed82-4c9d-9d33-6f4c3f60b2da"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:32:18.632309 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:32:18.632279    4461 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b4448d3-ed82-4c9d-9d33-6f4c3f60b2da-tmp-dir" pod "8b4448d3-ed82-4c9d-9d33-6f4c3f60b2da" (UID: "8b4448d3-ed82-4c9d-9d33-6f4c3f60b2da"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:32:18.632394 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:32:18.632301    4461 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b4448d3-ed82-4c9d-9d33-6f4c3f60b2da-home" pod "8b4448d3-ed82-4c9d-9d33-6f4c3f60b2da" (UID: "8b4448d3-ed82-4c9d-9d33-6f4c3f60b2da"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:32:18.632394 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:32:18.632307    4461 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8b4448d3-ed82-4c9d-9d33-6f4c3f60b2da-model-cache\") on node \"ip-10-0-143-156.ec2.internal\" DevicePath \"\""
Oct 30 03:32:18.634191 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:32:18.634168    4461 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b4448d3-ed82-4c9d-9d33-6f4c3f60b2da-dshm" pod "8b4448d3-ed82-4c9d-9d33-6f4c3f60b2da" (UID: "8b4448d3-ed82-4c9d-9d33-6f4c3f60b2da"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:32:18.634288 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:32:18.634215    4461 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8b4448d3-ed82-4c9d-9d33-6f4c3f60b2da-tls-certs" pod "8b4448d3-ed82-4c9d-9d33-6f4c3f60b2da" (UID: "8b4448d3-ed82-4c9d-9d33-6f4c3f60b2da"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:32:18.634288 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:32:18.634219    4461 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8b4448d3-ed82-4c9d-9d33-6f4c3f60b2da-kube-api-access-mf48d" pod "8b4448d3-ed82-4c9d-9d33-6f4c3f60b2da" (UID: "8b4448d3-ed82-4c9d-9d33-6f4c3f60b2da"). InnerVolumeSpecName "kube-api-access-mf48d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:32:18.733421 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:32:18.733385    4461 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mf48d\" (UniqueName: \"kubernetes.io/projected/8b4448d3-ed82-4c9d-9d33-6f4c3f60b2da-kube-api-access-mf48d\") on node \"ip-10-0-143-156.ec2.internal\" DevicePath \"\""
Oct 30 03:32:18.733421 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:32:18.733415    4461 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8b4448d3-ed82-4c9d-9d33-6f4c3f60b2da-tls-certs\") on node \"ip-10-0-143-156.ec2.internal\" DevicePath \"\""
Oct 30 03:32:18.733421 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:32:18.733425    4461 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8b4448d3-ed82-4c9d-9d33-6f4c3f60b2da-tmp-dir\") on node \"ip-10-0-143-156.ec2.internal\" DevicePath \"\""
Oct 30 03:32:18.733652 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:32:18.733432    4461 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8b4448d3-ed82-4c9d-9d33-6f4c3f60b2da-home\") on node \"ip-10-0-143-156.ec2.internal\" DevicePath \"\""
Oct 30 03:32:18.733652 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:32:18.733440    4461 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8b4448d3-ed82-4c9d-9d33-6f4c3f60b2da-dshm\") on node \"ip-10-0-143-156.ec2.internal\" DevicePath \"\""
Oct 30 03:32:19.151655 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:32:19.151619    4461 generic.go:356] "Generic (PLEG): container finished" podID="8b4448d3-ed82-4c9d-9d33-6f4c3f60b2da" containerID="fc4b37072f4c3e36ea5c2e56dc6dd8e4ddc0c2b3075b218688777ab576782123" exitCode=0
Oct 30 03:32:19.151655 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:32:19.151647    4461 generic.go:356] "Generic (PLEG): container finished" podID="8b4448d3-ed82-4c9d-9d33-6f4c3f60b2da" containerID="876145dd46a988cbab18acf578a7c21be721ce48df6fbbf0acdab68999c72f18" exitCode=0
Oct 30 03:32:19.152191 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:32:19.151689    4461 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-bc5ffbf5b-nvgcr"
Oct 30 03:32:19.152191 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:32:19.151711    4461 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-bc5ffbf5b-nvgcr" event={"ID":"8b4448d3-ed82-4c9d-9d33-6f4c3f60b2da","Type":"ContainerDied","Data":"fc4b37072f4c3e36ea5c2e56dc6dd8e4ddc0c2b3075b218688777ab576782123"}
Oct 30 03:32:19.152191 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:32:19.151782    4461 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-bc5ffbf5b-nvgcr" event={"ID":"8b4448d3-ed82-4c9d-9d33-6f4c3f60b2da","Type":"ContainerDied","Data":"876145dd46a988cbab18acf578a7c21be721ce48df6fbbf0acdab68999c72f18"}
Oct 30 03:32:19.152191 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:32:19.151812    4461 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-bc5ffbf5b-nvgcr" event={"ID":"8b4448d3-ed82-4c9d-9d33-6f4c3f60b2da","Type":"ContainerDied","Data":"69b4062758ba627ca08660dd2ec04e9369262d5ab15641567bda4b2295e0643d"}
Oct 30 03:32:19.152191 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:32:19.151845    4461 scope.go:122] "RemoveContainer" containerID="fc4b37072f4c3e36ea5c2e56dc6dd8e4ddc0c2b3075b218688777ab576782123"
Oct 30 03:32:19.160299 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:32:19.160282    4461 scope.go:122] "RemoveContainer" containerID="876145dd46a988cbab18acf578a7c21be721ce48df6fbbf0acdab68999c72f18"
Oct 30 03:32:19.169504 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:32:19.169488    4461 scope.go:122] "RemoveContainer" containerID="fc4b37072f4c3e36ea5c2e56dc6dd8e4ddc0c2b3075b218688777ab576782123"
Oct 30 03:32:19.169753 ip-10-0-143-156 kubenswrapper[4461]: E1030 03:32:19.169731    4461 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fc4b37072f4c3e36ea5c2e56dc6dd8e4ddc0c2b3075b218688777ab576782123\": container with ID starting with fc4b37072f4c3e36ea5c2e56dc6dd8e4ddc0c2b3075b218688777ab576782123 not found: ID does not exist" containerID="fc4b37072f4c3e36ea5c2e56dc6dd8e4ddc0c2b3075b218688777ab576782123"
Oct 30 03:32:19.169830 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:32:19.169759    4461 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fc4b37072f4c3e36ea5c2e56dc6dd8e4ddc0c2b3075b218688777ab576782123"} err="failed to get container status \"fc4b37072f4c3e36ea5c2e56dc6dd8e4ddc0c2b3075b218688777ab576782123\": rpc error: code = NotFound desc = could not find container \"fc4b37072f4c3e36ea5c2e56dc6dd8e4ddc0c2b3075b218688777ab576782123\": container with ID starting with fc4b37072f4c3e36ea5c2e56dc6dd8e4ddc0c2b3075b218688777ab576782123 not found: ID does not exist"
Oct 30 03:32:19.169830 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:32:19.169776    4461 scope.go:122] "RemoveContainer" containerID="876145dd46a988cbab18acf578a7c21be721ce48df6fbbf0acdab68999c72f18"
Oct 30 03:32:19.170021 ip-10-0-143-156 kubenswrapper[4461]: E1030 03:32:19.170006    4461 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"876145dd46a988cbab18acf578a7c21be721ce48df6fbbf0acdab68999c72f18\": container with ID starting with 876145dd46a988cbab18acf578a7c21be721ce48df6fbbf0acdab68999c72f18 not found: ID does not exist" containerID="876145dd46a988cbab18acf578a7c21be721ce48df6fbbf0acdab68999c72f18"
Oct 30 03:32:19.170067 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:32:19.170024    4461 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"876145dd46a988cbab18acf578a7c21be721ce48df6fbbf0acdab68999c72f18"} err="failed to get container status \"876145dd46a988cbab18acf578a7c21be721ce48df6fbbf0acdab68999c72f18\": rpc error: code = NotFound desc = could not find container \"876145dd46a988cbab18acf578a7c21be721ce48df6fbbf0acdab68999c72f18\": container with ID starting with 876145dd46a988cbab18acf578a7c21be721ce48df6fbbf0acdab68999c72f18 not found: ID does not exist"
Oct 30 03:32:19.170067 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:32:19.170036    4461 scope.go:122] "RemoveContainer" containerID="fc4b37072f4c3e36ea5c2e56dc6dd8e4ddc0c2b3075b218688777ab576782123"
Oct 30 03:32:19.170291 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:32:19.170271    4461 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fc4b37072f4c3e36ea5c2e56dc6dd8e4ddc0c2b3075b218688777ab576782123"} err="failed to get container status \"fc4b37072f4c3e36ea5c2e56dc6dd8e4ddc0c2b3075b218688777ab576782123\": rpc error: code = NotFound desc = could not find container \"fc4b37072f4c3e36ea5c2e56dc6dd8e4ddc0c2b3075b218688777ab576782123\": container with ID starting with fc4b37072f4c3e36ea5c2e56dc6dd8e4ddc0c2b3075b218688777ab576782123 not found: ID does not exist"
Oct 30 03:32:19.170332 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:32:19.170293    4461 scope.go:122] "RemoveContainer" containerID="876145dd46a988cbab18acf578a7c21be721ce48df6fbbf0acdab68999c72f18"
Oct 30 03:32:19.170525 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:32:19.170503    4461 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"876145dd46a988cbab18acf578a7c21be721ce48df6fbbf0acdab68999c72f18"} err="failed to get container status \"876145dd46a988cbab18acf578a7c21be721ce48df6fbbf0acdab68999c72f18\": rpc error: code = NotFound desc = could not find container \"876145dd46a988cbab18acf578a7c21be721ce48df6fbbf0acdab68999c72f18\": container with ID starting with 876145dd46a988cbab18acf578a7c21be721ce48df6fbbf0acdab68999c72f18 not found: ID does not exist"
Oct 30 03:32:19.216594 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:32:19.216568    4461 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-bc5ffbf5b-nvgcr"]
Oct 30 03:32:19.229162 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:32:19.229136    4461 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-bc5ffbf5b-nvgcr"]
Oct 30 03:32:20.113935 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:32:20.113902    4461 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8b4448d3-ed82-4c9d-9d33-6f4c3f60b2da" path="/var/lib/kubelet/pods/8b4448d3-ed82-4c9d-9d33-6f4c3f60b2da/volumes"
Oct 30 03:32:20.448357 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:32:20.448291    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:32:21.422721 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:32:21.422690    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:32:24.503361 ip-10-0-143-156 kubenswrapper[4461]: E1030 03:32:24.503328    4461 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2k85b" containerName="extractor"
Oct 30 03:32:25.018306 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:32:25.018266    4461 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedp57qq" podUID="6c81b022-5c85-4c62-8c0a-f3d4d86db0b8" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 03:32:33.057262 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:32:33.057198    4461 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedp57qq" podUID="6c81b022-5c85-4c62-8c0a-f3d4d86db0b8" containerName="main" containerID="cri-o://518e0fc52121f9c63abc447f94adf888648f93d39e3fc3ac1627236e8b610b42" gracePeriod=45
Oct 30 03:32:35.019142 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:32:35.019084    4461 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedp57qq" podUID="6c81b022-5c85-4c62-8c0a-f3d4d86db0b8" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 03:32:45.018681 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:32:45.018634    4461 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedp57qq" podUID="6c81b022-5c85-4c62-8c0a-f3d4d86db0b8" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 03:32:45.019166 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:32:45.018768    4461 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedp57qq"
Oct 30 03:32:45.344346 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:32:45.344264    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:32:54.502832 ip-10-0-143-156 kubenswrapper[4461]: E1030 03:32:54.502794    4461 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2k85b" containerName="extractor"
Oct 30 03:32:55.018947 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:32:55.018908    4461 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedp57qq" podUID="6c81b022-5c85-4c62-8c0a-f3d4d86db0b8" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 03:32:55.344514 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:32:55.344431    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:33:03.299086 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:33:03.299062    4461 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedp57qq"
Oct 30 03:33:03.307303 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:33:03.307273    4461 generic.go:356] "Generic (PLEG): container finished" podID="6c81b022-5c85-4c62-8c0a-f3d4d86db0b8" containerID="518e0fc52121f9c63abc447f94adf888648f93d39e3fc3ac1627236e8b610b42" exitCode=0
Oct 30 03:33:03.307408 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:33:03.307313    4461 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedp57qq"
Oct 30 03:33:03.307408 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:33:03.307345    4461 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedp57qq" event={"ID":"6c81b022-5c85-4c62-8c0a-f3d4d86db0b8","Type":"ContainerDied","Data":"518e0fc52121f9c63abc447f94adf888648f93d39e3fc3ac1627236e8b610b42"}
Oct 30 03:33:03.307408 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:33:03.307390    4461 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedp57qq" event={"ID":"6c81b022-5c85-4c62-8c0a-f3d4d86db0b8","Type":"ContainerDied","Data":"9cfadde3e1227508fbcd67b4e3fb005ea5c921f005ba7d046fdea6f5ce475723"}
Oct 30 03:33:03.307511 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:33:03.307413    4461 scope.go:122] "RemoveContainer" containerID="518e0fc52121f9c63abc447f94adf888648f93d39e3fc3ac1627236e8b610b42"
Oct 30 03:33:03.314832 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:33:03.314815    4461 scope.go:122] "RemoveContainer" containerID="518e0fc52121f9c63abc447f94adf888648f93d39e3fc3ac1627236e8b610b42"
Oct 30 03:33:03.315145 ip-10-0-143-156 kubenswrapper[4461]: E1030 03:33:03.315121    4461 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"518e0fc52121f9c63abc447f94adf888648f93d39e3fc3ac1627236e8b610b42\": container with ID starting with 518e0fc52121f9c63abc447f94adf888648f93d39e3fc3ac1627236e8b610b42 not found: ID does not exist" containerID="518e0fc52121f9c63abc447f94adf888648f93d39e3fc3ac1627236e8b610b42"
Oct 30 03:33:03.315199 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:33:03.315157    4461 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"518e0fc52121f9c63abc447f94adf888648f93d39e3fc3ac1627236e8b610b42"} err="failed to get container status \"518e0fc52121f9c63abc447f94adf888648f93d39e3fc3ac1627236e8b610b42\": rpc error: code = NotFound desc = could not find container \"518e0fc52121f9c63abc447f94adf888648f93d39e3fc3ac1627236e8b610b42\": container with ID starting with 518e0fc52121f9c63abc447f94adf888648f93d39e3fc3ac1627236e8b610b42 not found: ID does not exist"
Oct 30 03:33:03.401653 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:33:03.401621    4461 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6c81b022-5c85-4c62-8c0a-f3d4d86db0b8-kube-api-access-wms9g\" (UniqueName: \"kubernetes.io/projected/6c81b022-5c85-4c62-8c0a-f3d4d86db0b8-kube-api-access-wms9g\") pod \"6c81b022-5c85-4c62-8c0a-f3d4d86db0b8\" (UID: \"6c81b022-5c85-4c62-8c0a-f3d4d86db0b8\") "
Oct 30 03:33:03.401784 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:33:03.401666    4461 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6c81b022-5c85-4c62-8c0a-f3d4d86db0b8-tls-certs\" (UniqueName: \"kubernetes.io/secret/6c81b022-5c85-4c62-8c0a-f3d4d86db0b8-tls-certs\") pod \"6c81b022-5c85-4c62-8c0a-f3d4d86db0b8\" (UID: \"6c81b022-5c85-4c62-8c0a-f3d4d86db0b8\") "
Oct 30 03:33:03.403610 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:33:03.403588    4461 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6c81b022-5c85-4c62-8c0a-f3d4d86db0b8-tls-certs" pod "6c81b022-5c85-4c62-8c0a-f3d4d86db0b8" (UID: "6c81b022-5c85-4c62-8c0a-f3d4d86db0b8"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:33:03.403763 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:33:03.403743    4461 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6c81b022-5c85-4c62-8c0a-f3d4d86db0b8-kube-api-access-wms9g" pod "6c81b022-5c85-4c62-8c0a-f3d4d86db0b8" (UID: "6c81b022-5c85-4c62-8c0a-f3d4d86db0b8"). InnerVolumeSpecName "kube-api-access-wms9g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:33:03.502587 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:33:03.502563    4461 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wms9g\" (UniqueName: \"kubernetes.io/projected/6c81b022-5c85-4c62-8c0a-f3d4d86db0b8-kube-api-access-wms9g\") on node \"ip-10-0-143-156.ec2.internal\" DevicePath \"\""
Oct 30 03:33:03.502587 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:33:03.502587    4461 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6c81b022-5c85-4c62-8c0a-f3d4d86db0b8-tls-certs\") on node \"ip-10-0-143-156.ec2.internal\" DevicePath \"\""
Oct 30 03:33:03.652414 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:33:03.652383    4461 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedp57qq"]
Oct 30 03:33:03.664970 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:33:03.664944    4461 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedp57qq"]
Oct 30 03:33:04.114300 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:33:04.114265    4461 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6c81b022-5c85-4c62-8c0a-f3d4d86db0b8" path="/var/lib/kubelet/pods/6c81b022-5c85-4c62-8c0a-f3d4d86db0b8/volumes"
Oct 30 03:33:08.159659 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:33:08.159624    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:33:08.340331 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:33:08.340301    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:33:12.397415 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:33:12.397378    4461 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-846f48577-wlt5j"]
Oct 30 03:33:12.397920 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:33:12.397682    4461 cpu_manager.go:414] "containerMap: removing container" podUID="8b4448d3-ed82-4c9d-9d33-6f4c3f60b2da" containerName="llm-d-routing-sidecar"
Oct 30 03:33:12.397920 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:33:12.397693    4461 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8b4448d3-ed82-4c9d-9d33-6f4c3f60b2da" containerName="llm-d-routing-sidecar"
Oct 30 03:33:12.397920 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:33:12.397710    4461 cpu_manager.go:414] "containerMap: removing container" podUID="8b4448d3-ed82-4c9d-9d33-6f4c3f60b2da" containerName="main"
Oct 30 03:33:12.397920 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:33:12.397715    4461 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8b4448d3-ed82-4c9d-9d33-6f4c3f60b2da" containerName="main"
Oct 30 03:33:12.397920 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:33:12.397729    4461 cpu_manager.go:414] "containerMap: removing container" podUID="6c81b022-5c85-4c62-8c0a-f3d4d86db0b8" containerName="main"
Oct 30 03:33:12.397920 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:33:12.397736    4461 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6c81b022-5c85-4c62-8c0a-f3d4d86db0b8" containerName="main"
Oct 30 03:33:12.397920 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:33:12.397780    4461 memory_manager.go:365] "RemoveStaleState removing state" podUID="8b4448d3-ed82-4c9d-9d33-6f4c3f60b2da" containerName="llm-d-routing-sidecar"
Oct 30 03:33:12.397920 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:33:12.397788    4461 memory_manager.go:365] "RemoveStaleState removing state" podUID="8b4448d3-ed82-4c9d-9d33-6f4c3f60b2da" containerName="main"
Oct 30 03:33:12.397920 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:33:12.397796    4461 memory_manager.go:365] "RemoveStaleState removing state" podUID="6c81b022-5c85-4c62-8c0a-f3d4d86db0b8" containerName="main"
Oct 30 03:33:12.401703 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:33:12.401678    4461 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-846f48577-wlt5j"
Oct 30 03:33:12.423014 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:33:12.422988    4461 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-846f48577-wlt5j"]
Oct 30 03:33:12.429461 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:33:12.429438    4461 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"tokenizer-clean-path-test-kserve-self-signed-certs\""
Oct 30 03:33:12.429572 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:33:12.429471    4461 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"default-dockercfg-q6h96\""
Oct 30 03:33:12.429572 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:33:12.429483    4461 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"seaweedfs-s3-creds\""
Oct 30 03:33:12.480965 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:33:12.480932    4461 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e859bb9c-e8e2-4221-a6aa-4f56545c70c6-tls-certs\") pod \"tokenizer-clean-path-test-kserve-846f48577-wlt5j\" (UID: \"e859bb9c-e8e2-4221-a6aa-4f56545c70c6\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-846f48577-wlt5j"
Oct 30 03:33:12.481154 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:33:12.480977    4461 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9wfbk\" (UniqueName: \"kubernetes.io/projected/e859bb9c-e8e2-4221-a6aa-4f56545c70c6-kube-api-access-9wfbk\") pod \"tokenizer-clean-path-test-kserve-846f48577-wlt5j\" (UID: \"e859bb9c-e8e2-4221-a6aa-4f56545c70c6\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-846f48577-wlt5j"
Oct 30 03:33:12.481154 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:33:12.481122    4461 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e859bb9c-e8e2-4221-a6aa-4f56545c70c6-model-cache\") pod \"tokenizer-clean-path-test-kserve-846f48577-wlt5j\" (UID: \"e859bb9c-e8e2-4221-a6aa-4f56545c70c6\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-846f48577-wlt5j"
Oct 30 03:33:12.481294 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:33:12.481169    4461 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e859bb9c-e8e2-4221-a6aa-4f56545c70c6-home\") pod \"tokenizer-clean-path-test-kserve-846f48577-wlt5j\" (UID: \"e859bb9c-e8e2-4221-a6aa-4f56545c70c6\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-846f48577-wlt5j"
Oct 30 03:33:12.481294 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:33:12.481246    4461 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e859bb9c-e8e2-4221-a6aa-4f56545c70c6-dshm\") pod \"tokenizer-clean-path-test-kserve-846f48577-wlt5j\" (UID: \"e859bb9c-e8e2-4221-a6aa-4f56545c70c6\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-846f48577-wlt5j"
Oct 30 03:33:12.481294 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:33:12.481274    4461 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e859bb9c-e8e2-4221-a6aa-4f56545c70c6-tmp-dir\") pod \"tokenizer-clean-path-test-kserve-846f48577-wlt5j\" (UID: \"e859bb9c-e8e2-4221-a6aa-4f56545c70c6\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-846f48577-wlt5j"
Oct 30 03:33:12.481429 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:33:12.481300    4461 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e859bb9c-e8e2-4221-a6aa-4f56545c70c6-kserve-provision-location\") pod \"tokenizer-clean-path-test-kserve-846f48577-wlt5j\" (UID: \"e859bb9c-e8e2-4221-a6aa-4f56545c70c6\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-846f48577-wlt5j"
Oct 30 03:33:12.481429 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:33:12.481394    4461 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e859bb9c-e8e2-4221-a6aa-4f56545c70c6-cabundle-cert\") pod \"tokenizer-clean-path-test-kserve-846f48577-wlt5j\" (UID: \"e859bb9c-e8e2-4221-a6aa-4f56545c70c6\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-846f48577-wlt5j"
Oct 30 03:33:12.500270 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:33:12.500241    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:33:12.501732 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:33:12.501703    4461 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"kube-root-ca.crt\""
Oct 30 03:33:12.501842 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:33:12.501752    4461 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"odh-kserve-custom-ca-bundle\""
Oct 30 03:33:12.501842 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:33:12.501760    4461 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"openshift-service-ca.crt\""
Oct 30 03:33:12.582394 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:33:12.582361    4461 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e859bb9c-e8e2-4221-a6aa-4f56545c70c6-home\") pod \"tokenizer-clean-path-test-kserve-846f48577-wlt5j\" (UID: \"e859bb9c-e8e2-4221-a6aa-4f56545c70c6\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-846f48577-wlt5j"
Oct 30 03:33:12.582533 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:33:12.582401    4461 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e859bb9c-e8e2-4221-a6aa-4f56545c70c6-dshm\") pod \"tokenizer-clean-path-test-kserve-846f48577-wlt5j\" (UID: \"e859bb9c-e8e2-4221-a6aa-4f56545c70c6\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-846f48577-wlt5j"
Oct 30 03:33:12.582533 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:33:12.582419    4461 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e859bb9c-e8e2-4221-a6aa-4f56545c70c6-tmp-dir\") pod \"tokenizer-clean-path-test-kserve-846f48577-wlt5j\" (UID: \"e859bb9c-e8e2-4221-a6aa-4f56545c70c6\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-846f48577-wlt5j"
Oct 30 03:33:12.582533 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:33:12.582436    4461 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e859bb9c-e8e2-4221-a6aa-4f56545c70c6-kserve-provision-location\") pod \"tokenizer-clean-path-test-kserve-846f48577-wlt5j\" (UID: \"e859bb9c-e8e2-4221-a6aa-4f56545c70c6\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-846f48577-wlt5j"
Oct 30 03:33:12.582533 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:33:12.582457    4461 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e859bb9c-e8e2-4221-a6aa-4f56545c70c6-cabundle-cert\") pod \"tokenizer-clean-path-test-kserve-846f48577-wlt5j\" (UID: \"e859bb9c-e8e2-4221-a6aa-4f56545c70c6\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-846f48577-wlt5j"
Oct 30 03:33:12.582533 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:33:12.582484    4461 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e859bb9c-e8e2-4221-a6aa-4f56545c70c6-tls-certs\") pod \"tokenizer-clean-path-test-kserve-846f48577-wlt5j\" (UID: \"e859bb9c-e8e2-4221-a6aa-4f56545c70c6\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-846f48577-wlt5j"
Oct 30 03:33:12.582533 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:33:12.582516    4461 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9wfbk\" (UniqueName: \"kubernetes.io/projected/e859bb9c-e8e2-4221-a6aa-4f56545c70c6-kube-api-access-9wfbk\") pod \"tokenizer-clean-path-test-kserve-846f48577-wlt5j\" (UID: \"e859bb9c-e8e2-4221-a6aa-4f56545c70c6\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-846f48577-wlt5j"
Oct 30 03:33:12.582851 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:33:12.582652    4461 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e859bb9c-e8e2-4221-a6aa-4f56545c70c6-model-cache\") pod \"tokenizer-clean-path-test-kserve-846f48577-wlt5j\" (UID: \"e859bb9c-e8e2-4221-a6aa-4f56545c70c6\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-846f48577-wlt5j"
Oct 30 03:33:12.582851 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:33:12.582834    4461 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e859bb9c-e8e2-4221-a6aa-4f56545c70c6-kserve-provision-location\") pod \"tokenizer-clean-path-test-kserve-846f48577-wlt5j\" (UID: \"e859bb9c-e8e2-4221-a6aa-4f56545c70c6\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-846f48577-wlt5j"
Oct 30 03:33:12.582960 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:33:12.582905    4461 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e859bb9c-e8e2-4221-a6aa-4f56545c70c6-tmp-dir\") pod \"tokenizer-clean-path-test-kserve-846f48577-wlt5j\" (UID: \"e859bb9c-e8e2-4221-a6aa-4f56545c70c6\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-846f48577-wlt5j"
Oct 30 03:33:12.582960 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:33:12.582944    4461 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e859bb9c-e8e2-4221-a6aa-4f56545c70c6-home\") pod \"tokenizer-clean-path-test-kserve-846f48577-wlt5j\" (UID: \"e859bb9c-e8e2-4221-a6aa-4f56545c70c6\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-846f48577-wlt5j"
Oct 30 03:33:12.583047 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:33:12.582989    4461 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e859bb9c-e8e2-4221-a6aa-4f56545c70c6-model-cache\") pod \"tokenizer-clean-path-test-kserve-846f48577-wlt5j\" (UID: \"e859bb9c-e8e2-4221-a6aa-4f56545c70c6\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-846f48577-wlt5j"
Oct 30 03:33:12.583258 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:33:12.583236    4461 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e859bb9c-e8e2-4221-a6aa-4f56545c70c6-cabundle-cert\") pod \"tokenizer-clean-path-test-kserve-846f48577-wlt5j\" (UID: \"e859bb9c-e8e2-4221-a6aa-4f56545c70c6\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-846f48577-wlt5j"
Oct 30 03:33:12.584915 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:33:12.584890    4461 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e859bb9c-e8e2-4221-a6aa-4f56545c70c6-dshm\") pod \"tokenizer-clean-path-test-kserve-846f48577-wlt5j\" (UID: \"e859bb9c-e8e2-4221-a6aa-4f56545c70c6\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-846f48577-wlt5j"
Oct 30 03:33:12.585266 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:33:12.585247    4461 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e859bb9c-e8e2-4221-a6aa-4f56545c70c6-tls-certs\") pod \"tokenizer-clean-path-test-kserve-846f48577-wlt5j\" (UID: \"e859bb9c-e8e2-4221-a6aa-4f56545c70c6\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-846f48577-wlt5j"
Oct 30 03:33:12.606363 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:33:12.606340    4461 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9wfbk\" (UniqueName: \"kubernetes.io/projected/e859bb9c-e8e2-4221-a6aa-4f56545c70c6-kube-api-access-9wfbk\") pod \"tokenizer-clean-path-test-kserve-846f48577-wlt5j\" (UID: \"e859bb9c-e8e2-4221-a6aa-4f56545c70c6\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-846f48577-wlt5j"
Oct 30 03:33:12.716630 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:33:12.716601    4461 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-846f48577-wlt5j"
Oct 30 03:33:12.760767 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:33:12.760735    4461 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-84d6588cjcp5x"]
Oct 30 03:33:12.764554 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:33:12.764529    4461 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-84d6588cjcp5x"
Oct 30 03:33:12.783861 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:33:12.783837    4461 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"tokenizer-clean-path-test-epp-sa-dockercfg-sl5c6\""
Oct 30 03:33:12.784956 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:33:12.784928    4461 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-84d6588cjcp5x"]
Oct 30 03:33:12.824701 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:33:12.824666    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:33:12.884978 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:33:12.884943    4461 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2dx5j\" (UniqueName: \"kubernetes.io/projected/75002685-ae4a-4df3-a4ec-e51e3ec1dfea-kube-api-access-2dx5j\") pod \"tokenizer-clean-path-test-kserve-router-scheduler-84d6588cjcp5x\" (UID: \"75002685-ae4a-4df3-a4ec-e51e3ec1dfea\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-84d6588cjcp5x"
Oct 30 03:33:12.885176 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:33:12.885000    4461 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/75002685-ae4a-4df3-a4ec-e51e3ec1dfea-tls-certs\") pod \"tokenizer-clean-path-test-kserve-router-scheduler-84d6588cjcp5x\" (UID: \"75002685-ae4a-4df3-a4ec-e51e3ec1dfea\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-84d6588cjcp5x"
Oct 30 03:33:12.985815 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:33:12.985746    4461 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2dx5j\" (UniqueName: \"kubernetes.io/projected/75002685-ae4a-4df3-a4ec-e51e3ec1dfea-kube-api-access-2dx5j\") pod \"tokenizer-clean-path-test-kserve-router-scheduler-84d6588cjcp5x\" (UID: \"75002685-ae4a-4df3-a4ec-e51e3ec1dfea\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-84d6588cjcp5x"
Oct 30 03:33:12.985815 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:33:12.985805    4461 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/75002685-ae4a-4df3-a4ec-e51e3ec1dfea-tls-certs\") pod \"tokenizer-clean-path-test-kserve-router-scheduler-84d6588cjcp5x\" (UID: \"75002685-ae4a-4df3-a4ec-e51e3ec1dfea\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-84d6588cjcp5x"
Oct 30 03:33:12.988134 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:33:12.988113    4461 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/75002685-ae4a-4df3-a4ec-e51e3ec1dfea-tls-certs\") pod \"tokenizer-clean-path-test-kserve-router-scheduler-84d6588cjcp5x\" (UID: \"75002685-ae4a-4df3-a4ec-e51e3ec1dfea\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-84d6588cjcp5x"
Oct 30 03:33:13.011036 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:33:13.011011    4461 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2dx5j\" (UniqueName: \"kubernetes.io/projected/75002685-ae4a-4df3-a4ec-e51e3ec1dfea-kube-api-access-2dx5j\") pod \"tokenizer-clean-path-test-kserve-router-scheduler-84d6588cjcp5x\" (UID: \"75002685-ae4a-4df3-a4ec-e51e3ec1dfea\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-84d6588cjcp5x"
Oct 30 03:33:13.021052 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:33:13.021017    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:33:13.043343 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:33:13.043193    4461 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-846f48577-wlt5j"]
Oct 30 03:33:13.045861 ip-10-0-143-156 kubenswrapper[4461]: W1030 03:33:13.045837    4461 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode859bb9c_e8e2_4221_a6aa_4f56545c70c6.slice/crio-1085bcb20c8d196aa9cf5217c9518a98dd6d6d48c3e0fb0f4de4e49b23bfc614 WatchSource:0}: Error finding container 1085bcb20c8d196aa9cf5217c9518a98dd6d6d48c3e0fb0f4de4e49b23bfc614: Status 404 returned error can't find the container with id 1085bcb20c8d196aa9cf5217c9518a98dd6d6d48c3e0fb0f4de4e49b23bfc614
Oct 30 03:33:13.047703 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:33:13.047687    4461 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 03:33:13.079922 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:33:13.079895    4461 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-84d6588cjcp5x"
Oct 30 03:33:13.346208 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:33:13.346136    4461 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-846f48577-wlt5j" event={"ID":"e859bb9c-e8e2-4221-a6aa-4f56545c70c6","Type":"ContainerStarted","Data":"1085bcb20c8d196aa9cf5217c9518a98dd6d6d48c3e0fb0f4de4e49b23bfc614"}
Oct 30 03:33:13.377311 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:33:13.377281    4461 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-84d6588cjcp5x"]
Oct 30 03:33:13.380445 ip-10-0-143-156 kubenswrapper[4461]: W1030 03:33:13.380417    4461 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod75002685_ae4a_4df3_a4ec_e51e3ec1dfea.slice/crio-fd5957f06dce6ac61c2cc71fd40022d8173d3a344d52585117a34a5d2bc2c34b WatchSource:0}: Error finding container fd5957f06dce6ac61c2cc71fd40022d8173d3a344d52585117a34a5d2bc2c34b: Status 404 returned error can't find the container with id fd5957f06dce6ac61c2cc71fd40022d8173d3a344d52585117a34a5d2bc2c34b
Oct 30 03:33:14.363575 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:33:14.363532    4461 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-84d6588cjcp5x" event={"ID":"75002685-ae4a-4df3-a4ec-e51e3ec1dfea","Type":"ContainerStarted","Data":"5d37fa8eb9fd27546dbeca7bbda133eca033fd93507ff285c2e68cc807269d33"}
Oct 30 03:33:14.364048 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:33:14.363646    4461 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-84d6588cjcp5x" event={"ID":"75002685-ae4a-4df3-a4ec-e51e3ec1dfea","Type":"ContainerStarted","Data":"fd5957f06dce6ac61c2cc71fd40022d8173d3a344d52585117a34a5d2bc2c34b"}
Oct 30 03:33:14.364048 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:33:14.363683    4461 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-84d6588cjcp5x"
Oct 30 03:33:15.377937 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:33:15.377901    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:33:17.378884 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:33:17.378838    4461 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-846f48577-wlt5j" event={"ID":"e859bb9c-e8e2-4221-a6aa-4f56545c70c6","Type":"ContainerStarted","Data":"08862268ab730e180b0755ea12f8eb9214187e8164c0faa7f75e6eb6729d1112"}
Oct 30 03:33:17.424985 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:33:17.424930    4461 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-84d6588cjcp5x" podStartSLOduration=5.424915793 podStartE2EDuration="5.424915793s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:33:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:33:14.407176808 +0000 UTC m=+2262.456981983" watchObservedRunningTime="2026-10-30 03:33:17.424915793 +0000 UTC m=+2265.474720970"
Oct 30 03:33:19.110419 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:33:19.110387    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:33:19.389309 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:33:19.389223    4461 generic.go:356] "Generic (PLEG): container finished" podID="e859bb9c-e8e2-4221-a6aa-4f56545c70c6" containerID="08862268ab730e180b0755ea12f8eb9214187e8164c0faa7f75e6eb6729d1112" exitCode=0
Oct 30 03:33:19.389309 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:33:19.389277    4461 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-846f48577-wlt5j" event={"ID":"e859bb9c-e8e2-4221-a6aa-4f56545c70c6","Type":"ContainerDied","Data":"08862268ab730e180b0755ea12f8eb9214187e8164c0faa7f75e6eb6729d1112"}
Oct 30 03:33:20.395083 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:33:20.395051    4461 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-846f48577-wlt5j" event={"ID":"e859bb9c-e8e2-4221-a6aa-4f56545c70c6","Type":"ContainerStarted","Data":"9492954ddcd48f4850d37e23e3a34cae2fc639132464720ae11e0b7f17e55629"}
Oct 30 03:33:20.454306 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:33:20.454246    4461 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-846f48577-wlt5j" podStartSLOduration=2.851524582 podStartE2EDuration="8.454226897s" totalImagesPullingTime="3.490080299s" totalInitContainerRuntime="2.112622016s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:33:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:33:20.4532007 +0000 UTC m=+2268.503005878" watchObservedRunningTime="2026-10-30 03:33:20.454226897 +0000 UTC m=+2268.504032071"
Oct 30 03:33:20.717625 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:33:20.717591    4461 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-846f48577-wlt5j"
Oct 30 03:33:21.289438 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:33:21.289398    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:33:21.453580 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:33:21.453547    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:33:22.717665 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:33:22.717628    4461 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-846f48577-wlt5j"
Oct 30 03:33:22.730966 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:33:22.730939    4461 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-846f48577-wlt5j"
Oct 30 03:33:23.417933 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:33:23.417903    4461 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-846f48577-wlt5j"
Oct 30 03:33:24.504936 ip-10-0-143-156 kubenswrapper[4461]: E1030 03:33:24.504902    4461 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2k85b" containerName="extractor"
Oct 30 03:33:45.370480 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:33:45.370410    4461 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-84d6588cjcp5x"
Oct 30 03:33:45.465890 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:33:45.465860    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:33:54.503170 ip-10-0-143-156 kubenswrapper[4461]: E1030 03:33:54.503133    4461 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2k85b" containerName="extractor"
Oct 30 03:33:55.456589 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:33:55.456551    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:34:00.426621 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:34:00.426588    4461 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-846f48577-wlt5j"]
Oct 30 03:34:00.438880 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:34:00.438850    4461 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-84d6588cjcp5x"]
Oct 30 03:34:00.439945 ip-10-0-143-156 kubenswrapper[4461]: E1030 03:34:00.439904    4461 log.go:32] "ExecSync cmd from runtime service failed" err="rpc error: code = Unknown desc = command error: , stdout: , stderr: , exit code -1" containerID="9492954ddcd48f4850d37e23e3a34cae2fc639132464720ae11e0b7f17e55629" cmd=["/bin/sleep","15"]
Oct 30 03:34:00.440169 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:34:00.439965    4461 handlers.go:85] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-846f48577-wlt5j" message="" err="rpc error: code = Unknown desc = command error: , stdout: , stderr: , exit code -1"
Oct 30 03:34:00.440169 ip-10-0-143-156 kubenswrapper[4461]: E1030 03:34:00.439992    4461 kuberuntime_container.go:780] "PreStop hook failed" err="rpc error: code = Unknown desc = command error: , stdout: , stderr: , exit code -1" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-846f48577-wlt5j" podUID="e859bb9c-e8e2-4221-a6aa-4f56545c70c6" containerName="main" containerID="cri-o://9492954ddcd48f4850d37e23e3a34cae2fc639132464720ae11e0b7f17e55629"
Oct 30 03:34:00.440169 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:34:00.440043    4461 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-846f48577-wlt5j" podUID="e859bb9c-e8e2-4221-a6aa-4f56545c70c6" containerName="main" containerID="cri-o://9492954ddcd48f4850d37e23e3a34cae2fc639132464720ae11e0b7f17e55629" gracePeriod=60
Oct 30 03:34:00.515750 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:34:00.515713    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:34:00.679496 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:34:00.679424    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:34:01.182538 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:34:01.182509    4461 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-846f48577-wlt5j"
Oct 30 03:34:01.282720 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:34:01.282686    4461 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e859bb9c-e8e2-4221-a6aa-4f56545c70c6-dshm\" (UniqueName: \"kubernetes.io/empty-dir/e859bb9c-e8e2-4221-a6aa-4f56545c70c6-dshm\") pod \"e859bb9c-e8e2-4221-a6aa-4f56545c70c6\" (UID: \"e859bb9c-e8e2-4221-a6aa-4f56545c70c6\") "
Oct 30 03:34:01.282882 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:34:01.282737    4461 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e859bb9c-e8e2-4221-a6aa-4f56545c70c6-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e859bb9c-e8e2-4221-a6aa-4f56545c70c6-cabundle-cert\") pod \"e859bb9c-e8e2-4221-a6aa-4f56545c70c6\" (UID: \"e859bb9c-e8e2-4221-a6aa-4f56545c70c6\") "
Oct 30 03:34:01.282882 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:34:01.282786    4461 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e859bb9c-e8e2-4221-a6aa-4f56545c70c6-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e859bb9c-e8e2-4221-a6aa-4f56545c70c6-model-cache\") pod \"e859bb9c-e8e2-4221-a6aa-4f56545c70c6\" (UID: \"e859bb9c-e8e2-4221-a6aa-4f56545c70c6\") "
Oct 30 03:34:01.282882 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:34:01.282807    4461 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e859bb9c-e8e2-4221-a6aa-4f56545c70c6-kube-api-access-9wfbk\" (UniqueName: \"kubernetes.io/projected/e859bb9c-e8e2-4221-a6aa-4f56545c70c6-kube-api-access-9wfbk\") pod \"e859bb9c-e8e2-4221-a6aa-4f56545c70c6\" (UID: \"e859bb9c-e8e2-4221-a6aa-4f56545c70c6\") "
Oct 30 03:34:01.282882 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:34:01.282824    4461 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e859bb9c-e8e2-4221-a6aa-4f56545c70c6-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e859bb9c-e8e2-4221-a6aa-4f56545c70c6-tmp-dir\") pod \"e859bb9c-e8e2-4221-a6aa-4f56545c70c6\" (UID: \"e859bb9c-e8e2-4221-a6aa-4f56545c70c6\") "
Oct 30 03:34:01.282882 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:34:01.282860    4461 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e859bb9c-e8e2-4221-a6aa-4f56545c70c6-home\" (UniqueName: \"kubernetes.io/empty-dir/e859bb9c-e8e2-4221-a6aa-4f56545c70c6-home\") pod \"e859bb9c-e8e2-4221-a6aa-4f56545c70c6\" (UID: \"e859bb9c-e8e2-4221-a6aa-4f56545c70c6\") "
Oct 30 03:34:01.283185 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:34:01.282883    4461 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e859bb9c-e8e2-4221-a6aa-4f56545c70c6-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e859bb9c-e8e2-4221-a6aa-4f56545c70c6-kserve-provision-location\") pod \"e859bb9c-e8e2-4221-a6aa-4f56545c70c6\" (UID: \"e859bb9c-e8e2-4221-a6aa-4f56545c70c6\") "
Oct 30 03:34:01.283185 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:34:01.282928    4461 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e859bb9c-e8e2-4221-a6aa-4f56545c70c6-tls-certs\" (UniqueName: \"kubernetes.io/secret/e859bb9c-e8e2-4221-a6aa-4f56545c70c6-tls-certs\") pod \"e859bb9c-e8e2-4221-a6aa-4f56545c70c6\" (UID: \"e859bb9c-e8e2-4221-a6aa-4f56545c70c6\") "
Oct 30 03:34:01.283185 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:34:01.283078    4461 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e859bb9c-e8e2-4221-a6aa-4f56545c70c6-model-cache" pod "e859bb9c-e8e2-4221-a6aa-4f56545c70c6" (UID: "e859bb9c-e8e2-4221-a6aa-4f56545c70c6"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:34:01.283185 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:34:01.283163    4461 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e859bb9c-e8e2-4221-a6aa-4f56545c70c6-tmp-dir" pod "e859bb9c-e8e2-4221-a6aa-4f56545c70c6" (UID: "e859bb9c-e8e2-4221-a6aa-4f56545c70c6"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:34:01.283359 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:34:01.283183    4461 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e859bb9c-e8e2-4221-a6aa-4f56545c70c6-model-cache\") on node \"ip-10-0-143-156.ec2.internal\" DevicePath \"\""
Oct 30 03:34:01.283359 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:34:01.283191    4461 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e859bb9c-e8e2-4221-a6aa-4f56545c70c6-cabundle-cert" pod "e859bb9c-e8e2-4221-a6aa-4f56545c70c6" (UID: "e859bb9c-e8e2-4221-a6aa-4f56545c70c6"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:34:01.283359 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:34:01.283288    4461 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e859bb9c-e8e2-4221-a6aa-4f56545c70c6-home" pod "e859bb9c-e8e2-4221-a6aa-4f56545c70c6" (UID: "e859bb9c-e8e2-4221-a6aa-4f56545c70c6"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:34:01.285280 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:34:01.285254    4461 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e859bb9c-e8e2-4221-a6aa-4f56545c70c6-tls-certs" pod "e859bb9c-e8e2-4221-a6aa-4f56545c70c6" (UID: "e859bb9c-e8e2-4221-a6aa-4f56545c70c6"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:34:01.285428 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:34:01.285402    4461 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e859bb9c-e8e2-4221-a6aa-4f56545c70c6-dshm" pod "e859bb9c-e8e2-4221-a6aa-4f56545c70c6" (UID: "e859bb9c-e8e2-4221-a6aa-4f56545c70c6"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:34:01.285525 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:34:01.285487    4461 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e859bb9c-e8e2-4221-a6aa-4f56545c70c6-kube-api-access-9wfbk" pod "e859bb9c-e8e2-4221-a6aa-4f56545c70c6" (UID: "e859bb9c-e8e2-4221-a6aa-4f56545c70c6"). InnerVolumeSpecName "kube-api-access-9wfbk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:34:01.309237 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:34:01.309175    4461 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e859bb9c-e8e2-4221-a6aa-4f56545c70c6-kserve-provision-location" pod "e859bb9c-e8e2-4221-a6aa-4f56545c70c6" (UID: "e859bb9c-e8e2-4221-a6aa-4f56545c70c6"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:34:01.384584 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:34:01.384563    4461 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e859bb9c-e8e2-4221-a6aa-4f56545c70c6-cabundle-cert\") on node \"ip-10-0-143-156.ec2.internal\" DevicePath \"\""
Oct 30 03:34:01.384716 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:34:01.384586    4461 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9wfbk\" (UniqueName: \"kubernetes.io/projected/e859bb9c-e8e2-4221-a6aa-4f56545c70c6-kube-api-access-9wfbk\") on node \"ip-10-0-143-156.ec2.internal\" DevicePath \"\""
Oct 30 03:34:01.384716 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:34:01.384602    4461 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e859bb9c-e8e2-4221-a6aa-4f56545c70c6-tmp-dir\") on node \"ip-10-0-143-156.ec2.internal\" DevicePath \"\""
Oct 30 03:34:01.384716 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:34:01.384613    4461 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e859bb9c-e8e2-4221-a6aa-4f56545c70c6-home\") on node \"ip-10-0-143-156.ec2.internal\" DevicePath \"\""
Oct 30 03:34:01.384716 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:34:01.384625    4461 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e859bb9c-e8e2-4221-a6aa-4f56545c70c6-kserve-provision-location\") on node \"ip-10-0-143-156.ec2.internal\" DevicePath \"\""
Oct 30 03:34:01.384716 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:34:01.384636    4461 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e859bb9c-e8e2-4221-a6aa-4f56545c70c6-tls-certs\") on node \"ip-10-0-143-156.ec2.internal\" DevicePath \"\""
Oct 30 03:34:01.384716 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:34:01.384647    4461 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e859bb9c-e8e2-4221-a6aa-4f56545c70c6-dshm\") on node \"ip-10-0-143-156.ec2.internal\" DevicePath \"\""
Oct 30 03:34:01.553714 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:34:01.553684    4461 generic.go:356] "Generic (PLEG): container finished" podID="e859bb9c-e8e2-4221-a6aa-4f56545c70c6" containerID="9492954ddcd48f4850d37e23e3a34cae2fc639132464720ae11e0b7f17e55629" exitCode=0
Oct 30 03:34:01.554133 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:34:01.553734    4461 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-846f48577-wlt5j"
Oct 30 03:34:01.554133 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:34:01.553741    4461 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-846f48577-wlt5j" event={"ID":"e859bb9c-e8e2-4221-a6aa-4f56545c70c6","Type":"ContainerDied","Data":"9492954ddcd48f4850d37e23e3a34cae2fc639132464720ae11e0b7f17e55629"}
Oct 30 03:34:01.554133 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:34:01.553781    4461 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-846f48577-wlt5j" event={"ID":"e859bb9c-e8e2-4221-a6aa-4f56545c70c6","Type":"ContainerDied","Data":"1085bcb20c8d196aa9cf5217c9518a98dd6d6d48c3e0fb0f4de4e49b23bfc614"}
Oct 30 03:34:01.554133 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:34:01.553803    4461 scope.go:122] "RemoveContainer" containerID="9492954ddcd48f4850d37e23e3a34cae2fc639132464720ae11e0b7f17e55629"
Oct 30 03:34:01.563927 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:34:01.563910    4461 scope.go:122] "RemoveContainer" containerID="08862268ab730e180b0755ea12f8eb9214187e8164c0faa7f75e6eb6729d1112"
Oct 30 03:34:01.571390 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:34:01.571375    4461 scope.go:122] "RemoveContainer" containerID="9492954ddcd48f4850d37e23e3a34cae2fc639132464720ae11e0b7f17e55629"
Oct 30 03:34:01.571639 ip-10-0-143-156 kubenswrapper[4461]: E1030 03:34:01.571619    4461 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9492954ddcd48f4850d37e23e3a34cae2fc639132464720ae11e0b7f17e55629\": container with ID starting with 9492954ddcd48f4850d37e23e3a34cae2fc639132464720ae11e0b7f17e55629 not found: ID does not exist" containerID="9492954ddcd48f4850d37e23e3a34cae2fc639132464720ae11e0b7f17e55629"
Oct 30 03:34:01.571720 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:34:01.571650    4461 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9492954ddcd48f4850d37e23e3a34cae2fc639132464720ae11e0b7f17e55629"} err="failed to get container status \"9492954ddcd48f4850d37e23e3a34cae2fc639132464720ae11e0b7f17e55629\": rpc error: code = NotFound desc = could not find container \"9492954ddcd48f4850d37e23e3a34cae2fc639132464720ae11e0b7f17e55629\": container with ID starting with 9492954ddcd48f4850d37e23e3a34cae2fc639132464720ae11e0b7f17e55629 not found: ID does not exist"
Oct 30 03:34:01.571720 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:34:01.571674    4461 scope.go:122] "RemoveContainer" containerID="08862268ab730e180b0755ea12f8eb9214187e8164c0faa7f75e6eb6729d1112"
Oct 30 03:34:01.571928 ip-10-0-143-156 kubenswrapper[4461]: E1030 03:34:01.571911    4461 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"08862268ab730e180b0755ea12f8eb9214187e8164c0faa7f75e6eb6729d1112\": container with ID starting with 08862268ab730e180b0755ea12f8eb9214187e8164c0faa7f75e6eb6729d1112 not found: ID does not exist" containerID="08862268ab730e180b0755ea12f8eb9214187e8164c0faa7f75e6eb6729d1112"
Oct 30 03:34:01.571975 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:34:01.571935    4461 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"08862268ab730e180b0755ea12f8eb9214187e8164c0faa7f75e6eb6729d1112"} err="failed to get container status \"08862268ab730e180b0755ea12f8eb9214187e8164c0faa7f75e6eb6729d1112\": rpc error: code = NotFound desc = could not find container \"08862268ab730e180b0755ea12f8eb9214187e8164c0faa7f75e6eb6729d1112\": container with ID starting with 08862268ab730e180b0755ea12f8eb9214187e8164c0faa7f75e6eb6729d1112 not found: ID does not exist"
Oct 30 03:34:01.606514 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:34:01.606491    4461 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-846f48577-wlt5j"]
Oct 30 03:34:01.619128 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:34:01.619092    4461 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-846f48577-wlt5j"]
Oct 30 03:34:02.114006 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:34:02.113973    4461 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e859bb9c-e8e2-4221-a6aa-4f56545c70c6" path="/var/lib/kubelet/pods/e859bb9c-e8e2-4221-a6aa-4f56545c70c6/volumes"
Oct 30 03:34:05.370578 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:34:05.370540    4461 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-84d6588cjcp5x" podUID="75002685-ae4a-4df3-a4ec-e51e3ec1dfea" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 03:34:08.053008 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:34:08.052972    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:34:08.245713 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:34:08.245674    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:34:15.369985 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:34:15.369946    4461 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-84d6588cjcp5x" podUID="75002685-ae4a-4df3-a4ec-e51e3ec1dfea" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 03:34:15.459007 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:34:15.458971    4461 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-84d6588cjcp5x" podUID="75002685-ae4a-4df3-a4ec-e51e3ec1dfea" containerName="main" containerID="cri-o://5d37fa8eb9fd27546dbeca7bbda133eca033fd93507ff285c2e68cc807269d33" gracePeriod=45
Oct 30 03:34:24.504941 ip-10-0-143-156 kubenswrapper[4461]: E1030 03:34:24.504896    4461 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2k85b" containerName="extractor"
Oct 30 03:34:25.370653 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:34:25.370618    4461 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-84d6588cjcp5x" podUID="75002685-ae4a-4df3-a4ec-e51e3ec1dfea" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 03:34:25.370817 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:34:25.370728    4461 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-84d6588cjcp5x"
Oct 30 03:34:35.370195 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:34:35.370143    4461 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-84d6588cjcp5x" podUID="75002685-ae4a-4df3-a4ec-e51e3ec1dfea" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 03:34:45.369879 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:34:45.369841    4461 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-84d6588cjcp5x" podUID="75002685-ae4a-4df3-a4ec-e51e3ec1dfea" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 03:34:45.587643 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:34:45.587599    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:34:45.702875 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:34:45.702849    4461 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-84d6588cjcp5x"
Oct 30 03:34:45.721827 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:34:45.721801    4461 generic.go:356] "Generic (PLEG): container finished" podID="75002685-ae4a-4df3-a4ec-e51e3ec1dfea" containerID="5d37fa8eb9fd27546dbeca7bbda133eca033fd93507ff285c2e68cc807269d33" exitCode=0
Oct 30 03:34:45.721932 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:34:45.721841    4461 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-84d6588cjcp5x"
Oct 30 03:34:45.721932 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:34:45.721860    4461 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-84d6588cjcp5x" event={"ID":"75002685-ae4a-4df3-a4ec-e51e3ec1dfea","Type":"ContainerDied","Data":"5d37fa8eb9fd27546dbeca7bbda133eca033fd93507ff285c2e68cc807269d33"}
Oct 30 03:34:45.721932 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:34:45.721900    4461 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-84d6588cjcp5x" event={"ID":"75002685-ae4a-4df3-a4ec-e51e3ec1dfea","Type":"ContainerDied","Data":"fd5957f06dce6ac61c2cc71fd40022d8173d3a344d52585117a34a5d2bc2c34b"}
Oct 30 03:34:45.721932 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:34:45.721920    4461 scope.go:122] "RemoveContainer" containerID="5d37fa8eb9fd27546dbeca7bbda133eca033fd93507ff285c2e68cc807269d33"
Oct 30 03:34:45.731380 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:34:45.731362    4461 scope.go:122] "RemoveContainer" containerID="5d37fa8eb9fd27546dbeca7bbda133eca033fd93507ff285c2e68cc807269d33"
Oct 30 03:34:45.731669 ip-10-0-143-156 kubenswrapper[4461]: E1030 03:34:45.731649    4461 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5d37fa8eb9fd27546dbeca7bbda133eca033fd93507ff285c2e68cc807269d33\": container with ID starting with 5d37fa8eb9fd27546dbeca7bbda133eca033fd93507ff285c2e68cc807269d33 not found: ID does not exist" containerID="5d37fa8eb9fd27546dbeca7bbda133eca033fd93507ff285c2e68cc807269d33"
Oct 30 03:34:45.731743 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:34:45.731677    4461 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5d37fa8eb9fd27546dbeca7bbda133eca033fd93507ff285c2e68cc807269d33"} err="failed to get container status \"5d37fa8eb9fd27546dbeca7bbda133eca033fd93507ff285c2e68cc807269d33\": rpc error: code = NotFound desc = could not find container \"5d37fa8eb9fd27546dbeca7bbda133eca033fd93507ff285c2e68cc807269d33\": container with ID starting with 5d37fa8eb9fd27546dbeca7bbda133eca033fd93507ff285c2e68cc807269d33 not found: ID does not exist"
Oct 30 03:34:45.755162 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:34:45.755140    4461 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/75002685-ae4a-4df3-a4ec-e51e3ec1dfea-kube-api-access-2dx5j\" (UniqueName: \"kubernetes.io/projected/75002685-ae4a-4df3-a4ec-e51e3ec1dfea-kube-api-access-2dx5j\") pod \"75002685-ae4a-4df3-a4ec-e51e3ec1dfea\" (UID: \"75002685-ae4a-4df3-a4ec-e51e3ec1dfea\") "
Oct 30 03:34:45.755252 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:34:45.755177    4461 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/75002685-ae4a-4df3-a4ec-e51e3ec1dfea-tls-certs\" (UniqueName: \"kubernetes.io/secret/75002685-ae4a-4df3-a4ec-e51e3ec1dfea-tls-certs\") pod \"75002685-ae4a-4df3-a4ec-e51e3ec1dfea\" (UID: \"75002685-ae4a-4df3-a4ec-e51e3ec1dfea\") "
Oct 30 03:34:45.757203 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:34:45.757175    4461 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/75002685-ae4a-4df3-a4ec-e51e3ec1dfea-tls-certs" pod "75002685-ae4a-4df3-a4ec-e51e3ec1dfea" (UID: "75002685-ae4a-4df3-a4ec-e51e3ec1dfea"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:34:45.757304 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:34:45.757265    4461 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/75002685-ae4a-4df3-a4ec-e51e3ec1dfea-kube-api-access-2dx5j" pod "75002685-ae4a-4df3-a4ec-e51e3ec1dfea" (UID: "75002685-ae4a-4df3-a4ec-e51e3ec1dfea"). InnerVolumeSpecName "kube-api-access-2dx5j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:34:45.855724 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:34:45.855695    4461 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2dx5j\" (UniqueName: \"kubernetes.io/projected/75002685-ae4a-4df3-a4ec-e51e3ec1dfea-kube-api-access-2dx5j\") on node \"ip-10-0-143-156.ec2.internal\" DevicePath \"\""
Oct 30 03:34:45.855724 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:34:45.855721    4461 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/75002685-ae4a-4df3-a4ec-e51e3ec1dfea-tls-certs\") on node \"ip-10-0-143-156.ec2.internal\" DevicePath \"\""
Oct 30 03:34:46.066263 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:34:46.066232    4461 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-84d6588cjcp5x"]
Oct 30 03:34:46.078943 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:34:46.078915    4461 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-84d6588cjcp5x"]
Oct 30 03:34:46.114327 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:34:46.114298    4461 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="75002685-ae4a-4df3-a4ec-e51e3ec1dfea" path="/var/lib/kubelet/pods/75002685-ae4a-4df3-a4ec-e51e3ec1dfea/volumes"
Oct 30 03:34:50.691966 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:34:50.691893    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:34:50.872396 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:34:50.872360    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:34:54.503825 ip-10-0-143-156 kubenswrapper[4461]: E1030 03:34:54.503788    4461 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2k85b" containerName="extractor"
Oct 30 03:34:55.571414 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:34:55.571383    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:34:55.700538 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:34:55.700507    4461 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-bd846f858-gwv2g"]
Oct 30 03:34:55.700808 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:34:55.700795    4461 cpu_manager.go:414] "containerMap: removing container" podUID="e859bb9c-e8e2-4221-a6aa-4f56545c70c6" containerName="main"
Oct 30 03:34:55.700850 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:34:55.700809    4461 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e859bb9c-e8e2-4221-a6aa-4f56545c70c6" containerName="main"
Oct 30 03:34:55.700850 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:34:55.700820    4461 cpu_manager.go:414] "containerMap: removing container" podUID="75002685-ae4a-4df3-a4ec-e51e3ec1dfea" containerName="main"
Oct 30 03:34:55.700850 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:34:55.700825    4461 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="75002685-ae4a-4df3-a4ec-e51e3ec1dfea" containerName="main"
Oct 30 03:34:55.700850 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:34:55.700833    4461 cpu_manager.go:414] "containerMap: removing container" podUID="e859bb9c-e8e2-4221-a6aa-4f56545c70c6" containerName="storage-initializer"
Oct 30 03:34:55.700850 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:34:55.700837    4461 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e859bb9c-e8e2-4221-a6aa-4f56545c70c6" containerName="storage-initializer"
Oct 30 03:34:55.701031 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:34:55.700886    4461 memory_manager.go:365] "RemoveStaleState removing state" podUID="e859bb9c-e8e2-4221-a6aa-4f56545c70c6" containerName="main"
Oct 30 03:34:55.701031 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:34:55.700894    4461 memory_manager.go:365] "RemoveStaleState removing state" podUID="75002685-ae4a-4df3-a4ec-e51e3ec1dfea" containerName="main"
Oct 30 03:34:55.705192 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:34:55.705171    4461 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-bd846f858-gwv2g"
Oct 30 03:34:55.722271 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:34:55.722249    4461 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"tokenizer-migration-path-test-kserve-self-signed-certs\""
Oct 30 03:34:55.736044 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:34:55.736022    4461 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-bd846f858-gwv2g"]
Oct 30 03:34:55.791545 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:34:55.791510    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:34:55.804667 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:34:55.804641    4461 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"openshift-service-ca.crt\""
Oct 30 03:34:55.804898 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:34:55.804880    4461 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"odh-kserve-custom-ca-bundle\""
Oct 30 03:34:55.804946 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:34:55.804881    4461 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"kube-root-ca.crt\""
Oct 30 03:34:55.804946 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:34:55.804913    4461 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"seaweedfs-s3-creds\""
Oct 30 03:34:55.805027 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:34:55.805001    4461 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"default-dockercfg-tplmt\""
Oct 30 03:34:55.839593 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:34:55.839525    4461 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/cb97f849-7c86-4f17-978c-f8e77298c1aa-cabundle-cert\") pod \"tokenizer-migration-path-test-kserve-bd846f858-gwv2g\" (UID: \"cb97f849-7c86-4f17-978c-f8e77298c1aa\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-bd846f858-gwv2g"
Oct 30 03:34:55.839593 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:34:55.839569    4461 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cb97f849-7c86-4f17-978c-f8e77298c1aa-kserve-provision-location\") pod \"tokenizer-migration-path-test-kserve-bd846f858-gwv2g\" (UID: \"cb97f849-7c86-4f17-978c-f8e77298c1aa\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-bd846f858-gwv2g"
Oct 30 03:34:55.839593 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:34:55.839588    4461 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j79rq\" (UniqueName: \"kubernetes.io/projected/cb97f849-7c86-4f17-978c-f8e77298c1aa-kube-api-access-j79rq\") pod \"tokenizer-migration-path-test-kserve-bd846f858-gwv2g\" (UID: \"cb97f849-7c86-4f17-978c-f8e77298c1aa\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-bd846f858-gwv2g"
Oct 30 03:34:55.839776 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:34:55.839729    4461 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cb97f849-7c86-4f17-978c-f8e77298c1aa-home\") pod \"tokenizer-migration-path-test-kserve-bd846f858-gwv2g\" (UID: \"cb97f849-7c86-4f17-978c-f8e77298c1aa\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-bd846f858-gwv2g"
Oct 30 03:34:55.839776 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:34:55.839758    4461 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cb97f849-7c86-4f17-978c-f8e77298c1aa-tmp-dir\") pod \"tokenizer-migration-path-test-kserve-bd846f858-gwv2g\" (UID: \"cb97f849-7c86-4f17-978c-f8e77298c1aa\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-bd846f858-gwv2g"
Oct 30 03:34:55.839851 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:34:55.839787    4461 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cb97f849-7c86-4f17-978c-f8e77298c1aa-model-cache\") pod \"tokenizer-migration-path-test-kserve-bd846f858-gwv2g\" (UID: \"cb97f849-7c86-4f17-978c-f8e77298c1aa\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-bd846f858-gwv2g"
Oct 30 03:34:55.839905 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:34:55.839888    4461 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cb97f849-7c86-4f17-978c-f8e77298c1aa-dshm\") pod \"tokenizer-migration-path-test-kserve-bd846f858-gwv2g\" (UID: \"cb97f849-7c86-4f17-978c-f8e77298c1aa\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-bd846f858-gwv2g"
Oct 30 03:34:55.839942 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:34:55.839920    4461 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cb97f849-7c86-4f17-978c-f8e77298c1aa-tls-certs\") pod \"tokenizer-migration-path-test-kserve-bd846f858-gwv2g\" (UID: \"cb97f849-7c86-4f17-978c-f8e77298c1aa\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-bd846f858-gwv2g"
Oct 30 03:34:55.941027 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:34:55.940994    4461 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cb97f849-7c86-4f17-978c-f8e77298c1aa-home\") pod \"tokenizer-migration-path-test-kserve-bd846f858-gwv2g\" (UID: \"cb97f849-7c86-4f17-978c-f8e77298c1aa\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-bd846f858-gwv2g"
Oct 30 03:34:55.941027 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:34:55.941028    4461 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cb97f849-7c86-4f17-978c-f8e77298c1aa-tmp-dir\") pod \"tokenizer-migration-path-test-kserve-bd846f858-gwv2g\" (UID: \"cb97f849-7c86-4f17-978c-f8e77298c1aa\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-bd846f858-gwv2g"
Oct 30 03:34:55.941249 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:34:55.941052    4461 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cb97f849-7c86-4f17-978c-f8e77298c1aa-model-cache\") pod \"tokenizer-migration-path-test-kserve-bd846f858-gwv2g\" (UID: \"cb97f849-7c86-4f17-978c-f8e77298c1aa\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-bd846f858-gwv2g"
Oct 30 03:34:55.941249 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:34:55.941073    4461 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cb97f849-7c86-4f17-978c-f8e77298c1aa-dshm\") pod \"tokenizer-migration-path-test-kserve-bd846f858-gwv2g\" (UID: \"cb97f849-7c86-4f17-978c-f8e77298c1aa\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-bd846f858-gwv2g"
Oct 30 03:34:55.941249 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:34:55.941139    4461 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cb97f849-7c86-4f17-978c-f8e77298c1aa-tls-certs\") pod \"tokenizer-migration-path-test-kserve-bd846f858-gwv2g\" (UID: \"cb97f849-7c86-4f17-978c-f8e77298c1aa\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-bd846f858-gwv2g"
Oct 30 03:34:55.941249 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:34:55.941187    4461 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/cb97f849-7c86-4f17-978c-f8e77298c1aa-cabundle-cert\") pod \"tokenizer-migration-path-test-kserve-bd846f858-gwv2g\" (UID: \"cb97f849-7c86-4f17-978c-f8e77298c1aa\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-bd846f858-gwv2g"
Oct 30 03:34:55.941249 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:34:55.941237    4461 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cb97f849-7c86-4f17-978c-f8e77298c1aa-kserve-provision-location\") pod \"tokenizer-migration-path-test-kserve-bd846f858-gwv2g\" (UID: \"cb97f849-7c86-4f17-978c-f8e77298c1aa\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-bd846f858-gwv2g"
Oct 30 03:34:55.941498 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:34:55.941259    4461 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j79rq\" (UniqueName: \"kubernetes.io/projected/cb97f849-7c86-4f17-978c-f8e77298c1aa-kube-api-access-j79rq\") pod \"tokenizer-migration-path-test-kserve-bd846f858-gwv2g\" (UID: \"cb97f849-7c86-4f17-978c-f8e77298c1aa\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-bd846f858-gwv2g"
Oct 30 03:34:55.941498 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:34:55.941455    4461 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cb97f849-7c86-4f17-978c-f8e77298c1aa-tmp-dir\") pod \"tokenizer-migration-path-test-kserve-bd846f858-gwv2g\" (UID: \"cb97f849-7c86-4f17-978c-f8e77298c1aa\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-bd846f858-gwv2g"
Oct 30 03:34:55.941617 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:34:55.941529    4461 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cb97f849-7c86-4f17-978c-f8e77298c1aa-model-cache\") pod \"tokenizer-migration-path-test-kserve-bd846f858-gwv2g\" (UID: \"cb97f849-7c86-4f17-978c-f8e77298c1aa\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-bd846f858-gwv2g"
Oct 30 03:34:55.941617 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:34:55.941562    4461 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cb97f849-7c86-4f17-978c-f8e77298c1aa-home\") pod \"tokenizer-migration-path-test-kserve-bd846f858-gwv2g\" (UID: \"cb97f849-7c86-4f17-978c-f8e77298c1aa\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-bd846f858-gwv2g"
Oct 30 03:34:55.941725 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:34:55.941681    4461 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cb97f849-7c86-4f17-978c-f8e77298c1aa-kserve-provision-location\") pod \"tokenizer-migration-path-test-kserve-bd846f858-gwv2g\" (UID: \"cb97f849-7c86-4f17-978c-f8e77298c1aa\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-bd846f858-gwv2g"
Oct 30 03:34:55.941968 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:34:55.941945    4461 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/cb97f849-7c86-4f17-978c-f8e77298c1aa-cabundle-cert\") pod \"tokenizer-migration-path-test-kserve-bd846f858-gwv2g\" (UID: \"cb97f849-7c86-4f17-978c-f8e77298c1aa\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-bd846f858-gwv2g"
Oct 30 03:34:55.943358 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:34:55.943340    4461 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cb97f849-7c86-4f17-978c-f8e77298c1aa-dshm\") pod \"tokenizer-migration-path-test-kserve-bd846f858-gwv2g\" (UID: \"cb97f849-7c86-4f17-978c-f8e77298c1aa\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-bd846f858-gwv2g"
Oct 30 03:34:55.943682 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:34:55.943663    4461 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cb97f849-7c86-4f17-978c-f8e77298c1aa-tls-certs\") pod \"tokenizer-migration-path-test-kserve-bd846f858-gwv2g\" (UID: \"cb97f849-7c86-4f17-978c-f8e77298c1aa\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-bd846f858-gwv2g"
Oct 30 03:34:55.970660 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:34:55.970634    4461 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j79rq\" (UniqueName: \"kubernetes.io/projected/cb97f849-7c86-4f17-978c-f8e77298c1aa-kube-api-access-j79rq\") pod \"tokenizer-migration-path-test-kserve-bd846f858-gwv2g\" (UID: \"cb97f849-7c86-4f17-978c-f8e77298c1aa\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-bd846f858-gwv2g"
Oct 30 03:34:56.020953 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:34:56.020895    4461 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-bd846f858-gwv2g"
Oct 30 03:34:56.195418 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:34:56.195331    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:34:56.341298 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:34:56.341266    4461 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-bd846f858-gwv2g"]
Oct 30 03:34:56.345371 ip-10-0-143-156 kubenswrapper[4461]: W1030 03:34:56.345343    4461 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb97f849_7c86_4f17_978c_f8e77298c1aa.slice/crio-2493ed7a1f92d2c81412ef8f8cbe5e9404cf757425a60cc97689962e79abfac6 WatchSource:0}: Error finding container 2493ed7a1f92d2c81412ef8f8cbe5e9404cf757425a60cc97689962e79abfac6: Status 404 returned error can't find the container with id 2493ed7a1f92d2c81412ef8f8cbe5e9404cf757425a60cc97689962e79abfac6
Oct 30 03:34:56.366502 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:34:56.366477    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:34:56.762551 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:34:56.762517    4461 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-bd846f858-gwv2g" event={"ID":"cb97f849-7c86-4f17-978c-f8e77298c1aa","Type":"ContainerStarted","Data":"090b723af52947bb7662da589d7ac86c493b7fe40c3d6dba0b8a6ae4ef4d849d"}
Oct 30 03:34:56.762930 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:34:56.762571    4461 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-bd846f858-gwv2g" event={"ID":"cb97f849-7c86-4f17-978c-f8e77298c1aa","Type":"ContainerStarted","Data":"2493ed7a1f92d2c81412ef8f8cbe5e9404cf757425a60cc97689962e79abfac6"}
Oct 30 03:34:58.123294 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:34:58.123260    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:34:58.298430 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:34:58.298372    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:34:58.574396 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:34:58.574363    4461 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-854b944bd8-mbwh6"]
Oct 30 03:34:58.577580 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:34:58.577555    4461 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-854b944bd8-mbwh6"
Oct 30 03:34:58.593580 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:34:58.593536    4461 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-6f0aa991\"/\"openshift-service-ca.crt\""
Oct 30 03:34:58.593580 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:34:58.593549    4461 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-6f0aa991\"/\"router-gateway-1-openshift-default-dockercfg-q5ffm\""
Oct 30 03:34:58.593580 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:34:58.593542    4461 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-6f0aa991\"/\"openshift-gw-ca-root-cert\""
Oct 30 03:34:58.593827 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:34:58.593584    4461 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-6f0aa991\"/\"kube-root-ca.crt\""
Oct 30 03:34:58.611053 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:34:58.611026    4461 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-854b944bd8-mbwh6"]
Oct 30 03:34:58.665401 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:34:58.665365    4461 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/06762940-e16c-4ab3-b9c6-b65345ba80a1-istio-envoy\") pod \"router-gateway-1-openshift-default-854b944bd8-mbwh6\" (UID: \"06762940-e16c-4ab3-b9c6-b65345ba80a1\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-854b944bd8-mbwh6"
Oct 30 03:34:58.665562 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:34:58.665407    4461 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/06762940-e16c-4ab3-b9c6-b65345ba80a1-istio-podinfo\") pod \"router-gateway-1-openshift-default-854b944bd8-mbwh6\" (UID: \"06762940-e16c-4ab3-b9c6-b65345ba80a1\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-854b944bd8-mbwh6"
Oct 30 03:34:58.665562 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:34:58.665485    4461 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/06762940-e16c-4ab3-b9c6-b65345ba80a1-istio-token\") pod \"router-gateway-1-openshift-default-854b944bd8-mbwh6\" (UID: \"06762940-e16c-4ab3-b9c6-b65345ba80a1\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-854b944bd8-mbwh6"
Oct 30 03:34:58.665686 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:34:58.665570    4461 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/06762940-e16c-4ab3-b9c6-b65345ba80a1-istio-data\") pod \"router-gateway-1-openshift-default-854b944bd8-mbwh6\" (UID: \"06762940-e16c-4ab3-b9c6-b65345ba80a1\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-854b944bd8-mbwh6"
Oct 30 03:34:58.665686 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:34:58.665598    4461 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/06762940-e16c-4ab3-b9c6-b65345ba80a1-credential-socket\") pod \"router-gateway-1-openshift-default-854b944bd8-mbwh6\" (UID: \"06762940-e16c-4ab3-b9c6-b65345ba80a1\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-854b944bd8-mbwh6"
Oct 30 03:34:58.665686 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:34:58.665660    4461 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d7sl4\" (UniqueName: \"kubernetes.io/projected/06762940-e16c-4ab3-b9c6-b65345ba80a1-kube-api-access-d7sl4\") pod \"router-gateway-1-openshift-default-854b944bd8-mbwh6\" (UID: \"06762940-e16c-4ab3-b9c6-b65345ba80a1\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-854b944bd8-mbwh6"
Oct 30 03:34:58.665813 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:34:58.665699    4461 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/06762940-e16c-4ab3-b9c6-b65345ba80a1-workload-socket\") pod \"router-gateway-1-openshift-default-854b944bd8-mbwh6\" (UID: \"06762940-e16c-4ab3-b9c6-b65345ba80a1\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-854b944bd8-mbwh6"
Oct 30 03:34:58.665813 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:34:58.665732    4461 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/06762940-e16c-4ab3-b9c6-b65345ba80a1-workload-certs\") pod \"router-gateway-1-openshift-default-854b944bd8-mbwh6\" (UID: \"06762940-e16c-4ab3-b9c6-b65345ba80a1\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-854b944bd8-mbwh6"
Oct 30 03:34:58.665914 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:34:58.665838    4461 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/06762940-e16c-4ab3-b9c6-b65345ba80a1-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-854b944bd8-mbwh6\" (UID: \"06762940-e16c-4ab3-b9c6-b65345ba80a1\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-854b944bd8-mbwh6"
Oct 30 03:34:58.713909 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:34:58.713874    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:34:58.766625 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:34:58.766538    4461 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/06762940-e16c-4ab3-b9c6-b65345ba80a1-workload-socket\") pod \"router-gateway-1-openshift-default-854b944bd8-mbwh6\" (UID: \"06762940-e16c-4ab3-b9c6-b65345ba80a1\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-854b944bd8-mbwh6"
Oct 30 03:34:58.766625 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:34:58.766574    4461 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/06762940-e16c-4ab3-b9c6-b65345ba80a1-workload-certs\") pod \"router-gateway-1-openshift-default-854b944bd8-mbwh6\" (UID: \"06762940-e16c-4ab3-b9c6-b65345ba80a1\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-854b944bd8-mbwh6"
Oct 30 03:34:58.766625 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:34:58.766599    4461 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/06762940-e16c-4ab3-b9c6-b65345ba80a1-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-854b944bd8-mbwh6\" (UID: \"06762940-e16c-4ab3-b9c6-b65345ba80a1\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-854b944bd8-mbwh6"
Oct 30 03:34:58.766909 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:34:58.766643    4461 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/06762940-e16c-4ab3-b9c6-b65345ba80a1-istio-envoy\") pod \"router-gateway-1-openshift-default-854b944bd8-mbwh6\" (UID: \"06762940-e16c-4ab3-b9c6-b65345ba80a1\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-854b944bd8-mbwh6"
Oct 30 03:34:58.766909 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:34:58.766666    4461 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/06762940-e16c-4ab3-b9c6-b65345ba80a1-istio-podinfo\") pod \"router-gateway-1-openshift-default-854b944bd8-mbwh6\" (UID: \"06762940-e16c-4ab3-b9c6-b65345ba80a1\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-854b944bd8-mbwh6"
Oct 30 03:34:58.766909 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:34:58.766698    4461 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/06762940-e16c-4ab3-b9c6-b65345ba80a1-istio-token\") pod \"router-gateway-1-openshift-default-854b944bd8-mbwh6\" (UID: \"06762940-e16c-4ab3-b9c6-b65345ba80a1\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-854b944bd8-mbwh6"
Oct 30 03:34:58.766909 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:34:58.766752    4461 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/06762940-e16c-4ab3-b9c6-b65345ba80a1-istio-data\") pod \"router-gateway-1-openshift-default-854b944bd8-mbwh6\" (UID: \"06762940-e16c-4ab3-b9c6-b65345ba80a1\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-854b944bd8-mbwh6"
Oct 30 03:34:58.766909 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:34:58.766780    4461 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/06762940-e16c-4ab3-b9c6-b65345ba80a1-credential-socket\") pod \"router-gateway-1-openshift-default-854b944bd8-mbwh6\" (UID: \"06762940-e16c-4ab3-b9c6-b65345ba80a1\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-854b944bd8-mbwh6"
Oct 30 03:34:58.766909 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:34:58.766802    4461 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d7sl4\" (UniqueName: \"kubernetes.io/projected/06762940-e16c-4ab3-b9c6-b65345ba80a1-kube-api-access-d7sl4\") pod \"router-gateway-1-openshift-default-854b944bd8-mbwh6\" (UID: \"06762940-e16c-4ab3-b9c6-b65345ba80a1\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-854b944bd8-mbwh6"
Oct 30 03:34:58.767234 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:34:58.767078    4461 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/06762940-e16c-4ab3-b9c6-b65345ba80a1-workload-certs\") pod \"router-gateway-1-openshift-default-854b944bd8-mbwh6\" (UID: \"06762940-e16c-4ab3-b9c6-b65345ba80a1\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-854b944bd8-mbwh6"
Oct 30 03:34:58.767415 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:34:58.767298    4461 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/06762940-e16c-4ab3-b9c6-b65345ba80a1-istio-data\") pod \"router-gateway-1-openshift-default-854b944bd8-mbwh6\" (UID: \"06762940-e16c-4ab3-b9c6-b65345ba80a1\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-854b944bd8-mbwh6"
Oct 30 03:34:58.767415 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:34:58.767333    4461 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/06762940-e16c-4ab3-b9c6-b65345ba80a1-workload-socket\") pod \"router-gateway-1-openshift-default-854b944bd8-mbwh6\" (UID: \"06762940-e16c-4ab3-b9c6-b65345ba80a1\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-854b944bd8-mbwh6"
Oct 30 03:34:58.767415 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:34:58.767333    4461 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/06762940-e16c-4ab3-b9c6-b65345ba80a1-credential-socket\") pod \"router-gateway-1-openshift-default-854b944bd8-mbwh6\" (UID: \"06762940-e16c-4ab3-b9c6-b65345ba80a1\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-854b944bd8-mbwh6"
Oct 30 03:34:58.767634 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:34:58.767450    4461 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/06762940-e16c-4ab3-b9c6-b65345ba80a1-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-854b944bd8-mbwh6\" (UID: \"06762940-e16c-4ab3-b9c6-b65345ba80a1\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-854b944bd8-mbwh6"
Oct 30 03:34:58.769797 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:34:58.769773    4461 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/06762940-e16c-4ab3-b9c6-b65345ba80a1-istio-envoy\") pod \"router-gateway-1-openshift-default-854b944bd8-mbwh6\" (UID: \"06762940-e16c-4ab3-b9c6-b65345ba80a1\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-854b944bd8-mbwh6"
Oct 30 03:34:58.769973 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:34:58.769953    4461 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/06762940-e16c-4ab3-b9c6-b65345ba80a1-istio-podinfo\") pod \"router-gateway-1-openshift-default-854b944bd8-mbwh6\" (UID: \"06762940-e16c-4ab3-b9c6-b65345ba80a1\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-854b944bd8-mbwh6"
Oct 30 03:34:58.780035 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:34:58.780005    4461 generic.go:356] "Generic (PLEG): container finished" podID="cb97f849-7c86-4f17-978c-f8e77298c1aa" containerID="090b723af52947bb7662da589d7ac86c493b7fe40c3d6dba0b8a6ae4ef4d849d" exitCode=0
Oct 30 03:34:58.780172 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:34:58.780061    4461 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-bd846f858-gwv2g" event={"ID":"cb97f849-7c86-4f17-978c-f8e77298c1aa","Type":"ContainerDied","Data":"090b723af52947bb7662da589d7ac86c493b7fe40c3d6dba0b8a6ae4ef4d849d"}
Oct 30 03:34:58.790996 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:34:58.790973    4461 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/06762940-e16c-4ab3-b9c6-b65345ba80a1-istio-token\") pod \"router-gateway-1-openshift-default-854b944bd8-mbwh6\" (UID: \"06762940-e16c-4ab3-b9c6-b65345ba80a1\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-854b944bd8-mbwh6"
Oct 30 03:34:58.791152 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:34:58.791133    4461 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d7sl4\" (UniqueName: \"kubernetes.io/projected/06762940-e16c-4ab3-b9c6-b65345ba80a1-kube-api-access-d7sl4\") pod \"router-gateway-1-openshift-default-854b944bd8-mbwh6\" (UID: \"06762940-e16c-4ab3-b9c6-b65345ba80a1\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-854b944bd8-mbwh6"
Oct 30 03:34:58.884960 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:34:58.884925    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:34:58.894913 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:34:58.894885    4461 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-854b944bd8-mbwh6"
Oct 30 03:34:59.206949 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:34:59.206866    4461 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-854b944bd8-mbwh6"]
Oct 30 03:34:59.209645 ip-10-0-143-156 kubenswrapper[4461]: W1030 03:34:59.209611    4461 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod06762940_e16c_4ab3_b9c6_b65345ba80a1.slice/crio-0f6e8ebd8772e5345e44d36229391b5093a890ec3a55d4388d210a1e4e9be60e WatchSource:0}: Error finding container 0f6e8ebd8772e5345e44d36229391b5093a890ec3a55d4388d210a1e4e9be60e: Status 404 returned error can't find the container with id 0f6e8ebd8772e5345e44d36229391b5093a890ec3a55d4388d210a1e4e9be60e
Oct 30 03:34:59.211710 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:34:59.211674    4461 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 03:34:59.211797 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:34:59.211749    4461 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 03:34:59.211797 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:34:59.211780    4461 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 03:34:59.786064 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:34:59.786028    4461 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-bd846f858-gwv2g" event={"ID":"cb97f849-7c86-4f17-978c-f8e77298c1aa","Type":"ContainerStarted","Data":"bb40283fd8cc8588057039a772a112328ea7e7e14665fc42fd5ae8c218407b22"}
Oct 30 03:34:59.787613 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:34:59.787589    4461 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-854b944bd8-mbwh6" event={"ID":"06762940-e16c-4ab3-b9c6-b65345ba80a1","Type":"ContainerStarted","Data":"a309c1aec24f7b79e5fae550841f8da7167453ceda31b232a615a7aefa6a60a4"}
Oct 30 03:34:59.787737 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:34:59.787627    4461 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-854b944bd8-mbwh6" event={"ID":"06762940-e16c-4ab3-b9c6-b65345ba80a1","Type":"ContainerStarted","Data":"0f6e8ebd8772e5345e44d36229391b5093a890ec3a55d4388d210a1e4e9be60e"}
Oct 30 03:34:59.834233 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:34:59.834169    4461 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-bd846f858-gwv2g" podStartSLOduration=2.748908807 podStartE2EDuration="4.834155728s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.085246921s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:34:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:34:59.833347979 +0000 UTC m=+2367.883153183" watchObservedRunningTime="2026-10-30 03:34:59.834155728 +0000 UTC m=+2367.883960883"
Oct 30 03:34:59.881001 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:34:59.880955    4461 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-854b944bd8-mbwh6" podStartSLOduration=1.880942286 podStartE2EDuration="1.880942286s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:34:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:34:59.880509723 +0000 UTC m=+2367.930314899" watchObservedRunningTime="2026-10-30 03:34:59.880942286 +0000 UTC m=+2367.930747462"
Oct 30 03:34:59.895631 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:34:59.895606    4461 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-854b944bd8-mbwh6"
Oct 30 03:35:00.021476 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:35:00.021440    4461 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-bd846f858-gwv2g"
Oct 30 03:35:00.900054 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:35:00.900024    4461 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-854b944bd8-mbwh6"
Oct 30 03:35:01.796737 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:35:01.796706    4461 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-854b944bd8-mbwh6"
Oct 30 03:35:01.797462 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:35:01.797440    4461 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-854b944bd8-mbwh6"
Oct 30 03:35:02.107424 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:35:02.107351    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:35:02.736050 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:35:02.736015    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:35:04.555517 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:35:04.555482    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:35:04.722952 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:35:04.722913    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:35:05.190862 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:35:05.190827    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:35:06.021900 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:35:06.021862    4461 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-bd846f858-gwv2g"
Oct 30 03:35:06.034448 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:35:06.034424    4461 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-bd846f858-gwv2g"
Oct 30 03:35:06.826238 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:35:06.826204    4461 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-bd846f858-gwv2g"
Oct 30 03:35:06.926773 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:35:06.926737    4461 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5577b665bbml9xz"]
Oct 30 03:35:06.931004 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:35:06.930979    4461 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5577b665bbml9xz"
Oct 30 03:35:06.952401 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:35:06.952371    4461 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-6f0aa991\"/\"default-dockercfg-h6445\""
Oct 30 03:35:06.952531 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:35:06.952413    4461 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-6f0aa991\"/\"llmisv122f03714c5bdf915a2917fdf1262b98-kserve-self-signed-certs\""
Oct 30 03:35:06.953758 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:35:06.953736    4461 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5577b665bbml9xz"]
Oct 30 03:35:07.038878 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:35:07.038845    4461 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f36784f0-4807-48e8-8185-79bdec0fa85a-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5577b665bbml9xz\" (UID: \"f36784f0-4807-48e8-8185-79bdec0fa85a\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5577b665bbml9xz"
Oct 30 03:35:07.039295 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:35:07.038901    4461 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f36784f0-4807-48e8-8185-79bdec0fa85a-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5577b665bbml9xz\" (UID: \"f36784f0-4807-48e8-8185-79bdec0fa85a\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5577b665bbml9xz"
Oct 30 03:35:07.039295 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:35:07.038982    4461 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f36784f0-4807-48e8-8185-79bdec0fa85a-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5577b665bbml9xz\" (UID: \"f36784f0-4807-48e8-8185-79bdec0fa85a\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5577b665bbml9xz"
Oct 30 03:35:07.039295 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:35:07.039035    4461 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f36784f0-4807-48e8-8185-79bdec0fa85a-dshm\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5577b665bbml9xz\" (UID: \"f36784f0-4807-48e8-8185-79bdec0fa85a\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5577b665bbml9xz"
Oct 30 03:35:07.039295 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:35:07.039075    4461 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ffsz9\" (UniqueName: \"kubernetes.io/projected/f36784f0-4807-48e8-8185-79bdec0fa85a-kube-api-access-ffsz9\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5577b665bbml9xz\" (UID: \"f36784f0-4807-48e8-8185-79bdec0fa85a\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5577b665bbml9xz"
Oct 30 03:35:07.039295 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:35:07.039158    4461 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f36784f0-4807-48e8-8185-79bdec0fa85a-home\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5577b665bbml9xz\" (UID: \"f36784f0-4807-48e8-8185-79bdec0fa85a\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5577b665bbml9xz"
Oct 30 03:35:07.076866 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:35:07.076791    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:35:07.140472 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:35:07.140440    4461 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f36784f0-4807-48e8-8185-79bdec0fa85a-dshm\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5577b665bbml9xz\" (UID: \"f36784f0-4807-48e8-8185-79bdec0fa85a\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5577b665bbml9xz"
Oct 30 03:35:07.140622 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:35:07.140485    4461 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ffsz9\" (UniqueName: \"kubernetes.io/projected/f36784f0-4807-48e8-8185-79bdec0fa85a-kube-api-access-ffsz9\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5577b665bbml9xz\" (UID: \"f36784f0-4807-48e8-8185-79bdec0fa85a\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5577b665bbml9xz"
Oct 30 03:35:07.140622 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:35:07.140518    4461 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f36784f0-4807-48e8-8185-79bdec0fa85a-home\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5577b665bbml9xz\" (UID: \"f36784f0-4807-48e8-8185-79bdec0fa85a\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5577b665bbml9xz"
Oct 30 03:35:07.140622 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:35:07.140544    4461 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f36784f0-4807-48e8-8185-79bdec0fa85a-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5577b665bbml9xz\" (UID: \"f36784f0-4807-48e8-8185-79bdec0fa85a\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5577b665bbml9xz"
Oct 30 03:35:07.140622 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:35:07.140606    4461 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f36784f0-4807-48e8-8185-79bdec0fa85a-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5577b665bbml9xz\" (UID: \"f36784f0-4807-48e8-8185-79bdec0fa85a\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5577b665bbml9xz"
Oct 30 03:35:07.140782 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:35:07.140628    4461 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f36784f0-4807-48e8-8185-79bdec0fa85a-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5577b665bbml9xz\" (UID: \"f36784f0-4807-48e8-8185-79bdec0fa85a\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5577b665bbml9xz"
Oct 30 03:35:07.141001 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:35:07.140972    4461 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f36784f0-4807-48e8-8185-79bdec0fa85a-home\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5577b665bbml9xz\" (UID: \"f36784f0-4807-48e8-8185-79bdec0fa85a\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5577b665bbml9xz"
Oct 30 03:35:07.141060 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:35:07.141006    4461 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f36784f0-4807-48e8-8185-79bdec0fa85a-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5577b665bbml9xz\" (UID: \"f36784f0-4807-48e8-8185-79bdec0fa85a\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5577b665bbml9xz"
Oct 30 03:35:07.141060 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:35:07.141033    4461 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f36784f0-4807-48e8-8185-79bdec0fa85a-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5577b665bbml9xz\" (UID: \"f36784f0-4807-48e8-8185-79bdec0fa85a\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5577b665bbml9xz"
Oct 30 03:35:07.142746 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:35:07.142727    4461 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f36784f0-4807-48e8-8185-79bdec0fa85a-dshm\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5577b665bbml9xz\" (UID: \"f36784f0-4807-48e8-8185-79bdec0fa85a\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5577b665bbml9xz"
Oct 30 03:35:07.143056 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:35:07.143037    4461 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f36784f0-4807-48e8-8185-79bdec0fa85a-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5577b665bbml9xz\" (UID: \"f36784f0-4807-48e8-8185-79bdec0fa85a\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5577b665bbml9xz"
Oct 30 03:35:07.164267 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:35:07.164237    4461 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ffsz9\" (UniqueName: \"kubernetes.io/projected/f36784f0-4807-48e8-8185-79bdec0fa85a-kube-api-access-ffsz9\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5577b665bbml9xz\" (UID: \"f36784f0-4807-48e8-8185-79bdec0fa85a\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5577b665bbml9xz"
Oct 30 03:35:07.246732 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:35:07.246686    4461 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5577b665bbml9xz"
Oct 30 03:35:07.545115 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:35:07.545054    4461 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5577b665bbml9xz"]
Oct 30 03:35:07.549451 ip-10-0-143-156 kubenswrapper[4461]: W1030 03:35:07.549412    4461 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf36784f0_4807_48e8_8185_79bdec0fa85a.slice/crio-06a4332a5c652ffc9003982b082fc6b52c33316acea35aa7b519df4dad15a6aa WatchSource:0}: Error finding container 06a4332a5c652ffc9003982b082fc6b52c33316acea35aa7b519df4dad15a6aa: Status 404 returned error can't find the container with id 06a4332a5c652ffc9003982b082fc6b52c33316acea35aa7b519df4dad15a6aa
Oct 30 03:35:07.820797 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:35:07.820710    4461 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5577b665bbml9xz" event={"ID":"f36784f0-4807-48e8-8185-79bdec0fa85a","Type":"ContainerStarted","Data":"34ba4b6a609f5ae1ec99663a470bc744d000ae291f1ac1196da9756320da9527"}
Oct 30 03:35:07.820797 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:35:07.820777    4461 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5577b665bbml9xz" event={"ID":"f36784f0-4807-48e8-8185-79bdec0fa85a","Type":"ContainerStarted","Data":"06a4332a5c652ffc9003982b082fc6b52c33316acea35aa7b519df4dad15a6aa"}
Oct 30 03:35:07.864427 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:35:07.864377    4461 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5577b665bbml9xz" podStartSLOduration=1.864362206 podStartE2EDuration="1.864362206s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:35:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:35:07.863035382 +0000 UTC m=+2375.912840569" watchObservedRunningTime="2026-10-30 03:35:07.864362206 +0000 UTC m=+2375.914167381"
Oct 30 03:35:08.247584 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:35:08.247548    4461 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5577b665bbml9xz"
Oct 30 03:35:11.302367 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:35:11.302336    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:35:11.484120 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:35:11.483348    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:35:16.427145 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:35:16.427112    4461 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-h4qh8"]
Oct 30 03:35:16.431194 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:35:16.431170    4461 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-h4qh8"
Oct 30 03:35:16.433822 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:35:16.433795    4461 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f83d1219-fd7f-47f2-af01-40fb3a4cd2a0-workload-certs\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-h4qh8\" (UID: \"f83d1219-fd7f-47f2-af01-40fb3a4cd2a0\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-h4qh8"
Oct 30 03:35:16.433968 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:35:16.433834    4461 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f83d1219-fd7f-47f2-af01-40fb3a4cd2a0-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-h4qh8\" (UID: \"f83d1219-fd7f-47f2-af01-40fb3a4cd2a0\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-h4qh8"
Oct 30 03:35:16.433968 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:35:16.433866    4461 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f83d1219-fd7f-47f2-af01-40fb3a4cd2a0-istio-data\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-h4qh8\" (UID: \"f83d1219-fd7f-47f2-af01-40fb3a4cd2a0\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-h4qh8"
Oct 30 03:35:16.434111 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:35:16.433969    4461 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-klgb7\" (UniqueName: \"kubernetes.io/projected/f83d1219-fd7f-47f2-af01-40fb3a4cd2a0-kube-api-access-klgb7\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-h4qh8\" (UID: \"f83d1219-fd7f-47f2-af01-40fb3a4cd2a0\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-h4qh8"
Oct 30 03:35:16.434111 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:35:16.434001    4461 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f83d1219-fd7f-47f2-af01-40fb3a4cd2a0-istio-envoy\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-h4qh8\" (UID: \"f83d1219-fd7f-47f2-af01-40fb3a4cd2a0\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-h4qh8"
Oct 30 03:35:16.434111 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:35:16.434067    4461 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f83d1219-fd7f-47f2-af01-40fb3a4cd2a0-credential-socket\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-h4qh8\" (UID: \"f83d1219-fd7f-47f2-af01-40fb3a4cd2a0\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-h4qh8"
Oct 30 03:35:16.434318 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:35:16.434128    4461 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f83d1219-fd7f-47f2-af01-40fb3a4cd2a0-workload-socket\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-h4qh8\" (UID: \"f83d1219-fd7f-47f2-af01-40fb3a4cd2a0\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-h4qh8"
Oct 30 03:35:16.434318 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:35:16.434171    4461 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f83d1219-fd7f-47f2-af01-40fb3a4cd2a0-istio-podinfo\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-h4qh8\" (UID: \"f83d1219-fd7f-47f2-af01-40fb3a4cd2a0\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-h4qh8"
Oct 30 03:35:16.434318 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:35:16.434233    4461 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f83d1219-fd7f-47f2-af01-40fb3a4cd2a0-istio-token\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-h4qh8\" (UID: \"f83d1219-fd7f-47f2-af01-40fb3a4cd2a0\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-h4qh8"
Oct 30 03:35:16.444685 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:35:16.444661    4461 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-wbk94"]
Oct 30 03:35:16.447720 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:35:16.447701    4461 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-wbk94"
Oct 30 03:35:16.456879 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:35:16.456260    4461 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Oct 30 03:35:16.456879 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:35:16.456286    4461 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Oct 30 03:35:16.456879 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:35:16.456286    4461 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"openshift-ai-inference-openshift-default-dockercfg-pzw79\""
Oct 30 03:35:16.456879 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:35:16.456378    4461 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Oct 30 03:35:16.460632 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:35:16.460609    4461 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-h4qh8"]
Oct 30 03:35:16.490254 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:35:16.490230    4461 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-wbk94"]
Oct 30 03:35:16.534600 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:35:16.534568    4461 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1e409e01-826e-4317-af23-194073c1c6a1-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-wbk94\" (UID: \"1e409e01-826e-4317-af23-194073c1c6a1\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-wbk94"
Oct 30 03:35:16.534710 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:35:16.534608    4461 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f83d1219-fd7f-47f2-af01-40fb3a4cd2a0-istio-podinfo\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-h4qh8\" (UID: \"f83d1219-fd7f-47f2-af01-40fb3a4cd2a0\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-h4qh8"
Oct 30 03:35:16.534710 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:35:16.534634    4461 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f83d1219-fd7f-47f2-af01-40fb3a4cd2a0-istio-token\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-h4qh8\" (UID: \"f83d1219-fd7f-47f2-af01-40fb3a4cd2a0\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-h4qh8"
Oct 30 03:35:16.534710 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:35:16.534681    4461 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1e409e01-826e-4317-af23-194073c1c6a1-istio-envoy\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-wbk94\" (UID: \"1e409e01-826e-4317-af23-194073c1c6a1\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-wbk94"
Oct 30 03:35:16.534828 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:35:16.534708    4461 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f83d1219-fd7f-47f2-af01-40fb3a4cd2a0-workload-certs\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-h4qh8\" (UID: \"f83d1219-fd7f-47f2-af01-40fb3a4cd2a0\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-h4qh8"
Oct 30 03:35:16.534828 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:35:16.534732    4461 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1e409e01-826e-4317-af23-194073c1c6a1-istio-podinfo\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-wbk94\" (UID: \"1e409e01-826e-4317-af23-194073c1c6a1\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-wbk94"
Oct 30 03:35:16.534923 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:35:16.534903    4461 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e409e01-826e-4317-af23-194073c1c6a1-workload-socket\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-wbk94\" (UID: \"1e409e01-826e-4317-af23-194073c1c6a1\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-wbk94"
Oct 30 03:35:16.534969 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:35:16.534958    4461 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f83d1219-fd7f-47f2-af01-40fb3a4cd2a0-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-h4qh8\" (UID: \"f83d1219-fd7f-47f2-af01-40fb3a4cd2a0\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-h4qh8"
Oct 30 03:35:16.535011 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:35:16.534997    4461 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6dgp4\" (UniqueName: \"kubernetes.io/projected/1e409e01-826e-4317-af23-194073c1c6a1-kube-api-access-6dgp4\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-wbk94\" (UID: \"1e409e01-826e-4317-af23-194073c1c6a1\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-wbk94"
Oct 30 03:35:16.535073 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:35:16.535053    4461 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f83d1219-fd7f-47f2-af01-40fb3a4cd2a0-istio-data\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-h4qh8\" (UID: \"f83d1219-fd7f-47f2-af01-40fb3a4cd2a0\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-h4qh8"
Oct 30 03:35:16.535144 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:35:16.535113    4461 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f83d1219-fd7f-47f2-af01-40fb3a4cd2a0-workload-certs\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-h4qh8\" (UID: \"f83d1219-fd7f-47f2-af01-40fb3a4cd2a0\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-h4qh8"
Oct 30 03:35:16.535144 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:35:16.535119    4461 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-klgb7\" (UniqueName: \"kubernetes.io/projected/f83d1219-fd7f-47f2-af01-40fb3a4cd2a0-kube-api-access-klgb7\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-h4qh8\" (UID: \"f83d1219-fd7f-47f2-af01-40fb3a4cd2a0\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-h4qh8"
Oct 30 03:35:16.535328 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:35:16.535308    4461 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f83d1219-fd7f-47f2-af01-40fb3a4cd2a0-istio-envoy\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-h4qh8\" (UID: \"f83d1219-fd7f-47f2-af01-40fb3a4cd2a0\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-h4qh8"
Oct 30 03:35:16.535415 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:35:16.535352    4461 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f83d1219-fd7f-47f2-af01-40fb3a4cd2a0-credential-socket\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-h4qh8\" (UID: \"f83d1219-fd7f-47f2-af01-40fb3a4cd2a0\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-h4qh8"
Oct 30 03:35:16.535473 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:35:16.535412    4461 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f83d1219-fd7f-47f2-af01-40fb3a4cd2a0-istio-data\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-h4qh8\" (UID: \"f83d1219-fd7f-47f2-af01-40fb3a4cd2a0\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-h4qh8"
Oct 30 03:35:16.535528 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:35:16.535450    4461 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e409e01-826e-4317-af23-194073c1c6a1-credential-socket\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-wbk94\" (UID: \"1e409e01-826e-4317-af23-194073c1c6a1\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-wbk94"
Oct 30 03:35:16.535528 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:35:16.535518    4461 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1e409e01-826e-4317-af23-194073c1c6a1-istio-data\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-wbk94\" (UID: \"1e409e01-826e-4317-af23-194073c1c6a1\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-wbk94"
Oct 30 03:35:16.535649 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:35:16.535613    4461 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1e409e01-826e-4317-af23-194073c1c6a1-workload-certs\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-wbk94\" (UID: \"1e409e01-826e-4317-af23-194073c1c6a1\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-wbk94"
Oct 30 03:35:16.535692 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:35:16.535642    4461 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f83d1219-fd7f-47f2-af01-40fb3a4cd2a0-credential-socket\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-h4qh8\" (UID: \"f83d1219-fd7f-47f2-af01-40fb3a4cd2a0\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-h4qh8"
Oct 30 03:35:16.535692 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:35:16.535681    4461 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1e409e01-826e-4317-af23-194073c1c6a1-istio-token\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-wbk94\" (UID: \"1e409e01-826e-4317-af23-194073c1c6a1\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-wbk94"
Oct 30 03:35:16.535771 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:35:16.535700    4461 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f83d1219-fd7f-47f2-af01-40fb3a4cd2a0-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-h4qh8\" (UID: \"f83d1219-fd7f-47f2-af01-40fb3a4cd2a0\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-h4qh8"
Oct 30 03:35:16.535771 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:35:16.535714    4461 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f83d1219-fd7f-47f2-af01-40fb3a4cd2a0-workload-socket\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-h4qh8\" (UID: \"f83d1219-fd7f-47f2-af01-40fb3a4cd2a0\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-h4qh8"
Oct 30 03:35:16.535970 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:35:16.535953    4461 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f83d1219-fd7f-47f2-af01-40fb3a4cd2a0-workload-socket\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-h4qh8\" (UID: \"f83d1219-fd7f-47f2-af01-40fb3a4cd2a0\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-h4qh8"
Oct 30 03:35:16.537638 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:35:16.537618    4461 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f83d1219-fd7f-47f2-af01-40fb3a4cd2a0-istio-envoy\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-h4qh8\" (UID: \"f83d1219-fd7f-47f2-af01-40fb3a4cd2a0\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-h4qh8"
Oct 30 03:35:16.537990 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:35:16.537971    4461 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f83d1219-fd7f-47f2-af01-40fb3a4cd2a0-istio-podinfo\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-h4qh8\" (UID: \"f83d1219-fd7f-47f2-af01-40fb3a4cd2a0\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-h4qh8"
Oct 30 03:35:16.558395 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:35:16.558374    4461 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f83d1219-fd7f-47f2-af01-40fb3a4cd2a0-istio-token\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-h4qh8\" (UID: \"f83d1219-fd7f-47f2-af01-40fb3a4cd2a0\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-h4qh8"
Oct 30 03:35:16.558760 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:35:16.558732    4461 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-klgb7\" (UniqueName: \"kubernetes.io/projected/f83d1219-fd7f-47f2-af01-40fb3a4cd2a0-kube-api-access-klgb7\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-h4qh8\" (UID: \"f83d1219-fd7f-47f2-af01-40fb3a4cd2a0\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-h4qh8"
Oct 30 03:35:16.637080 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:35:16.637053    4461 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e409e01-826e-4317-af23-194073c1c6a1-workload-socket\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-wbk94\" (UID: \"1e409e01-826e-4317-af23-194073c1c6a1\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-wbk94"
Oct 30 03:35:16.637202 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:35:16.637118    4461 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6dgp4\" (UniqueName: \"kubernetes.io/projected/1e409e01-826e-4317-af23-194073c1c6a1-kube-api-access-6dgp4\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-wbk94\" (UID: \"1e409e01-826e-4317-af23-194073c1c6a1\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-wbk94"
Oct 30 03:35:16.637202 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:35:16.637163    4461 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e409e01-826e-4317-af23-194073c1c6a1-credential-socket\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-wbk94\" (UID: \"1e409e01-826e-4317-af23-194073c1c6a1\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-wbk94"
Oct 30 03:35:16.637202 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:35:16.637192    4461 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1e409e01-826e-4317-af23-194073c1c6a1-istio-data\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-wbk94\" (UID: \"1e409e01-826e-4317-af23-194073c1c6a1\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-wbk94"
Oct 30 03:35:16.637368 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:35:16.637210    4461 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1e409e01-826e-4317-af23-194073c1c6a1-workload-certs\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-wbk94\" (UID: \"1e409e01-826e-4317-af23-194073c1c6a1\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-wbk94"
Oct 30 03:35:16.637368 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:35:16.637225    4461 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1e409e01-826e-4317-af23-194073c1c6a1-istio-token\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-wbk94\" (UID: \"1e409e01-826e-4317-af23-194073c1c6a1\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-wbk94"
Oct 30 03:35:16.637368 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:35:16.637259    4461 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1e409e01-826e-4317-af23-194073c1c6a1-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-wbk94\" (UID: \"1e409e01-826e-4317-af23-194073c1c6a1\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-wbk94"
Oct 30 03:35:16.637368 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:35:16.637289    4461 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1e409e01-826e-4317-af23-194073c1c6a1-istio-envoy\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-wbk94\" (UID: \"1e409e01-826e-4317-af23-194073c1c6a1\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-wbk94"
Oct 30 03:35:16.637368 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:35:16.637309    4461 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1e409e01-826e-4317-af23-194073c1c6a1-istio-podinfo\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-wbk94\" (UID: \"1e409e01-826e-4317-af23-194073c1c6a1\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-wbk94"
Oct 30 03:35:16.637626 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:35:16.637491    4461 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e409e01-826e-4317-af23-194073c1c6a1-credential-socket\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-wbk94\" (UID: \"1e409e01-826e-4317-af23-194073c1c6a1\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-wbk94"
Oct 30 03:35:16.637626 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:35:16.637538    4461 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e409e01-826e-4317-af23-194073c1c6a1-workload-socket\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-wbk94\" (UID: \"1e409e01-826e-4317-af23-194073c1c6a1\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-wbk94"
Oct 30 03:35:16.637626 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:35:16.637605    4461 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1e409e01-826e-4317-af23-194073c1c6a1-istio-data\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-wbk94\" (UID: \"1e409e01-826e-4317-af23-194073c1c6a1\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-wbk94"
Oct 30 03:35:16.637804 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:35:16.637780    4461 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1e409e01-826e-4317-af23-194073c1c6a1-workload-certs\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-wbk94\" (UID: \"1e409e01-826e-4317-af23-194073c1c6a1\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-wbk94"
Oct 30 03:35:16.638092 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:35:16.638075    4461 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1e409e01-826e-4317-af23-194073c1c6a1-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-wbk94\" (UID: \"1e409e01-826e-4317-af23-194073c1c6a1\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-wbk94"
Oct 30 03:35:16.640137 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:35:16.640088    4461 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1e409e01-826e-4317-af23-194073c1c6a1-istio-envoy\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-wbk94\" (UID: \"1e409e01-826e-4317-af23-194073c1c6a1\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-wbk94"
Oct 30 03:35:16.640223 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:35:16.640210    4461 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1e409e01-826e-4317-af23-194073c1c6a1-istio-podinfo\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-wbk94\" (UID: \"1e409e01-826e-4317-af23-194073c1c6a1\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-wbk94"
Oct 30 03:35:16.661028 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:35:16.661001    4461 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1e409e01-826e-4317-af23-194073c1c6a1-istio-token\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-wbk94\" (UID: \"1e409e01-826e-4317-af23-194073c1c6a1\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-wbk94"
Oct 30 03:35:16.661143 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:35:16.661067    4461 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6dgp4\" (UniqueName: \"kubernetes.io/projected/1e409e01-826e-4317-af23-194073c1c6a1-kube-api-access-6dgp4\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-wbk94\" (UID: \"1e409e01-826e-4317-af23-194073c1c6a1\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-wbk94"
Oct 30 03:35:16.749044 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:35:16.749021    4461 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-h4qh8"
Oct 30 03:35:16.763292 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:35:16.763271    4461 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-wbk94"
Oct 30 03:35:17.094366 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:35:17.094332    4461 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-wbk94"]
Oct 30 03:35:17.096265 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:35:17.096110    4461 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-h4qh8"]
Oct 30 03:35:17.096386 ip-10-0-143-156 kubenswrapper[4461]: W1030 03:35:17.096341    4461 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1e409e01_826e_4317_af23_194073c1c6a1.slice/crio-d8e22a34e2ae1e4fcb9fca2bbdfe0740e6781314cb8709994fe70e46eba2b46c WatchSource:0}: Error finding container d8e22a34e2ae1e4fcb9fca2bbdfe0740e6781314cb8709994fe70e46eba2b46c: Status 404 returned error can't find the container with id d8e22a34e2ae1e4fcb9fca2bbdfe0740e6781314cb8709994fe70e46eba2b46c
Oct 30 03:35:17.097247 ip-10-0-143-156 kubenswrapper[4461]: W1030 03:35:17.097221    4461 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf83d1219_fd7f_47f2_af01_40fb3a4cd2a0.slice/crio-f359120081b80ce0e3fd9c157d3046362f8961e9e9ae37f4366d36a110e6cc66 WatchSource:0}: Error finding container f359120081b80ce0e3fd9c157d3046362f8961e9e9ae37f4366d36a110e6cc66: Status 404 returned error can't find the container with id f359120081b80ce0e3fd9c157d3046362f8961e9e9ae37f4366d36a110e6cc66
Oct 30 03:35:17.098549 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:35:17.098512    4461 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 03:35:17.098657 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:35:17.098587    4461 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 03:35:17.098657 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:35:17.098617    4461 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 03:35:17.099122 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:35:17.099078    4461 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 03:35:17.099170 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:35:17.099153    4461 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 03:35:17.099210 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:35:17.099179    4461 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 03:35:17.247289 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:35:17.247249    4461 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5577b665bbml9xz"
Oct 30 03:35:17.266708 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:35:17.266680    4461 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5577b665bbml9xz"
Oct 30 03:35:17.839235 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:35:17.839195    4461 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-854b944bd8-mbwh6" podUID="06762940-e16c-4ab3-b9c6-b65345ba80a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.34:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 03:35:17.885716 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:35:17.870223    4461 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-h4qh8" event={"ID":"f83d1219-fd7f-47f2-af01-40fb3a4cd2a0","Type":"ContainerStarted","Data":"100b56d0a786b9a8beeea46f62ad206f2fb7f3e5426ce3abfffe84093f0fd11a"}
Oct 30 03:35:17.885716 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:35:17.870287    4461 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-h4qh8" event={"ID":"f83d1219-fd7f-47f2-af01-40fb3a4cd2a0","Type":"ContainerStarted","Data":"f359120081b80ce0e3fd9c157d3046362f8961e9e9ae37f4366d36a110e6cc66"}
Oct 30 03:35:17.917118 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:35:17.914688    4461 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-wbk94" event={"ID":"1e409e01-826e-4317-af23-194073c1c6a1","Type":"ContainerStarted","Data":"603df059ee07b50290a7b02ac07fd4009814e118193758d252e5769594ae563c"}
Oct 30 03:35:17.917118 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:35:17.914795    4461 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-wbk94" event={"ID":"1e409e01-826e-4317-af23-194073c1c6a1","Type":"ContainerStarted","Data":"d8e22a34e2ae1e4fcb9fca2bbdfe0740e6781314cb8709994fe70e46eba2b46c"}
Oct 30 03:35:17.926425 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:35:17.926372    4461 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5577b665bbml9xz"
Oct 30 03:35:17.943918 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:35:17.943856    4461 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-h4qh8" podStartSLOduration=1.943839586 podStartE2EDuration="1.943839586s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:35:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:35:17.942191068 +0000 UTC m=+2385.991996263" watchObservedRunningTime="2026-10-30 03:35:17.943839586 +0000 UTC m=+2385.993644764"
Oct 30 03:35:18.039305 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:35:18.039232    4461 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-wbk94" podStartSLOduration=2.039212566 podStartE2EDuration="2.039212566s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:35:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:35:18.037794603 +0000 UTC m=+2386.087599798" watchObservedRunningTime="2026-10-30 03:35:18.039212566 +0000 UTC m=+2386.089017743"
Oct 30 03:35:18.751194 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:35:18.751159    4461 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-h4qh8"
Oct 30 03:35:18.751352 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:35:18.751266    4461 patch_prober.go:29] interesting pod/openshift-ai-inference-openshift-default-5b4f8cc59b-h4qh8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" start-of-body=
Oct 30 03:35:18.751352 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:35:18.751297    4461 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-h4qh8" podUID="f83d1219-fd7f-47f2-af01-40fb3a4cd2a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused"
Oct 30 03:35:18.769121 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:35:18.766394    4461 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-wbk94"
Oct 30 03:35:18.769121 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:35:18.766491    4461 patch_prober.go:29] interesting pod/openshift-ai-inference-openshift-default-5b4f8cc59b-wbk94 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" start-of-body=
Oct 30 03:35:18.769121 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:35:18.766527    4461 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-wbk94" podUID="1e409e01-826e-4317-af23-194073c1c6a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused"
Oct 30 03:35:18.919188 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:35:18.919153    4461 generic.go:356] "Generic (PLEG): container finished" podID="06762940-e16c-4ab3-b9c6-b65345ba80a1" containerID="a309c1aec24f7b79e5fae550841f8da7167453ceda31b232a615a7aefa6a60a4" exitCode=0
Oct 30 03:35:18.919928 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:35:18.919905    4461 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-854b944bd8-mbwh6" event={"ID":"06762940-e16c-4ab3-b9c6-b65345ba80a1","Type":"ContainerDied","Data":"a309c1aec24f7b79e5fae550841f8da7167453ceda31b232a615a7aefa6a60a4"}
Oct 30 03:35:18.920276 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:35:18.920245    4461 scope.go:122] "RemoveContainer" containerID="a309c1aec24f7b79e5fae550841f8da7167453ceda31b232a615a7aefa6a60a4"
Oct 30 03:35:18.921291 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:35:18.921261    4461 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 03:35:18.921385 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:35:18.921343    4461 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 03:35:18.921452 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:35:18.921394    4461 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 03:35:19.750227 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:35:19.750193    4461 patch_prober.go:29] interesting pod/openshift-ai-inference-openshift-default-5b4f8cc59b-h4qh8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" start-of-body=
Oct 30 03:35:19.750417 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:35:19.750247    4461 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-h4qh8" podUID="f83d1219-fd7f-47f2-af01-40fb3a4cd2a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused"
Oct 30 03:35:19.776122 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:35:19.765454    4461 patch_prober.go:29] interesting pod/openshift-ai-inference-openshift-default-5b4f8cc59b-wbk94 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" start-of-body=
Oct 30 03:35:19.776122 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:35:19.765519    4461 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-wbk94" podUID="1e409e01-826e-4317-af23-194073c1c6a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused"
Oct 30 03:35:19.949222 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:35:19.949168    4461 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-854b944bd8-mbwh6" event={"ID":"06762940-e16c-4ab3-b9c6-b65345ba80a1","Type":"ContainerStarted","Data":"888fde87d4e38dafc9c8e5e545760ec5b2e1dc306f08940b5629a4f1d16c075d"}
Oct 30 03:35:20.749689 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:35:20.749652    4461 patch_prober.go:29] interesting pod/openshift-ai-inference-openshift-default-5b4f8cc59b-h4qh8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" start-of-body=
Oct 30 03:35:20.749869 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:35:20.749702    4461 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-h4qh8" podUID="f83d1219-fd7f-47f2-af01-40fb3a4cd2a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused"
Oct 30 03:35:20.769120 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:35:20.765467    4461 patch_prober.go:29] interesting pod/openshift-ai-inference-openshift-default-5b4f8cc59b-wbk94 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" start-of-body=
Oct 30 03:35:20.769120 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:35:20.765527    4461 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-wbk94" podUID="1e409e01-826e-4317-af23-194073c1c6a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused"
Oct 30 03:35:20.895902 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:35:20.895492    4461 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-854b944bd8-mbwh6"
Oct 30 03:35:20.895902 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:35:20.895635    4461 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-854b944bd8-mbwh6" podUID="06762940-e16c-4ab3-b9c6-b65345ba80a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused"
Oct 30 03:35:21.749874 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:35:21.749837    4461 patch_prober.go:29] interesting pod/openshift-ai-inference-openshift-default-5b4f8cc59b-h4qh8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" start-of-body=
Oct 30 03:35:21.750325 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:35:21.749890    4461 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-h4qh8" podUID="f83d1219-fd7f-47f2-af01-40fb3a4cd2a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused"
Oct 30 03:35:21.765453 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:35:21.765422    4461 patch_prober.go:29] interesting pod/openshift-ai-inference-openshift-default-5b4f8cc59b-wbk94 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" start-of-body=
Oct 30 03:35:21.765688 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:35:21.765654    4461 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-wbk94" podUID="1e409e01-826e-4317-af23-194073c1c6a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused"
Oct 30 03:35:21.898641 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:35:21.898597    4461 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-854b944bd8-mbwh6" podUID="06762940-e16c-4ab3-b9c6-b65345ba80a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused"
Oct 30 03:35:22.749540 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:35:22.749500    4461 patch_prober.go:29] interesting pod/openshift-ai-inference-openshift-default-5b4f8cc59b-h4qh8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" start-of-body=
Oct 30 03:35:22.749729 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:35:22.749556    4461 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-h4qh8" podUID="f83d1219-fd7f-47f2-af01-40fb3a4cd2a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused"
Oct 30 03:35:22.774219 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:35:22.774184    4461 patch_prober.go:29] interesting pod/openshift-ai-inference-openshift-default-5b4f8cc59b-wbk94 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" start-of-body=
Oct 30 03:35:22.774959 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:35:22.774931    4461 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-wbk94" podUID="1e409e01-826e-4317-af23-194073c1c6a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused"
Oct 30 03:35:22.895513 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:35:22.895470    4461 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-854b944bd8-mbwh6" podUID="06762940-e16c-4ab3-b9c6-b65345ba80a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused"
Oct 30 03:35:23.753255 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:35:23.753224    4461 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-h4qh8"
Oct 30 03:35:23.753500 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:35:23.753478    4461 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-h4qh8"
Oct 30 03:35:23.754198 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:35:23.754182    4461 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-h4qh8"
Oct 30 03:35:23.783120 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:35:23.782669    4461 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-wbk94"
Oct 30 03:35:23.785779 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:35:23.783658    4461 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-wbk94"
Oct 30 03:35:23.785779 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:35:23.784916    4461 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-wbk94"
Oct 30 03:35:23.896036 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:35:23.895996    4461 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-854b944bd8-mbwh6" podUID="06762940-e16c-4ab3-b9c6-b65345ba80a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused"
Oct 30 03:35:24.510211 ip-10-0-143-156 kubenswrapper[4461]: E1030 03:35:24.510175    4461 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2k85b" containerName="extractor"
Oct 30 03:35:24.895796 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:35:24.895697    4461 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-854b944bd8-mbwh6" podUID="06762940-e16c-4ab3-b9c6-b65345ba80a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused"
Oct 30 03:35:25.899014 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:35:25.898980    4461 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-854b944bd8-mbwh6"
Oct 30 03:35:25.899479 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:35:25.899320    4461 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-854b944bd8-mbwh6"
Oct 30 03:35:25.899895 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:35:25.899875    4461 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-854b944bd8-mbwh6"
Oct 30 03:35:31.487620 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:35:31.487577    4461 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-rx6vm"]
Oct 30 03:35:31.491252 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:35:31.491222    4461 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-rx6vm"
Oct 30 03:35:31.525476 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:35:31.525446    4461 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-rx6vm"]
Oct 30 03:35:31.560142 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:35:31.560082    4461 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/35b8f310-5db9-4cf0-99ea-0baca82d3cda-credential-socket\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-rx6vm\" (UID: \"35b8f310-5db9-4cf0-99ea-0baca82d3cda\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-rx6vm"
Oct 30 03:35:31.560300 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:35:31.560203    4461 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/35b8f310-5db9-4cf0-99ea-0baca82d3cda-istio-data\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-rx6vm\" (UID: \"35b8f310-5db9-4cf0-99ea-0baca82d3cda\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-rx6vm"
Oct 30 03:35:31.560300 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:35:31.560275    4461 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/35b8f310-5db9-4cf0-99ea-0baca82d3cda-istio-podinfo\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-rx6vm\" (UID: \"35b8f310-5db9-4cf0-99ea-0baca82d3cda\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-rx6vm"
Oct 30 03:35:31.560300 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:35:31.560296    4461 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/35b8f310-5db9-4cf0-99ea-0baca82d3cda-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-rx6vm\" (UID: \"35b8f310-5db9-4cf0-99ea-0baca82d3cda\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-rx6vm"
Oct 30 03:35:31.560425 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:35:31.560409    4461 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/35b8f310-5db9-4cf0-99ea-0baca82d3cda-workload-certs\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-rx6vm\" (UID: \"35b8f310-5db9-4cf0-99ea-0baca82d3cda\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-rx6vm"
Oct 30 03:35:31.560462 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:35:31.560441    4461 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h46qf\" (UniqueName: \"kubernetes.io/projected/35b8f310-5db9-4cf0-99ea-0baca82d3cda-kube-api-access-h46qf\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-rx6vm\" (UID: \"35b8f310-5db9-4cf0-99ea-0baca82d3cda\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-rx6vm"
Oct 30 03:35:31.560510 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:35:31.560487    4461 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/35b8f310-5db9-4cf0-99ea-0baca82d3cda-workload-socket\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-rx6vm\" (UID: \"35b8f310-5db9-4cf0-99ea-0baca82d3cda\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-rx6vm"
Oct 30 03:35:31.560561 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:35:31.560537    4461 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/35b8f310-5db9-4cf0-99ea-0baca82d3cda-istio-envoy\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-rx6vm\" (UID: \"35b8f310-5db9-4cf0-99ea-0baca82d3cda\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-rx6vm"
Oct 30 03:35:31.560605 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:35:31.560579    4461 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/35b8f310-5db9-4cf0-99ea-0baca82d3cda-istio-token\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-rx6vm\" (UID: \"35b8f310-5db9-4cf0-99ea-0baca82d3cda\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-rx6vm"
Oct 30 03:35:31.661801 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:35:31.661764    4461 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/35b8f310-5db9-4cf0-99ea-0baca82d3cda-istio-podinfo\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-rx6vm\" (UID: \"35b8f310-5db9-4cf0-99ea-0baca82d3cda\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-rx6vm"
Oct 30 03:35:31.661801 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:35:31.661799    4461 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/35b8f310-5db9-4cf0-99ea-0baca82d3cda-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-rx6vm\" (UID: \"35b8f310-5db9-4cf0-99ea-0baca82d3cda\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-rx6vm"
Oct 30 03:35:31.662051 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:35:31.661823    4461 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/35b8f310-5db9-4cf0-99ea-0baca82d3cda-workload-certs\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-rx6vm\" (UID: \"35b8f310-5db9-4cf0-99ea-0baca82d3cda\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-rx6vm"
Oct 30 03:35:31.662051 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:35:31.661934    4461 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h46qf\" (UniqueName: \"kubernetes.io/projected/35b8f310-5db9-4cf0-99ea-0baca82d3cda-kube-api-access-h46qf\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-rx6vm\" (UID: \"35b8f310-5db9-4cf0-99ea-0baca82d3cda\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-rx6vm"
Oct 30 03:35:31.662051 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:35:31.661987    4461 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/35b8f310-5db9-4cf0-99ea-0baca82d3cda-workload-socket\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-rx6vm\" (UID: \"35b8f310-5db9-4cf0-99ea-0baca82d3cda\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-rx6vm"
Oct 30 03:35:31.662051 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:35:31.662011    4461 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/35b8f310-5db9-4cf0-99ea-0baca82d3cda-istio-envoy\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-rx6vm\" (UID: \"35b8f310-5db9-4cf0-99ea-0baca82d3cda\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-rx6vm"
Oct 30 03:35:31.662051 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:35:31.662035    4461 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/35b8f310-5db9-4cf0-99ea-0baca82d3cda-istio-token\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-rx6vm\" (UID: \"35b8f310-5db9-4cf0-99ea-0baca82d3cda\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-rx6vm"
Oct 30 03:35:31.662324 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:35:31.662119    4461 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/35b8f310-5db9-4cf0-99ea-0baca82d3cda-credential-socket\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-rx6vm\" (UID: \"35b8f310-5db9-4cf0-99ea-0baca82d3cda\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-rx6vm"
Oct 30 03:35:31.662324 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:35:31.662187    4461 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/35b8f310-5db9-4cf0-99ea-0baca82d3cda-istio-data\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-rx6vm\" (UID: \"35b8f310-5db9-4cf0-99ea-0baca82d3cda\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-rx6vm"
Oct 30 03:35:31.662430 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:35:31.662341    4461 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/35b8f310-5db9-4cf0-99ea-0baca82d3cda-workload-certs\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-rx6vm\" (UID: \"35b8f310-5db9-4cf0-99ea-0baca82d3cda\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-rx6vm"
Oct 30 03:35:31.662495 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:35:31.662474    4461 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/35b8f310-5db9-4cf0-99ea-0baca82d3cda-workload-socket\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-rx6vm\" (UID: \"35b8f310-5db9-4cf0-99ea-0baca82d3cda\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-rx6vm"
Oct 30 03:35:31.662546 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:35:31.662490    4461 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/35b8f310-5db9-4cf0-99ea-0baca82d3cda-istio-data\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-rx6vm\" (UID: \"35b8f310-5db9-4cf0-99ea-0baca82d3cda\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-rx6vm"
Oct 30 03:35:31.662611 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:35:31.662597    4461 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/35b8f310-5db9-4cf0-99ea-0baca82d3cda-credential-socket\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-rx6vm\" (UID: \"35b8f310-5db9-4cf0-99ea-0baca82d3cda\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-rx6vm"
Oct 30 03:35:31.662665 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:35:31.662650    4461 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/35b8f310-5db9-4cf0-99ea-0baca82d3cda-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-rx6vm\" (UID: \"35b8f310-5db9-4cf0-99ea-0baca82d3cda\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-rx6vm"
Oct 30 03:35:31.664304 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:35:31.664282    4461 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/35b8f310-5db9-4cf0-99ea-0baca82d3cda-istio-envoy\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-rx6vm\" (UID: \"35b8f310-5db9-4cf0-99ea-0baca82d3cda\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-rx6vm"
Oct 30 03:35:31.664405 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:35:31.664394    4461 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/35b8f310-5db9-4cf0-99ea-0baca82d3cda-istio-podinfo\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-rx6vm\" (UID: \"35b8f310-5db9-4cf0-99ea-0baca82d3cda\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-rx6vm"
Oct 30 03:35:31.685895 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:35:31.685865    4461 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/35b8f310-5db9-4cf0-99ea-0baca82d3cda-istio-token\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-rx6vm\" (UID: \"35b8f310-5db9-4cf0-99ea-0baca82d3cda\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-rx6vm"
Oct 30 03:35:31.686009 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:35:31.685960    4461 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h46qf\" (UniqueName: \"kubernetes.io/projected/35b8f310-5db9-4cf0-99ea-0baca82d3cda-kube-api-access-h46qf\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-rx6vm\" (UID: \"35b8f310-5db9-4cf0-99ea-0baca82d3cda\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-rx6vm"
Oct 30 03:35:31.804702 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:35:31.804617    4461 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-rx6vm"
Oct 30 03:35:32.119768 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:35:32.119745    4461 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-rx6vm"]
Oct 30 03:35:32.122109 ip-10-0-143-156 kubenswrapper[4461]: W1030 03:35:32.122066    4461 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod35b8f310_5db9_4cf0_99ea_0baca82d3cda.slice/crio-18556be2d41ad1803c95525eef5e836910cac49066a9bb5a14fbca8d4d8f989a WatchSource:0}: Error finding container 18556be2d41ad1803c95525eef5e836910cac49066a9bb5a14fbca8d4d8f989a: Status 404 returned error can't find the container with id 18556be2d41ad1803c95525eef5e836910cac49066a9bb5a14fbca8d4d8f989a
Oct 30 03:35:32.124001 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:35:32.123964    4461 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 03:35:32.124068 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:35:32.124052    4461 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 03:35:32.124128 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:35:32.124107    4461 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 03:35:33.035875 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:35:33.035843    4461 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-rx6vm" event={"ID":"35b8f310-5db9-4cf0-99ea-0baca82d3cda","Type":"ContainerStarted","Data":"ad636ca1d87a5ca50776401658029cf4714c3cbeae5268ca0c5eb6e9007bfec0"}
Oct 30 03:35:33.036309 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:35:33.035889    4461 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-rx6vm" event={"ID":"35b8f310-5db9-4cf0-99ea-0baca82d3cda","Type":"ContainerStarted","Data":"18556be2d41ad1803c95525eef5e836910cac49066a9bb5a14fbca8d4d8f989a"}
Oct 30 03:35:33.102052 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:35:33.102001    4461 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-rx6vm" podStartSLOduration=2.101985547 podStartE2EDuration="2.101985547s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:35:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:35:33.10092218 +0000 UTC m=+2401.150727381" watchObservedRunningTime="2026-10-30 03:35:33.101985547 +0000 UTC m=+2401.151790723"
Oct 30 03:35:33.805589 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:35:33.805549    4461 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-rx6vm"
Oct 30 03:35:33.806953 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:35:33.806932    4461 patch_prober.go:29] interesting pod/openshift-ai-inference-openshift-default-5b4f8cc59b-rx6vm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" start-of-body=
Oct 30 03:35:33.807023 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:35:33.806978    4461 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-rx6vm" podUID="35b8f310-5db9-4cf0-99ea-0baca82d3cda" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused"
Oct 30 03:35:33.990937 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:35:33.990897    4461 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-bd846f858-gwv2g"]
Oct 30 03:35:34.001809 ip-10-0-143-156 kubenswrapper[4461]: E1030 03:35:34.001768    4461 log.go:32] "ExecSync cmd from runtime service failed" err="rpc error: code = Unknown desc = command error: , stdout: , stderr: , exit code -1" containerID="bb40283fd8cc8588057039a772a112328ea7e7e14665fc42fd5ae8c218407b22" cmd=["/bin/sleep","15"]
Oct 30 03:35:34.001966 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:35:34.001827    4461 handlers.go:85] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-bd846f858-gwv2g" message="" err="rpc error: code = Unknown desc = command error: , stdout: , stderr: , exit code -1"
Oct 30 03:35:34.001966 ip-10-0-143-156 kubenswrapper[4461]: E1030 03:35:34.001846    4461 kuberuntime_container.go:780] "PreStop hook failed" err="rpc error: code = Unknown desc = command error: , stdout: , stderr: , exit code -1" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-bd846f858-gwv2g" podUID="cb97f849-7c86-4f17-978c-f8e77298c1aa" containerName="main" containerID="cri-o://bb40283fd8cc8588057039a772a112328ea7e7e14665fc42fd5ae8c218407b22"
Oct 30 03:35:34.001966 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:35:34.001885    4461 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-bd846f858-gwv2g" podUID="cb97f849-7c86-4f17-978c-f8e77298c1aa" containerName="main" containerID="cri-o://bb40283fd8cc8588057039a772a112328ea7e7e14665fc42fd5ae8c218407b22" gracePeriod=60
Oct 30 03:35:34.111233 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:35:34.111185    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:35:34.286740 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:35:34.286642    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:35:34.428748 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:35:34.428722    4461 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-bd846f858-gwv2g"
Oct 30 03:35:34.606232 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:35:34.606201    4461 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cb97f849-7c86-4f17-978c-f8e77298c1aa-kube-api-access-j79rq\" (UniqueName: \"kubernetes.io/projected/cb97f849-7c86-4f17-978c-f8e77298c1aa-kube-api-access-j79rq\") pod \"cb97f849-7c86-4f17-978c-f8e77298c1aa\" (UID: \"cb97f849-7c86-4f17-978c-f8e77298c1aa\") "
Oct 30 03:35:34.606386 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:35:34.606238    4461 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cb97f849-7c86-4f17-978c-f8e77298c1aa-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cb97f849-7c86-4f17-978c-f8e77298c1aa-kserve-provision-location\") pod \"cb97f849-7c86-4f17-978c-f8e77298c1aa\" (UID: \"cb97f849-7c86-4f17-978c-f8e77298c1aa\") "
Oct 30 03:35:34.606386 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:35:34.606278    4461 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cb97f849-7c86-4f17-978c-f8e77298c1aa-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cb97f849-7c86-4f17-978c-f8e77298c1aa-model-cache\") pod \"cb97f849-7c86-4f17-978c-f8e77298c1aa\" (UID: \"cb97f849-7c86-4f17-978c-f8e77298c1aa\") "
Oct 30 03:35:34.606499 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:35:34.606409    4461 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cb97f849-7c86-4f17-978c-f8e77298c1aa-tls-certs\" (UniqueName: \"kubernetes.io/secret/cb97f849-7c86-4f17-978c-f8e77298c1aa-tls-certs\") pod \"cb97f849-7c86-4f17-978c-f8e77298c1aa\" (UID: \"cb97f849-7c86-4f17-978c-f8e77298c1aa\") "
Oct 30 03:35:34.606499 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:35:34.606480    4461 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cb97f849-7c86-4f17-978c-f8e77298c1aa-dshm\" (UniqueName: \"kubernetes.io/empty-dir/cb97f849-7c86-4f17-978c-f8e77298c1aa-dshm\") pod \"cb97f849-7c86-4f17-978c-f8e77298c1aa\" (UID: \"cb97f849-7c86-4f17-978c-f8e77298c1aa\") "
Oct 30 03:35:34.606609 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:35:34.606518    4461 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cb97f849-7c86-4f17-978c-f8e77298c1aa-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/cb97f849-7c86-4f17-978c-f8e77298c1aa-cabundle-cert\") pod \"cb97f849-7c86-4f17-978c-f8e77298c1aa\" (UID: \"cb97f849-7c86-4f17-978c-f8e77298c1aa\") "
Oct 30 03:35:34.606609 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:35:34.606540    4461 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb97f849-7c86-4f17-978c-f8e77298c1aa-model-cache" pod "cb97f849-7c86-4f17-978c-f8e77298c1aa" (UID: "cb97f849-7c86-4f17-978c-f8e77298c1aa"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:35:34.606609 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:35:34.606574    4461 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cb97f849-7c86-4f17-978c-f8e77298c1aa-home\" (UniqueName: \"kubernetes.io/empty-dir/cb97f849-7c86-4f17-978c-f8e77298c1aa-home\") pod \"cb97f849-7c86-4f17-978c-f8e77298c1aa\" (UID: \"cb97f849-7c86-4f17-978c-f8e77298c1aa\") "
Oct 30 03:35:34.606609 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:35:34.606602    4461 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cb97f849-7c86-4f17-978c-f8e77298c1aa-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cb97f849-7c86-4f17-978c-f8e77298c1aa-tmp-dir\") pod \"cb97f849-7c86-4f17-978c-f8e77298c1aa\" (UID: \"cb97f849-7c86-4f17-978c-f8e77298c1aa\") "
Oct 30 03:35:34.606817 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:35:34.606767    4461 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb97f849-7c86-4f17-978c-f8e77298c1aa-home" pod "cb97f849-7c86-4f17-978c-f8e77298c1aa" (UID: "cb97f849-7c86-4f17-978c-f8e77298c1aa"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:35:34.606870 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:35:34.606838    4461 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cb97f849-7c86-4f17-978c-f8e77298c1aa-cabundle-cert" pod "cb97f849-7c86-4f17-978c-f8e77298c1aa" (UID: "cb97f849-7c86-4f17-978c-f8e77298c1aa"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:35:34.606948 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:35:34.606926    4461 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cb97f849-7c86-4f17-978c-f8e77298c1aa-model-cache\") on node \"ip-10-0-143-156.ec2.internal\" DevicePath \"\""
Oct 30 03:35:34.607013 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:35:34.606943    4461 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb97f849-7c86-4f17-978c-f8e77298c1aa-tmp-dir" pod "cb97f849-7c86-4f17-978c-f8e77298c1aa" (UID: "cb97f849-7c86-4f17-978c-f8e77298c1aa"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:35:34.607013 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:35:34.606953    4461 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/cb97f849-7c86-4f17-978c-f8e77298c1aa-cabundle-cert\") on node \"ip-10-0-143-156.ec2.internal\" DevicePath \"\""
Oct 30 03:35:34.607013 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:35:34.606993    4461 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cb97f849-7c86-4f17-978c-f8e77298c1aa-home\") on node \"ip-10-0-143-156.ec2.internal\" DevicePath \"\""
Oct 30 03:35:34.608735 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:35:34.608702    4461 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cb97f849-7c86-4f17-978c-f8e77298c1aa-tls-certs" pod "cb97f849-7c86-4f17-978c-f8e77298c1aa" (UID: "cb97f849-7c86-4f17-978c-f8e77298c1aa"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:35:34.608735 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:35:34.608717    4461 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb97f849-7c86-4f17-978c-f8e77298c1aa-dshm" pod "cb97f849-7c86-4f17-978c-f8e77298c1aa" (UID: "cb97f849-7c86-4f17-978c-f8e77298c1aa"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:35:34.608901 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:35:34.608840    4461 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cb97f849-7c86-4f17-978c-f8e77298c1aa-kube-api-access-j79rq" pod "cb97f849-7c86-4f17-978c-f8e77298c1aa" (UID: "cb97f849-7c86-4f17-978c-f8e77298c1aa"). InnerVolumeSpecName "kube-api-access-j79rq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:35:34.624630 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:35:34.624609    4461 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb97f849-7c86-4f17-978c-f8e77298c1aa-kserve-provision-location" pod "cb97f849-7c86-4f17-978c-f8e77298c1aa" (UID: "cb97f849-7c86-4f17-978c-f8e77298c1aa"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:35:34.707537 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:35:34.707461    4461 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cb97f849-7c86-4f17-978c-f8e77298c1aa-tmp-dir\") on node \"ip-10-0-143-156.ec2.internal\" DevicePath \"\""
Oct 30 03:35:34.707537 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:35:34.707489    4461 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j79rq\" (UniqueName: \"kubernetes.io/projected/cb97f849-7c86-4f17-978c-f8e77298c1aa-kube-api-access-j79rq\") on node \"ip-10-0-143-156.ec2.internal\" DevicePath \"\""
Oct 30 03:35:34.707537 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:35:34.707498    4461 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cb97f849-7c86-4f17-978c-f8e77298c1aa-kserve-provision-location\") on node \"ip-10-0-143-156.ec2.internal\" DevicePath \"\""
Oct 30 03:35:34.707537 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:35:34.707506    4461 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cb97f849-7c86-4f17-978c-f8e77298c1aa-tls-certs\") on node \"ip-10-0-143-156.ec2.internal\" DevicePath \"\""
Oct 30 03:35:34.707537 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:35:34.707515    4461 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cb97f849-7c86-4f17-978c-f8e77298c1aa-dshm\") on node \"ip-10-0-143-156.ec2.internal\" DevicePath \"\""
Oct 30 03:35:34.805658 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:35:34.805623    4461 patch_prober.go:29] interesting pod/openshift-ai-inference-openshift-default-5b4f8cc59b-rx6vm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" start-of-body=
Oct 30 03:35:34.805834 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:35:34.805676    4461 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-rx6vm" podUID="35b8f310-5db9-4cf0-99ea-0baca82d3cda" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused"
Oct 30 03:35:35.045198 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:35:35.045164    4461 generic.go:356] "Generic (PLEG): container finished" podID="cb97f849-7c86-4f17-978c-f8e77298c1aa" containerID="bb40283fd8cc8588057039a772a112328ea7e7e14665fc42fd5ae8c218407b22" exitCode=0
Oct 30 03:35:35.045351 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:35:35.045213    4461 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-bd846f858-gwv2g"
Oct 30 03:35:35.045351 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:35:35.045238    4461 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-bd846f858-gwv2g" event={"ID":"cb97f849-7c86-4f17-978c-f8e77298c1aa","Type":"ContainerDied","Data":"bb40283fd8cc8588057039a772a112328ea7e7e14665fc42fd5ae8c218407b22"}
Oct 30 03:35:35.045351 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:35:35.045291    4461 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-bd846f858-gwv2g" event={"ID":"cb97f849-7c86-4f17-978c-f8e77298c1aa","Type":"ContainerDied","Data":"2493ed7a1f92d2c81412ef8f8cbe5e9404cf757425a60cc97689962e79abfac6"}
Oct 30 03:35:35.045351 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:35:35.045314    4461 scope.go:122] "RemoveContainer" containerID="bb40283fd8cc8588057039a772a112328ea7e7e14665fc42fd5ae8c218407b22"
Oct 30 03:35:35.057108 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:35:35.057074    4461 scope.go:122] "RemoveContainer" containerID="090b723af52947bb7662da589d7ac86c493b7fe40c3d6dba0b8a6ae4ef4d849d"
Oct 30 03:35:35.092611 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:35:35.092583    4461 scope.go:122] "RemoveContainer" containerID="bb40283fd8cc8588057039a772a112328ea7e7e14665fc42fd5ae8c218407b22"
Oct 30 03:35:35.092963 ip-10-0-143-156 kubenswrapper[4461]: E1030 03:35:35.092942    4461 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bb40283fd8cc8588057039a772a112328ea7e7e14665fc42fd5ae8c218407b22\": container with ID starting with bb40283fd8cc8588057039a772a112328ea7e7e14665fc42fd5ae8c218407b22 not found: ID does not exist" containerID="bb40283fd8cc8588057039a772a112328ea7e7e14665fc42fd5ae8c218407b22"
Oct 30 03:35:35.093058 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:35:35.092972    4461 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bb40283fd8cc8588057039a772a112328ea7e7e14665fc42fd5ae8c218407b22"} err="failed to get container status \"bb40283fd8cc8588057039a772a112328ea7e7e14665fc42fd5ae8c218407b22\": rpc error: code = NotFound desc = could not find container \"bb40283fd8cc8588057039a772a112328ea7e7e14665fc42fd5ae8c218407b22\": container with ID starting with bb40283fd8cc8588057039a772a112328ea7e7e14665fc42fd5ae8c218407b22 not found: ID does not exist"
Oct 30 03:35:35.093058 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:35:35.092991    4461 scope.go:122] "RemoveContainer" containerID="090b723af52947bb7662da589d7ac86c493b7fe40c3d6dba0b8a6ae4ef4d849d"
Oct 30 03:35:35.093518 ip-10-0-143-156 kubenswrapper[4461]: E1030 03:35:35.093493    4461 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"090b723af52947bb7662da589d7ac86c493b7fe40c3d6dba0b8a6ae4ef4d849d\": container with ID starting with 090b723af52947bb7662da589d7ac86c493b7fe40c3d6dba0b8a6ae4ef4d849d not found: ID does not exist" containerID="090b723af52947bb7662da589d7ac86c493b7fe40c3d6dba0b8a6ae4ef4d849d"
Oct 30 03:35:35.093585 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:35:35.093526    4461 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"090b723af52947bb7662da589d7ac86c493b7fe40c3d6dba0b8a6ae4ef4d849d"} err="failed to get container status \"090b723af52947bb7662da589d7ac86c493b7fe40c3d6dba0b8a6ae4ef4d849d\": rpc error: code = NotFound desc = could not find container \"090b723af52947bb7662da589d7ac86c493b7fe40c3d6dba0b8a6ae4ef4d849d\": container with ID starting with 090b723af52947bb7662da589d7ac86c493b7fe40c3d6dba0b8a6ae4ef4d849d not found: ID does not exist"
Oct 30 03:35:35.108269 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:35:35.108243    4461 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-bd846f858-gwv2g"]
Oct 30 03:35:35.112028 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:35:35.111993    4461 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-bd846f858-gwv2g"]
Oct 30 03:35:35.805656 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:35:35.805617    4461 patch_prober.go:29] interesting pod/openshift-ai-inference-openshift-default-5b4f8cc59b-rx6vm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" start-of-body=
Oct 30 03:35:35.805840 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:35:35.805700    4461 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-rx6vm" podUID="35b8f310-5db9-4cf0-99ea-0baca82d3cda" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused"
Oct 30 03:35:36.115689 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:35:36.115604    4461 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cb97f849-7c86-4f17-978c-f8e77298c1aa" path="/var/lib/kubelet/pods/cb97f849-7c86-4f17-978c-f8e77298c1aa/volumes"
Oct 30 03:35:36.805664 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:35:36.805629    4461 patch_prober.go:29] interesting pod/openshift-ai-inference-openshift-default-5b4f8cc59b-rx6vm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" start-of-body=
Oct 30 03:35:36.805857 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:35:36.805684    4461 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-rx6vm" podUID="35b8f310-5db9-4cf0-99ea-0baca82d3cda" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused"
Oct 30 03:35:37.805268 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:35:37.805229    4461 patch_prober.go:29] interesting pod/openshift-ai-inference-openshift-default-5b4f8cc59b-rx6vm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" start-of-body=
Oct 30 03:35:37.805670 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:35:37.805284    4461 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-rx6vm" podUID="35b8f310-5db9-4cf0-99ea-0baca82d3cda" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused"
Oct 30 03:35:38.809067 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:35:38.809034    4461 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-rx6vm"
Oct 30 03:35:38.809462 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:35:38.809325    4461 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-rx6vm"
Oct 30 03:35:38.809919 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:35:38.809901    4461 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-rx6vm"
Oct 30 03:35:45.705685 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:35:45.705648    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:35:54.503944 ip-10-0-143-156 kubenswrapper[4461]: E1030 03:35:54.503905    4461 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2k85b" containerName="extractor"
Oct 30 03:35:55.692760 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:35:55.692725    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:36:06.242242 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:36:06.242207    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:36:08.300363 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:36:08.300323    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:36:24.503827 ip-10-0-143-156 kubenswrapper[4461]: E1030 03:36:24.503789    4461 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2k85b" containerName="extractor"
Oct 30 03:36:46.008256 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:36:46.008162    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:36:54.504369 ip-10-0-143-156 kubenswrapper[4461]: E1030 03:36:54.504328    4461 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2k85b" containerName="extractor"
Oct 30 03:36:55.749459 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:36:55.749412    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:37:24.504358 ip-10-0-143-156 kubenswrapper[4461]: E1030 03:37:24.504320    4461 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2k85b" containerName="extractor"
Oct 30 03:37:42.278267 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:37:42.278230    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:37:42.452397 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:37:42.452366    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:37:44.570004 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:37:44.569973    4461 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-76c447b64-v4jqz"]
Oct 30 03:37:44.570450 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:37:44.570354    4461 cpu_manager.go:414] "containerMap: removing container" podUID="cb97f849-7c86-4f17-978c-f8e77298c1aa" containerName="main"
Oct 30 03:37:44.570450 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:37:44.570370    4461 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cb97f849-7c86-4f17-978c-f8e77298c1aa" containerName="main"
Oct 30 03:37:44.570450 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:37:44.570383    4461 cpu_manager.go:414] "containerMap: removing container" podUID="cb97f849-7c86-4f17-978c-f8e77298c1aa" containerName="storage-initializer"
Oct 30 03:37:44.570450 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:37:44.570389    4461 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cb97f849-7c86-4f17-978c-f8e77298c1aa" containerName="storage-initializer"
Oct 30 03:37:44.570615 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:37:44.570466    4461 memory_manager.go:365] "RemoveStaleState removing state" podUID="cb97f849-7c86-4f17-978c-f8e77298c1aa" containerName="main"
Oct 30 03:37:44.573494 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:37:44.573475    4461 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-76c447b64-v4jqz"
Oct 30 03:37:44.591575 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:37:44.591549    4461 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-0eb24714\"/\"seaweedfs-s3-creds\""
Oct 30 03:37:44.591722 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:37:44.591600    4461 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-0eb24714\"/\"default-dockercfg-hm8vv\""
Oct 30 03:37:44.604332 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:37:44.604305    4461 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-0eb24714\"/\"tokenizer-template-args-test-kserve-self-signed-certs\""
Oct 30 03:37:44.605684 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:37:44.605663    4461 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-76c447b64-v4jqz"]
Oct 30 03:37:44.642112 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:37:44.642075    4461 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/19127377-19af-4a38-956d-0fab260b8b2d-tls-certs\") pod \"tokenizer-template-args-test-kserve-76c447b64-v4jqz\" (UID: \"19127377-19af-4a38-956d-0fab260b8b2d\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-76c447b64-v4jqz"
Oct 30 03:37:44.642226 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:37:44.642144    4461 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bf5p4\" (UniqueName: \"kubernetes.io/projected/19127377-19af-4a38-956d-0fab260b8b2d-kube-api-access-bf5p4\") pod \"tokenizer-template-args-test-kserve-76c447b64-v4jqz\" (UID: \"19127377-19af-4a38-956d-0fab260b8b2d\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-76c447b64-v4jqz"
Oct 30 03:37:44.642226 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:37:44.642210    4461 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/19127377-19af-4a38-956d-0fab260b8b2d-cabundle-cert\") pod \"tokenizer-template-args-test-kserve-76c447b64-v4jqz\" (UID: \"19127377-19af-4a38-956d-0fab260b8b2d\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-76c447b64-v4jqz"
Oct 30 03:37:44.642298 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:37:44.642244    4461 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/19127377-19af-4a38-956d-0fab260b8b2d-kserve-provision-location\") pod \"tokenizer-template-args-test-kserve-76c447b64-v4jqz\" (UID: \"19127377-19af-4a38-956d-0fab260b8b2d\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-76c447b64-v4jqz"
Oct 30 03:37:44.642298 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:37:44.642277    4461 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/19127377-19af-4a38-956d-0fab260b8b2d-home\") pod \"tokenizer-template-args-test-kserve-76c447b64-v4jqz\" (UID: \"19127377-19af-4a38-956d-0fab260b8b2d\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-76c447b64-v4jqz"
Oct 30 03:37:44.642364 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:37:44.642350    4461 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/19127377-19af-4a38-956d-0fab260b8b2d-tmp-dir\") pod \"tokenizer-template-args-test-kserve-76c447b64-v4jqz\" (UID: \"19127377-19af-4a38-956d-0fab260b8b2d\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-76c447b64-v4jqz"
Oct 30 03:37:44.642406 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:37:44.642388    4461 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/19127377-19af-4a38-956d-0fab260b8b2d-dshm\") pod \"tokenizer-template-args-test-kserve-76c447b64-v4jqz\" (UID: \"19127377-19af-4a38-956d-0fab260b8b2d\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-76c447b64-v4jqz"
Oct 30 03:37:44.642441 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:37:44.642417    4461 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/19127377-19af-4a38-956d-0fab260b8b2d-model-cache\") pod \"tokenizer-template-args-test-kserve-76c447b64-v4jqz\" (UID: \"19127377-19af-4a38-956d-0fab260b8b2d\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-76c447b64-v4jqz"
Oct 30 03:37:44.651579 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:37:44.651554    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:37:44.675982 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:37:44.675961    4461 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-0eb24714\"/\"kube-root-ca.crt\""
Oct 30 03:37:44.676090 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:37:44.676047    4461 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-0eb24714\"/\"odh-kserve-custom-ca-bundle\""
Oct 30 03:37:44.676090 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:37:44.676068    4461 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-0eb24714\"/\"openshift-service-ca.crt\""
Oct 30 03:37:44.743342 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:37:44.743308    4461 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/19127377-19af-4a38-956d-0fab260b8b2d-tls-certs\") pod \"tokenizer-template-args-test-kserve-76c447b64-v4jqz\" (UID: \"19127377-19af-4a38-956d-0fab260b8b2d\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-76c447b64-v4jqz"
Oct 30 03:37:44.743515 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:37:44.743350    4461 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bf5p4\" (UniqueName: \"kubernetes.io/projected/19127377-19af-4a38-956d-0fab260b8b2d-kube-api-access-bf5p4\") pod \"tokenizer-template-args-test-kserve-76c447b64-v4jqz\" (UID: \"19127377-19af-4a38-956d-0fab260b8b2d\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-76c447b64-v4jqz"
Oct 30 03:37:44.743515 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:37:44.743390    4461 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/19127377-19af-4a38-956d-0fab260b8b2d-cabundle-cert\") pod \"tokenizer-template-args-test-kserve-76c447b64-v4jqz\" (UID: \"19127377-19af-4a38-956d-0fab260b8b2d\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-76c447b64-v4jqz"
Oct 30 03:37:44.743515 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:37:44.743437    4461 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/19127377-19af-4a38-956d-0fab260b8b2d-kserve-provision-location\") pod \"tokenizer-template-args-test-kserve-76c447b64-v4jqz\" (UID: \"19127377-19af-4a38-956d-0fab260b8b2d\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-76c447b64-v4jqz"
Oct 30 03:37:44.743515 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:37:44.743484    4461 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/19127377-19af-4a38-956d-0fab260b8b2d-home\") pod \"tokenizer-template-args-test-kserve-76c447b64-v4jqz\" (UID: \"19127377-19af-4a38-956d-0fab260b8b2d\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-76c447b64-v4jqz"
Oct 30 03:37:44.743727 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:37:44.743519    4461 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/19127377-19af-4a38-956d-0fab260b8b2d-tmp-dir\") pod \"tokenizer-template-args-test-kserve-76c447b64-v4jqz\" (UID: \"19127377-19af-4a38-956d-0fab260b8b2d\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-76c447b64-v4jqz"
Oct 30 03:37:44.743727 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:37:44.743557    4461 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/19127377-19af-4a38-956d-0fab260b8b2d-dshm\") pod \"tokenizer-template-args-test-kserve-76c447b64-v4jqz\" (UID: \"19127377-19af-4a38-956d-0fab260b8b2d\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-76c447b64-v4jqz"
Oct 30 03:37:44.743727 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:37:44.743586    4461 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/19127377-19af-4a38-956d-0fab260b8b2d-model-cache\") pod \"tokenizer-template-args-test-kserve-76c447b64-v4jqz\" (UID: \"19127377-19af-4a38-956d-0fab260b8b2d\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-76c447b64-v4jqz"
Oct 30 03:37:44.743974 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:37:44.743946    4461 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/19127377-19af-4a38-956d-0fab260b8b2d-home\") pod \"tokenizer-template-args-test-kserve-76c447b64-v4jqz\" (UID: \"19127377-19af-4a38-956d-0fab260b8b2d\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-76c447b64-v4jqz"
Oct 30 03:37:44.744069 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:37:44.744046    4461 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/19127377-19af-4a38-956d-0fab260b8b2d-kserve-provision-location\") pod \"tokenizer-template-args-test-kserve-76c447b64-v4jqz\" (UID: \"19127377-19af-4a38-956d-0fab260b8b2d\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-76c447b64-v4jqz"
Oct 30 03:37:44.744164 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:37:44.744129    4461 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/19127377-19af-4a38-956d-0fab260b8b2d-model-cache\") pod \"tokenizer-template-args-test-kserve-76c447b64-v4jqz\" (UID: \"19127377-19af-4a38-956d-0fab260b8b2d\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-76c447b64-v4jqz"
Oct 30 03:37:44.744221 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:37:44.744161    4461 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/19127377-19af-4a38-956d-0fab260b8b2d-tmp-dir\") pod \"tokenizer-template-args-test-kserve-76c447b64-v4jqz\" (UID: \"19127377-19af-4a38-956d-0fab260b8b2d\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-76c447b64-v4jqz"
Oct 30 03:37:44.744289 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:37:44.744268    4461 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/19127377-19af-4a38-956d-0fab260b8b2d-cabundle-cert\") pod \"tokenizer-template-args-test-kserve-76c447b64-v4jqz\" (UID: \"19127377-19af-4a38-956d-0fab260b8b2d\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-76c447b64-v4jqz"
Oct 30 03:37:44.745908 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:37:44.745886    4461 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/19127377-19af-4a38-956d-0fab260b8b2d-dshm\") pod \"tokenizer-template-args-test-kserve-76c447b64-v4jqz\" (UID: \"19127377-19af-4a38-956d-0fab260b8b2d\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-76c447b64-v4jqz"
Oct 30 03:37:44.746002 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:37:44.745990    4461 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/19127377-19af-4a38-956d-0fab260b8b2d-tls-certs\") pod \"tokenizer-template-args-test-kserve-76c447b64-v4jqz\" (UID: \"19127377-19af-4a38-956d-0fab260b8b2d\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-76c447b64-v4jqz"
Oct 30 03:37:44.766908 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:37:44.766883    4461 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bf5p4\" (UniqueName: \"kubernetes.io/projected/19127377-19af-4a38-956d-0fab260b8b2d-kube-api-access-bf5p4\") pod \"tokenizer-template-args-test-kserve-76c447b64-v4jqz\" (UID: \"19127377-19af-4a38-956d-0fab260b8b2d\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-76c447b64-v4jqz"
Oct 30 03:37:44.888333 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:37:44.888255    4461 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-76c447b64-v4jqz"
Oct 30 03:37:44.996160 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:37:44.996128    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:37:45.171172 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:37:45.171085    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:37:45.207434 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:37:45.207407    4461 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-76c447b64-v4jqz"]
Oct 30 03:37:45.211168 ip-10-0-143-156 kubenswrapper[4461]: W1030 03:37:45.211132    4461 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod19127377_19af_4a38_956d_0fab260b8b2d.slice/crio-df0ed2263c95755fc9aa6cc60b1fba8603c84eef01501b92c202ff26277d5acb WatchSource:0}: Error finding container df0ed2263c95755fc9aa6cc60b1fba8603c84eef01501b92c202ff26277d5acb: Status 404 returned error can't find the container with id df0ed2263c95755fc9aa6cc60b1fba8603c84eef01501b92c202ff26277d5acb
Oct 30 03:37:45.557463 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:37:45.557429    4461 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-76c447b64-v4jqz" event={"ID":"19127377-19af-4a38-956d-0fab260b8b2d","Type":"ContainerStarted","Data":"e236125cf249d5443e57b9904fd1e572d26067394b51301c48e40ecc567e806f"}
Oct 30 03:37:45.557672 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:37:45.557479    4461 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-76c447b64-v4jqz" event={"ID":"19127377-19af-4a38-956d-0fab260b8b2d","Type":"ContainerStarted","Data":"df0ed2263c95755fc9aa6cc60b1fba8603c84eef01501b92c202ff26277d5acb"}
Oct 30 03:37:46.099259 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:37:46.099221    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:37:48.614120 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:37:48.613319    4461 generic.go:356] "Generic (PLEG): container finished" podID="19127377-19af-4a38-956d-0fab260b8b2d" containerID="e236125cf249d5443e57b9904fd1e572d26067394b51301c48e40ecc567e806f" exitCode=0
Oct 30 03:37:48.614120 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:37:48.613400    4461 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-76c447b64-v4jqz" event={"ID":"19127377-19af-4a38-956d-0fab260b8b2d","Type":"ContainerDied","Data":"e236125cf249d5443e57b9904fd1e572d26067394b51301c48e40ecc567e806f"}
Oct 30 03:37:48.950564 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:37:48.950481    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:37:49.638713 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:37:49.638662    4461 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-76c447b64-v4jqz" event={"ID":"19127377-19af-4a38-956d-0fab260b8b2d","Type":"ContainerStarted","Data":"ce256bc90b4c9b9608fe8d7b2e617ad27cef4436909f38d2adde9491a37878a9"}
Oct 30 03:37:49.723120 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:37:49.722504    4461 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-76c447b64-v4jqz" podStartSLOduration=2.965823314 podStartE2EDuration="5.722469551s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.756646237s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:37:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:37:49.72068814 +0000 UTC m=+2537.770493318" watchObservedRunningTime="2026-10-30 03:37:49.722469551 +0000 UTC m=+2537.772274727"
Oct 30 03:37:49.893178 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:37:49.892187    4461 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-76c447b64-v4jqz"
Oct 30 03:37:50.923041 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:37:50.922959    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:37:51.142934 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:37:51.142903    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:37:54.503613 ip-10-0-143-156 kubenswrapper[4461]: E1030 03:37:54.503570    4461 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2k85b" containerName="extractor"
Oct 30 03:37:54.888739 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:37:54.888659    4461 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-76c447b64-v4jqz"
Oct 30 03:37:54.901053 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:37:54.901031    4461 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-76c447b64-v4jqz"
Oct 30 03:37:54.939264 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:37:54.939232    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:37:55.117617 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:37:55.117581    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:37:55.677008 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:37:55.676980    4461 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-76c447b64-v4jqz"
Oct 30 03:38:15.968685 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:38:15.968581    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:38:19.050271 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:38:19.050237    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:38:19.226787 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:38:19.226753    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:38:24.503013 ip-10-0-143-156 kubenswrapper[4461]: E1030 03:38:24.502970    4461 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2k85b" containerName="extractor"
Oct 30 03:38:34.079901 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:38:34.079863    4461 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-76c447b64-v4jqz"]
Oct 30 03:38:34.088906 ip-10-0-143-156 kubenswrapper[4461]: E1030 03:38:34.088870    4461 log.go:32] "ExecSync cmd from runtime service failed" err="rpc error: code = Unknown desc = command error: , stdout: , stderr: , exit code -1" containerID="ce256bc90b4c9b9608fe8d7b2e617ad27cef4436909f38d2adde9491a37878a9" cmd=["/bin/sleep","15"]
Oct 30 03:38:34.089063 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:38:34.088930    4461 handlers.go:85] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-76c447b64-v4jqz" message="" err="rpc error: code = Unknown desc = command error: , stdout: , stderr: , exit code -1"
Oct 30 03:38:34.089063 ip-10-0-143-156 kubenswrapper[4461]: E1030 03:38:34.088946    4461 kuberuntime_container.go:780] "PreStop hook failed" err="rpc error: code = Unknown desc = command error: , stdout: , stderr: , exit code -1" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-76c447b64-v4jqz" podUID="19127377-19af-4a38-956d-0fab260b8b2d" containerName="main" containerID="cri-o://ce256bc90b4c9b9608fe8d7b2e617ad27cef4436909f38d2adde9491a37878a9"
Oct 30 03:38:34.089063 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:38:34.088988    4461 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-76c447b64-v4jqz" podUID="19127377-19af-4a38-956d-0fab260b8b2d" containerName="main" containerID="cri-o://ce256bc90b4c9b9608fe8d7b2e617ad27cef4436909f38d2adde9491a37878a9" gracePeriod=60
Oct 30 03:38:34.111235 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:38:34.111205    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:38:34.298280 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:38:34.298251    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:38:34.430924 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:38:34.430899    4461 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-76c447b64-v4jqz"
Oct 30 03:38:34.501905 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:38:34.501876    4461 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/19127377-19af-4a38-956d-0fab260b8b2d-dshm\" (UniqueName: \"kubernetes.io/empty-dir/19127377-19af-4a38-956d-0fab260b8b2d-dshm\") pod \"19127377-19af-4a38-956d-0fab260b8b2d\" (UID: \"19127377-19af-4a38-956d-0fab260b8b2d\") "
Oct 30 03:38:34.501905 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:38:34.501910    4461 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/19127377-19af-4a38-956d-0fab260b8b2d-tls-certs\" (UniqueName: \"kubernetes.io/secret/19127377-19af-4a38-956d-0fab260b8b2d-tls-certs\") pod \"19127377-19af-4a38-956d-0fab260b8b2d\" (UID: \"19127377-19af-4a38-956d-0fab260b8b2d\") "
Oct 30 03:38:34.502135 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:38:34.501929    4461 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/19127377-19af-4a38-956d-0fab260b8b2d-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/19127377-19af-4a38-956d-0fab260b8b2d-cabundle-cert\") pod \"19127377-19af-4a38-956d-0fab260b8b2d\" (UID: \"19127377-19af-4a38-956d-0fab260b8b2d\") "
Oct 30 03:38:34.502135 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:38:34.501959    4461 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/19127377-19af-4a38-956d-0fab260b8b2d-home\" (UniqueName: \"kubernetes.io/empty-dir/19127377-19af-4a38-956d-0fab260b8b2d-home\") pod \"19127377-19af-4a38-956d-0fab260b8b2d\" (UID: \"19127377-19af-4a38-956d-0fab260b8b2d\") "
Oct 30 03:38:34.502135 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:38:34.501976    4461 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/19127377-19af-4a38-956d-0fab260b8b2d-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/19127377-19af-4a38-956d-0fab260b8b2d-tmp-dir\") pod \"19127377-19af-4a38-956d-0fab260b8b2d\" (UID: \"19127377-19af-4a38-956d-0fab260b8b2d\") "
Oct 30 03:38:34.502135 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:38:34.502003    4461 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/19127377-19af-4a38-956d-0fab260b8b2d-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/19127377-19af-4a38-956d-0fab260b8b2d-kserve-provision-location\") pod \"19127377-19af-4a38-956d-0fab260b8b2d\" (UID: \"19127377-19af-4a38-956d-0fab260b8b2d\") "
Oct 30 03:38:34.502135 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:38:34.502068    4461 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/19127377-19af-4a38-956d-0fab260b8b2d-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/19127377-19af-4a38-956d-0fab260b8b2d-model-cache\") pod \"19127377-19af-4a38-956d-0fab260b8b2d\" (UID: \"19127377-19af-4a38-956d-0fab260b8b2d\") "
Oct 30 03:38:34.502135 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:38:34.502127    4461 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/19127377-19af-4a38-956d-0fab260b8b2d-kube-api-access-bf5p4\" (UniqueName: \"kubernetes.io/projected/19127377-19af-4a38-956d-0fab260b8b2d-kube-api-access-bf5p4\") pod \"19127377-19af-4a38-956d-0fab260b8b2d\" (UID: \"19127377-19af-4a38-956d-0fab260b8b2d\") "
Oct 30 03:38:34.502417 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:38:34.502250    4461 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19127377-19af-4a38-956d-0fab260b8b2d-home" pod "19127377-19af-4a38-956d-0fab260b8b2d" (UID: "19127377-19af-4a38-956d-0fab260b8b2d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:38:34.502417 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:38:34.502365    4461 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19127377-19af-4a38-956d-0fab260b8b2d-tmp-dir" pod "19127377-19af-4a38-956d-0fab260b8b2d" (UID: "19127377-19af-4a38-956d-0fab260b8b2d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:38:34.502722 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:38:34.502517    4461 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/19127377-19af-4a38-956d-0fab260b8b2d-tmp-dir\") on node \"ip-10-0-143-156.ec2.internal\" DevicePath \"\""
Oct 30 03:38:34.502722 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:38:34.502556    4461 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/19127377-19af-4a38-956d-0fab260b8b2d-home\") on node \"ip-10-0-143-156.ec2.internal\" DevicePath \"\""
Oct 30 03:38:34.502722 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:38:34.502626    4461 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19127377-19af-4a38-956d-0fab260b8b2d-model-cache" pod "19127377-19af-4a38-956d-0fab260b8b2d" (UID: "19127377-19af-4a38-956d-0fab260b8b2d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:38:34.502722 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:38:34.502713    4461 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/19127377-19af-4a38-956d-0fab260b8b2d-cabundle-cert" pod "19127377-19af-4a38-956d-0fab260b8b2d" (UID: "19127377-19af-4a38-956d-0fab260b8b2d"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:38:34.504661 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:38:34.504638    4461 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19127377-19af-4a38-956d-0fab260b8b2d-dshm" pod "19127377-19af-4a38-956d-0fab260b8b2d" (UID: "19127377-19af-4a38-956d-0fab260b8b2d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:38:34.504771 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:38:34.504755    4461 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/19127377-19af-4a38-956d-0fab260b8b2d-kube-api-access-bf5p4" pod "19127377-19af-4a38-956d-0fab260b8b2d" (UID: "19127377-19af-4a38-956d-0fab260b8b2d"). InnerVolumeSpecName "kube-api-access-bf5p4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:38:34.504831 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:38:34.504803    4461 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/19127377-19af-4a38-956d-0fab260b8b2d-tls-certs" pod "19127377-19af-4a38-956d-0fab260b8b2d" (UID: "19127377-19af-4a38-956d-0fab260b8b2d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:38:34.521943 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:38:34.521910    4461 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19127377-19af-4a38-956d-0fab260b8b2d-kserve-provision-location" pod "19127377-19af-4a38-956d-0fab260b8b2d" (UID: "19127377-19af-4a38-956d-0fab260b8b2d"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:38:34.603872 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:38:34.603791    4461 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/19127377-19af-4a38-956d-0fab260b8b2d-kserve-provision-location\") on node \"ip-10-0-143-156.ec2.internal\" DevicePath \"\""
Oct 30 03:38:34.603872 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:38:34.603822    4461 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/19127377-19af-4a38-956d-0fab260b8b2d-model-cache\") on node \"ip-10-0-143-156.ec2.internal\" DevicePath \"\""
Oct 30 03:38:34.603872 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:38:34.603834    4461 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bf5p4\" (UniqueName: \"kubernetes.io/projected/19127377-19af-4a38-956d-0fab260b8b2d-kube-api-access-bf5p4\") on node \"ip-10-0-143-156.ec2.internal\" DevicePath \"\""
Oct 30 03:38:34.603872 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:38:34.603843    4461 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/19127377-19af-4a38-956d-0fab260b8b2d-dshm\") on node \"ip-10-0-143-156.ec2.internal\" DevicePath \"\""
Oct 30 03:38:34.603872 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:38:34.603852    4461 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/19127377-19af-4a38-956d-0fab260b8b2d-tls-certs\") on node \"ip-10-0-143-156.ec2.internal\" DevicePath \"\""
Oct 30 03:38:34.603872 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:38:34.603862    4461 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/19127377-19af-4a38-956d-0fab260b8b2d-cabundle-cert\") on node \"ip-10-0-143-156.ec2.internal\" DevicePath \"\""
Oct 30 03:38:34.824355 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:38:34.824322    4461 generic.go:356] "Generic (PLEG): container finished" podID="19127377-19af-4a38-956d-0fab260b8b2d" containerID="ce256bc90b4c9b9608fe8d7b2e617ad27cef4436909f38d2adde9491a37878a9" exitCode=0
Oct 30 03:38:34.824533 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:38:34.824374    4461 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-76c447b64-v4jqz"
Oct 30 03:38:34.824533 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:38:34.824374    4461 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-76c447b64-v4jqz" event={"ID":"19127377-19af-4a38-956d-0fab260b8b2d","Type":"ContainerDied","Data":"ce256bc90b4c9b9608fe8d7b2e617ad27cef4436909f38d2adde9491a37878a9"}
Oct 30 03:38:34.824533 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:38:34.824502    4461 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-76c447b64-v4jqz" event={"ID":"19127377-19af-4a38-956d-0fab260b8b2d","Type":"ContainerDied","Data":"df0ed2263c95755fc9aa6cc60b1fba8603c84eef01501b92c202ff26277d5acb"}
Oct 30 03:38:34.824533 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:38:34.824527    4461 scope.go:122] "RemoveContainer" containerID="ce256bc90b4c9b9608fe8d7b2e617ad27cef4436909f38d2adde9491a37878a9"
Oct 30 03:38:34.833663 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:38:34.833646    4461 scope.go:122] "RemoveContainer" containerID="e236125cf249d5443e57b9904fd1e572d26067394b51301c48e40ecc567e806f"
Oct 30 03:38:34.840900 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:38:34.840883    4461 scope.go:122] "RemoveContainer" containerID="ce256bc90b4c9b9608fe8d7b2e617ad27cef4436909f38d2adde9491a37878a9"
Oct 30 03:38:34.841165 ip-10-0-143-156 kubenswrapper[4461]: E1030 03:38:34.841146    4461 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ce256bc90b4c9b9608fe8d7b2e617ad27cef4436909f38d2adde9491a37878a9\": container with ID starting with ce256bc90b4c9b9608fe8d7b2e617ad27cef4436909f38d2adde9491a37878a9 not found: ID does not exist" containerID="ce256bc90b4c9b9608fe8d7b2e617ad27cef4436909f38d2adde9491a37878a9"
Oct 30 03:38:34.841227 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:38:34.841174    4461 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ce256bc90b4c9b9608fe8d7b2e617ad27cef4436909f38d2adde9491a37878a9"} err="failed to get container status \"ce256bc90b4c9b9608fe8d7b2e617ad27cef4436909f38d2adde9491a37878a9\": rpc error: code = NotFound desc = could not find container \"ce256bc90b4c9b9608fe8d7b2e617ad27cef4436909f38d2adde9491a37878a9\": container with ID starting with ce256bc90b4c9b9608fe8d7b2e617ad27cef4436909f38d2adde9491a37878a9 not found: ID does not exist"
Oct 30 03:38:34.841227 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:38:34.841193    4461 scope.go:122] "RemoveContainer" containerID="e236125cf249d5443e57b9904fd1e572d26067394b51301c48e40ecc567e806f"
Oct 30 03:38:34.841388 ip-10-0-143-156 kubenswrapper[4461]: E1030 03:38:34.841372    4461 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e236125cf249d5443e57b9904fd1e572d26067394b51301c48e40ecc567e806f\": container with ID starting with e236125cf249d5443e57b9904fd1e572d26067394b51301c48e40ecc567e806f not found: ID does not exist" containerID="e236125cf249d5443e57b9904fd1e572d26067394b51301c48e40ecc567e806f"
Oct 30 03:38:34.841427 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:38:34.841391    4461 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e236125cf249d5443e57b9904fd1e572d26067394b51301c48e40ecc567e806f"} err="failed to get container status \"e236125cf249d5443e57b9904fd1e572d26067394b51301c48e40ecc567e806f\": rpc error: code = NotFound desc = could not find container \"e236125cf249d5443e57b9904fd1e572d26067394b51301c48e40ecc567e806f\": container with ID starting with e236125cf249d5443e57b9904fd1e572d26067394b51301c48e40ecc567e806f not found: ID does not exist"
Oct 30 03:38:34.877653 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:38:34.877590    4461 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-76c447b64-v4jqz"]
Oct 30 03:38:34.890299 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:38:34.890276    4461 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-76c447b64-v4jqz"]
Oct 30 03:38:36.121952 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:38:36.121917    4461 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="19127377-19af-4a38-956d-0fab260b8b2d" path="/var/lib/kubelet/pods/19127377-19af-4a38-956d-0fab260b8b2d/volumes"
Oct 30 03:38:42.560899 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:38:42.560861    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:38:43.539003 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:38:43.538968    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:38:46.246281 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:38:46.246249    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:38:51.044866 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:38:51.044829    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:38:54.503493 ip-10-0-143-156 kubenswrapper[4461]: E1030 03:38:54.503456    4461 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2k85b" containerName="extractor"
Oct 30 03:39:16.094988 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:39:16.094945    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:39:17.229823 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:39:17.229793    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:39:24.185867 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:39:24.185832    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:39:24.355025 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:39:24.354988    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:39:24.504276 ip-10-0-143-156 kubenswrapper[4461]: E1030 03:39:24.504243    4461 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2k85b" containerName="extractor"
Oct 30 03:39:30.637011 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:39:30.636978    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:39:31.035291 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:39:31.035255    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:39:31.220234 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:39:31.220202    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:39:33.694223 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:39:33.694167    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:39:36.737566 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:39:36.737528    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:39:36.960136 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:39:36.960079    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:39:46.377053 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:39:46.376968    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:39:51.185339 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:39:51.185304    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:39:54.503641 ip-10-0-143-156 kubenswrapper[4461]: E1030 03:39:54.503600    4461 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2k85b" containerName="extractor"
Oct 30 03:40:06.081007 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:40:06.080976    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:40:06.250766 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:40:06.250731    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:40:16.188995 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:40:16.188956    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:40:24.503220 ip-10-0-143-156 kubenswrapper[4461]: E1030 03:40:24.503185    4461 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2k85b" containerName="extractor"
Oct 30 03:40:46.470449 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:40:46.470412    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:40:51.284241 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:40:51.284206    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:40:54.504283 ip-10-0-143-156 kubenswrapper[4461]: E1030 03:40:54.504245    4461 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2k85b" containerName="extractor"
Oct 30 03:40:56.168235 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:40:56.168149    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:40:56.335304 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:40:56.335269    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:41:01.356522 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:41:01.356487    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:41:01.655404 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:41:01.655308    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:41:03.736667 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:41:03.736623    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:41:09.853452 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:41:09.853416    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:41:10.021652 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:41:10.021617    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:41:11.504153 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:41:11.504120    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:41:16.302287 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:41:16.302189    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:41:24.502978 ip-10-0-143-156 kubenswrapper[4461]: E1030 03:41:24.502942    4461 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2k85b" containerName="extractor"
Oct 30 03:41:38.848767 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:41:38.848732    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:41:39.089444 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:41:39.089407    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:41:42.180676 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:41:42.180643    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:41:46.619726 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:41:46.619692    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:41:51.401067 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:41:51.401025    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:41:54.503314 ip-10-0-143-156 kubenswrapper[4461]: E1030 03:41:54.503269    4461 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2k85b" containerName="extractor"
Oct 30 03:42:13.550600 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:42:13.550557    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:42:13.732648 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:42:13.732591    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:42:16.432178 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:42:16.432142    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:42:24.503736 ip-10-0-143-156 kubenswrapper[4461]: E1030 03:42:24.503701    4461 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2k85b" containerName="extractor"
Oct 30 03:42:28.821654 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:42:28.821621    4461 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5577b665bbml9xz"]
Oct 30 03:42:28.830448 ip-10-0-143-156 kubenswrapper[4461]: E1030 03:42:28.830396    4461 log.go:32] "ExecSync cmd from runtime service failed" err="rpc error: code = Unknown desc = command error: , stdout: , stderr: , exit code -1" containerID="34ba4b6a609f5ae1ec99663a470bc744d000ae291f1ac1196da9756320da9527" cmd=["/bin/sleep","15"]
Oct 30 03:42:28.830582 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:42:28.830464    4461 handlers.go:85] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5577b665bbml9xz" message="" err="rpc error: code = Unknown desc = command error: , stdout: , stderr: , exit code -1"
Oct 30 03:42:28.830582 ip-10-0-143-156 kubenswrapper[4461]: E1030 03:42:28.830477    4461 kuberuntime_container.go:780] "PreStop hook failed" err="rpc error: code = Unknown desc = command error: , stdout: , stderr: , exit code -1" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5577b665bbml9xz" podUID="f36784f0-4807-48e8-8185-79bdec0fa85a" containerName="main" containerID="cri-o://34ba4b6a609f5ae1ec99663a470bc744d000ae291f1ac1196da9756320da9527"
Oct 30 03:42:28.830582 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:42:28.830524    4461 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5577b665bbml9xz" podUID="f36784f0-4807-48e8-8185-79bdec0fa85a" containerName="main" containerID="cri-o://34ba4b6a609f5ae1ec99663a470bc744d000ae291f1ac1196da9756320da9527" gracePeriod=60
Oct 30 03:42:28.914663 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:42:28.914626    4461 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5577b665bbml9xz" podUID="f36784f0-4807-48e8-8185-79bdec0fa85a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.35:8000/health\": dial tcp 10.133.0.35:8000: connect: connection refused"
Oct 30 03:42:28.939788 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:42:28.939759    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:42:29.170704 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:42:29.170675    4461 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5577b665bbml9xz"
Oct 30 03:42:29.282224 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:42:29.282172    4461 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f36784f0-4807-48e8-8185-79bdec0fa85a-tls-certs\" (UniqueName: \"kubernetes.io/secret/f36784f0-4807-48e8-8185-79bdec0fa85a-tls-certs\") pod \"f36784f0-4807-48e8-8185-79bdec0fa85a\" (UID: \"f36784f0-4807-48e8-8185-79bdec0fa85a\") "
Oct 30 03:42:29.282397 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:42:29.282242    4461 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f36784f0-4807-48e8-8185-79bdec0fa85a-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f36784f0-4807-48e8-8185-79bdec0fa85a-model-cache\") pod \"f36784f0-4807-48e8-8185-79bdec0fa85a\" (UID: \"f36784f0-4807-48e8-8185-79bdec0fa85a\") "
Oct 30 03:42:29.282397 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:42:29.282356    4461 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f36784f0-4807-48e8-8185-79bdec0fa85a-kube-api-access-ffsz9\" (UniqueName: \"kubernetes.io/projected/f36784f0-4807-48e8-8185-79bdec0fa85a-kube-api-access-ffsz9\") pod \"f36784f0-4807-48e8-8185-79bdec0fa85a\" (UID: \"f36784f0-4807-48e8-8185-79bdec0fa85a\") "
Oct 30 03:42:29.282498 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:42:29.282413    4461 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f36784f0-4807-48e8-8185-79bdec0fa85a-home\" (UniqueName: \"kubernetes.io/empty-dir/f36784f0-4807-48e8-8185-79bdec0fa85a-home\") pod \"f36784f0-4807-48e8-8185-79bdec0fa85a\" (UID: \"f36784f0-4807-48e8-8185-79bdec0fa85a\") "
Oct 30 03:42:29.282498 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:42:29.282477    4461 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f36784f0-4807-48e8-8185-79bdec0fa85a-dshm\" (UniqueName: \"kubernetes.io/empty-dir/f36784f0-4807-48e8-8185-79bdec0fa85a-dshm\") pod \"f36784f0-4807-48e8-8185-79bdec0fa85a\" (UID: \"f36784f0-4807-48e8-8185-79bdec0fa85a\") "
Oct 30 03:42:29.282498 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:42:29.282490    4461 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f36784f0-4807-48e8-8185-79bdec0fa85a-model-cache" pod "f36784f0-4807-48e8-8185-79bdec0fa85a" (UID: "f36784f0-4807-48e8-8185-79bdec0fa85a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:42:29.282660 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:42:29.282525    4461 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f36784f0-4807-48e8-8185-79bdec0fa85a-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f36784f0-4807-48e8-8185-79bdec0fa85a-tmp-dir\") pod \"f36784f0-4807-48e8-8185-79bdec0fa85a\" (UID: \"f36784f0-4807-48e8-8185-79bdec0fa85a\") "
Oct 30 03:42:29.282754 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:42:29.282716    4461 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f36784f0-4807-48e8-8185-79bdec0fa85a-home" pod "f36784f0-4807-48e8-8185-79bdec0fa85a" (UID: "f36784f0-4807-48e8-8185-79bdec0fa85a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:42:29.282854 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:42:29.282831    4461 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f36784f0-4807-48e8-8185-79bdec0fa85a-tmp-dir" pod "f36784f0-4807-48e8-8185-79bdec0fa85a" (UID: "f36784f0-4807-48e8-8185-79bdec0fa85a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:42:29.283060 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:42:29.283001    4461 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f36784f0-4807-48e8-8185-79bdec0fa85a-model-cache\") on node \"ip-10-0-143-156.ec2.internal\" DevicePath \"\""
Oct 30 03:42:29.283060 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:42:29.283024    4461 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f36784f0-4807-48e8-8185-79bdec0fa85a-home\") on node \"ip-10-0-143-156.ec2.internal\" DevicePath \"\""
Oct 30 03:42:29.283060 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:42:29.283037    4461 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f36784f0-4807-48e8-8185-79bdec0fa85a-tmp-dir\") on node \"ip-10-0-143-156.ec2.internal\" DevicePath \"\""
Oct 30 03:42:29.284664 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:42:29.284629    4461 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f36784f0-4807-48e8-8185-79bdec0fa85a-dshm" pod "f36784f0-4807-48e8-8185-79bdec0fa85a" (UID: "f36784f0-4807-48e8-8185-79bdec0fa85a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:42:29.284769 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:42:29.284713    4461 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f36784f0-4807-48e8-8185-79bdec0fa85a-kube-api-access-ffsz9" pod "f36784f0-4807-48e8-8185-79bdec0fa85a" (UID: "f36784f0-4807-48e8-8185-79bdec0fa85a"). InnerVolumeSpecName "kube-api-access-ffsz9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:42:29.284769 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:42:29.284764    4461 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f36784f0-4807-48e8-8185-79bdec0fa85a-tls-certs" pod "f36784f0-4807-48e8-8185-79bdec0fa85a" (UID: "f36784f0-4807-48e8-8185-79bdec0fa85a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:42:29.384490 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:42:29.384413    4461 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ffsz9\" (UniqueName: \"kubernetes.io/projected/f36784f0-4807-48e8-8185-79bdec0fa85a-kube-api-access-ffsz9\") on node \"ip-10-0-143-156.ec2.internal\" DevicePath \"\""
Oct 30 03:42:29.384490 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:42:29.384440    4461 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f36784f0-4807-48e8-8185-79bdec0fa85a-dshm\") on node \"ip-10-0-143-156.ec2.internal\" DevicePath \"\""
Oct 30 03:42:29.384490 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:42:29.384449    4461 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f36784f0-4807-48e8-8185-79bdec0fa85a-tls-certs\") on node \"ip-10-0-143-156.ec2.internal\" DevicePath \"\""
Oct 30 03:42:29.785737 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:42:29.785074    4461 generic.go:356] "Generic (PLEG): container finished" podID="f36784f0-4807-48e8-8185-79bdec0fa85a" containerID="34ba4b6a609f5ae1ec99663a470bc744d000ae291f1ac1196da9756320da9527" exitCode=0
Oct 30 03:42:29.785737 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:42:29.785202    4461 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5577b665bbml9xz" event={"ID":"f36784f0-4807-48e8-8185-79bdec0fa85a","Type":"ContainerDied","Data":"34ba4b6a609f5ae1ec99663a470bc744d000ae291f1ac1196da9756320da9527"}
Oct 30 03:42:29.785737 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:42:29.785297    4461 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5577b665bbml9xz" event={"ID":"f36784f0-4807-48e8-8185-79bdec0fa85a","Type":"ContainerDied","Data":"06a4332a5c652ffc9003982b082fc6b52c33316acea35aa7b519df4dad15a6aa"}
Oct 30 03:42:29.785737 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:42:29.785337    4461 scope.go:122] "RemoveContainer" containerID="34ba4b6a609f5ae1ec99663a470bc744d000ae291f1ac1196da9756320da9527"
Oct 30 03:42:29.785737 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:42:29.785534    4461 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5577b665bbml9xz"
Oct 30 03:42:29.795646 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:42:29.795626    4461 scope.go:122] "RemoveContainer" containerID="34ba4b6a609f5ae1ec99663a470bc744d000ae291f1ac1196da9756320da9527"
Oct 30 03:42:29.795910 ip-10-0-143-156 kubenswrapper[4461]: E1030 03:42:29.795892    4461 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"34ba4b6a609f5ae1ec99663a470bc744d000ae291f1ac1196da9756320da9527\": container with ID starting with 34ba4b6a609f5ae1ec99663a470bc744d000ae291f1ac1196da9756320da9527 not found: ID does not exist" containerID="34ba4b6a609f5ae1ec99663a470bc744d000ae291f1ac1196da9756320da9527"
Oct 30 03:42:29.795946 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:42:29.795919    4461 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"34ba4b6a609f5ae1ec99663a470bc744d000ae291f1ac1196da9756320da9527"} err="failed to get container status \"34ba4b6a609f5ae1ec99663a470bc744d000ae291f1ac1196da9756320da9527\": rpc error: code = NotFound desc = could not find container \"34ba4b6a609f5ae1ec99663a470bc744d000ae291f1ac1196da9756320da9527\": container with ID starting with 34ba4b6a609f5ae1ec99663a470bc744d000ae291f1ac1196da9756320da9527 not found: ID does not exist"
Oct 30 03:42:29.837744 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:42:29.837706    4461 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5577b665bbml9xz"]
Oct 30 03:42:29.850351 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:42:29.850326    4461 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5577b665bbml9xz"]
Oct 30 03:42:30.115157 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:42:30.115067    4461 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f36784f0-4807-48e8-8185-79bdec0fa85a" path="/var/lib/kubelet/pods/f36784f0-4807-48e8-8185-79bdec0fa85a/volumes"
Oct 30 03:42:33.891715 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:42:33.891680    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:42:34.191956 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:42:34.191825    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:42:38.756608 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:42:38.756570    4461 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-854b944bd8-mbwh6"]
Oct 30 03:42:38.757031 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:42:38.756843    4461 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-854b944bd8-mbwh6" podUID="06762940-e16c-4ab3-b9c6-b65345ba80a1" containerName="istio-proxy" containerID="cri-o://888fde87d4e38dafc9c8e5e545760ec5b2e1dc306f08940b5629a4f1d16c075d" gracePeriod=30
Oct 30 03:42:38.768447 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:42:38.768423    4461 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-668c779fc6ddcqx"]
Oct 30 03:42:38.768770 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:42:38.768758    4461 cpu_manager.go:414] "containerMap: removing container" podUID="19127377-19af-4a38-956d-0fab260b8b2d" containerName="storage-initializer"
Oct 30 03:42:38.768812 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:42:38.768772    4461 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="19127377-19af-4a38-956d-0fab260b8b2d" containerName="storage-initializer"
Oct 30 03:42:38.768812 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:42:38.768790    4461 cpu_manager.go:414] "containerMap: removing container" podUID="f36784f0-4807-48e8-8185-79bdec0fa85a" containerName="main"
Oct 30 03:42:38.768812 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:42:38.768796    4461 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f36784f0-4807-48e8-8185-79bdec0fa85a" containerName="main"
Oct 30 03:42:38.768812 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:42:38.768808    4461 cpu_manager.go:414] "containerMap: removing container" podUID="19127377-19af-4a38-956d-0fab260b8b2d" containerName="main"
Oct 30 03:42:38.768952 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:42:38.768814    4461 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="19127377-19af-4a38-956d-0fab260b8b2d" containerName="main"
Oct 30 03:42:38.768952 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:42:38.768896    4461 memory_manager.go:365] "RemoveStaleState removing state" podUID="f36784f0-4807-48e8-8185-79bdec0fa85a" containerName="main"
Oct 30 03:42:38.768952 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:42:38.768910    4461 memory_manager.go:365] "RemoveStaleState removing state" podUID="19127377-19af-4a38-956d-0fab260b8b2d" containerName="main"
Oct 30 03:42:38.773336 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:42:38.773316    4461 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-668c779fc6ddcqx"
Oct 30 03:42:38.788438 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:42:38.788401    4461 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-079cb970\"/\"odh-kserve-custom-ca-bundle\""
Oct 30 03:42:38.788438 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:42:38.788416    4461 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-079cb970\"/\"default-dockercfg-2gzq8\""
Oct 30 03:42:38.788438 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:42:38.788412    4461 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-079cb970\"/\"seaweedfs-s3-creds\""
Oct 30 03:42:38.788656 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:42:38.788454    4461 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-079cb970\"/\"kube-root-ca.crt\""
Oct 30 03:42:38.788656 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:42:38.788502    4461 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-079cb970\"/\"openshift-service-ca.crt\""
Oct 30 03:42:38.788656 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:42:38.788590    4461 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-079cb970\"/\"llmisva690bbc929faec8bc98c767f16c003c1-kserve-self-signed-certs\""
Oct 30 03:42:38.803137 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:42:38.803111    4461 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-668c779fc6ddcqx"]
Oct 30 03:42:38.868567 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:42:38.868536    4461 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/032898e3-3bbf-4e02-a24e-851329178601-home\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-668c779fc6ddcqx\" (UID: \"032898e3-3bbf-4e02-a24e-851329178601\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-668c779fc6ddcqx"
Oct 30 03:42:38.868702 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:42:38.868580    4461 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/032898e3-3bbf-4e02-a24e-851329178601-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-668c779fc6ddcqx\" (UID: \"032898e3-3bbf-4e02-a24e-851329178601\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-668c779fc6ddcqx"
Oct 30 03:42:38.868702 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:42:38.868650    4461 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/032898e3-3bbf-4e02-a24e-851329178601-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-668c779fc6ddcqx\" (UID: \"032898e3-3bbf-4e02-a24e-851329178601\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-668c779fc6ddcqx"
Oct 30 03:42:38.868702 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:42:38.868694    4461 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/032898e3-3bbf-4e02-a24e-851329178601-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-668c779fc6ddcqx\" (UID: \"032898e3-3bbf-4e02-a24e-851329178601\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-668c779fc6ddcqx"
Oct 30 03:42:38.868850 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:42:38.868715    4461 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/032898e3-3bbf-4e02-a24e-851329178601-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-668c779fc6ddcqx\" (UID: \"032898e3-3bbf-4e02-a24e-851329178601\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-668c779fc6ddcqx"
Oct 30 03:42:38.868850 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:42:38.868773    4461 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-49xff\" (UniqueName: \"kubernetes.io/projected/032898e3-3bbf-4e02-a24e-851329178601-kube-api-access-49xff\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-668c779fc6ddcqx\" (UID: \"032898e3-3bbf-4e02-a24e-851329178601\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-668c779fc6ddcqx"
Oct 30 03:42:38.868981 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:42:38.868955    4461 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/032898e3-3bbf-4e02-a24e-851329178601-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-668c779fc6ddcqx\" (UID: \"032898e3-3bbf-4e02-a24e-851329178601\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-668c779fc6ddcqx"
Oct 30 03:42:38.869082 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:42:38.869031    4461 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/032898e3-3bbf-4e02-a24e-851329178601-dshm\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-668c779fc6ddcqx\" (UID: \"032898e3-3bbf-4e02-a24e-851329178601\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-668c779fc6ddcqx"
Oct 30 03:42:38.869353 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:42:38.869334    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:42:38.969464 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:42:38.969430    4461 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/032898e3-3bbf-4e02-a24e-851329178601-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-668c779fc6ddcqx\" (UID: \"032898e3-3bbf-4e02-a24e-851329178601\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-668c779fc6ddcqx"
Oct 30 03:42:38.969640 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:42:38.969479    4461 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/032898e3-3bbf-4e02-a24e-851329178601-dshm\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-668c779fc6ddcqx\" (UID: \"032898e3-3bbf-4e02-a24e-851329178601\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-668c779fc6ddcqx"
Oct 30 03:42:38.969640 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:42:38.969511    4461 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/032898e3-3bbf-4e02-a24e-851329178601-home\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-668c779fc6ddcqx\" (UID: \"032898e3-3bbf-4e02-a24e-851329178601\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-668c779fc6ddcqx"
Oct 30 03:42:38.969640 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:42:38.969536    4461 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/032898e3-3bbf-4e02-a24e-851329178601-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-668c779fc6ddcqx\" (UID: \"032898e3-3bbf-4e02-a24e-851329178601\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-668c779fc6ddcqx"
Oct 30 03:42:38.969640 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:42:38.969558    4461 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/032898e3-3bbf-4e02-a24e-851329178601-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-668c779fc6ddcqx\" (UID: \"032898e3-3bbf-4e02-a24e-851329178601\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-668c779fc6ddcqx"
Oct 30 03:42:38.969640 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:42:38.969597    4461 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/032898e3-3bbf-4e02-a24e-851329178601-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-668c779fc6ddcqx\" (UID: \"032898e3-3bbf-4e02-a24e-851329178601\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-668c779fc6ddcqx"
Oct 30 03:42:38.969640 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:42:38.969622    4461 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/032898e3-3bbf-4e02-a24e-851329178601-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-668c779fc6ddcqx\" (UID: \"032898e3-3bbf-4e02-a24e-851329178601\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-668c779fc6ddcqx"
Oct 30 03:42:38.969925 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:42:38.969649    4461 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-49xff\" (UniqueName: \"kubernetes.io/projected/032898e3-3bbf-4e02-a24e-851329178601-kube-api-access-49xff\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-668c779fc6ddcqx\" (UID: \"032898e3-3bbf-4e02-a24e-851329178601\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-668c779fc6ddcqx"
Oct 30 03:42:38.969925 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:42:38.969896    4461 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/032898e3-3bbf-4e02-a24e-851329178601-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-668c779fc6ddcqx\" (UID: \"032898e3-3bbf-4e02-a24e-851329178601\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-668c779fc6ddcqx"
Oct 30 03:42:38.970038 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:42:38.969970    4461 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/032898e3-3bbf-4e02-a24e-851329178601-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-668c779fc6ddcqx\" (UID: \"032898e3-3bbf-4e02-a24e-851329178601\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-668c779fc6ddcqx"
Oct 30 03:42:38.970147 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:42:38.970116    4461 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/032898e3-3bbf-4e02-a24e-851329178601-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-668c779fc6ddcqx\" (UID: \"032898e3-3bbf-4e02-a24e-851329178601\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-668c779fc6ddcqx"
Oct 30 03:42:38.970262 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:42:38.970239    4461 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/032898e3-3bbf-4e02-a24e-851329178601-home\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-668c779fc6ddcqx\" (UID: \"032898e3-3bbf-4e02-a24e-851329178601\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-668c779fc6ddcqx"
Oct 30 03:42:38.970449 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:42:38.970433    4461 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/032898e3-3bbf-4e02-a24e-851329178601-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-668c779fc6ddcqx\" (UID: \"032898e3-3bbf-4e02-a24e-851329178601\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-668c779fc6ddcqx"
Oct 30 03:42:38.972081 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:42:38.972059    4461 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/032898e3-3bbf-4e02-a24e-851329178601-dshm\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-668c779fc6ddcqx\" (UID: \"032898e3-3bbf-4e02-a24e-851329178601\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-668c779fc6ddcqx"
Oct 30 03:42:38.972266 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:42:38.972248    4461 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/032898e3-3bbf-4e02-a24e-851329178601-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-668c779fc6ddcqx\" (UID: \"032898e3-3bbf-4e02-a24e-851329178601\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-668c779fc6ddcqx"
Oct 30 03:42:38.993672 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:42:38.993644    4461 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-49xff\" (UniqueName: \"kubernetes.io/projected/032898e3-3bbf-4e02-a24e-851329178601-kube-api-access-49xff\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-668c779fc6ddcqx\" (UID: \"032898e3-3bbf-4e02-a24e-851329178601\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-668c779fc6ddcqx"
Oct 30 03:42:39.047286 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:42:39.047206    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:42:39.053655 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:42:39.053626    4461 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schepp6g7"]
Oct 30 03:42:39.058134 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:42:39.058088    4461 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schepp6g7"
Oct 30 03:42:39.082965 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:42:39.082943    4461 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-079cb970\"/\"llmisvc-model-fb-opt-125m-route-87882a8e-epp-sa-dockercfg-jtrfh\""
Oct 30 03:42:39.088344 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:42:39.088310    4461 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schepp6g7"]
Oct 30 03:42:39.088527 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:42:39.088505    4461 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-668c779fc6ddcqx"
Oct 30 03:42:39.171863 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:42:39.171817    4461 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dfbxg\" (UniqueName: \"kubernetes.io/projected/9816db6e-ae8b-49a8-960f-7362ed59e5d6-kube-api-access-dfbxg\") pod \"llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schepp6g7\" (UID: \"9816db6e-ae8b-49a8-960f-7362ed59e5d6\") " pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schepp6g7"
Oct 30 03:42:39.172044 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:42:39.172022    4461 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9816db6e-ae8b-49a8-960f-7362ed59e5d6-tls-certs\") pod \"llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schepp6g7\" (UID: \"9816db6e-ae8b-49a8-960f-7362ed59e5d6\") " pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schepp6g7"
Oct 30 03:42:39.217383 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:42:39.217345    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:42:39.272696 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:42:39.272660    4461 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9816db6e-ae8b-49a8-960f-7362ed59e5d6-tls-certs\") pod \"llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schepp6g7\" (UID: \"9816db6e-ae8b-49a8-960f-7362ed59e5d6\") " pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schepp6g7"
Oct 30 03:42:39.272850 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:42:39.272714    4461 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dfbxg\" (UniqueName: \"kubernetes.io/projected/9816db6e-ae8b-49a8-960f-7362ed59e5d6-kube-api-access-dfbxg\") pod \"llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schepp6g7\" (UID: \"9816db6e-ae8b-49a8-960f-7362ed59e5d6\") " pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schepp6g7"
Oct 30 03:42:39.275262 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:42:39.275240    4461 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9816db6e-ae8b-49a8-960f-7362ed59e5d6-tls-certs\") pod \"llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schepp6g7\" (UID: \"9816db6e-ae8b-49a8-960f-7362ed59e5d6\") " pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schepp6g7"
Oct 30 03:42:39.296127 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:42:39.296109    4461 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dfbxg\" (UniqueName: \"kubernetes.io/projected/9816db6e-ae8b-49a8-960f-7362ed59e5d6-kube-api-access-dfbxg\") pod \"llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schepp6g7\" (UID: \"9816db6e-ae8b-49a8-960f-7362ed59e5d6\") " pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schepp6g7"
Oct 30 03:42:39.373817 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:42:39.373750    4461 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schepp6g7"
Oct 30 03:42:39.411660 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:42:39.411615    4461 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-668c779fc6ddcqx"]
Oct 30 03:42:39.686698 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:42:39.686673    4461 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schepp6g7"]
Oct 30 03:42:39.689336 ip-10-0-143-156 kubenswrapper[4461]: W1030 03:42:39.689309    4461 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9816db6e_ae8b_49a8_960f_7362ed59e5d6.slice/crio-c7548f99a21fb3d68ddcf52eee87dce90e81bf7062f08dc0802c9b5c62dfaeee WatchSource:0}: Error finding container c7548f99a21fb3d68ddcf52eee87dce90e81bf7062f08dc0802c9b5c62dfaeee: Status 404 returned error can't find the container with id c7548f99a21fb3d68ddcf52eee87dce90e81bf7062f08dc0802c9b5c62dfaeee
Oct 30 03:42:39.827125 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:42:39.827075    4461 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-668c779fc6ddcqx" event={"ID":"032898e3-3bbf-4e02-a24e-851329178601","Type":"ContainerStarted","Data":"31c0a2dbe10622d2140722878c4346e98bfa7d74421d6f98a4eef6103160e170"}
Oct 30 03:42:39.827531 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:42:39.827145    4461 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-668c779fc6ddcqx" event={"ID":"032898e3-3bbf-4e02-a24e-851329178601","Type":"ContainerStarted","Data":"12aad70add8bc245dc59978069f8e4727dd3a157e7198cd8b5cfbbc7c213b4ab"}
Oct 30 03:42:39.828604 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:42:39.828580    4461 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schepp6g7" event={"ID":"9816db6e-ae8b-49a8-960f-7362ed59e5d6","Type":"ContainerStarted","Data":"6711ce0a8c321be8bedadf3e1310b233644f9364c667e6abf48031a8b01c737b"}
Oct 30 03:42:39.828720 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:42:39.828619    4461 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schepp6g7" event={"ID":"9816db6e-ae8b-49a8-960f-7362ed59e5d6","Type":"ContainerStarted","Data":"c7548f99a21fb3d68ddcf52eee87dce90e81bf7062f08dc0802c9b5c62dfaeee"}
Oct 30 03:42:39.828720 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:42:39.828714    4461 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schepp6g7"
Oct 30 03:42:39.914938 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:42:39.914884    4461 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schepp6g7" podStartSLOduration=0.914871389 podStartE2EDuration="914.871389ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:42:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:42:39.913597803 +0000 UTC m=+2827.963403000" watchObservedRunningTime="2026-10-30 03:42:39.914871389 +0000 UTC m=+2827.964676565"
Oct 30 03:42:40.901813 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:42:40.901770    4461 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-854b944bd8-mbwh6" podUID="06762940-e16c-4ab3-b9c6-b65345ba80a1" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 03:42:41.472533 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:42:41.472479    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:42:41.838905 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:42:41.838827    4461 generic.go:356] "Generic (PLEG): container finished" podID="032898e3-3bbf-4e02-a24e-851329178601" containerID="31c0a2dbe10622d2140722878c4346e98bfa7d74421d6f98a4eef6103160e170" exitCode=0
Oct 30 03:42:41.838905 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:42:41.838876    4461 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-668c779fc6ddcqx" event={"ID":"032898e3-3bbf-4e02-a24e-851329178601","Type":"ContainerDied","Data":"31c0a2dbe10622d2140722878c4346e98bfa7d74421d6f98a4eef6103160e170"}
Oct 30 03:42:41.840306 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:42:41.840289    4461 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 03:42:44.062119 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:42:44.061960    4461 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-854b944bd8-mbwh6"
Oct 30 03:42:44.127451 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:42:44.127415    4461 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/06762940-e16c-4ab3-b9c6-b65345ba80a1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/06762940-e16c-4ab3-b9c6-b65345ba80a1-credential-socket\") pod \"06762940-e16c-4ab3-b9c6-b65345ba80a1\" (UID: \"06762940-e16c-4ab3-b9c6-b65345ba80a1\") "
Oct 30 03:42:44.127632 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:42:44.127473    4461 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/06762940-e16c-4ab3-b9c6-b65345ba80a1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/06762940-e16c-4ab3-b9c6-b65345ba80a1-istio-envoy\") pod \"06762940-e16c-4ab3-b9c6-b65345ba80a1\" (UID: \"06762940-e16c-4ab3-b9c6-b65345ba80a1\") "
Oct 30 03:42:44.127632 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:42:44.127504    4461 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/06762940-e16c-4ab3-b9c6-b65345ba80a1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/06762940-e16c-4ab3-b9c6-b65345ba80a1-workload-certs\") pod \"06762940-e16c-4ab3-b9c6-b65345ba80a1\" (UID: \"06762940-e16c-4ab3-b9c6-b65345ba80a1\") "
Oct 30 03:42:44.127632 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:42:44.127560    4461 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/06762940-e16c-4ab3-b9c6-b65345ba80a1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/06762940-e16c-4ab3-b9c6-b65345ba80a1-workload-socket\") pod \"06762940-e16c-4ab3-b9c6-b65345ba80a1\" (UID: \"06762940-e16c-4ab3-b9c6-b65345ba80a1\") "
Oct 30 03:42:44.127632 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:42:44.127610    4461 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/06762940-e16c-4ab3-b9c6-b65345ba80a1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/06762940-e16c-4ab3-b9c6-b65345ba80a1-istio-podinfo\") pod \"06762940-e16c-4ab3-b9c6-b65345ba80a1\" (UID: \"06762940-e16c-4ab3-b9c6-b65345ba80a1\") "
Oct 30 03:42:44.127861 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:42:44.127638    4461 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/06762940-e16c-4ab3-b9c6-b65345ba80a1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/06762940-e16c-4ab3-b9c6-b65345ba80a1-istiod-ca-cert\") pod \"06762940-e16c-4ab3-b9c6-b65345ba80a1\" (UID: \"06762940-e16c-4ab3-b9c6-b65345ba80a1\") "
Oct 30 03:42:44.127861 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:42:44.127673    4461 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06762940-e16c-4ab3-b9c6-b65345ba80a1-credential-socket" pod "06762940-e16c-4ab3-b9c6-b65345ba80a1" (UID: "06762940-e16c-4ab3-b9c6-b65345ba80a1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:42:44.127861 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:42:44.127713    4461 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/06762940-e16c-4ab3-b9c6-b65345ba80a1-istio-token\" (UniqueName: \"kubernetes.io/projected/06762940-e16c-4ab3-b9c6-b65345ba80a1-istio-token\") pod \"06762940-e16c-4ab3-b9c6-b65345ba80a1\" (UID: \"06762940-e16c-4ab3-b9c6-b65345ba80a1\") "
Oct 30 03:42:44.127861 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:42:44.127768    4461 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/06762940-e16c-4ab3-b9c6-b65345ba80a1-kube-api-access-d7sl4\" (UniqueName: \"kubernetes.io/projected/06762940-e16c-4ab3-b9c6-b65345ba80a1-kube-api-access-d7sl4\") pod \"06762940-e16c-4ab3-b9c6-b65345ba80a1\" (UID: \"06762940-e16c-4ab3-b9c6-b65345ba80a1\") "
Oct 30 03:42:44.127861 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:42:44.127816    4461 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/06762940-e16c-4ab3-b9c6-b65345ba80a1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/06762940-e16c-4ab3-b9c6-b65345ba80a1-istio-data\") pod \"06762940-e16c-4ab3-b9c6-b65345ba80a1\" (UID: \"06762940-e16c-4ab3-b9c6-b65345ba80a1\") "
Oct 30 03:42:44.127861 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:42:44.127853    4461 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06762940-e16c-4ab3-b9c6-b65345ba80a1-workload-certs" pod "06762940-e16c-4ab3-b9c6-b65345ba80a1" (UID: "06762940-e16c-4ab3-b9c6-b65345ba80a1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:42:44.128270 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:42:44.127882    4461 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06762940-e16c-4ab3-b9c6-b65345ba80a1-workload-socket" pod "06762940-e16c-4ab3-b9c6-b65345ba80a1" (UID: "06762940-e16c-4ab3-b9c6-b65345ba80a1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:42:44.128270 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:42:44.128080    4461 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/06762940-e16c-4ab3-b9c6-b65345ba80a1-credential-socket\") on node \"ip-10-0-143-156.ec2.internal\" DevicePath \"\""
Oct 30 03:42:44.128270 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:42:44.128116    4461 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/06762940-e16c-4ab3-b9c6-b65345ba80a1-workload-certs\") on node \"ip-10-0-143-156.ec2.internal\" DevicePath \"\""
Oct 30 03:42:44.128270 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:42:44.128130    4461 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/06762940-e16c-4ab3-b9c6-b65345ba80a1-workload-socket\") on node \"ip-10-0-143-156.ec2.internal\" DevicePath \"\""
Oct 30 03:42:44.128270 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:42:44.128160    4461 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/06762940-e16c-4ab3-b9c6-b65345ba80a1-istiod-ca-cert" pod "06762940-e16c-4ab3-b9c6-b65345ba80a1" (UID: "06762940-e16c-4ab3-b9c6-b65345ba80a1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:42:44.128701 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:42:44.128644    4461 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06762940-e16c-4ab3-b9c6-b65345ba80a1-istio-data" pod "06762940-e16c-4ab3-b9c6-b65345ba80a1" (UID: "06762940-e16c-4ab3-b9c6-b65345ba80a1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:42:44.130646 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:42:44.130618    4461 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/06762940-e16c-4ab3-b9c6-b65345ba80a1-istio-podinfo" pod "06762940-e16c-4ab3-b9c6-b65345ba80a1" (UID: "06762940-e16c-4ab3-b9c6-b65345ba80a1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 03:42:44.130797 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:42:44.130747    4461 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06762940-e16c-4ab3-b9c6-b65345ba80a1-istio-envoy" pod "06762940-e16c-4ab3-b9c6-b65345ba80a1" (UID: "06762940-e16c-4ab3-b9c6-b65345ba80a1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:42:44.132087 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:42:44.132055    4461 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/06762940-e16c-4ab3-b9c6-b65345ba80a1-kube-api-access-d7sl4" pod "06762940-e16c-4ab3-b9c6-b65345ba80a1" (UID: "06762940-e16c-4ab3-b9c6-b65345ba80a1"). InnerVolumeSpecName "kube-api-access-d7sl4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:42:44.134422 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:42:44.134328    4461 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/06762940-e16c-4ab3-b9c6-b65345ba80a1-istio-token" pod "06762940-e16c-4ab3-b9c6-b65345ba80a1" (UID: "06762940-e16c-4ab3-b9c6-b65345ba80a1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:42:44.229484 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:42:44.229304    4461 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/06762940-e16c-4ab3-b9c6-b65345ba80a1-istio-token\") on node \"ip-10-0-143-156.ec2.internal\" DevicePath \"\""
Oct 30 03:42:44.229484 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:42:44.229344    4461 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d7sl4\" (UniqueName: \"kubernetes.io/projected/06762940-e16c-4ab3-b9c6-b65345ba80a1-kube-api-access-d7sl4\") on node \"ip-10-0-143-156.ec2.internal\" DevicePath \"\""
Oct 30 03:42:44.229484 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:42:44.229364    4461 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/06762940-e16c-4ab3-b9c6-b65345ba80a1-istio-data\") on node \"ip-10-0-143-156.ec2.internal\" DevicePath \"\""
Oct 30 03:42:44.229484 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:42:44.229379    4461 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/06762940-e16c-4ab3-b9c6-b65345ba80a1-istio-envoy\") on node \"ip-10-0-143-156.ec2.internal\" DevicePath \"\""
Oct 30 03:42:44.229484 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:42:44.229392    4461 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/06762940-e16c-4ab3-b9c6-b65345ba80a1-istio-podinfo\") on node \"ip-10-0-143-156.ec2.internal\" DevicePath \"\""
Oct 30 03:42:44.229484 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:42:44.229405    4461 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/06762940-e16c-4ab3-b9c6-b65345ba80a1-istiod-ca-cert\") on node \"ip-10-0-143-156.ec2.internal\" DevicePath \"\""
Oct 30 03:42:44.859318 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:42:44.859280    4461 generic.go:356] "Generic (PLEG): container finished" podID="06762940-e16c-4ab3-b9c6-b65345ba80a1" containerID="888fde87d4e38dafc9c8e5e545760ec5b2e1dc306f08940b5629a4f1d16c075d" exitCode=0
Oct 30 03:42:44.859499 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:42:44.859335    4461 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-854b944bd8-mbwh6" event={"ID":"06762940-e16c-4ab3-b9c6-b65345ba80a1","Type":"ContainerDied","Data":"888fde87d4e38dafc9c8e5e545760ec5b2e1dc306f08940b5629a4f1d16c075d"}
Oct 30 03:42:44.859499 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:42:44.859383    4461 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-854b944bd8-mbwh6" event={"ID":"06762940-e16c-4ab3-b9c6-b65345ba80a1","Type":"ContainerDied","Data":"0f6e8ebd8772e5345e44d36229391b5093a890ec3a55d4388d210a1e4e9be60e"}
Oct 30 03:42:44.859499 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:42:44.859410    4461 scope.go:122] "RemoveContainer" containerID="888fde87d4e38dafc9c8e5e545760ec5b2e1dc306f08940b5629a4f1d16c075d"
Oct 30 03:42:44.859716 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:42:44.859578    4461 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-854b944bd8-mbwh6"
Oct 30 03:42:44.880708 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:42:44.880480    4461 scope.go:122] "RemoveContainer" containerID="a309c1aec24f7b79e5fae550841f8da7167453ceda31b232a615a7aefa6a60a4"
Oct 30 03:42:44.890279 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:42:44.890193    4461 scope.go:122] "RemoveContainer" containerID="888fde87d4e38dafc9c8e5e545760ec5b2e1dc306f08940b5629a4f1d16c075d"
Oct 30 03:42:44.890577 ip-10-0-143-156 kubenswrapper[4461]: E1030 03:42:44.890552    4461 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"888fde87d4e38dafc9c8e5e545760ec5b2e1dc306f08940b5629a4f1d16c075d\": container with ID starting with 888fde87d4e38dafc9c8e5e545760ec5b2e1dc306f08940b5629a4f1d16c075d not found: ID does not exist" containerID="888fde87d4e38dafc9c8e5e545760ec5b2e1dc306f08940b5629a4f1d16c075d"
Oct 30 03:42:44.890694 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:42:44.890588    4461 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"888fde87d4e38dafc9c8e5e545760ec5b2e1dc306f08940b5629a4f1d16c075d"} err="failed to get container status \"888fde87d4e38dafc9c8e5e545760ec5b2e1dc306f08940b5629a4f1d16c075d\": rpc error: code = NotFound desc = could not find container \"888fde87d4e38dafc9c8e5e545760ec5b2e1dc306f08940b5629a4f1d16c075d\": container with ID starting with 888fde87d4e38dafc9c8e5e545760ec5b2e1dc306f08940b5629a4f1d16c075d not found: ID does not exist"
Oct 30 03:42:44.890694 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:42:44.890607    4461 scope.go:122] "RemoveContainer" containerID="a309c1aec24f7b79e5fae550841f8da7167453ceda31b232a615a7aefa6a60a4"
Oct 30 03:42:44.890905 ip-10-0-143-156 kubenswrapper[4461]: E1030 03:42:44.890878    4461 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a309c1aec24f7b79e5fae550841f8da7167453ceda31b232a615a7aefa6a60a4\": container with ID starting with a309c1aec24f7b79e5fae550841f8da7167453ceda31b232a615a7aefa6a60a4 not found: ID does not exist" containerID="a309c1aec24f7b79e5fae550841f8da7167453ceda31b232a615a7aefa6a60a4"
Oct 30 03:42:44.890979 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:42:44.890915    4461 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a309c1aec24f7b79e5fae550841f8da7167453ceda31b232a615a7aefa6a60a4"} err="failed to get container status \"a309c1aec24f7b79e5fae550841f8da7167453ceda31b232a615a7aefa6a60a4\": rpc error: code = NotFound desc = could not find container \"a309c1aec24f7b79e5fae550841f8da7167453ceda31b232a615a7aefa6a60a4\": container with ID starting with a309c1aec24f7b79e5fae550841f8da7167453ceda31b232a615a7aefa6a60a4 not found: ID does not exist"
Oct 30 03:42:44.928154 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:42:44.928122    4461 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-854b944bd8-mbwh6"]
Oct 30 03:42:44.940780 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:42:44.940752    4461 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-854b944bd8-mbwh6"]
Oct 30 03:42:46.116761 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:42:46.116672    4461 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="06762940-e16c-4ab3-b9c6-b65345ba80a1" path="/var/lib/kubelet/pods/06762940-e16c-4ab3-b9c6-b65345ba80a1/volumes"
Oct 30 03:42:46.724819 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:42:46.724779    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:42:51.522058 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:42:51.522022    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:42:54.509349 ip-10-0-143-156 kubenswrapper[4461]: E1030 03:42:54.509309    4461 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2k85b" containerName="extractor"
Oct 30 03:43:00.644371 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:43:00.644318    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:43:01.643362 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:43:01.643273    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:43:03.416493 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:43:03.416458    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:43:03.585340 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:43:03.585305    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:43:07.530110 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:43:07.530073    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:43:09.982132 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:43:09.982081    4461 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-668c779fc6ddcqx" event={"ID":"032898e3-3bbf-4e02-a24e-851329178601","Type":"ContainerStarted","Data":"80b1a361f8a95fa00e843d35a741834033d9b2da927e3ba7f5facc926034322e"}
Oct 30 03:43:10.058786 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:43:10.058736    4461 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-668c779fc6ddcqx" podStartSLOduration=2.704728059 podStartE2EDuration="32.058721225s" totalImagesPullingTime="27.158990503s" totalInitContainerRuntime="2.195002663s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:42:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:43:10.044680505 +0000 UTC m=+2858.094485701" watchObservedRunningTime="2026-10-30 03:43:10.058721225 +0000 UTC m=+2858.108526401"
Oct 30 03:43:10.088706 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:43:10.088661    4461 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-668c779fc6ddcqx"
Oct 30 03:43:10.833572 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:43:10.833543    4461 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schepp6g7"
Oct 30 03:43:16.579029 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:43:16.578978    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:43:17.196028 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:43:17.195983    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:43:19.089326 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:43:19.089291    4461 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-668c779fc6ddcqx"
Oct 30 03:43:19.090891 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:43:19.090859    4461 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-668c779fc6ddcqx" podUID="032898e3-3bbf-4e02-a24e-851329178601" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/health\": dial tcp 10.133.0.40:8000: connect: connection refused"
Oct 30 03:43:22.566500 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:43:22.566456    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:43:22.729653 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:43:22.729619    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:43:24.503746 ip-10-0-143-156 kubenswrapper[4461]: E1030 03:43:24.503708    4461 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2k85b" containerName="extractor"
Oct 30 03:43:29.088978 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:43:29.088932    4461 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-668c779fc6ddcqx" podUID="032898e3-3bbf-4e02-a24e-851329178601" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/health\": dial tcp 10.133.0.40:8000: connect: connection refused"
Oct 30 03:43:39.089761 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:43:39.089711    4461 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-668c779fc6ddcqx" podUID="032898e3-3bbf-4e02-a24e-851329178601" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/health\": dial tcp 10.133.0.40:8000: connect: connection refused"
Oct 30 03:43:46.959310 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:43:46.959276    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:43:49.089436 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:43:49.089378    4461 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-668c779fc6ddcqx" podUID="032898e3-3bbf-4e02-a24e-851329178601" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/health\": dial tcp 10.133.0.40:8000: connect: connection refused"
Oct 30 03:43:53.492904 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:43:53.492870    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:43:53.668725 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:43:53.668694    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:43:54.503881 ip-10-0-143-156 kubenswrapper[4461]: E1030 03:43:54.503842    4461 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2k85b" containerName="extractor"
Oct 30 03:43:59.089728 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:43:59.089681    4461 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-668c779fc6ddcqx" podUID="032898e3-3bbf-4e02-a24e-851329178601" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/health\": dial tcp 10.133.0.40:8000: connect: connection refused"
Oct 30 03:44:00.156269 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:44:00.156237    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:44:01.669286 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:44:01.669249    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:44:02.745685 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:44:02.745648    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:44:03.048559 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:44:03.048472    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:44:06.065752 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:44:06.065717    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:44:06.235494 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:44:06.235457    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:44:09.088956 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:44:09.088911    4461 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-668c779fc6ddcqx" podUID="032898e3-3bbf-4e02-a24e-851329178601" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/health\": dial tcp 10.133.0.40:8000: connect: connection refused"
Oct 30 03:44:12.303332 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:44:12.303287    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:44:19.088987 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:44:19.088892    4461 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-668c779fc6ddcqx" podUID="032898e3-3bbf-4e02-a24e-851329178601" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/health\": dial tcp 10.133.0.40:8000: connect: connection refused"
Oct 30 03:44:23.862537 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:44:23.862499    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:44:24.504887 ip-10-0-143-156 kubenswrapper[4461]: E1030 03:44:24.504844    4461 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2k85b" containerName="extractor"
Oct 30 03:44:24.910843 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:44:24.910766    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:44:26.708373 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:44:26.708335    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:44:29.089611 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:44:29.089568    4461 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-668c779fc6ddcqx" podUID="032898e3-3bbf-4e02-a24e-851329178601" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/health\": dial tcp 10.133.0.40:8000: connect: connection refused"
Oct 30 03:44:39.089754 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:44:39.089702    4461 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-668c779fc6ddcqx" podUID="032898e3-3bbf-4e02-a24e-851329178601" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/health\": dial tcp 10.133.0.40:8000: connect: connection refused"
Oct 30 03:44:46.350406 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:44:46.350372    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:44:46.644081 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:44:46.643992    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:44:46.978789 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:44:46.978756    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:44:48.856925 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:44:48.856882    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:44:49.089170 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:44:49.089124    4461 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-668c779fc6ddcqx" podUID="032898e3-3bbf-4e02-a24e-851329178601" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/health\": dial tcp 10.133.0.40:8000: connect: connection refused"
Oct 30 03:44:54.503376 ip-10-0-143-156 kubenswrapper[4461]: E1030 03:44:54.503326    4461 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2k85b" containerName="extractor"
Oct 30 03:44:59.099107 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:44:59.099059    4461 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-668c779fc6ddcqx"
Oct 30 03:44:59.107022 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:44:59.106992    4461 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-668c779fc6ddcqx"
Oct 30 03:45:01.790719 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:45:01.790685    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:45:04.482761 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:45:04.482730    4461 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schepp6g7"]
Oct 30 03:45:04.496272 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:45:04.496238    4461 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-668c779fc6ddcqx"]
Oct 30 03:45:04.547131 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:45:04.547084    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:45:04.715809 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:45:04.715772    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:45:10.834088 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:45:10.834053    4461 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schepp6g7" podUID="9816db6e-ae8b-49a8-960f-7362ed59e5d6" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 03:45:19.502689 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:45:19.502646    4461 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schepp6g7" podUID="9816db6e-ae8b-49a8-960f-7362ed59e5d6" containerName="main" containerID="cri-o://6711ce0a8c321be8bedadf3e1310b233644f9364c667e6abf48031a8b01c737b" gracePeriod=45
Oct 30 03:45:19.519807 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:45:19.519768    4461 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-668c779fc6ddcqx" podUID="032898e3-3bbf-4e02-a24e-851329178601" containerName="main" containerID="cri-o://80b1a361f8a95fa00e843d35a741834033d9b2da927e3ba7f5facc926034322e" gracePeriod=45
Oct 30 03:45:20.100675 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:45:20.100631    4461 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-668c779fc6ddcqx" podUID="032898e3-3bbf-4e02-a24e-851329178601" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/health\": dial tcp 10.133.0.40:8000: connect: connection refused"
Oct 30 03:45:20.191497 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:45:20.191472    4461 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-668c779fc6ddcqx"
Oct 30 03:45:20.369558 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:45:20.369466    4461 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/032898e3-3bbf-4e02-a24e-851329178601-tls-certs\" (UniqueName: \"kubernetes.io/secret/032898e3-3bbf-4e02-a24e-851329178601-tls-certs\") pod \"032898e3-3bbf-4e02-a24e-851329178601\" (UID: \"032898e3-3bbf-4e02-a24e-851329178601\") "
Oct 30 03:45:20.369558 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:45:20.369520    4461 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/032898e3-3bbf-4e02-a24e-851329178601-kube-api-access-49xff\" (UniqueName: \"kubernetes.io/projected/032898e3-3bbf-4e02-a24e-851329178601-kube-api-access-49xff\") pod \"032898e3-3bbf-4e02-a24e-851329178601\" (UID: \"032898e3-3bbf-4e02-a24e-851329178601\") "
Oct 30 03:45:20.369558 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:45:20.369557    4461 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/032898e3-3bbf-4e02-a24e-851329178601-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/032898e3-3bbf-4e02-a24e-851329178601-kserve-provision-location\") pod \"032898e3-3bbf-4e02-a24e-851329178601\" (UID: \"032898e3-3bbf-4e02-a24e-851329178601\") "
Oct 30 03:45:20.369846 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:45:20.369597    4461 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/032898e3-3bbf-4e02-a24e-851329178601-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/032898e3-3bbf-4e02-a24e-851329178601-model-cache\") pod \"032898e3-3bbf-4e02-a24e-851329178601\" (UID: \"032898e3-3bbf-4e02-a24e-851329178601\") "
Oct 30 03:45:20.369846 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:45:20.369625    4461 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/032898e3-3bbf-4e02-a24e-851329178601-home\" (UniqueName: \"kubernetes.io/empty-dir/032898e3-3bbf-4e02-a24e-851329178601-home\") pod \"032898e3-3bbf-4e02-a24e-851329178601\" (UID: \"032898e3-3bbf-4e02-a24e-851329178601\") "
Oct 30 03:45:20.369846 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:45:20.369647    4461 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/032898e3-3bbf-4e02-a24e-851329178601-dshm\" (UniqueName: \"kubernetes.io/empty-dir/032898e3-3bbf-4e02-a24e-851329178601-dshm\") pod \"032898e3-3bbf-4e02-a24e-851329178601\" (UID: \"032898e3-3bbf-4e02-a24e-851329178601\") "
Oct 30 03:45:20.369846 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:45:20.369686    4461 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/032898e3-3bbf-4e02-a24e-851329178601-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/032898e3-3bbf-4e02-a24e-851329178601-tmp-dir\") pod \"032898e3-3bbf-4e02-a24e-851329178601\" (UID: \"032898e3-3bbf-4e02-a24e-851329178601\") "
Oct 30 03:45:20.369846 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:45:20.369739    4461 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/032898e3-3bbf-4e02-a24e-851329178601-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/032898e3-3bbf-4e02-a24e-851329178601-cabundle-cert\") pod \"032898e3-3bbf-4e02-a24e-851329178601\" (UID: \"032898e3-3bbf-4e02-a24e-851329178601\") "
Oct 30 03:45:20.370376 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:45:20.370340    4461 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/032898e3-3bbf-4e02-a24e-851329178601-model-cache" pod "032898e3-3bbf-4e02-a24e-851329178601" (UID: "032898e3-3bbf-4e02-a24e-851329178601"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:45:20.370494 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:45:20.370373    4461 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/032898e3-3bbf-4e02-a24e-851329178601-cabundle-cert" pod "032898e3-3bbf-4e02-a24e-851329178601" (UID: "032898e3-3bbf-4e02-a24e-851329178601"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:45:20.370494 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:45:20.370451    4461 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/032898e3-3bbf-4e02-a24e-851329178601-home" pod "032898e3-3bbf-4e02-a24e-851329178601" (UID: "032898e3-3bbf-4e02-a24e-851329178601"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:45:20.371916 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:45:20.371888    4461 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/032898e3-3bbf-4e02-a24e-851329178601-tls-certs" pod "032898e3-3bbf-4e02-a24e-851329178601" (UID: "032898e3-3bbf-4e02-a24e-851329178601"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:45:20.372141 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:45:20.372120    4461 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/032898e3-3bbf-4e02-a24e-851329178601-kube-api-access-49xff" pod "032898e3-3bbf-4e02-a24e-851329178601" (UID: "032898e3-3bbf-4e02-a24e-851329178601"). InnerVolumeSpecName "kube-api-access-49xff". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:45:20.372274 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:45:20.372253    4461 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/032898e3-3bbf-4e02-a24e-851329178601-dshm" pod "032898e3-3bbf-4e02-a24e-851329178601" (UID: "032898e3-3bbf-4e02-a24e-851329178601"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:45:20.382790 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:45:20.382757    4461 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/032898e3-3bbf-4e02-a24e-851329178601-tmp-dir" pod "032898e3-3bbf-4e02-a24e-851329178601" (UID: "032898e3-3bbf-4e02-a24e-851329178601"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:45:20.390045 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:45:20.390023    4461 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/032898e3-3bbf-4e02-a24e-851329178601-kserve-provision-location" pod "032898e3-3bbf-4e02-a24e-851329178601" (UID: "032898e3-3bbf-4e02-a24e-851329178601"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:45:20.471377 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:45:20.471350    4461 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/032898e3-3bbf-4e02-a24e-851329178601-tls-certs\") on node \"ip-10-0-143-156.ec2.internal\" DevicePath \"\""
Oct 30 03:45:20.471377 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:45:20.471375    4461 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-49xff\" (UniqueName: \"kubernetes.io/projected/032898e3-3bbf-4e02-a24e-851329178601-kube-api-access-49xff\") on node \"ip-10-0-143-156.ec2.internal\" DevicePath \"\""
Oct 30 03:45:20.471377 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:45:20.471384    4461 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/032898e3-3bbf-4e02-a24e-851329178601-kserve-provision-location\") on node \"ip-10-0-143-156.ec2.internal\" DevicePath \"\""
Oct 30 03:45:20.471604 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:45:20.471392    4461 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/032898e3-3bbf-4e02-a24e-851329178601-model-cache\") on node \"ip-10-0-143-156.ec2.internal\" DevicePath \"\""
Oct 30 03:45:20.471604 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:45:20.471400    4461 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/032898e3-3bbf-4e02-a24e-851329178601-home\") on node \"ip-10-0-143-156.ec2.internal\" DevicePath \"\""
Oct 30 03:45:20.471604 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:45:20.471408    4461 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/032898e3-3bbf-4e02-a24e-851329178601-dshm\") on node \"ip-10-0-143-156.ec2.internal\" DevicePath \"\""
Oct 30 03:45:20.471604 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:45:20.471415    4461 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/032898e3-3bbf-4e02-a24e-851329178601-tmp-dir\") on node \"ip-10-0-143-156.ec2.internal\" DevicePath \"\""
Oct 30 03:45:20.471604 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:45:20.471422    4461 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/032898e3-3bbf-4e02-a24e-851329178601-cabundle-cert\") on node \"ip-10-0-143-156.ec2.internal\" DevicePath \"\""
Oct 30 03:45:20.543630 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:45:20.543599    4461 generic.go:356] "Generic (PLEG): container finished" podID="032898e3-3bbf-4e02-a24e-851329178601" containerID="80b1a361f8a95fa00e843d35a741834033d9b2da927e3ba7f5facc926034322e" exitCode=0
Oct 30 03:45:20.544075 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:45:20.543666    4461 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-668c779fc6ddcqx"
Oct 30 03:45:20.544075 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:45:20.543699    4461 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-668c779fc6ddcqx" event={"ID":"032898e3-3bbf-4e02-a24e-851329178601","Type":"ContainerDied","Data":"80b1a361f8a95fa00e843d35a741834033d9b2da927e3ba7f5facc926034322e"}
Oct 30 03:45:20.544075 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:45:20.543752    4461 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-668c779fc6ddcqx" event={"ID":"032898e3-3bbf-4e02-a24e-851329178601","Type":"ContainerDied","Data":"12aad70add8bc245dc59978069f8e4727dd3a157e7198cd8b5cfbbc7c213b4ab"}
Oct 30 03:45:20.544075 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:45:20.543786    4461 scope.go:122] "RemoveContainer" containerID="80b1a361f8a95fa00e843d35a741834033d9b2da927e3ba7f5facc926034322e"
Oct 30 03:45:20.559301 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:45:20.559279    4461 scope.go:122] "RemoveContainer" containerID="31c0a2dbe10622d2140722878c4346e98bfa7d74421d6f98a4eef6103160e170"
Oct 30 03:45:20.568634 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:45:20.568613    4461 scope.go:122] "RemoveContainer" containerID="80b1a361f8a95fa00e843d35a741834033d9b2da927e3ba7f5facc926034322e"
Oct 30 03:45:20.568946 ip-10-0-143-156 kubenswrapper[4461]: E1030 03:45:20.568921    4461 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"80b1a361f8a95fa00e843d35a741834033d9b2da927e3ba7f5facc926034322e\": container with ID starting with 80b1a361f8a95fa00e843d35a741834033d9b2da927e3ba7f5facc926034322e not found: ID does not exist" containerID="80b1a361f8a95fa00e843d35a741834033d9b2da927e3ba7f5facc926034322e"
Oct 30 03:45:20.569043 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:45:20.568959    4461 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"80b1a361f8a95fa00e843d35a741834033d9b2da927e3ba7f5facc926034322e"} err="failed to get container status \"80b1a361f8a95fa00e843d35a741834033d9b2da927e3ba7f5facc926034322e\": rpc error: code = NotFound desc = could not find container \"80b1a361f8a95fa00e843d35a741834033d9b2da927e3ba7f5facc926034322e\": container with ID starting with 80b1a361f8a95fa00e843d35a741834033d9b2da927e3ba7f5facc926034322e not found: ID does not exist"
Oct 30 03:45:20.569043 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:45:20.568983    4461 scope.go:122] "RemoveContainer" containerID="31c0a2dbe10622d2140722878c4346e98bfa7d74421d6f98a4eef6103160e170"
Oct 30 03:45:20.569283 ip-10-0-143-156 kubenswrapper[4461]: E1030 03:45:20.569250    4461 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"31c0a2dbe10622d2140722878c4346e98bfa7d74421d6f98a4eef6103160e170\": container with ID starting with 31c0a2dbe10622d2140722878c4346e98bfa7d74421d6f98a4eef6103160e170 not found: ID does not exist" containerID="31c0a2dbe10622d2140722878c4346e98bfa7d74421d6f98a4eef6103160e170"
Oct 30 03:45:20.569361 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:45:20.569290    4461 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"31c0a2dbe10622d2140722878c4346e98bfa7d74421d6f98a4eef6103160e170"} err="failed to get container status \"31c0a2dbe10622d2140722878c4346e98bfa7d74421d6f98a4eef6103160e170\": rpc error: code = NotFound desc = could not find container \"31c0a2dbe10622d2140722878c4346e98bfa7d74421d6f98a4eef6103160e170\": container with ID starting with 31c0a2dbe10622d2140722878c4346e98bfa7d74421d6f98a4eef6103160e170 not found: ID does not exist"
Oct 30 03:45:20.598997 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:45:20.598964    4461 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-668c779fc6ddcqx"]
Oct 30 03:45:20.612950 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:45:20.612922    4461 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-668c779fc6ddcqx"]
Oct 30 03:45:20.670290 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:45:20.670214    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:45:20.834595 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:45:20.834557    4461 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schepp6g7" podUID="9816db6e-ae8b-49a8-960f-7362ed59e5d6" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 03:45:20.856134 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:45:20.855719    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:45:22.116108 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:45:22.116062    4461 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="032898e3-3bbf-4e02-a24e-851329178601" path="/var/lib/kubelet/pods/032898e3-3bbf-4e02-a24e-851329178601/volumes"
Oct 30 03:45:24.503807 ip-10-0-143-156 kubenswrapper[4461]: E1030 03:45:24.503703    4461 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2k85b" containerName="extractor"
Oct 30 03:45:26.835675 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:45:26.835632    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:45:30.834395 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:45:30.834358    4461 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schepp6g7" podUID="9816db6e-ae8b-49a8-960f-7362ed59e5d6" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 03:45:30.834770 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:45:30.834468    4461 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schepp6g7"
Oct 30 03:45:40.834297 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:45:40.834262    4461 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schepp6g7" podUID="9816db6e-ae8b-49a8-960f-7362ed59e5d6" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 03:45:43.756183 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:45:43.756147    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:45:44.901390 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:45:44.901299    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:45:47.131920 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:45:47.131867    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:45:49.664501 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:45:49.662601    4461 generic.go:356] "Generic (PLEG): container finished" podID="9816db6e-ae8b-49a8-960f-7362ed59e5d6" containerID="6711ce0a8c321be8bedadf3e1310b233644f9364c667e6abf48031a8b01c737b" exitCode=0
Oct 30 03:45:49.664501 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:45:49.662754    4461 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schepp6g7" event={"ID":"9816db6e-ae8b-49a8-960f-7362ed59e5d6","Type":"ContainerDied","Data":"6711ce0a8c321be8bedadf3e1310b233644f9364c667e6abf48031a8b01c737b"}
Oct 30 03:45:49.754863 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:45:49.754838    4461 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schepp6g7"
Oct 30 03:45:49.824274 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:45:49.824215    4461 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9816db6e-ae8b-49a8-960f-7362ed59e5d6-kube-api-access-dfbxg\" (UniqueName: \"kubernetes.io/projected/9816db6e-ae8b-49a8-960f-7362ed59e5d6-kube-api-access-dfbxg\") pod \"9816db6e-ae8b-49a8-960f-7362ed59e5d6\" (UID: \"9816db6e-ae8b-49a8-960f-7362ed59e5d6\") "
Oct 30 03:45:49.824403 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:45:49.824278    4461 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9816db6e-ae8b-49a8-960f-7362ed59e5d6-tls-certs\" (UniqueName: \"kubernetes.io/secret/9816db6e-ae8b-49a8-960f-7362ed59e5d6-tls-certs\") pod \"9816db6e-ae8b-49a8-960f-7362ed59e5d6\" (UID: \"9816db6e-ae8b-49a8-960f-7362ed59e5d6\") "
Oct 30 03:45:49.826246 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:45:49.826219    4461 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9816db6e-ae8b-49a8-960f-7362ed59e5d6-tls-certs" pod "9816db6e-ae8b-49a8-960f-7362ed59e5d6" (UID: "9816db6e-ae8b-49a8-960f-7362ed59e5d6"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:45:49.826345 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:45:49.826219    4461 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9816db6e-ae8b-49a8-960f-7362ed59e5d6-kube-api-access-dfbxg" pod "9816db6e-ae8b-49a8-960f-7362ed59e5d6" (UID: "9816db6e-ae8b-49a8-960f-7362ed59e5d6"). InnerVolumeSpecName "kube-api-access-dfbxg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:45:49.925840 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:45:49.925808    4461 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dfbxg\" (UniqueName: \"kubernetes.io/projected/9816db6e-ae8b-49a8-960f-7362ed59e5d6-kube-api-access-dfbxg\") on node \"ip-10-0-143-156.ec2.internal\" DevicePath \"\""
Oct 30 03:45:49.925840 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:45:49.925835    4461 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9816db6e-ae8b-49a8-960f-7362ed59e5d6-tls-certs\") on node \"ip-10-0-143-156.ec2.internal\" DevicePath \"\""
Oct 30 03:45:50.668340 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:45:50.668311    4461 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schepp6g7"
Oct 30 03:45:50.668340 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:45:50.668325    4461 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schepp6g7" event={"ID":"9816db6e-ae8b-49a8-960f-7362ed59e5d6","Type":"ContainerDied","Data":"c7548f99a21fb3d68ddcf52eee87dce90e81bf7062f08dc0802c9b5c62dfaeee"}
Oct 30 03:45:50.668870 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:45:50.668386    4461 scope.go:122] "RemoveContainer" containerID="6711ce0a8c321be8bedadf3e1310b233644f9364c667e6abf48031a8b01c737b"
Oct 30 03:45:50.722389 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:45:50.722360    4461 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schepp6g7"]
Oct 30 03:45:50.734742 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:45:50.734717    4461 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schepp6g7"]
Oct 30 03:45:52.115503 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:45:52.115472    4461 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9816db6e-ae8b-49a8-960f-7362ed59e5d6" path="/var/lib/kubelet/pods/9816db6e-ae8b-49a8-960f-7362ed59e5d6/volumes"
Oct 30 03:45:54.502678 ip-10-0-143-156 kubenswrapper[4461]: E1030 03:45:54.502646    4461 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2k85b" containerName="extractor"
Oct 30 03:45:54.686620 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:45:54.686590    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:45:55.131921 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:45:55.131887    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:46:01.435717 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:46:01.435672    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:46:01.883301 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:46:01.883270    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:46:07.519055 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:46:07.519018    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:46:07.698602 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:46:07.698569    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:46:10.837064 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:46:10.837027    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:46:11.034299 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:46:11.034265    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:46:24.504404 ip-10-0-143-156 kubenswrapper[4461]: E1030 03:46:24.504370    4461 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2k85b" containerName="extractor"
Oct 30 03:46:26.955587 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:46:26.955550    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:46:39.911330 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:46:39.911294    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:46:40.083035 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:46:40.083003    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:46:47.239089 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:46:47.239053    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:46:54.503622 ip-10-0-143-156 kubenswrapper[4461]: E1030 03:46:54.503580    4461 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2k85b" containerName="extractor"
Oct 30 03:47:02.017981 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:47:02.017946    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:47:15.327869 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:47:15.327781    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:47:24.504199 ip-10-0-143-156 kubenswrapper[4461]: E1030 03:47:24.504161    4461 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2k85b" containerName="extractor"
Oct 30 03:47:27.067736 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:47:27.067704    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:47:47.541955 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:47:47.541914    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:47:54.504535 ip-10-0-143-156 kubenswrapper[4461]: E1030 03:47:54.504493    4461 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2k85b" containerName="extractor"
Oct 30 03:48:02.138846 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:48:02.138805    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:48:06.881682 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:48:06.881646    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:48:07.051789 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:48:07.051757    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:48:11.661838 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:48:11.661800    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:48:11.902586 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:48:11.902540    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:48:16.909591 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:48:16.909552    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:48:21.265204 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:48:21.265165    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:48:21.469240 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:48:21.469202    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:48:24.504328 ip-10-0-143-156 kubenswrapper[4461]: E1030 03:48:24.504294    4461 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2k85b" containerName="extractor"
Oct 30 03:48:27.214189 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:48:27.214082    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:48:27.826877 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:48:27.826842    4461 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-dcdf746d6-zzv4v"]
Oct 30 03:48:27.827251 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:48:27.827234    4461 cpu_manager.go:414] "containerMap: removing container" podUID="9816db6e-ae8b-49a8-960f-7362ed59e5d6" containerName="main"
Oct 30 03:48:27.827353 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:48:27.827252    4461 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9816db6e-ae8b-49a8-960f-7362ed59e5d6" containerName="main"
Oct 30 03:48:27.827353 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:48:27.827265    4461 cpu_manager.go:414] "containerMap: removing container" podUID="032898e3-3bbf-4e02-a24e-851329178601" containerName="main"
Oct 30 03:48:27.827353 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:48:27.827274    4461 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="032898e3-3bbf-4e02-a24e-851329178601" containerName="main"
Oct 30 03:48:27.827353 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:48:27.827297    4461 cpu_manager.go:414] "containerMap: removing container" podUID="032898e3-3bbf-4e02-a24e-851329178601" containerName="storage-initializer"
Oct 30 03:48:27.827353 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:48:27.827305    4461 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="032898e3-3bbf-4e02-a24e-851329178601" containerName="storage-initializer"
Oct 30 03:48:27.827353 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:48:27.827327    4461 cpu_manager.go:414] "containerMap: removing container" podUID="06762940-e16c-4ab3-b9c6-b65345ba80a1" containerName="istio-proxy"
Oct 30 03:48:27.827353 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:48:27.827335    4461 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="06762940-e16c-4ab3-b9c6-b65345ba80a1" containerName="istio-proxy"
Oct 30 03:48:27.827710 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:48:27.827429    4461 memory_manager.go:365] "RemoveStaleState removing state" podUID="06762940-e16c-4ab3-b9c6-b65345ba80a1" containerName="istio-proxy"
Oct 30 03:48:27.827710 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:48:27.827442    4461 memory_manager.go:365] "RemoveStaleState removing state" podUID="9816db6e-ae8b-49a8-960f-7362ed59e5d6" containerName="main"
Oct 30 03:48:27.827710 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:48:27.827451    4461 memory_manager.go:365] "RemoveStaleState removing state" podUID="032898e3-3bbf-4e02-a24e-851329178601" containerName="main"
Oct 30 03:48:27.827710 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:48:27.827545    4461 cpu_manager.go:414] "containerMap: removing container" podUID="06762940-e16c-4ab3-b9c6-b65345ba80a1" containerName="istio-proxy"
Oct 30 03:48:27.827710 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:48:27.827554    4461 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="06762940-e16c-4ab3-b9c6-b65345ba80a1" containerName="istio-proxy"
Oct 30 03:48:27.827710 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:48:27.827638    4461 memory_manager.go:365] "RemoveStaleState removing state" podUID="06762940-e16c-4ab3-b9c6-b65345ba80a1" containerName="istio-proxy"
Oct 30 03:48:27.830733 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:48:27.830712    4461 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-dcdf746d6-zzv4v"
Oct 30 03:48:27.849656 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:48:27.849626    4461 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-dcdf746d6-zzv4v"]
Oct 30 03:48:27.861066 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:48:27.861029    4461 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-5bc7dccd\"/\"seaweedfs-s3-creds\""
Oct 30 03:48:27.861066 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:48:27.861053    4461 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-5bc7dccd\"/\"kube-root-ca.crt\""
Oct 30 03:48:27.861066 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:48:27.861067    4461 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-5bc7dccd\"/\"llmisv3e414c2ba058a022dfd694dbcbac5b51-kserve-self-signed-certs\""
Oct 30 03:48:27.861335 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:48:27.861045    4461 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-5bc7dccd\"/\"odh-kserve-custom-ca-bundle\""
Oct 30 03:48:27.861335 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:48:27.861175    4461 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-5bc7dccd\"/\"default-dockercfg-kgjd6\""
Oct 30 03:48:27.861444 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:48:27.861380    4461 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-5bc7dccd\"/\"openshift-service-ca.crt\""
Oct 30 03:48:27.924194 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:48:27.924162    4461 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/14296c92-55c6-44b3-8597-211a0e81e424-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-dcdf746d6-zzv4v\" (UID: \"14296c92-55c6-44b3-8597-211a0e81e424\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-dcdf746d6-zzv4v"
Oct 30 03:48:27.924372 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:48:27.924205    4461 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/14296c92-55c6-44b3-8597-211a0e81e424-model-cache\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-dcdf746d6-zzv4v\" (UID: \"14296c92-55c6-44b3-8597-211a0e81e424\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-dcdf746d6-zzv4v"
Oct 30 03:48:27.924372 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:48:27.924230    4461 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/14296c92-55c6-44b3-8597-211a0e81e424-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-dcdf746d6-zzv4v\" (UID: \"14296c92-55c6-44b3-8597-211a0e81e424\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-dcdf746d6-zzv4v"
Oct 30 03:48:27.924372 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:48:27.924327    4461 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/14296c92-55c6-44b3-8597-211a0e81e424-home\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-dcdf746d6-zzv4v\" (UID: \"14296c92-55c6-44b3-8597-211a0e81e424\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-dcdf746d6-zzv4v"
Oct 30 03:48:27.924372 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:48:27.924352    4461 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/14296c92-55c6-44b3-8597-211a0e81e424-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-dcdf746d6-zzv4v\" (UID: \"14296c92-55c6-44b3-8597-211a0e81e424\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-dcdf746d6-zzv4v"
Oct 30 03:48:27.924372 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:48:27.924369    4461 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/14296c92-55c6-44b3-8597-211a0e81e424-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-dcdf746d6-zzv4v\" (UID: \"14296c92-55c6-44b3-8597-211a0e81e424\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-dcdf746d6-zzv4v"
Oct 30 03:48:27.924564 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:48:27.924386    4461 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/14296c92-55c6-44b3-8597-211a0e81e424-dshm\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-dcdf746d6-zzv4v\" (UID: \"14296c92-55c6-44b3-8597-211a0e81e424\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-dcdf746d6-zzv4v"
Oct 30 03:48:27.924564 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:48:27.924408    4461 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d2kh8\" (UniqueName: \"kubernetes.io/projected/14296c92-55c6-44b3-8597-211a0e81e424-kube-api-access-d2kh8\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-dcdf746d6-zzv4v\" (UID: \"14296c92-55c6-44b3-8597-211a0e81e424\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-dcdf746d6-zzv4v"
Oct 30 03:48:27.931060 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:48:27.931015    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:48:28.025615 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:48:28.025576    4461 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/14296c92-55c6-44b3-8597-211a0e81e424-model-cache\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-dcdf746d6-zzv4v\" (UID: \"14296c92-55c6-44b3-8597-211a0e81e424\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-dcdf746d6-zzv4v"
Oct 30 03:48:28.025615 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:48:28.025618    4461 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/14296c92-55c6-44b3-8597-211a0e81e424-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-dcdf746d6-zzv4v\" (UID: \"14296c92-55c6-44b3-8597-211a0e81e424\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-dcdf746d6-zzv4v"
Oct 30 03:48:28.025859 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:48:28.025658    4461 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/14296c92-55c6-44b3-8597-211a0e81e424-home\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-dcdf746d6-zzv4v\" (UID: \"14296c92-55c6-44b3-8597-211a0e81e424\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-dcdf746d6-zzv4v"
Oct 30 03:48:28.025859 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:48:28.025684    4461 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/14296c92-55c6-44b3-8597-211a0e81e424-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-dcdf746d6-zzv4v\" (UID: \"14296c92-55c6-44b3-8597-211a0e81e424\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-dcdf746d6-zzv4v"
Oct 30 03:48:28.025859 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:48:28.025701    4461 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/14296c92-55c6-44b3-8597-211a0e81e424-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-dcdf746d6-zzv4v\" (UID: \"14296c92-55c6-44b3-8597-211a0e81e424\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-dcdf746d6-zzv4v"
Oct 30 03:48:28.025859 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:48:28.025726    4461 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/14296c92-55c6-44b3-8597-211a0e81e424-dshm\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-dcdf746d6-zzv4v\" (UID: \"14296c92-55c6-44b3-8597-211a0e81e424\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-dcdf746d6-zzv4v"
Oct 30 03:48:28.025859 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:48:28.025758    4461 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d2kh8\" (UniqueName: \"kubernetes.io/projected/14296c92-55c6-44b3-8597-211a0e81e424-kube-api-access-d2kh8\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-dcdf746d6-zzv4v\" (UID: \"14296c92-55c6-44b3-8597-211a0e81e424\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-dcdf746d6-zzv4v"
Oct 30 03:48:28.026130 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:48:28.025877    4461 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/14296c92-55c6-44b3-8597-211a0e81e424-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-dcdf746d6-zzv4v\" (UID: \"14296c92-55c6-44b3-8597-211a0e81e424\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-dcdf746d6-zzv4v"
Oct 30 03:48:28.026130 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:48:28.026068    4461 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/14296c92-55c6-44b3-8597-211a0e81e424-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-dcdf746d6-zzv4v\" (UID: \"14296c92-55c6-44b3-8597-211a0e81e424\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-dcdf746d6-zzv4v"
Oct 30 03:48:28.026242 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:48:28.026164    4461 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/14296c92-55c6-44b3-8597-211a0e81e424-model-cache\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-dcdf746d6-zzv4v\" (UID: \"14296c92-55c6-44b3-8597-211a0e81e424\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-dcdf746d6-zzv4v"
Oct 30 03:48:28.026242 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:48:28.026201    4461 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/14296c92-55c6-44b3-8597-211a0e81e424-home\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-dcdf746d6-zzv4v\" (UID: \"14296c92-55c6-44b3-8597-211a0e81e424\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-dcdf746d6-zzv4v"
Oct 30 03:48:28.026322 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:48:28.026249    4461 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/14296c92-55c6-44b3-8597-211a0e81e424-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-dcdf746d6-zzv4v\" (UID: \"14296c92-55c6-44b3-8597-211a0e81e424\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-dcdf746d6-zzv4v"
Oct 30 03:48:28.026424 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:48:28.026407    4461 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/14296c92-55c6-44b3-8597-211a0e81e424-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-dcdf746d6-zzv4v\" (UID: \"14296c92-55c6-44b3-8597-211a0e81e424\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-dcdf746d6-zzv4v"
Oct 30 03:48:28.028330 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:48:28.028301    4461 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/14296c92-55c6-44b3-8597-211a0e81e424-dshm\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-dcdf746d6-zzv4v\" (UID: \"14296c92-55c6-44b3-8597-211a0e81e424\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-dcdf746d6-zzv4v"
Oct 30 03:48:28.028527 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:48:28.028508    4461 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/14296c92-55c6-44b3-8597-211a0e81e424-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-dcdf746d6-zzv4v\" (UID: \"14296c92-55c6-44b3-8597-211a0e81e424\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-dcdf746d6-zzv4v"
Oct 30 03:48:28.049875 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:48:28.049850    4461 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d2kh8\" (UniqueName: \"kubernetes.io/projected/14296c92-55c6-44b3-8597-211a0e81e424-kube-api-access-d2kh8\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-dcdf746d6-zzv4v\" (UID: \"14296c92-55c6-44b3-8597-211a0e81e424\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-dcdf746d6-zzv4v"
Oct 30 03:48:28.145392 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:48:28.145310    4461 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-dcdf746d6-zzv4v"
Oct 30 03:48:28.265013 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:48:28.264977    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:48:28.475249 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:48:28.475201    4461 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-dcdf746d6-zzv4v"]
Oct 30 03:48:28.477654 ip-10-0-143-156 kubenswrapper[4461]: W1030 03:48:28.477627    4461 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod14296c92_55c6_44b3_8597_211a0e81e424.slice/crio-cc4715815d4122b8419f61b6e9b654da99c13846635a8eac960c570ace01279d WatchSource:0}: Error finding container cc4715815d4122b8419f61b6e9b654da99c13846635a8eac960c570ace01279d: Status 404 returned error can't find the container with id cc4715815d4122b8419f61b6e9b654da99c13846635a8eac960c570ace01279d
Oct 30 03:48:28.660311 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:48:28.660281    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:48:29.299087 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:48:29.299047    4461 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-dcdf746d6-zzv4v" event={"ID":"14296c92-55c6-44b3-8597-211a0e81e424","Type":"ContainerStarted","Data":"bd91548ecd9a6d5a6bf65f395b498ff964dac8d935749ac5730e47c493a1b8e1"}
Oct 30 03:48:29.299584 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:48:29.299124    4461 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-dcdf746d6-zzv4v" event={"ID":"14296c92-55c6-44b3-8597-211a0e81e424","Type":"ContainerStarted","Data":"cc4715815d4122b8419f61b6e9b654da99c13846635a8eac960c570ace01279d"}
Oct 30 03:48:31.314916 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:48:31.314883    4461 generic.go:356] "Generic (PLEG): container finished" podID="14296c92-55c6-44b3-8597-211a0e81e424" containerID="bd91548ecd9a6d5a6bf65f395b498ff964dac8d935749ac5730e47c493a1b8e1" exitCode=0
Oct 30 03:48:31.315313 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:48:31.314961    4461 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-dcdf746d6-zzv4v" event={"ID":"14296c92-55c6-44b3-8597-211a0e81e424","Type":"ContainerDied","Data":"bd91548ecd9a6d5a6bf65f395b498ff964dac8d935749ac5730e47c493a1b8e1"}
Oct 30 03:48:32.320948 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:48:32.320917    4461 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-dcdf746d6-zzv4v" event={"ID":"14296c92-55c6-44b3-8597-211a0e81e424","Type":"ContainerStarted","Data":"17e91a6412ad1c2606b22e6e9e39c6c4ee05ca77c7ceb293fd3c70a131cba508"}
Oct 30 03:48:32.394738 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:48:32.394673    4461 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-dcdf746d6-zzv4v" podStartSLOduration=2.6923319660000002 podStartE2EDuration="5.394658861s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.702326895s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:48:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:48:32.381503022 +0000 UTC m=+3180.431308198" watchObservedRunningTime="2026-10-30 03:48:32.394658861 +0000 UTC m=+3180.444464096"
Oct 30 03:48:33.030565 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:48:33.030526    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:48:33.145397 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:48:33.145362    4461 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-dcdf746d6-zzv4v"
Oct 30 03:48:35.056475 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:48:35.056437    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:48:35.233195 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:48:35.233158    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:48:38.146389 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:48:38.146353    4461 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-dcdf746d6-zzv4v"
Oct 30 03:48:38.147957 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:48:38.147923    4461 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-dcdf746d6-zzv4v" podUID="14296c92-55c6-44b3-8597-211a0e81e424" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/health\": dial tcp 10.133.0.42:8000: connect: connection refused"
Oct 30 03:48:47.484371 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:48:47.484285    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:48:48.146068 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:48:48.146024    4461 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-dcdf746d6-zzv4v" podUID="14296c92-55c6-44b3-8597-211a0e81e424" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/health\": dial tcp 10.133.0.42:8000: connect: connection refused"
Oct 30 03:48:54.504442 ip-10-0-143-156 kubenswrapper[4461]: E1030 03:48:54.504397    4461 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2k85b" containerName="extractor"
Oct 30 03:48:58.146238 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:48:58.146193    4461 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-dcdf746d6-zzv4v" podUID="14296c92-55c6-44b3-8597-211a0e81e424" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/health\": dial tcp 10.133.0.42:8000: connect: connection refused"
Oct 30 03:49:02.268473 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:49:02.268436    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:49:08.145774 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:49:08.145731    4461 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-dcdf746d6-zzv4v" podUID="14296c92-55c6-44b3-8597-211a0e81e424" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/health\": dial tcp 10.133.0.42:8000: connect: connection refused"
Oct 30 03:49:18.145628 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:49:18.145582    4461 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-dcdf746d6-zzv4v" podUID="14296c92-55c6-44b3-8597-211a0e81e424" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/health\": dial tcp 10.133.0.42:8000: connect: connection refused"
Oct 30 03:49:24.503317 ip-10-0-143-156 kubenswrapper[4461]: E1030 03:49:24.503274    4461 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2k85b" containerName="extractor"
Oct 30 03:49:27.316882 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:49:27.316847    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:49:28.146076 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:49:28.146028    4461 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-dcdf746d6-zzv4v" podUID="14296c92-55c6-44b3-8597-211a0e81e424" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/health\": dial tcp 10.133.0.42:8000: connect: connection refused"
Oct 30 03:49:38.146052 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:49:38.146005    4461 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-dcdf746d6-zzv4v" podUID="14296c92-55c6-44b3-8597-211a0e81e424" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/health\": dial tcp 10.133.0.42:8000: connect: connection refused"
Oct 30 03:49:42.469377 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:49:42.469332    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:49:47.619515 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:49:47.619476    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:49:48.156164 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:49:48.156131    4461 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-dcdf746d6-zzv4v"
Oct 30 03:49:48.163974 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:49:48.163947    4461 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-dcdf746d6-zzv4v"
Oct 30 03:49:54.504648 ip-10-0-143-156 kubenswrapper[4461]: E1030 03:49:54.504608    4461 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2k85b" containerName="extractor"
Oct 30 03:50:02.383374 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:50:02.383338    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:50:13.089566 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:50:13.089534    4461 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-dcdf746d6-zzv4v"]
Oct 30 03:50:13.181775 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:50:13.181740    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:50:13.371705 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:50:13.371624    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:50:18.410809 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:50:18.410725    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:50:18.582151 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:50:18.582116    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:50:24.503121 ip-10-0-143-156 kubenswrapper[4461]: E1030 03:50:24.503051    4461 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2k85b" containerName="extractor"
Oct 30 03:50:27.345667 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:50:27.345628    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:50:27.528795 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:50:27.528753    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:50:28.114000 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:50:28.113942    4461 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-dcdf746d6-zzv4v" podUID="14296c92-55c6-44b3-8597-211a0e81e424" containerName="main" containerID="cri-o://17e91a6412ad1c2606b22e6e9e39c6c4ee05ca77c7ceb293fd3c70a131cba508" gracePeriod=45
Oct 30 03:50:28.156711 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:50:28.156672    4461 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-dcdf746d6-zzv4v" podUID="14296c92-55c6-44b3-8597-211a0e81e424" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/health\": dial tcp 10.133.0.42:8000: connect: connection refused"
Oct 30 03:50:28.768230 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:50:28.768206    4461 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-dcdf746d6-zzv4v"
Oct 30 03:50:28.816351 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:50:28.816283    4461 generic.go:356] "Generic (PLEG): container finished" podID="14296c92-55c6-44b3-8597-211a0e81e424" containerID="17e91a6412ad1c2606b22e6e9e39c6c4ee05ca77c7ceb293fd3c70a131cba508" exitCode=0
Oct 30 03:50:28.816351 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:50:28.816335    4461 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-dcdf746d6-zzv4v"
Oct 30 03:50:28.816519 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:50:28.816334    4461 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-dcdf746d6-zzv4v" event={"ID":"14296c92-55c6-44b3-8597-211a0e81e424","Type":"ContainerDied","Data":"17e91a6412ad1c2606b22e6e9e39c6c4ee05ca77c7ceb293fd3c70a131cba508"}
Oct 30 03:50:28.816519 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:50:28.816466    4461 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-dcdf746d6-zzv4v" event={"ID":"14296c92-55c6-44b3-8597-211a0e81e424","Type":"ContainerDied","Data":"cc4715815d4122b8419f61b6e9b654da99c13846635a8eac960c570ace01279d"}
Oct 30 03:50:28.816519 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:50:28.816495    4461 scope.go:122] "RemoveContainer" containerID="17e91a6412ad1c2606b22e6e9e39c6c4ee05ca77c7ceb293fd3c70a131cba508"
Oct 30 03:50:28.824337 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:50:28.824317    4461 scope.go:122] "RemoveContainer" containerID="bd91548ecd9a6d5a6bf65f395b498ff964dac8d935749ac5730e47c493a1b8e1"
Oct 30 03:50:28.832472 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:50:28.832456    4461 scope.go:122] "RemoveContainer" containerID="17e91a6412ad1c2606b22e6e9e39c6c4ee05ca77c7ceb293fd3c70a131cba508"
Oct 30 03:50:28.832742 ip-10-0-143-156 kubenswrapper[4461]: E1030 03:50:28.832726    4461 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"17e91a6412ad1c2606b22e6e9e39c6c4ee05ca77c7ceb293fd3c70a131cba508\": container with ID starting with 17e91a6412ad1c2606b22e6e9e39c6c4ee05ca77c7ceb293fd3c70a131cba508 not found: ID does not exist" containerID="17e91a6412ad1c2606b22e6e9e39c6c4ee05ca77c7ceb293fd3c70a131cba508"
Oct 30 03:50:28.832781 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:50:28.832761    4461 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"17e91a6412ad1c2606b22e6e9e39c6c4ee05ca77c7ceb293fd3c70a131cba508"} err="failed to get container status \"17e91a6412ad1c2606b22e6e9e39c6c4ee05ca77c7ceb293fd3c70a131cba508\": rpc error: code = NotFound desc = could not find container \"17e91a6412ad1c2606b22e6e9e39c6c4ee05ca77c7ceb293fd3c70a131cba508\": container with ID starting with 17e91a6412ad1c2606b22e6e9e39c6c4ee05ca77c7ceb293fd3c70a131cba508 not found: ID does not exist"
Oct 30 03:50:28.832781 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:50:28.832778    4461 scope.go:122] "RemoveContainer" containerID="bd91548ecd9a6d5a6bf65f395b498ff964dac8d935749ac5730e47c493a1b8e1"
Oct 30 03:50:28.833030 ip-10-0-143-156 kubenswrapper[4461]: E1030 03:50:28.833016    4461 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bd91548ecd9a6d5a6bf65f395b498ff964dac8d935749ac5730e47c493a1b8e1\": container with ID starting with bd91548ecd9a6d5a6bf65f395b498ff964dac8d935749ac5730e47c493a1b8e1 not found: ID does not exist" containerID="bd91548ecd9a6d5a6bf65f395b498ff964dac8d935749ac5730e47c493a1b8e1"
Oct 30 03:50:28.833066 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:50:28.833036    4461 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bd91548ecd9a6d5a6bf65f395b498ff964dac8d935749ac5730e47c493a1b8e1"} err="failed to get container status \"bd91548ecd9a6d5a6bf65f395b498ff964dac8d935749ac5730e47c493a1b8e1\": rpc error: code = NotFound desc = could not find container \"bd91548ecd9a6d5a6bf65f395b498ff964dac8d935749ac5730e47c493a1b8e1\": container with ID starting with bd91548ecd9a6d5a6bf65f395b498ff964dac8d935749ac5730e47c493a1b8e1 not found: ID does not exist"
Oct 30 03:50:28.914284 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:50:28.914253    4461 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/14296c92-55c6-44b3-8597-211a0e81e424-kube-api-access-d2kh8\" (UniqueName: \"kubernetes.io/projected/14296c92-55c6-44b3-8597-211a0e81e424-kube-api-access-d2kh8\") pod \"14296c92-55c6-44b3-8597-211a0e81e424\" (UID: \"14296c92-55c6-44b3-8597-211a0e81e424\") "
Oct 30 03:50:28.914457 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:50:28.914311    4461 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/14296c92-55c6-44b3-8597-211a0e81e424-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/14296c92-55c6-44b3-8597-211a0e81e424-cabundle-cert\") pod \"14296c92-55c6-44b3-8597-211a0e81e424\" (UID: \"14296c92-55c6-44b3-8597-211a0e81e424\") "
Oct 30 03:50:28.914457 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:50:28.914381    4461 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/14296c92-55c6-44b3-8597-211a0e81e424-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/14296c92-55c6-44b3-8597-211a0e81e424-tmp-dir\") pod \"14296c92-55c6-44b3-8597-211a0e81e424\" (UID: \"14296c92-55c6-44b3-8597-211a0e81e424\") "
Oct 30 03:50:28.914457 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:50:28.914436    4461 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/14296c92-55c6-44b3-8597-211a0e81e424-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/14296c92-55c6-44b3-8597-211a0e81e424-kserve-provision-location\") pod \"14296c92-55c6-44b3-8597-211a0e81e424\" (UID: \"14296c92-55c6-44b3-8597-211a0e81e424\") "
Oct 30 03:50:28.914577 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:50:28.914493    4461 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/14296c92-55c6-44b3-8597-211a0e81e424-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/14296c92-55c6-44b3-8597-211a0e81e424-model-cache\") pod \"14296c92-55c6-44b3-8597-211a0e81e424\" (UID: \"14296c92-55c6-44b3-8597-211a0e81e424\") "
Oct 30 03:50:28.914577 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:50:28.914550    4461 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/14296c92-55c6-44b3-8597-211a0e81e424-dshm\" (UniqueName: \"kubernetes.io/empty-dir/14296c92-55c6-44b3-8597-211a0e81e424-dshm\") pod \"14296c92-55c6-44b3-8597-211a0e81e424\" (UID: \"14296c92-55c6-44b3-8597-211a0e81e424\") "
Oct 30 03:50:28.914683 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:50:28.914591    4461 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/14296c92-55c6-44b3-8597-211a0e81e424-home\" (UniqueName: \"kubernetes.io/empty-dir/14296c92-55c6-44b3-8597-211a0e81e424-home\") pod \"14296c92-55c6-44b3-8597-211a0e81e424\" (UID: \"14296c92-55c6-44b3-8597-211a0e81e424\") "
Oct 30 03:50:28.914683 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:50:28.914623    4461 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/14296c92-55c6-44b3-8597-211a0e81e424-tls-certs\" (UniqueName: \"kubernetes.io/secret/14296c92-55c6-44b3-8597-211a0e81e424-tls-certs\") pod \"14296c92-55c6-44b3-8597-211a0e81e424\" (UID: \"14296c92-55c6-44b3-8597-211a0e81e424\") "
Oct 30 03:50:28.914811 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:50:28.914705    4461 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/14296c92-55c6-44b3-8597-211a0e81e424-cabundle-cert" pod "14296c92-55c6-44b3-8597-211a0e81e424" (UID: "14296c92-55c6-44b3-8597-211a0e81e424"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:50:28.914811 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:50:28.914781    4461 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/14296c92-55c6-44b3-8597-211a0e81e424-model-cache" pod "14296c92-55c6-44b3-8597-211a0e81e424" (UID: "14296c92-55c6-44b3-8597-211a0e81e424"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:50:28.914935 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:50:28.914914    4461 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/14296c92-55c6-44b3-8597-211a0e81e424-cabundle-cert\") on node \"ip-10-0-143-156.ec2.internal\" DevicePath \"\""
Oct 30 03:50:28.914992 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:50:28.914939    4461 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/14296c92-55c6-44b3-8597-211a0e81e424-model-cache\") on node \"ip-10-0-143-156.ec2.internal\" DevicePath \"\""
Oct 30 03:50:28.915089 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:50:28.915059    4461 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/14296c92-55c6-44b3-8597-211a0e81e424-home" pod "14296c92-55c6-44b3-8597-211a0e81e424" (UID: "14296c92-55c6-44b3-8597-211a0e81e424"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:50:28.916901 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:50:28.916879    4461 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/14296c92-55c6-44b3-8597-211a0e81e424-kube-api-access-d2kh8" pod "14296c92-55c6-44b3-8597-211a0e81e424" (UID: "14296c92-55c6-44b3-8597-211a0e81e424"). InnerVolumeSpecName "kube-api-access-d2kh8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:50:28.917000 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:50:28.916938    4461 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/14296c92-55c6-44b3-8597-211a0e81e424-dshm" pod "14296c92-55c6-44b3-8597-211a0e81e424" (UID: "14296c92-55c6-44b3-8597-211a0e81e424"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:50:28.917110 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:50:28.917083    4461 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/14296c92-55c6-44b3-8597-211a0e81e424-tls-certs" pod "14296c92-55c6-44b3-8597-211a0e81e424" (UID: "14296c92-55c6-44b3-8597-211a0e81e424"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:50:28.926394 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:50:28.926370    4461 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/14296c92-55c6-44b3-8597-211a0e81e424-tmp-dir" pod "14296c92-55c6-44b3-8597-211a0e81e424" (UID: "14296c92-55c6-44b3-8597-211a0e81e424"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:50:28.936599 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:50:28.936569    4461 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/14296c92-55c6-44b3-8597-211a0e81e424-kserve-provision-location" pod "14296c92-55c6-44b3-8597-211a0e81e424" (UID: "14296c92-55c6-44b3-8597-211a0e81e424"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:50:29.016385 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:50:29.016349    4461 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/14296c92-55c6-44b3-8597-211a0e81e424-tmp-dir\") on node \"ip-10-0-143-156.ec2.internal\" DevicePath \"\""
Oct 30 03:50:29.016385 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:50:29.016378    4461 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/14296c92-55c6-44b3-8597-211a0e81e424-kserve-provision-location\") on node \"ip-10-0-143-156.ec2.internal\" DevicePath \"\""
Oct 30 03:50:29.016385 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:50:29.016388    4461 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/14296c92-55c6-44b3-8597-211a0e81e424-dshm\") on node \"ip-10-0-143-156.ec2.internal\" DevicePath \"\""
Oct 30 03:50:29.016385 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:50:29.016396    4461 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/14296c92-55c6-44b3-8597-211a0e81e424-home\") on node \"ip-10-0-143-156.ec2.internal\" DevicePath \"\""
Oct 30 03:50:29.016651 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:50:29.016403    4461 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/14296c92-55c6-44b3-8597-211a0e81e424-tls-certs\") on node \"ip-10-0-143-156.ec2.internal\" DevicePath \"\""
Oct 30 03:50:29.016651 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:50:29.016411    4461 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d2kh8\" (UniqueName: \"kubernetes.io/projected/14296c92-55c6-44b3-8597-211a0e81e424-kube-api-access-d2kh8\") on node \"ip-10-0-143-156.ec2.internal\" DevicePath \"\""
Oct 30 03:50:29.163475 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:50:29.163439    4461 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-dcdf746d6-zzv4v"]
Oct 30 03:50:29.176006 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:50:29.175981    4461 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-dcdf746d6-zzv4v"]
Oct 30 03:50:30.114572 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:50:30.114541    4461 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="14296c92-55c6-44b3-8597-211a0e81e424" path="/var/lib/kubelet/pods/14296c92-55c6-44b3-8597-211a0e81e424/volumes"
Oct 30 03:50:47.778864 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:50:47.778829    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:50:54.502448 ip-10-0-143-156 kubenswrapper[4461]: E1030 03:50:54.502407    4461 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2k85b" containerName="extractor"
Oct 30 03:51:02.499980 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:51:02.499944    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:51:03.254209 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:51:03.254175    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:51:03.430698 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:51:03.430664    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:51:07.818745 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:51:07.818711    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:51:07.933318 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:51:07.933287    4461 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schepq7gw"]
Oct 30 03:51:07.933735 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:51:07.933715    4461 cpu_manager.go:414] "containerMap: removing container" podUID="14296c92-55c6-44b3-8597-211a0e81e424" containerName="storage-initializer"
Oct 30 03:51:07.933735 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:51:07.933734    4461 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="14296c92-55c6-44b3-8597-211a0e81e424" containerName="storage-initializer"
Oct 30 03:51:07.933891 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:51:07.933754    4461 cpu_manager.go:414] "containerMap: removing container" podUID="14296c92-55c6-44b3-8597-211a0e81e424" containerName="main"
Oct 30 03:51:07.933891 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:51:07.933759    4461 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="14296c92-55c6-44b3-8597-211a0e81e424" containerName="main"
Oct 30 03:51:07.933891 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:51:07.933819    4461 memory_manager.go:365] "RemoveStaleState removing state" podUID="14296c92-55c6-44b3-8597-211a0e81e424" containerName="main"
Oct 30 03:51:07.937418 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:51:07.937398    4461 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schepq7gw"
Oct 30 03:51:07.952726 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:51:07.952703    4461 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-733055c7\"/\"openshift-service-ca.crt\""
Oct 30 03:51:07.952917 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:51:07.952900    4461 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-733055c7\"/\"default-dockercfg-7cnqp\""
Oct 30 03:51:07.952964 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:51:07.952926    4461 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-733055c7\"/\"kube-root-ca.crt\""
Oct 30 03:51:07.965451 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:51:07.965431    4461 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-733055c7\"/\"llmisvc-model-fb-opt-125m-with-ba4d693a-epp-sa-dockercfg-rdgvc\""
Oct 30 03:51:07.965796 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:51:07.965777    4461 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-733055c7\"/\"llmisv77ff2528d3e9b4972cd9335229fce9f0-kserve-self-signed-certs\""
Oct 30 03:51:07.967076 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:51:07.967058    4461 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schepq7gw"]
Oct 30 03:51:08.034946 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:51:08.034917    4461 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vfj27\" (UniqueName: \"kubernetes.io/projected/eba97ad2-2fda-4441-ae03-08ff71bd32e5-kube-api-access-vfj27\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schepq7gw\" (UID: \"eba97ad2-2fda-4441-ae03-08ff71bd32e5\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schepq7gw"
Oct 30 03:51:08.035121 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:51:08.034974    4461 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eba97ad2-2fda-4441-ae03-08ff71bd32e5-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schepq7gw\" (UID: \"eba97ad2-2fda-4441-ae03-08ff71bd32e5\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schepq7gw"
Oct 30 03:51:08.071557 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:51:08.071493    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:51:08.135473 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:51:08.135445    4461 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eba97ad2-2fda-4441-ae03-08ff71bd32e5-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schepq7gw\" (UID: \"eba97ad2-2fda-4441-ae03-08ff71bd32e5\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schepq7gw"
Oct 30 03:51:08.135590 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:51:08.135513    4461 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vfj27\" (UniqueName: \"kubernetes.io/projected/eba97ad2-2fda-4441-ae03-08ff71bd32e5-kube-api-access-vfj27\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schepq7gw\" (UID: \"eba97ad2-2fda-4441-ae03-08ff71bd32e5\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schepq7gw"
Oct 30 03:51:08.137983 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:51:08.137956    4461 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eba97ad2-2fda-4441-ae03-08ff71bd32e5-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schepq7gw\" (UID: \"eba97ad2-2fda-4441-ae03-08ff71bd32e5\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schepq7gw"
Oct 30 03:51:08.159416 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:51:08.159386    4461 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vfj27\" (UniqueName: \"kubernetes.io/projected/eba97ad2-2fda-4441-ae03-08ff71bd32e5-kube-api-access-vfj27\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schepq7gw\" (UID: \"eba97ad2-2fda-4441-ae03-08ff71bd32e5\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schepq7gw"
Oct 30 03:51:08.251015 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:51:08.250983    4461 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schepq7gw"
Oct 30 03:51:08.548472 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:51:08.548385    4461 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schepq7gw"]
Oct 30 03:51:08.550743 ip-10-0-143-156 kubenswrapper[4461]: W1030 03:51:08.550716    4461 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeba97ad2_2fda_4441_ae03_08ff71bd32e5.slice/crio-64da5cd3c4a0421b893fdfa1fc0f8727c59df3293058104c16ed43acaea117fc WatchSource:0}: Error finding container 64da5cd3c4a0421b893fdfa1fc0f8727c59df3293058104c16ed43acaea117fc: Status 404 returned error can't find the container with id 64da5cd3c4a0421b893fdfa1fc0f8727c59df3293058104c16ed43acaea117fc
Oct 30 03:51:08.981166 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:51:08.981122    4461 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schepq7gw" event={"ID":"eba97ad2-2fda-4441-ae03-08ff71bd32e5","Type":"ContainerStarted","Data":"c37263503065a344c8e985ea2ab5178810c30c2e507744c7a424607c66248b68"}
Oct 30 03:51:08.981556 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:51:08.981184    4461 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schepq7gw" event={"ID":"eba97ad2-2fda-4441-ae03-08ff71bd32e5","Type":"ContainerStarted","Data":"64da5cd3c4a0421b893fdfa1fc0f8727c59df3293058104c16ed43acaea117fc"}
Oct 30 03:51:08.981556 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:51:08.981237    4461 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schepq7gw"
Oct 30 03:51:09.023710 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:51:09.023657    4461 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schepq7gw" podStartSLOduration=2.023642234 podStartE2EDuration="2.023642234s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:51:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:51:09.022402703 +0000 UTC m=+3337.072207884" watchObservedRunningTime="2026-10-30 03:51:09.023642234 +0000 UTC m=+3337.073447409"
Oct 30 03:51:10.632145 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:51:10.632079    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:51:14.356227 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:51:14.356190    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:51:16.776319 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:51:16.776262    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:51:17.065627 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:51:17.065539    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:51:17.461964 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:51:17.461869    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:51:17.661427 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:51:17.661392    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:51:21.292455 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:51:21.292407    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:51:21.526943 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:51:21.526909    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:51:23.923478 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:51:23.923444    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:51:24.088333 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:51:24.088297    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:51:24.504566 ip-10-0-143-156 kubenswrapper[4461]: E1030 03:51:24.504524    4461 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2k85b" containerName="extractor"
Oct 30 03:51:27.563429 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:51:27.563396    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:51:29.755859 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:51:29.755820    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:51:29.952001 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:51:29.951970    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:51:39.986483 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:51:39.986451    4461 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schepq7gw"
Oct 30 03:51:47.870297 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:51:47.870218    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:51:54.503215 ip-10-0-143-156 kubenswrapper[4461]: E1030 03:51:54.503177    4461 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2k85b" containerName="extractor"
Oct 30 03:52:02.707710 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:52:02.707671    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:52:24.503863 ip-10-0-143-156 kubenswrapper[4461]: E1030 03:52:24.503823    4461 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2k85b" containerName="extractor"
Oct 30 03:52:27.705954 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:52:27.705919    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:52:40.481621 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:52:40.481581    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:52:40.486563 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:52:40.486535    4461 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schepq7gw"]
Oct 30 03:52:40.677501 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:52:40.677464    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:52:47.977473 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:52:47.977434    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:52:49.986378 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:52:49.986344    4461 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schepq7gw" podUID="eba97ad2-2fda-4441-ae03-08ff71bd32e5" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 03:52:54.506357 ip-10-0-143-156 kubenswrapper[4461]: E1030 03:52:54.506318    4461 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2k85b" containerName="extractor"
Oct 30 03:52:55.505575 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:52:55.505519    4461 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schepq7gw" podUID="eba97ad2-2fda-4441-ae03-08ff71bd32e5" containerName="main" containerID="cri-o://c37263503065a344c8e985ea2ab5178810c30c2e507744c7a424607c66248b68" gracePeriod=45
Oct 30 03:52:59.987121 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:52:59.987065    4461 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schepq7gw" podUID="eba97ad2-2fda-4441-ae03-08ff71bd32e5" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 03:53:02.760588 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:53:02.760551    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:53:09.986710 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:53:09.986671    4461 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schepq7gw" podUID="eba97ad2-2fda-4441-ae03-08ff71bd32e5" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 03:53:09.987118 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:53:09.986770    4461 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schepq7gw"
Oct 30 03:53:19.987333 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:53:19.987252    4461 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schepq7gw" podUID="eba97ad2-2fda-4441-ae03-08ff71bd32e5" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 03:53:24.505015 ip-10-0-143-156 kubenswrapper[4461]: E1030 03:53:24.504959    4461 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2k85b" containerName="extractor"
Oct 30 03:53:25.743631 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:53:25.743609    4461 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schepq7gw"
Oct 30 03:53:25.833423 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:53:25.833351    4461 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eba97ad2-2fda-4441-ae03-08ff71bd32e5-kube-api-access-vfj27\" (UniqueName: \"kubernetes.io/projected/eba97ad2-2fda-4441-ae03-08ff71bd32e5-kube-api-access-vfj27\") pod \"eba97ad2-2fda-4441-ae03-08ff71bd32e5\" (UID: \"eba97ad2-2fda-4441-ae03-08ff71bd32e5\") "
Oct 30 03:53:25.833573 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:53:25.833485    4461 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/eba97ad2-2fda-4441-ae03-08ff71bd32e5-tls-certs\" (UniqueName: \"kubernetes.io/secret/eba97ad2-2fda-4441-ae03-08ff71bd32e5-tls-certs\") pod \"eba97ad2-2fda-4441-ae03-08ff71bd32e5\" (UID: \"eba97ad2-2fda-4441-ae03-08ff71bd32e5\") "
Oct 30 03:53:25.835558 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:53:25.835529    4461 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eba97ad2-2fda-4441-ae03-08ff71bd32e5-kube-api-access-vfj27" pod "eba97ad2-2fda-4441-ae03-08ff71bd32e5" (UID: "eba97ad2-2fda-4441-ae03-08ff71bd32e5"). InnerVolumeSpecName "kube-api-access-vfj27". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:53:25.835666 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:53:25.835562    4461 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eba97ad2-2fda-4441-ae03-08ff71bd32e5-tls-certs" pod "eba97ad2-2fda-4441-ae03-08ff71bd32e5" (UID: "eba97ad2-2fda-4441-ae03-08ff71bd32e5"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:53:25.934452 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:53:25.934421    4461 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eba97ad2-2fda-4441-ae03-08ff71bd32e5-tls-certs\") on node \"ip-10-0-143-156.ec2.internal\" DevicePath \"\""
Oct 30 03:53:25.934452 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:53:25.934446    4461 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vfj27\" (UniqueName: \"kubernetes.io/projected/eba97ad2-2fda-4441-ae03-08ff71bd32e5-kube-api-access-vfj27\") on node \"ip-10-0-143-156.ec2.internal\" DevicePath \"\""
Oct 30 03:53:26.555040 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:53:26.554300    4461 generic.go:356] "Generic (PLEG): container finished" podID="eba97ad2-2fda-4441-ae03-08ff71bd32e5" containerID="c37263503065a344c8e985ea2ab5178810c30c2e507744c7a424607c66248b68" exitCode=0
Oct 30 03:53:26.555040 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:53:26.554420    4461 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schepq7gw" event={"ID":"eba97ad2-2fda-4441-ae03-08ff71bd32e5","Type":"ContainerDied","Data":"c37263503065a344c8e985ea2ab5178810c30c2e507744c7a424607c66248b68"}
Oct 30 03:53:26.555040 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:53:26.554467    4461 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schepq7gw" event={"ID":"eba97ad2-2fda-4441-ae03-08ff71bd32e5","Type":"ContainerDied","Data":"64da5cd3c4a0421b893fdfa1fc0f8727c59df3293058104c16ed43acaea117fc"}
Oct 30 03:53:26.555040 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:53:26.554496    4461 scope.go:122] "RemoveContainer" containerID="c37263503065a344c8e985ea2ab5178810c30c2e507744c7a424607c66248b68"
Oct 30 03:53:26.555040 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:53:26.554649    4461 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schepq7gw"
Oct 30 03:53:26.563804 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:53:26.563784    4461 scope.go:122] "RemoveContainer" containerID="c37263503065a344c8e985ea2ab5178810c30c2e507744c7a424607c66248b68"
Oct 30 03:53:26.564058 ip-10-0-143-156 kubenswrapper[4461]: E1030 03:53:26.564043    4461 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c37263503065a344c8e985ea2ab5178810c30c2e507744c7a424607c66248b68\": container with ID starting with c37263503065a344c8e985ea2ab5178810c30c2e507744c7a424607c66248b68 not found: ID does not exist" containerID="c37263503065a344c8e985ea2ab5178810c30c2e507744c7a424607c66248b68"
Oct 30 03:53:26.564118 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:53:26.564067    4461 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c37263503065a344c8e985ea2ab5178810c30c2e507744c7a424607c66248b68"} err="failed to get container status \"c37263503065a344c8e985ea2ab5178810c30c2e507744c7a424607c66248b68\": rpc error: code = NotFound desc = could not find container \"c37263503065a344c8e985ea2ab5178810c30c2e507744c7a424607c66248b68\": container with ID starting with c37263503065a344c8e985ea2ab5178810c30c2e507744c7a424607c66248b68 not found: ID does not exist"
Oct 30 03:53:26.606226 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:53:26.606202    4461 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schepq7gw"]
Oct 30 03:53:26.618871 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:53:26.618847    4461 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schepq7gw"]
Oct 30 03:53:27.798847 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:53:27.798812    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:53:28.114794 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:53:28.114720    4461 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eba97ad2-2fda-4441-ae03-08ff71bd32e5" path="/var/lib/kubelet/pods/eba97ad2-2fda-4441-ae03-08ff71bd32e5/volumes"
Oct 30 03:53:30.205840 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:53:30.205800    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:53:30.377588 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:53:30.377555    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:53:36.869950 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:53:36.869907    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:53:36.879938 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:53:36.879910    4461 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-85bf77f655-872lw"]
Oct 30 03:53:36.880288 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:53:36.880273    4461 cpu_manager.go:414] "containerMap: removing container" podUID="eba97ad2-2fda-4441-ae03-08ff71bd32e5" containerName="main"
Oct 30 03:53:36.880336 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:53:36.880290    4461 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eba97ad2-2fda-4441-ae03-08ff71bd32e5" containerName="main"
Oct 30 03:53:36.880372 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:53:36.880355    4461 memory_manager.go:365] "RemoveStaleState removing state" podUID="eba97ad2-2fda-4441-ae03-08ff71bd32e5" containerName="main"
Oct 30 03:53:36.884749 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:53:36.884733    4461 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-85bf77f655-872lw"
Oct 30 03:53:36.901512 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:53:36.901489    4461 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-85bf77f655-872lw"]
Oct 30 03:53:36.904031 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:53:36.904011    4461 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"seaweedfs-s3-creds\""
Oct 30 03:53:36.904143 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:53:36.904011    4461 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"openshift-service-ca.crt\""
Oct 30 03:53:36.912869 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:53:36.912832    4461 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"odh-kserve-custom-ca-bundle\""
Oct 30 03:53:36.912869 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:53:36.912847    4461 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"kube-root-ca.crt\""
Oct 30 03:53:36.913005 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:53:36.912853    4461 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"stop-feature-test-kserve-self-signed-certs\""
Oct 30 03:53:36.913005 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:53:36.912886    4461 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"default-dockercfg-n4xll\""
Oct 30 03:53:37.031652 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:53:37.031616    4461 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q7dg8\" (UniqueName: \"kubernetes.io/projected/73a08e53-4ce0-49a3-b08a-2a66076123c4-kube-api-access-q7dg8\") pod \"stop-feature-test-kserve-85bf77f655-872lw\" (UID: \"73a08e53-4ce0-49a3-b08a-2a66076123c4\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-85bf77f655-872lw"
Oct 30 03:53:37.031819 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:53:37.031671    4461 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/73a08e53-4ce0-49a3-b08a-2a66076123c4-tmp-dir\") pod \"stop-feature-test-kserve-85bf77f655-872lw\" (UID: \"73a08e53-4ce0-49a3-b08a-2a66076123c4\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-85bf77f655-872lw"
Oct 30 03:53:37.031819 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:53:37.031698    4461 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/73a08e53-4ce0-49a3-b08a-2a66076123c4-kserve-provision-location\") pod \"stop-feature-test-kserve-85bf77f655-872lw\" (UID: \"73a08e53-4ce0-49a3-b08a-2a66076123c4\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-85bf77f655-872lw"
Oct 30 03:53:37.031819 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:53:37.031794    4461 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/73a08e53-4ce0-49a3-b08a-2a66076123c4-tls-certs\") pod \"stop-feature-test-kserve-85bf77f655-872lw\" (UID: \"73a08e53-4ce0-49a3-b08a-2a66076123c4\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-85bf77f655-872lw"
Oct 30 03:53:37.031944 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:53:37.031832    4461 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/73a08e53-4ce0-49a3-b08a-2a66076123c4-cabundle-cert\") pod \"stop-feature-test-kserve-85bf77f655-872lw\" (UID: \"73a08e53-4ce0-49a3-b08a-2a66076123c4\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-85bf77f655-872lw"
Oct 30 03:53:37.031944 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:53:37.031856    4461 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/73a08e53-4ce0-49a3-b08a-2a66076123c4-home\") pod \"stop-feature-test-kserve-85bf77f655-872lw\" (UID: \"73a08e53-4ce0-49a3-b08a-2a66076123c4\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-85bf77f655-872lw"
Oct 30 03:53:37.031944 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:53:37.031909    4461 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/73a08e53-4ce0-49a3-b08a-2a66076123c4-dshm\") pod \"stop-feature-test-kserve-85bf77f655-872lw\" (UID: \"73a08e53-4ce0-49a3-b08a-2a66076123c4\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-85bf77f655-872lw"
Oct 30 03:53:37.032063 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:53:37.031980    4461 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/73a08e53-4ce0-49a3-b08a-2a66076123c4-model-cache\") pod \"stop-feature-test-kserve-85bf77f655-872lw\" (UID: \"73a08e53-4ce0-49a3-b08a-2a66076123c4\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-85bf77f655-872lw"
Oct 30 03:53:37.048968 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:53:37.048940    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:53:37.133277 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:53:37.133197    4461 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q7dg8\" (UniqueName: \"kubernetes.io/projected/73a08e53-4ce0-49a3-b08a-2a66076123c4-kube-api-access-q7dg8\") pod \"stop-feature-test-kserve-85bf77f655-872lw\" (UID: \"73a08e53-4ce0-49a3-b08a-2a66076123c4\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-85bf77f655-872lw"
Oct 30 03:53:37.133277 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:53:37.133243    4461 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/73a08e53-4ce0-49a3-b08a-2a66076123c4-tmp-dir\") pod \"stop-feature-test-kserve-85bf77f655-872lw\" (UID: \"73a08e53-4ce0-49a3-b08a-2a66076123c4\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-85bf77f655-872lw"
Oct 30 03:53:37.133277 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:53:37.133271    4461 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/73a08e53-4ce0-49a3-b08a-2a66076123c4-kserve-provision-location\") pod \"stop-feature-test-kserve-85bf77f655-872lw\" (UID: \"73a08e53-4ce0-49a3-b08a-2a66076123c4\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-85bf77f655-872lw"
Oct 30 03:53:37.133539 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:53:37.133297    4461 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/73a08e53-4ce0-49a3-b08a-2a66076123c4-tls-certs\") pod \"stop-feature-test-kserve-85bf77f655-872lw\" (UID: \"73a08e53-4ce0-49a3-b08a-2a66076123c4\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-85bf77f655-872lw"
Oct 30 03:53:37.133539 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:53:37.133320    4461 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/73a08e53-4ce0-49a3-b08a-2a66076123c4-cabundle-cert\") pod \"stop-feature-test-kserve-85bf77f655-872lw\" (UID: \"73a08e53-4ce0-49a3-b08a-2a66076123c4\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-85bf77f655-872lw"
Oct 30 03:53:37.133539 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:53:37.133341    4461 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/73a08e53-4ce0-49a3-b08a-2a66076123c4-home\") pod \"stop-feature-test-kserve-85bf77f655-872lw\" (UID: \"73a08e53-4ce0-49a3-b08a-2a66076123c4\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-85bf77f655-872lw"
Oct 30 03:53:37.133539 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:53:37.133358    4461 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/73a08e53-4ce0-49a3-b08a-2a66076123c4-dshm\") pod \"stop-feature-test-kserve-85bf77f655-872lw\" (UID: \"73a08e53-4ce0-49a3-b08a-2a66076123c4\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-85bf77f655-872lw"
Oct 30 03:53:37.133539 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:53:37.133385    4461 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/73a08e53-4ce0-49a3-b08a-2a66076123c4-model-cache\") pod \"stop-feature-test-kserve-85bf77f655-872lw\" (UID: \"73a08e53-4ce0-49a3-b08a-2a66076123c4\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-85bf77f655-872lw"
Oct 30 03:53:37.133787 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:53:37.133724    4461 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/73a08e53-4ce0-49a3-b08a-2a66076123c4-tmp-dir\") pod \"stop-feature-test-kserve-85bf77f655-872lw\" (UID: \"73a08e53-4ce0-49a3-b08a-2a66076123c4\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-85bf77f655-872lw"
Oct 30 03:53:37.133787 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:53:37.133764    4461 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/73a08e53-4ce0-49a3-b08a-2a66076123c4-home\") pod \"stop-feature-test-kserve-85bf77f655-872lw\" (UID: \"73a08e53-4ce0-49a3-b08a-2a66076123c4\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-85bf77f655-872lw"
Oct 30 03:53:37.133896 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:53:37.133808    4461 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/73a08e53-4ce0-49a3-b08a-2a66076123c4-kserve-provision-location\") pod \"stop-feature-test-kserve-85bf77f655-872lw\" (UID: \"73a08e53-4ce0-49a3-b08a-2a66076123c4\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-85bf77f655-872lw"
Oct 30 03:53:37.133896 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:53:37.133847    4461 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/73a08e53-4ce0-49a3-b08a-2a66076123c4-model-cache\") pod \"stop-feature-test-kserve-85bf77f655-872lw\" (UID: \"73a08e53-4ce0-49a3-b08a-2a66076123c4\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-85bf77f655-872lw"
Oct 30 03:53:37.134158 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:53:37.134142    4461 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/73a08e53-4ce0-49a3-b08a-2a66076123c4-cabundle-cert\") pod \"stop-feature-test-kserve-85bf77f655-872lw\" (UID: \"73a08e53-4ce0-49a3-b08a-2a66076123c4\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-85bf77f655-872lw"
Oct 30 03:53:37.135760 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:53:37.135734    4461 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/73a08e53-4ce0-49a3-b08a-2a66076123c4-dshm\") pod \"stop-feature-test-kserve-85bf77f655-872lw\" (UID: \"73a08e53-4ce0-49a3-b08a-2a66076123c4\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-85bf77f655-872lw"
Oct 30 03:53:37.136038 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:53:37.136019    4461 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/73a08e53-4ce0-49a3-b08a-2a66076123c4-tls-certs\") pod \"stop-feature-test-kserve-85bf77f655-872lw\" (UID: \"73a08e53-4ce0-49a3-b08a-2a66076123c4\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-85bf77f655-872lw"
Oct 30 03:53:37.162993 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:53:37.162958    4461 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q7dg8\" (UniqueName: \"kubernetes.io/projected/73a08e53-4ce0-49a3-b08a-2a66076123c4-kube-api-access-q7dg8\") pod \"stop-feature-test-kserve-85bf77f655-872lw\" (UID: \"73a08e53-4ce0-49a3-b08a-2a66076123c4\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-85bf77f655-872lw"
Oct 30 03:53:37.199569 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:53:37.199539    4461 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-85bf77f655-872lw"
Oct 30 03:53:37.273355 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:53:37.273288    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:53:37.525809 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:53:37.525780    4461 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-85bf77f655-872lw"]
Oct 30 03:53:37.526586 ip-10-0-143-156 kubenswrapper[4461]: W1030 03:53:37.526559    4461 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod73a08e53_4ce0_49a3_b08a_2a66076123c4.slice/crio-ca7fdd8dd7b3945762247aac322acfe6ef06dee595fedf67ac9f8a5920ca5dfd WatchSource:0}: Error finding container ca7fdd8dd7b3945762247aac322acfe6ef06dee595fedf67ac9f8a5920ca5dfd: Status 404 returned error can't find the container with id ca7fdd8dd7b3945762247aac322acfe6ef06dee595fedf67ac9f8a5920ca5dfd
Oct 30 03:53:37.598732 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:53:37.598701    4461 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-85bf77f655-872lw" event={"ID":"73a08e53-4ce0-49a3-b08a-2a66076123c4","Type":"ContainerStarted","Data":"7ccabd5d3e448416b62f6aa44f7fcc3f09f332cd01d43f15df19f666f8453159"}
Oct 30 03:53:37.598858 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:53:37.598757    4461 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-85bf77f655-872lw" event={"ID":"73a08e53-4ce0-49a3-b08a-2a66076123c4","Type":"ContainerStarted","Data":"ca7fdd8dd7b3945762247aac322acfe6ef06dee595fedf67ac9f8a5920ca5dfd"}
Oct 30 03:53:39.206637 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:53:39.206571    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:53:39.380261 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:53:39.380224    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:53:39.540627 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:53:39.540596    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:53:40.613336 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:53:40.613298    4461 generic.go:356] "Generic (PLEG): container finished" podID="73a08e53-4ce0-49a3-b08a-2a66076123c4" containerID="7ccabd5d3e448416b62f6aa44f7fcc3f09f332cd01d43f15df19f666f8453159" exitCode=0
Oct 30 03:53:40.613749 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:53:40.613371    4461 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-85bf77f655-872lw" event={"ID":"73a08e53-4ce0-49a3-b08a-2a66076123c4","Type":"ContainerDied","Data":"7ccabd5d3e448416b62f6aa44f7fcc3f09f332cd01d43f15df19f666f8453159"}
Oct 30 03:53:41.620369 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:53:41.620335    4461 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-85bf77f655-872lw" event={"ID":"73a08e53-4ce0-49a3-b08a-2a66076123c4","Type":"ContainerStarted","Data":"f71f7bed9e197879b562d1ed1cbef997e1b385adb95acfd26cf44928014ff01c"}
Oct 30 03:53:41.693778 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:53:41.693717    4461 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-85bf77f655-872lw" podStartSLOduration=3.512310997 podStartE2EDuration="5.693700972s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.181389975s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:53:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:53:41.680374903 +0000 UTC m=+3489.730180079" watchObservedRunningTime="2026-10-30 03:53:41.693700972 +0000 UTC m=+3489.743506148"
Oct 30 03:53:42.200588 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:53:42.200550    4461 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-85bf77f655-872lw"
Oct 30 03:53:43.020462 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:53:43.020401    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:53:43.186749 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:53:43.186718    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:53:47.200046 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:53:47.200005    4461 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-85bf77f655-872lw"
Oct 30 03:53:47.201300 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:53:47.201275    4461 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-85bf77f655-872lw" podUID="73a08e53-4ce0-49a3-b08a-2a66076123c4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/health\": dial tcp 10.133.0.44:8000: connect: connection refused"
Oct 30 03:53:48.144082 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:53:48.144044    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:53:49.677919 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:53:49.677884    4461 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-rx6vm"]
Oct 30 03:53:49.678352 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:53:49.678149    4461 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-rx6vm" podUID="35b8f310-5db9-4cf0-99ea-0baca82d3cda" containerName="istio-proxy" containerID="cri-o://ad636ca1d87a5ca50776401658029cf4714c3cbeae5268ca0c5eb6e9007bfec0" gracePeriod=30
Oct 30 03:53:53.811642 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:53:53.811606    4461 patch_prober.go:29] interesting pod/openshift-ai-inference-openshift-default-5b4f8cc59b-rx6vm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 03:53:53.812051 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:53:53.811661    4461 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-rx6vm" podUID="35b8f310-5db9-4cf0-99ea-0baca82d3cda" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 03:53:54.504400 ip-10-0-143-156 kubenswrapper[4461]: E1030 03:53:54.504354    4461 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2k85b" containerName="extractor"
Oct 30 03:53:54.915615 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:53:54.915585    4461 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-rx6vm"
Oct 30 03:53:55.010844 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:53:55.010809    4461 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/35b8f310-5db9-4cf0-99ea-0baca82d3cda-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/35b8f310-5db9-4cf0-99ea-0baca82d3cda-workload-socket\") pod \"35b8f310-5db9-4cf0-99ea-0baca82d3cda\" (UID: \"35b8f310-5db9-4cf0-99ea-0baca82d3cda\") "
Oct 30 03:53:55.010844 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:53:55.010844    4461 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/35b8f310-5db9-4cf0-99ea-0baca82d3cda-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/35b8f310-5db9-4cf0-99ea-0baca82d3cda-istio-data\") pod \"35b8f310-5db9-4cf0-99ea-0baca82d3cda\" (UID: \"35b8f310-5db9-4cf0-99ea-0baca82d3cda\") "
Oct 30 03:53:55.011090 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:53:55.010890    4461 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/35b8f310-5db9-4cf0-99ea-0baca82d3cda-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/35b8f310-5db9-4cf0-99ea-0baca82d3cda-istiod-ca-cert\") pod \"35b8f310-5db9-4cf0-99ea-0baca82d3cda\" (UID: \"35b8f310-5db9-4cf0-99ea-0baca82d3cda\") "
Oct 30 03:53:55.011090 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:53:55.010918    4461 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/35b8f310-5db9-4cf0-99ea-0baca82d3cda-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/35b8f310-5db9-4cf0-99ea-0baca82d3cda-workload-certs\") pod \"35b8f310-5db9-4cf0-99ea-0baca82d3cda\" (UID: \"35b8f310-5db9-4cf0-99ea-0baca82d3cda\") "
Oct 30 03:53:55.011090 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:53:55.010947    4461 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/35b8f310-5db9-4cf0-99ea-0baca82d3cda-kube-api-access-h46qf\" (UniqueName: \"kubernetes.io/projected/35b8f310-5db9-4cf0-99ea-0baca82d3cda-kube-api-access-h46qf\") pod \"35b8f310-5db9-4cf0-99ea-0baca82d3cda\" (UID: \"35b8f310-5db9-4cf0-99ea-0baca82d3cda\") "
Oct 30 03:53:55.011090 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:53:55.010976    4461 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/35b8f310-5db9-4cf0-99ea-0baca82d3cda-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/35b8f310-5db9-4cf0-99ea-0baca82d3cda-istio-podinfo\") pod \"35b8f310-5db9-4cf0-99ea-0baca82d3cda\" (UID: \"35b8f310-5db9-4cf0-99ea-0baca82d3cda\") "
Oct 30 03:53:55.011090 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:53:55.011003    4461 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/35b8f310-5db9-4cf0-99ea-0baca82d3cda-istio-token\" (UniqueName: \"kubernetes.io/projected/35b8f310-5db9-4cf0-99ea-0baca82d3cda-istio-token\") pod \"35b8f310-5db9-4cf0-99ea-0baca82d3cda\" (UID: \"35b8f310-5db9-4cf0-99ea-0baca82d3cda\") "
Oct 30 03:53:55.011090 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:53:55.011034    4461 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/35b8f310-5db9-4cf0-99ea-0baca82d3cda-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/35b8f310-5db9-4cf0-99ea-0baca82d3cda-credential-socket\") pod \"35b8f310-5db9-4cf0-99ea-0baca82d3cda\" (UID: \"35b8f310-5db9-4cf0-99ea-0baca82d3cda\") "
Oct 30 03:53:55.011090 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:53:55.011085    4461 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/35b8f310-5db9-4cf0-99ea-0baca82d3cda-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/35b8f310-5db9-4cf0-99ea-0baca82d3cda-istio-envoy\") pod \"35b8f310-5db9-4cf0-99ea-0baca82d3cda\" (UID: \"35b8f310-5db9-4cf0-99ea-0baca82d3cda\") "
Oct 30 03:53:55.011465 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:53:55.011153    4461 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35b8f310-5db9-4cf0-99ea-0baca82d3cda-workload-socket" pod "35b8f310-5db9-4cf0-99ea-0baca82d3cda" (UID: "35b8f310-5db9-4cf0-99ea-0baca82d3cda"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:53:55.011465 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:53:55.011190    4461 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35b8f310-5db9-4cf0-99ea-0baca82d3cda-istio-data" pod "35b8f310-5db9-4cf0-99ea-0baca82d3cda" (UID: "35b8f310-5db9-4cf0-99ea-0baca82d3cda"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:53:55.011465 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:53:55.011280    4461 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35b8f310-5db9-4cf0-99ea-0baca82d3cda-workload-certs" pod "35b8f310-5db9-4cf0-99ea-0baca82d3cda" (UID: "35b8f310-5db9-4cf0-99ea-0baca82d3cda"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:53:55.011465 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:53:55.011382    4461 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35b8f310-5db9-4cf0-99ea-0baca82d3cda-credential-socket" pod "35b8f310-5db9-4cf0-99ea-0baca82d3cda" (UID: "35b8f310-5db9-4cf0-99ea-0baca82d3cda"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:53:55.011465 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:53:55.011435    4461 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/35b8f310-5db9-4cf0-99ea-0baca82d3cda-workload-socket\") on node \"ip-10-0-143-156.ec2.internal\" DevicePath \"\""
Oct 30 03:53:55.011465 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:53:55.011450    4461 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/35b8f310-5db9-4cf0-99ea-0baca82d3cda-istio-data\") on node \"ip-10-0-143-156.ec2.internal\" DevicePath \"\""
Oct 30 03:53:55.011465 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:53:55.011465    4461 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/35b8f310-5db9-4cf0-99ea-0baca82d3cda-workload-certs\") on node \"ip-10-0-143-156.ec2.internal\" DevicePath \"\""
Oct 30 03:53:55.011798 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:53:55.011774    4461 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/35b8f310-5db9-4cf0-99ea-0baca82d3cda-istiod-ca-cert" pod "35b8f310-5db9-4cf0-99ea-0baca82d3cda" (UID: "35b8f310-5db9-4cf0-99ea-0baca82d3cda"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:53:55.013416 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:53:55.013383    4461 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/35b8f310-5db9-4cf0-99ea-0baca82d3cda-istio-token" pod "35b8f310-5db9-4cf0-99ea-0baca82d3cda" (UID: "35b8f310-5db9-4cf0-99ea-0baca82d3cda"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:53:55.013535 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:53:55.013482    4461 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/35b8f310-5db9-4cf0-99ea-0baca82d3cda-kube-api-access-h46qf" pod "35b8f310-5db9-4cf0-99ea-0baca82d3cda" (UID: "35b8f310-5db9-4cf0-99ea-0baca82d3cda"). InnerVolumeSpecName "kube-api-access-h46qf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:53:55.013903 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:53:55.013883    4461 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35b8f310-5db9-4cf0-99ea-0baca82d3cda-istio-envoy" pod "35b8f310-5db9-4cf0-99ea-0baca82d3cda" (UID: "35b8f310-5db9-4cf0-99ea-0baca82d3cda"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:53:55.013975 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:53:55.013941    4461 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/35b8f310-5db9-4cf0-99ea-0baca82d3cda-istio-podinfo" pod "35b8f310-5db9-4cf0-99ea-0baca82d3cda" (UID: "35b8f310-5db9-4cf0-99ea-0baca82d3cda"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 03:53:55.111986 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:53:55.111905    4461 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/35b8f310-5db9-4cf0-99ea-0baca82d3cda-istio-envoy\") on node \"ip-10-0-143-156.ec2.internal\" DevicePath \"\""
Oct 30 03:53:55.111986 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:53:55.111930    4461 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/35b8f310-5db9-4cf0-99ea-0baca82d3cda-istiod-ca-cert\") on node \"ip-10-0-143-156.ec2.internal\" DevicePath \"\""
Oct 30 03:53:55.111986 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:53:55.111940    4461 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h46qf\" (UniqueName: \"kubernetes.io/projected/35b8f310-5db9-4cf0-99ea-0baca82d3cda-kube-api-access-h46qf\") on node \"ip-10-0-143-156.ec2.internal\" DevicePath \"\""
Oct 30 03:53:55.111986 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:53:55.111948    4461 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/35b8f310-5db9-4cf0-99ea-0baca82d3cda-istio-podinfo\") on node \"ip-10-0-143-156.ec2.internal\" DevicePath \"\""
Oct 30 03:53:55.111986 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:53:55.111957    4461 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/35b8f310-5db9-4cf0-99ea-0baca82d3cda-istio-token\") on node \"ip-10-0-143-156.ec2.internal\" DevicePath \"\""
Oct 30 03:53:55.111986 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:53:55.111966    4461 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/35b8f310-5db9-4cf0-99ea-0baca82d3cda-credential-socket\") on node \"ip-10-0-143-156.ec2.internal\" DevicePath \"\""
Oct 30 03:53:55.680360 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:53:55.680322    4461 generic.go:356] "Generic (PLEG): container finished" podID="35b8f310-5db9-4cf0-99ea-0baca82d3cda" containerID="ad636ca1d87a5ca50776401658029cf4714c3cbeae5268ca0c5eb6e9007bfec0" exitCode=0
Oct 30 03:53:55.680555 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:53:55.680375    4461 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-rx6vm"
Oct 30 03:53:55.680555 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:53:55.680375    4461 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-rx6vm" event={"ID":"35b8f310-5db9-4cf0-99ea-0baca82d3cda","Type":"ContainerDied","Data":"ad636ca1d87a5ca50776401658029cf4714c3cbeae5268ca0c5eb6e9007bfec0"}
Oct 30 03:53:55.680555 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:53:55.680484    4461 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-rx6vm" event={"ID":"35b8f310-5db9-4cf0-99ea-0baca82d3cda","Type":"ContainerDied","Data":"18556be2d41ad1803c95525eef5e836910cac49066a9bb5a14fbca8d4d8f989a"}
Oct 30 03:53:55.680555 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:53:55.680511    4461 scope.go:122] "RemoveContainer" containerID="ad636ca1d87a5ca50776401658029cf4714c3cbeae5268ca0c5eb6e9007bfec0"
Oct 30 03:53:55.689441 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:53:55.689423    4461 scope.go:122] "RemoveContainer" containerID="ad636ca1d87a5ca50776401658029cf4714c3cbeae5268ca0c5eb6e9007bfec0"
Oct 30 03:53:55.689709 ip-10-0-143-156 kubenswrapper[4461]: E1030 03:53:55.689690    4461 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ad636ca1d87a5ca50776401658029cf4714c3cbeae5268ca0c5eb6e9007bfec0\": container with ID starting with ad636ca1d87a5ca50776401658029cf4714c3cbeae5268ca0c5eb6e9007bfec0 not found: ID does not exist" containerID="ad636ca1d87a5ca50776401658029cf4714c3cbeae5268ca0c5eb6e9007bfec0"
Oct 30 03:53:55.689757 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:53:55.689717    4461 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ad636ca1d87a5ca50776401658029cf4714c3cbeae5268ca0c5eb6e9007bfec0"} err="failed to get container status \"ad636ca1d87a5ca50776401658029cf4714c3cbeae5268ca0c5eb6e9007bfec0\": rpc error: code = NotFound desc = could not find container \"ad636ca1d87a5ca50776401658029cf4714c3cbeae5268ca0c5eb6e9007bfec0\": container with ID starting with ad636ca1d87a5ca50776401658029cf4714c3cbeae5268ca0c5eb6e9007bfec0 not found: ID does not exist"
Oct 30 03:53:55.745993 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:53:55.745962    4461 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-rx6vm"]
Oct 30 03:53:55.758380 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:53:55.758351    4461 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-rx6vm"]
Oct 30 03:53:56.115361 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:53:56.115330    4461 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="35b8f310-5db9-4cf0-99ea-0baca82d3cda" path="/var/lib/kubelet/pods/35b8f310-5db9-4cf0-99ea-0baca82d3cda/volumes"
Oct 30 03:53:57.200444 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:53:57.200402    4461 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-85bf77f655-872lw" podUID="73a08e53-4ce0-49a3-b08a-2a66076123c4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/health\": dial tcp 10.133.0.44:8000: connect: connection refused"
Oct 30 03:54:07.200165 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:54:07.200120    4461 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-85bf77f655-872lw" podUID="73a08e53-4ce0-49a3-b08a-2a66076123c4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/health\": dial tcp 10.133.0.44:8000: connect: connection refused"
Oct 30 03:54:17.200564 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:54:17.200517    4461 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-85bf77f655-872lw" podUID="73a08e53-4ce0-49a3-b08a-2a66076123c4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/health\": dial tcp 10.133.0.44:8000: connect: connection refused"
Oct 30 03:54:18.929826 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:54:18.929788    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:54:21.726309 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:54:21.726280    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:54:22.699781 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:54:22.699746    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:54:24.503598 ip-10-0-143-156 kubenswrapper[4461]: E1030 03:54:24.503549    4461 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2k85b" containerName="extractor"
Oct 30 03:54:27.200805 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:54:27.200767    4461 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-85bf77f655-872lw" podUID="73a08e53-4ce0-49a3-b08a-2a66076123c4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/health\": dial tcp 10.133.0.44:8000: connect: connection refused"
Oct 30 03:54:27.931922 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:54:27.931886    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:54:28.779310 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:54:28.779272    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:54:28.998797 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:54:28.998765    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:54:29.169485 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:54:29.169381    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:54:31.316350 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:54:31.316315    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:54:31.844814 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:54:31.844783    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:54:32.026776 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:54:32.026745    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:54:32.584415 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:54:32.584381    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:54:35.180075 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:54:35.180040    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:54:36.040859 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:54:36.040821    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:54:36.221328 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:54:36.221293    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:54:37.200952 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:54:37.200913    4461 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-85bf77f655-872lw" podUID="73a08e53-4ce0-49a3-b08a-2a66076123c4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/health\": dial tcp 10.133.0.44:8000: connect: connection refused"
Oct 30 03:54:41.288876 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:54:41.288840    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:54:41.516449 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:54:41.516415    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:54:47.200894 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:54:47.200789    4461 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-85bf77f655-872lw" podUID="73a08e53-4ce0-49a3-b08a-2a66076123c4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/health\": dial tcp 10.133.0.44:8000: connect: connection refused"
Oct 30 03:54:48.216485 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:54:48.216419    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:54:54.504190 ip-10-0-143-156 kubenswrapper[4461]: E1030 03:54:54.504140    4461 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2k85b" containerName="extractor"
Oct 30 03:54:57.200837 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:54:57.200788    4461 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-85bf77f655-872lw" podUID="73a08e53-4ce0-49a3-b08a-2a66076123c4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/health\": dial tcp 10.133.0.44:8000: connect: connection refused"
Oct 30 03:55:07.200162 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:55:07.200120    4461 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-85bf77f655-872lw" podUID="73a08e53-4ce0-49a3-b08a-2a66076123c4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/health\": dial tcp 10.133.0.44:8000: connect: connection refused"
Oct 30 03:55:17.200190 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:55:17.200150    4461 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-85bf77f655-872lw" podUID="73a08e53-4ce0-49a3-b08a-2a66076123c4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/health\": dial tcp 10.133.0.44:8000: connect: connection refused"
Oct 30 03:55:17.885132 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:55:17.885064    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:55:24.506550 ip-10-0-143-156 kubenswrapper[4461]: E1030 03:55:24.506505    4461 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2k85b" containerName="extractor"
Oct 30 03:55:27.201051 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:55:27.201012    4461 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-85bf77f655-872lw" podUID="73a08e53-4ce0-49a3-b08a-2a66076123c4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/health\": dial tcp 10.133.0.44:8000: connect: connection refused"
Oct 30 03:55:28.042727 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:55:28.042693    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:55:37.210794 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:55:37.210765    4461 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-85bf77f655-872lw"
Oct 30 03:55:37.218522 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:55:37.218494    4461 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-85bf77f655-872lw"
Oct 30 03:55:39.283368 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:55:39.283332    4461 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-85bf77f655-872lw"]
Oct 30 03:55:39.392606 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:55:39.392566    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:55:39.565764 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:55:39.565682    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:55:46.502507 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:55:46.502470    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:55:46.612982 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:55:46.612946    4461 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6c5444664c-cm9zb"]
Oct 30 03:55:46.613303 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:55:46.613289    4461 cpu_manager.go:414] "containerMap: removing container" podUID="35b8f310-5db9-4cf0-99ea-0baca82d3cda" containerName="istio-proxy"
Oct 30 03:55:46.613349 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:55:46.613304    4461 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="35b8f310-5db9-4cf0-99ea-0baca82d3cda" containerName="istio-proxy"
Oct 30 03:55:46.613384 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:55:46.613350    4461 memory_manager.go:365] "RemoveStaleState removing state" podUID="35b8f310-5db9-4cf0-99ea-0baca82d3cda" containerName="istio-proxy"
Oct 30 03:55:46.616513 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:55:46.616492    4461 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6c5444664c-cm9zb"
Oct 30 03:55:46.637399 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:55:46.637369    4461 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6c5444664c-cm9zb"]
Oct 30 03:55:46.645685 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:55:46.645662    4461 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"stop-feature-test-epp-sa-dockercfg-ftc27\""
Oct 30 03:55:46.680048 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:55:46.680018    4461 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ms7dd\" (UniqueName: \"kubernetes.io/projected/43ff4be2-222a-44f2-8ceb-de4b4d97adaa-kube-api-access-ms7dd\") pod \"stop-feature-test-kserve-router-scheduler-6c5444664c-cm9zb\" (UID: \"43ff4be2-222a-44f2-8ceb-de4b4d97adaa\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6c5444664c-cm9zb"
Oct 30 03:55:46.680176 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:55:46.680058    4461 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/43ff4be2-222a-44f2-8ceb-de4b4d97adaa-tls-certs\") pod \"stop-feature-test-kserve-router-scheduler-6c5444664c-cm9zb\" (UID: \"43ff4be2-222a-44f2-8ceb-de4b4d97adaa\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6c5444664c-cm9zb"
Oct 30 03:55:46.752450 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:55:46.752421    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:55:46.780789 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:55:46.780719    4461 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/43ff4be2-222a-44f2-8ceb-de4b4d97adaa-tls-certs\") pod \"stop-feature-test-kserve-router-scheduler-6c5444664c-cm9zb\" (UID: \"43ff4be2-222a-44f2-8ceb-de4b4d97adaa\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6c5444664c-cm9zb"
Oct 30 03:55:46.780919 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:55:46.780798    4461 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ms7dd\" (UniqueName: \"kubernetes.io/projected/43ff4be2-222a-44f2-8ceb-de4b4d97adaa-kube-api-access-ms7dd\") pod \"stop-feature-test-kserve-router-scheduler-6c5444664c-cm9zb\" (UID: \"43ff4be2-222a-44f2-8ceb-de4b4d97adaa\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6c5444664c-cm9zb"
Oct 30 03:55:46.783176 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:55:46.783155    4461 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/43ff4be2-222a-44f2-8ceb-de4b4d97adaa-tls-certs\") pod \"stop-feature-test-kserve-router-scheduler-6c5444664c-cm9zb\" (UID: \"43ff4be2-222a-44f2-8ceb-de4b4d97adaa\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6c5444664c-cm9zb"
Oct 30 03:55:46.805636 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:55:46.805613    4461 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ms7dd\" (UniqueName: \"kubernetes.io/projected/43ff4be2-222a-44f2-8ceb-de4b4d97adaa-kube-api-access-ms7dd\") pod \"stop-feature-test-kserve-router-scheduler-6c5444664c-cm9zb\" (UID: \"43ff4be2-222a-44f2-8ceb-de4b4d97adaa\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6c5444664c-cm9zb"
Oct 30 03:55:46.930778 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:55:46.930746    4461 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6c5444664c-cm9zb"
Oct 30 03:55:47.234396 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:55:47.234366    4461 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6c5444664c-cm9zb"]
Oct 30 03:55:47.234854 ip-10-0-143-156 kubenswrapper[4461]: W1030 03:55:47.234826    4461 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod43ff4be2_222a_44f2_8ceb_de4b4d97adaa.slice/crio-95505f259f816074ef6def98181f7551d275f7e61878f8797532e13ce195c1a8 WatchSource:0}: Error finding container 95505f259f816074ef6def98181f7551d275f7e61878f8797532e13ce195c1a8: Status 404 returned error can't find the container with id 95505f259f816074ef6def98181f7551d275f7e61878f8797532e13ce195c1a8
Oct 30 03:55:48.142952 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:55:48.142918    4461 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6c5444664c-cm9zb" event={"ID":"43ff4be2-222a-44f2-8ceb-de4b4d97adaa","Type":"ContainerStarted","Data":"ef9e48d98ffb8c476d91f8c0ded72501b47f63c03010213900d043e8a5898def"}
Oct 30 03:55:48.143361 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:55:48.142966    4461 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6c5444664c-cm9zb" event={"ID":"43ff4be2-222a-44f2-8ceb-de4b4d97adaa","Type":"ContainerStarted","Data":"95505f259f816074ef6def98181f7551d275f7e61878f8797532e13ce195c1a8"}
Oct 30 03:55:48.143361 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:55:48.143010    4461 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6c5444664c-cm9zb"
Oct 30 03:55:48.185032 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:55:48.184985    4461 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6c5444664c-cm9zb" podStartSLOduration=2.184972311 podStartE2EDuration="2.184972311s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:55:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:55:48.184332769 +0000 UTC m=+3616.234137977" watchObservedRunningTime="2026-10-30 03:55:48.184972311 +0000 UTC m=+3616.234777488"
Oct 30 03:55:48.367933 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:55:48.367900    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:55:50.960142 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:55:50.960064    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:55:54.306569 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:55:54.306512    4461 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-85bf77f655-872lw" podUID="73a08e53-4ce0-49a3-b08a-2a66076123c4" containerName="main" containerID="cri-o://f71f7bed9e197879b562d1ed1cbef997e1b385adb95acfd26cf44928014ff01c" gracePeriod=45
Oct 30 03:55:54.503823 ip-10-0-143-156 kubenswrapper[4461]: E1030 03:55:54.503779    4461 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2k85b" containerName="extractor"
Oct 30 03:55:55.065191 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:55:55.065125    4461 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-85bf77f655-872lw"
Oct 30 03:55:55.153027 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:55:55.152990    4461 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/73a08e53-4ce0-49a3-b08a-2a66076123c4-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/73a08e53-4ce0-49a3-b08a-2a66076123c4-cabundle-cert\") pod \"73a08e53-4ce0-49a3-b08a-2a66076123c4\" (UID: \"73a08e53-4ce0-49a3-b08a-2a66076123c4\") "
Oct 30 03:55:55.153228 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:55:55.153042    4461 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/73a08e53-4ce0-49a3-b08a-2a66076123c4-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/73a08e53-4ce0-49a3-b08a-2a66076123c4-kserve-provision-location\") pod \"73a08e53-4ce0-49a3-b08a-2a66076123c4\" (UID: \"73a08e53-4ce0-49a3-b08a-2a66076123c4\") "
Oct 30 03:55:55.153228 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:55:55.153077    4461 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/73a08e53-4ce0-49a3-b08a-2a66076123c4-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/73a08e53-4ce0-49a3-b08a-2a66076123c4-model-cache\") pod \"73a08e53-4ce0-49a3-b08a-2a66076123c4\" (UID: \"73a08e53-4ce0-49a3-b08a-2a66076123c4\") "
Oct 30 03:55:55.153228 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:55:55.153149    4461 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/73a08e53-4ce0-49a3-b08a-2a66076123c4-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/73a08e53-4ce0-49a3-b08a-2a66076123c4-tmp-dir\") pod \"73a08e53-4ce0-49a3-b08a-2a66076123c4\" (UID: \"73a08e53-4ce0-49a3-b08a-2a66076123c4\") "
Oct 30 03:55:55.153228 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:55:55.153190    4461 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/73a08e53-4ce0-49a3-b08a-2a66076123c4-home\" (UniqueName: \"kubernetes.io/empty-dir/73a08e53-4ce0-49a3-b08a-2a66076123c4-home\") pod \"73a08e53-4ce0-49a3-b08a-2a66076123c4\" (UID: \"73a08e53-4ce0-49a3-b08a-2a66076123c4\") "
Oct 30 03:55:55.153451 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:55:55.153246    4461 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/73a08e53-4ce0-49a3-b08a-2a66076123c4-dshm\" (UniqueName: \"kubernetes.io/empty-dir/73a08e53-4ce0-49a3-b08a-2a66076123c4-dshm\") pod \"73a08e53-4ce0-49a3-b08a-2a66076123c4\" (UID: \"73a08e53-4ce0-49a3-b08a-2a66076123c4\") "
Oct 30 03:55:55.153451 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:55:55.153286    4461 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/73a08e53-4ce0-49a3-b08a-2a66076123c4-kube-api-access-q7dg8\" (UniqueName: \"kubernetes.io/projected/73a08e53-4ce0-49a3-b08a-2a66076123c4-kube-api-access-q7dg8\") pod \"73a08e53-4ce0-49a3-b08a-2a66076123c4\" (UID: \"73a08e53-4ce0-49a3-b08a-2a66076123c4\") "
Oct 30 03:55:55.153451 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:55:55.153329    4461 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/73a08e53-4ce0-49a3-b08a-2a66076123c4-tls-certs\" (UniqueName: \"kubernetes.io/secret/73a08e53-4ce0-49a3-b08a-2a66076123c4-tls-certs\") pod \"73a08e53-4ce0-49a3-b08a-2a66076123c4\" (UID: \"73a08e53-4ce0-49a3-b08a-2a66076123c4\") "
Oct 30 03:55:55.153600 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:55:55.153437    4461 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/73a08e53-4ce0-49a3-b08a-2a66076123c4-model-cache" pod "73a08e53-4ce0-49a3-b08a-2a66076123c4" (UID: "73a08e53-4ce0-49a3-b08a-2a66076123c4"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:55:55.153600 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:55:55.153445    4461 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/73a08e53-4ce0-49a3-b08a-2a66076123c4-cabundle-cert" pod "73a08e53-4ce0-49a3-b08a-2a66076123c4" (UID: "73a08e53-4ce0-49a3-b08a-2a66076123c4"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:55:55.153756 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:55:55.153734    4461 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/73a08e53-4ce0-49a3-b08a-2a66076123c4-cabundle-cert\") on node \"ip-10-0-143-156.ec2.internal\" DevicePath \"\""
Oct 30 03:55:55.153922 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:55:55.153907    4461 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/73a08e53-4ce0-49a3-b08a-2a66076123c4-model-cache\") on node \"ip-10-0-143-156.ec2.internal\" DevicePath \"\""
Oct 30 03:55:55.155440 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:55:55.155417    4461 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/73a08e53-4ce0-49a3-b08a-2a66076123c4-home" pod "73a08e53-4ce0-49a3-b08a-2a66076123c4" (UID: "73a08e53-4ce0-49a3-b08a-2a66076123c4"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:55:55.156286 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:55:55.156262    4461 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/73a08e53-4ce0-49a3-b08a-2a66076123c4-dshm" pod "73a08e53-4ce0-49a3-b08a-2a66076123c4" (UID: "73a08e53-4ce0-49a3-b08a-2a66076123c4"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:55:55.156371 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:55:55.156277    4461 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/73a08e53-4ce0-49a3-b08a-2a66076123c4-tls-certs" pod "73a08e53-4ce0-49a3-b08a-2a66076123c4" (UID: "73a08e53-4ce0-49a3-b08a-2a66076123c4"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:55:55.156466 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:55:55.156447    4461 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/73a08e53-4ce0-49a3-b08a-2a66076123c4-kube-api-access-q7dg8" pod "73a08e53-4ce0-49a3-b08a-2a66076123c4" (UID: "73a08e53-4ce0-49a3-b08a-2a66076123c4"). InnerVolumeSpecName "kube-api-access-q7dg8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:55:55.171815 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:55:55.171780    4461 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/73a08e53-4ce0-49a3-b08a-2a66076123c4-tmp-dir" pod "73a08e53-4ce0-49a3-b08a-2a66076123c4" (UID: "73a08e53-4ce0-49a3-b08a-2a66076123c4"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:55:55.173682 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:55:55.173651    4461 generic.go:356] "Generic (PLEG): container finished" podID="73a08e53-4ce0-49a3-b08a-2a66076123c4" containerID="f71f7bed9e197879b562d1ed1cbef997e1b385adb95acfd26cf44928014ff01c" exitCode=0
Oct 30 03:55:55.173810 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:55:55.173727    4461 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-85bf77f655-872lw" event={"ID":"73a08e53-4ce0-49a3-b08a-2a66076123c4","Type":"ContainerDied","Data":"f71f7bed9e197879b562d1ed1cbef997e1b385adb95acfd26cf44928014ff01c"}
Oct 30 03:55:55.173810 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:55:55.173772    4461 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-85bf77f655-872lw" event={"ID":"73a08e53-4ce0-49a3-b08a-2a66076123c4","Type":"ContainerDied","Data":"ca7fdd8dd7b3945762247aac322acfe6ef06dee595fedf67ac9f8a5920ca5dfd"}
Oct 30 03:55:55.173810 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:55:55.173793    4461 scope.go:122] "RemoveContainer" containerID="f71f7bed9e197879b562d1ed1cbef997e1b385adb95acfd26cf44928014ff01c"
Oct 30 03:55:55.173983 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:55:55.173727    4461 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-85bf77f655-872lw"
Oct 30 03:55:55.181133 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:55:55.181106    4461 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/73a08e53-4ce0-49a3-b08a-2a66076123c4-kserve-provision-location" pod "73a08e53-4ce0-49a3-b08a-2a66076123c4" (UID: "73a08e53-4ce0-49a3-b08a-2a66076123c4"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:55:55.182262 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:55:55.182243    4461 scope.go:122] "RemoveContainer" containerID="7ccabd5d3e448416b62f6aa44f7fcc3f09f332cd01d43f15df19f666f8453159"
Oct 30 03:55:55.190656 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:55:55.190626    4461 scope.go:122] "RemoveContainer" containerID="f71f7bed9e197879b562d1ed1cbef997e1b385adb95acfd26cf44928014ff01c"
Oct 30 03:55:55.190967 ip-10-0-143-156 kubenswrapper[4461]: E1030 03:55:55.190944    4461 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f71f7bed9e197879b562d1ed1cbef997e1b385adb95acfd26cf44928014ff01c\": container with ID starting with f71f7bed9e197879b562d1ed1cbef997e1b385adb95acfd26cf44928014ff01c not found: ID does not exist" containerID="f71f7bed9e197879b562d1ed1cbef997e1b385adb95acfd26cf44928014ff01c"
Oct 30 03:55:55.191048 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:55:55.190975    4461 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f71f7bed9e197879b562d1ed1cbef997e1b385adb95acfd26cf44928014ff01c"} err="failed to get container status \"f71f7bed9e197879b562d1ed1cbef997e1b385adb95acfd26cf44928014ff01c\": rpc error: code = NotFound desc = could not find container \"f71f7bed9e197879b562d1ed1cbef997e1b385adb95acfd26cf44928014ff01c\": container with ID starting with f71f7bed9e197879b562d1ed1cbef997e1b385adb95acfd26cf44928014ff01c not found: ID does not exist"
Oct 30 03:55:55.191048 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:55:55.190997    4461 scope.go:122] "RemoveContainer" containerID="7ccabd5d3e448416b62f6aa44f7fcc3f09f332cd01d43f15df19f666f8453159"
Oct 30 03:55:55.191321 ip-10-0-143-156 kubenswrapper[4461]: E1030 03:55:55.191300    4461 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7ccabd5d3e448416b62f6aa44f7fcc3f09f332cd01d43f15df19f666f8453159\": container with ID starting with 7ccabd5d3e448416b62f6aa44f7fcc3f09f332cd01d43f15df19f666f8453159 not found: ID does not exist" containerID="7ccabd5d3e448416b62f6aa44f7fcc3f09f332cd01d43f15df19f666f8453159"
Oct 30 03:55:55.191360 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:55:55.191328    4461 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7ccabd5d3e448416b62f6aa44f7fcc3f09f332cd01d43f15df19f666f8453159"} err="failed to get container status \"7ccabd5d3e448416b62f6aa44f7fcc3f09f332cd01d43f15df19f666f8453159\": rpc error: code = NotFound desc = could not find container \"7ccabd5d3e448416b62f6aa44f7fcc3f09f332cd01d43f15df19f666f8453159\": container with ID starting with 7ccabd5d3e448416b62f6aa44f7fcc3f09f332cd01d43f15df19f666f8453159 not found: ID does not exist"
Oct 30 03:55:55.254918 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:55:55.254884    4461 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/73a08e53-4ce0-49a3-b08a-2a66076123c4-home\") on node \"ip-10-0-143-156.ec2.internal\" DevicePath \"\""
Oct 30 03:55:55.254918 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:55:55.254910    4461 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/73a08e53-4ce0-49a3-b08a-2a66076123c4-dshm\") on node \"ip-10-0-143-156.ec2.internal\" DevicePath \"\""
Oct 30 03:55:55.254918 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:55:55.254920    4461 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q7dg8\" (UniqueName: \"kubernetes.io/projected/73a08e53-4ce0-49a3-b08a-2a66076123c4-kube-api-access-q7dg8\") on node \"ip-10-0-143-156.ec2.internal\" DevicePath \"\""
Oct 30 03:55:55.255219 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:55:55.254930    4461 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/73a08e53-4ce0-49a3-b08a-2a66076123c4-tls-certs\") on node \"ip-10-0-143-156.ec2.internal\" DevicePath \"\""
Oct 30 03:55:55.255219 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:55:55.254939    4461 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/73a08e53-4ce0-49a3-b08a-2a66076123c4-kserve-provision-location\") on node \"ip-10-0-143-156.ec2.internal\" DevicePath \"\""
Oct 30 03:55:55.255219 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:55:55.254948    4461 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/73a08e53-4ce0-49a3-b08a-2a66076123c4-tmp-dir\") on node \"ip-10-0-143-156.ec2.internal\" DevicePath \"\""
Oct 30 03:55:55.519080 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:55:55.519051    4461 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-85bf77f655-872lw"]
Oct 30 03:55:55.532077 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:55:55.532051    4461 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-85bf77f655-872lw"]
Oct 30 03:55:56.114527 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:55:56.114495    4461 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="73a08e53-4ce0-49a3-b08a-2a66076123c4" path="/var/lib/kubelet/pods/73a08e53-4ce0-49a3-b08a-2a66076123c4/volumes"
Oct 30 03:56:18.002411 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:56:18.002336    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:56:19.149682 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:56:19.149653    4461 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6c5444664c-cm9zb"
Oct 30 03:56:24.503279 ip-10-0-143-156 kubenswrapper[4461]: E1030 03:56:24.503244    4461 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2k85b" containerName="extractor"
Oct 30 03:56:28.180677 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:56:28.180645    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:56:37.696826 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:56:37.696786    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:56:37.872131 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:56:37.872083    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:56:46.177576 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:56:46.177542    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:56:46.351225 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:56:46.351188    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:56:48.467378 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:56:48.467345    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:56:49.951841 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:56:49.951804    4461 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-wbk94"]
Oct 30 03:56:49.952278 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:56:49.952038    4461 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-wbk94" podUID="1e409e01-826e-4317-af23-194073c1c6a1" containerName="istio-proxy" containerID="cri-o://603df059ee07b50290a7b02ac07fd4009814e118193758d252e5769594ae563c" gracePeriod=30
Oct 30 03:56:53.785396 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:56:53.785358    4461 patch_prober.go:29] interesting pod/openshift-ai-inference-openshift-default-5b4f8cc59b-wbk94 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 03:56:53.785819 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:56:53.785415    4461 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-wbk94" podUID="1e409e01-826e-4317-af23-194073c1c6a1" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 03:56:54.502547 ip-10-0-143-156 kubenswrapper[4461]: E1030 03:56:54.502506    4461 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2k85b" containerName="extractor"
Oct 30 03:56:55.200730 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:56:55.200702    4461 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-wbk94"
Oct 30 03:56:55.229268 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:56:55.229176    4461 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1e409e01-826e-4317-af23-194073c1c6a1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1e409e01-826e-4317-af23-194073c1c6a1-istio-podinfo\") pod \"1e409e01-826e-4317-af23-194073c1c6a1\" (UID: \"1e409e01-826e-4317-af23-194073c1c6a1\") "
Oct 30 03:56:55.229268 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:56:55.229229    4461 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1e409e01-826e-4317-af23-194073c1c6a1-istio-token\" (UniqueName: \"kubernetes.io/projected/1e409e01-826e-4317-af23-194073c1c6a1-istio-token\") pod \"1e409e01-826e-4317-af23-194073c1c6a1\" (UID: \"1e409e01-826e-4317-af23-194073c1c6a1\") "
Oct 30 03:56:55.229268 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:56:55.229260    4461 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1e409e01-826e-4317-af23-194073c1c6a1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1e409e01-826e-4317-af23-194073c1c6a1-istio-envoy\") pod \"1e409e01-826e-4317-af23-194073c1c6a1\" (UID: \"1e409e01-826e-4317-af23-194073c1c6a1\") "
Oct 30 03:56:55.229602 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:56:55.229289    4461 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1e409e01-826e-4317-af23-194073c1c6a1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1e409e01-826e-4317-af23-194073c1c6a1-workload-certs\") pod \"1e409e01-826e-4317-af23-194073c1c6a1\" (UID: \"1e409e01-826e-4317-af23-194073c1c6a1\") "
Oct 30 03:56:55.229602 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:56:55.229319    4461 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1e409e01-826e-4317-af23-194073c1c6a1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e409e01-826e-4317-af23-194073c1c6a1-workload-socket\") pod \"1e409e01-826e-4317-af23-194073c1c6a1\" (UID: \"1e409e01-826e-4317-af23-194073c1c6a1\") "
Oct 30 03:56:55.229602 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:56:55.229342    4461 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1e409e01-826e-4317-af23-194073c1c6a1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1e409e01-826e-4317-af23-194073c1c6a1-istio-data\") pod \"1e409e01-826e-4317-af23-194073c1c6a1\" (UID: \"1e409e01-826e-4317-af23-194073c1c6a1\") "
Oct 30 03:56:55.229602 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:56:55.229390    4461 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1e409e01-826e-4317-af23-194073c1c6a1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1e409e01-826e-4317-af23-194073c1c6a1-istiod-ca-cert\") pod \"1e409e01-826e-4317-af23-194073c1c6a1\" (UID: \"1e409e01-826e-4317-af23-194073c1c6a1\") "
Oct 30 03:56:55.229602 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:56:55.229422    4461 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1e409e01-826e-4317-af23-194073c1c6a1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e409e01-826e-4317-af23-194073c1c6a1-credential-socket\") pod \"1e409e01-826e-4317-af23-194073c1c6a1\" (UID: \"1e409e01-826e-4317-af23-194073c1c6a1\") "
Oct 30 03:56:55.229879 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:56:55.229659    4461 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e409e01-826e-4317-af23-194073c1c6a1-workload-certs" pod "1e409e01-826e-4317-af23-194073c1c6a1" (UID: "1e409e01-826e-4317-af23-194073c1c6a1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:56:55.229879 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:56:55.229673    4461 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e409e01-826e-4317-af23-194073c1c6a1-workload-socket" pod "1e409e01-826e-4317-af23-194073c1c6a1" (UID: "1e409e01-826e-4317-af23-194073c1c6a1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:56:55.229879 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:56:55.229859    4461 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e409e01-826e-4317-af23-194073c1c6a1-credential-socket" pod "1e409e01-826e-4317-af23-194073c1c6a1" (UID: "1e409e01-826e-4317-af23-194073c1c6a1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:56:55.230036 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:56:55.229843    4461 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e409e01-826e-4317-af23-194073c1c6a1-istio-data" pod "1e409e01-826e-4317-af23-194073c1c6a1" (UID: "1e409e01-826e-4317-af23-194073c1c6a1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:56:55.230036 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:56:55.229958    4461 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1e409e01-826e-4317-af23-194073c1c6a1-istiod-ca-cert" pod "1e409e01-826e-4317-af23-194073c1c6a1" (UID: "1e409e01-826e-4317-af23-194073c1c6a1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:56:55.231553 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:56:55.231526    4461 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1e409e01-826e-4317-af23-194073c1c6a1-istio-podinfo" pod "1e409e01-826e-4317-af23-194073c1c6a1" (UID: "1e409e01-826e-4317-af23-194073c1c6a1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 03:56:55.231686 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:56:55.231606    4461 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1e409e01-826e-4317-af23-194073c1c6a1-istio-token" pod "1e409e01-826e-4317-af23-194073c1c6a1" (UID: "1e409e01-826e-4317-af23-194073c1c6a1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:56:55.231686 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:56:55.231659    4461 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e409e01-826e-4317-af23-194073c1c6a1-istio-envoy" pod "1e409e01-826e-4317-af23-194073c1c6a1" (UID: "1e409e01-826e-4317-af23-194073c1c6a1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:56:55.330667 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:56:55.330637    4461 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1e409e01-826e-4317-af23-194073c1c6a1-kube-api-access-6dgp4\" (UniqueName: \"kubernetes.io/projected/1e409e01-826e-4317-af23-194073c1c6a1-kube-api-access-6dgp4\") pod \"1e409e01-826e-4317-af23-194073c1c6a1\" (UID: \"1e409e01-826e-4317-af23-194073c1c6a1\") "
Oct 30 03:56:55.330937 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:56:55.330921    4461 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1e409e01-826e-4317-af23-194073c1c6a1-istio-envoy\") on node \"ip-10-0-143-156.ec2.internal\" DevicePath \"\""
Oct 30 03:56:55.331025 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:56:55.330945    4461 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1e409e01-826e-4317-af23-194073c1c6a1-workload-certs\") on node \"ip-10-0-143-156.ec2.internal\" DevicePath \"\""
Oct 30 03:56:55.331025 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:56:55.330958    4461 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e409e01-826e-4317-af23-194073c1c6a1-workload-socket\") on node \"ip-10-0-143-156.ec2.internal\" DevicePath \"\""
Oct 30 03:56:55.331025 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:56:55.330971    4461 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1e409e01-826e-4317-af23-194073c1c6a1-istio-data\") on node \"ip-10-0-143-156.ec2.internal\" DevicePath \"\""
Oct 30 03:56:55.331025 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:56:55.330985    4461 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1e409e01-826e-4317-af23-194073c1c6a1-istiod-ca-cert\") on node \"ip-10-0-143-156.ec2.internal\" DevicePath \"\""
Oct 30 03:56:55.331025 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:56:55.330998    4461 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e409e01-826e-4317-af23-194073c1c6a1-credential-socket\") on node \"ip-10-0-143-156.ec2.internal\" DevicePath \"\""
Oct 30 03:56:55.331025 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:56:55.331011    4461 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1e409e01-826e-4317-af23-194073c1c6a1-istio-podinfo\") on node \"ip-10-0-143-156.ec2.internal\" DevicePath \"\""
Oct 30 03:56:55.331025 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:56:55.331025    4461 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1e409e01-826e-4317-af23-194073c1c6a1-istio-token\") on node \"ip-10-0-143-156.ec2.internal\" DevicePath \"\""
Oct 30 03:56:55.332686 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:56:55.332657    4461 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1e409e01-826e-4317-af23-194073c1c6a1-kube-api-access-6dgp4" pod "1e409e01-826e-4317-af23-194073c1c6a1" (UID: "1e409e01-826e-4317-af23-194073c1c6a1"). InnerVolumeSpecName "kube-api-access-6dgp4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:56:55.415407 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:56:55.415374    4461 generic.go:356] "Generic (PLEG): container finished" podID="1e409e01-826e-4317-af23-194073c1c6a1" containerID="603df059ee07b50290a7b02ac07fd4009814e118193758d252e5769594ae563c" exitCode=0
Oct 30 03:56:55.415586 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:56:55.415425    4461 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-wbk94"
Oct 30 03:56:55.415586 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:56:55.415456    4461 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-wbk94" event={"ID":"1e409e01-826e-4317-af23-194073c1c6a1","Type":"ContainerDied","Data":"603df059ee07b50290a7b02ac07fd4009814e118193758d252e5769594ae563c"}
Oct 30 03:56:55.415586 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:56:55.415516    4461 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-wbk94" event={"ID":"1e409e01-826e-4317-af23-194073c1c6a1","Type":"ContainerDied","Data":"d8e22a34e2ae1e4fcb9fca2bbdfe0740e6781314cb8709994fe70e46eba2b46c"}
Oct 30 03:56:55.415586 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:56:55.415565    4461 scope.go:122] "RemoveContainer" containerID="603df059ee07b50290a7b02ac07fd4009814e118193758d252e5769594ae563c"
Oct 30 03:56:55.424278 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:56:55.424258    4461 scope.go:122] "RemoveContainer" containerID="603df059ee07b50290a7b02ac07fd4009814e118193758d252e5769594ae563c"
Oct 30 03:56:55.424564 ip-10-0-143-156 kubenswrapper[4461]: E1030 03:56:55.424544    4461 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"603df059ee07b50290a7b02ac07fd4009814e118193758d252e5769594ae563c\": container with ID starting with 603df059ee07b50290a7b02ac07fd4009814e118193758d252e5769594ae563c not found: ID does not exist" containerID="603df059ee07b50290a7b02ac07fd4009814e118193758d252e5769594ae563c"
Oct 30 03:56:55.424629 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:56:55.424576    4461 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"603df059ee07b50290a7b02ac07fd4009814e118193758d252e5769594ae563c"} err="failed to get container status \"603df059ee07b50290a7b02ac07fd4009814e118193758d252e5769594ae563c\": rpc error: code = NotFound desc = could not find container \"603df059ee07b50290a7b02ac07fd4009814e118193758d252e5769594ae563c\": container with ID starting with 603df059ee07b50290a7b02ac07fd4009814e118193758d252e5769594ae563c not found: ID does not exist"
Oct 30 03:56:55.431618 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:56:55.431600    4461 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6dgp4\" (UniqueName: \"kubernetes.io/projected/1e409e01-826e-4317-af23-194073c1c6a1-kube-api-access-6dgp4\") on node \"ip-10-0-143-156.ec2.internal\" DevicePath \"\""
Oct 30 03:56:55.468632 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:56:55.468604    4461 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-wbk94"]
Oct 30 03:56:55.472926 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:56:55.472904    4461 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-wbk94"]
Oct 30 03:56:56.115236 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:56:56.115208    4461 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1e409e01-826e-4317-af23-194073c1c6a1" path="/var/lib/kubelet/pods/1e409e01-826e-4317-af23-194073c1c6a1/volumes"
Oct 30 03:57:18.151348 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:57:18.151310    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:57:24.503480 ip-10-0-143-156 kubenswrapper[4461]: E1030 03:57:24.503441    4461 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2k85b" containerName="extractor"
Oct 30 03:57:27.822391 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:57:27.822344    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:57:27.999013 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:57:27.998975    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:57:28.358248 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:57:28.358214    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:57:31.685521 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:57:31.685482    4461 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7f9654778bpjznv"]
Oct 30 03:57:31.685973 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:57:31.685771    4461 cpu_manager.go:414] "containerMap: removing container" podUID="73a08e53-4ce0-49a3-b08a-2a66076123c4" containerName="main"
Oct 30 03:57:31.685973 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:57:31.685783    4461 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="73a08e53-4ce0-49a3-b08a-2a66076123c4" containerName="main"
Oct 30 03:57:31.685973 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:57:31.685798    4461 cpu_manager.go:414] "containerMap: removing container" podUID="1e409e01-826e-4317-af23-194073c1c6a1" containerName="istio-proxy"
Oct 30 03:57:31.685973 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:57:31.685803    4461 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1e409e01-826e-4317-af23-194073c1c6a1" containerName="istio-proxy"
Oct 30 03:57:31.685973 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:57:31.685818    4461 cpu_manager.go:414] "containerMap: removing container" podUID="73a08e53-4ce0-49a3-b08a-2a66076123c4" containerName="storage-initializer"
Oct 30 03:57:31.685973 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:57:31.685824    4461 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="73a08e53-4ce0-49a3-b08a-2a66076123c4" containerName="storage-initializer"
Oct 30 03:57:31.685973 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:57:31.685874    4461 memory_manager.go:365] "RemoveStaleState removing state" podUID="73a08e53-4ce0-49a3-b08a-2a66076123c4" containerName="main"
Oct 30 03:57:31.685973 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:57:31.685882    4461 memory_manager.go:365] "RemoveStaleState removing state" podUID="1e409e01-826e-4317-af23-194073c1c6a1" containerName="istio-proxy"
Oct 30 03:57:31.689700 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:57:31.689666    4461 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7f9654778bpjznv"
Oct 30 03:57:31.712122 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:57:31.712081    4461 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"odh-kserve-custom-ca-bundle\""
Oct 30 03:57:31.712122 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:57:31.712081    4461 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"seaweedfs-s3-creds\""
Oct 30 03:57:31.719312 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:57:31.719288    4461 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"default-dockercfg-wst2f\""
Oct 30 03:57:31.719473 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:57:31.719332    4461 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"kube-root-ca.crt\""
Oct 30 03:57:31.719473 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:57:31.719355    4461 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"llmisv44d181485fad85e662eb092f3749502f-kserve-self-signed-certs\""
Oct 30 03:57:31.719583 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:57:31.719480    4461 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-dockercfg-jsxxx\""
Oct 30 03:57:31.721501 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:57:31.721478    4461 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7f9654778bpjznv"]
Oct 30 03:57:31.810992 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:57:31.810951    4461 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"openshift-service-ca.crt\""
Oct 30 03:57:31.829442 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:57:31.829412    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:57:31.838561 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:57:31.838535    4461 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d83b10c6-b612-447e-967d-f170f2a6170d-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7f9654778bpjznv\" (UID: \"d83b10c6-b612-447e-967d-f170f2a6170d\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7f9654778bpjznv"
Oct 30 03:57:31.838639 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:57:31.838571    4461 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d83b10c6-b612-447e-967d-f170f2a6170d-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7f9654778bpjznv\" (UID: \"d83b10c6-b612-447e-967d-f170f2a6170d\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7f9654778bpjznv"
Oct 30 03:57:31.838677 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:57:31.838649    4461 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d83b10c6-b612-447e-967d-f170f2a6170d-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7f9654778bpjznv\" (UID: \"d83b10c6-b612-447e-967d-f170f2a6170d\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7f9654778bpjznv"
Oct 30 03:57:31.838677 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:57:31.838670    4461 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d83b10c6-b612-447e-967d-f170f2a6170d-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7f9654778bpjznv\" (UID: \"d83b10c6-b612-447e-967d-f170f2a6170d\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7f9654778bpjznv"
Oct 30 03:57:31.838746 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:57:31.838689    4461 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gg8fl\" (UniqueName: \"kubernetes.io/projected/d83b10c6-b612-447e-967d-f170f2a6170d-kube-api-access-gg8fl\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7f9654778bpjznv\" (UID: \"d83b10c6-b612-447e-967d-f170f2a6170d\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7f9654778bpjznv"
Oct 30 03:57:31.838746 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:57:31.838708    4461 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d83b10c6-b612-447e-967d-f170f2a6170d-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7f9654778bpjznv\" (UID: \"d83b10c6-b612-447e-967d-f170f2a6170d\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7f9654778bpjznv"
Oct 30 03:57:31.838816 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:57:31.838772    4461 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d83b10c6-b612-447e-967d-f170f2a6170d-dshm\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7f9654778bpjznv\" (UID: \"d83b10c6-b612-447e-967d-f170f2a6170d\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7f9654778bpjznv"
Oct 30 03:57:31.838869 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:57:31.838849    4461 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d83b10c6-b612-447e-967d-f170f2a6170d-home\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7f9654778bpjznv\" (UID: \"d83b10c6-b612-447e-967d-f170f2a6170d\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7f9654778bpjznv"
Oct 30 03:57:31.939755 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:57:31.939678    4461 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gg8fl\" (UniqueName: \"kubernetes.io/projected/d83b10c6-b612-447e-967d-f170f2a6170d-kube-api-access-gg8fl\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7f9654778bpjznv\" (UID: \"d83b10c6-b612-447e-967d-f170f2a6170d\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7f9654778bpjznv"
Oct 30 03:57:31.939755 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:57:31.939717    4461 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d83b10c6-b612-447e-967d-f170f2a6170d-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7f9654778bpjznv\" (UID: \"d83b10c6-b612-447e-967d-f170f2a6170d\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7f9654778bpjznv"
Oct 30 03:57:31.939961 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:57:31.939761    4461 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d83b10c6-b612-447e-967d-f170f2a6170d-dshm\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7f9654778bpjznv\" (UID: \"d83b10c6-b612-447e-967d-f170f2a6170d\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7f9654778bpjznv"
Oct 30 03:57:31.939961 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:57:31.939806    4461 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d83b10c6-b612-447e-967d-f170f2a6170d-home\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7f9654778bpjznv\" (UID: \"d83b10c6-b612-447e-967d-f170f2a6170d\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7f9654778bpjznv"
Oct 30 03:57:31.939961 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:57:31.939849    4461 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d83b10c6-b612-447e-967d-f170f2a6170d-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7f9654778bpjznv\" (UID: \"d83b10c6-b612-447e-967d-f170f2a6170d\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7f9654778bpjznv"
Oct 30 03:57:31.939961 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:57:31.939878    4461 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d83b10c6-b612-447e-967d-f170f2a6170d-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7f9654778bpjznv\" (UID: \"d83b10c6-b612-447e-967d-f170f2a6170d\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7f9654778bpjznv"
Oct 30 03:57:31.939961 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:57:31.939932    4461 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d83b10c6-b612-447e-967d-f170f2a6170d-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7f9654778bpjznv\" (UID: \"d83b10c6-b612-447e-967d-f170f2a6170d\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7f9654778bpjznv"
Oct 30 03:57:31.939961 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:57:31.939959    4461 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d83b10c6-b612-447e-967d-f170f2a6170d-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7f9654778bpjznv\" (UID: \"d83b10c6-b612-447e-967d-f170f2a6170d\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7f9654778bpjznv"
Oct 30 03:57:31.940408 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:57:31.940378    4461 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d83b10c6-b612-447e-967d-f170f2a6170d-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7f9654778bpjznv\" (UID: \"d83b10c6-b612-447e-967d-f170f2a6170d\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7f9654778bpjznv"
Oct 30 03:57:31.940504 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:57:31.940407    4461 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d83b10c6-b612-447e-967d-f170f2a6170d-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7f9654778bpjznv\" (UID: \"d83b10c6-b612-447e-967d-f170f2a6170d\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7f9654778bpjznv"
Oct 30 03:57:31.940504 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:57:31.940450    4461 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d83b10c6-b612-447e-967d-f170f2a6170d-home\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7f9654778bpjznv\" (UID: \"d83b10c6-b612-447e-967d-f170f2a6170d\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7f9654778bpjznv"
Oct 30 03:57:31.940504 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:57:31.940466    4461 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d83b10c6-b612-447e-967d-f170f2a6170d-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7f9654778bpjznv\" (UID: \"d83b10c6-b612-447e-967d-f170f2a6170d\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7f9654778bpjznv"
Oct 30 03:57:31.940698 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:57:31.940676    4461 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d83b10c6-b612-447e-967d-f170f2a6170d-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7f9654778bpjznv\" (UID: \"d83b10c6-b612-447e-967d-f170f2a6170d\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7f9654778bpjznv"
Oct 30 03:57:31.942299 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:57:31.942272    4461 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d83b10c6-b612-447e-967d-f170f2a6170d-dshm\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7f9654778bpjznv\" (UID: \"d83b10c6-b612-447e-967d-f170f2a6170d\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7f9654778bpjznv"
Oct 30 03:57:31.942729 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:57:31.942707    4461 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d83b10c6-b612-447e-967d-f170f2a6170d-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7f9654778bpjznv\" (UID: \"d83b10c6-b612-447e-967d-f170f2a6170d\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7f9654778bpjznv"
Oct 30 03:57:31.963667 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:57:31.963643    4461 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gg8fl\" (UniqueName: \"kubernetes.io/projected/d83b10c6-b612-447e-967d-f170f2a6170d-kube-api-access-gg8fl\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7f9654778bpjznv\" (UID: \"d83b10c6-b612-447e-967d-f170f2a6170d\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7f9654778bpjznv"
Oct 30 03:57:32.008637 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:57:32.008603    4461 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7f9654778bpjznv"
Oct 30 03:57:32.336019 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:57:32.335986    4461 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7f9654778bpjznv"]
Oct 30 03:57:32.336568 ip-10-0-143-156 kubenswrapper[4461]: W1030 03:57:32.336542    4461 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd83b10c6_b612_447e_967d_f170f2a6170d.slice/crio-2057d4ef0b73aacdba9240e8523f81b82199d8313b67d1a4751f7b552935e877 WatchSource:0}: Error finding container 2057d4ef0b73aacdba9240e8523f81b82199d8313b67d1a4751f7b552935e877: Status 404 returned error can't find the container with id 2057d4ef0b73aacdba9240e8523f81b82199d8313b67d1a4751f7b552935e877
Oct 30 03:57:32.557776 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:57:32.557745    4461 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7f9654778bpjznv" event={"ID":"d83b10c6-b612-447e-967d-f170f2a6170d","Type":"ContainerStarted","Data":"69ea986a4c5b9fe5bd1ace8b5ea90ada6ba6eaba651ecd8e7cac21078e495fd0"}
Oct 30 03:57:32.557907 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:57:32.557795    4461 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7f9654778bpjznv" event={"ID":"d83b10c6-b612-447e-967d-f170f2a6170d","Type":"ContainerStarted","Data":"2057d4ef0b73aacdba9240e8523f81b82199d8313b67d1a4751f7b552935e877"}
Oct 30 03:57:32.557907 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:57:32.557861    4461 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7f9654778bpjznv"
Oct 30 03:57:32.572793 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:57:32.572768    4461 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"seaweedfs-s3-creds\""
Oct 30 03:57:33.564671 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:57:33.564631    4461 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7f9654778bpjznv" event={"ID":"d83b10c6-b612-447e-967d-f170f2a6170d","Type":"ContainerStarted","Data":"07ca7664c49981c00e7714c5194187ba168e250d6a5f04c4a27128912c24eea9"}
Oct 30 03:57:33.852862 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:57:33.852777    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:57:35.574712 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:57:35.574680    4461 generic.go:356] "Generic (PLEG): container finished" podID="d83b10c6-b612-447e-967d-f170f2a6170d" containerID="07ca7664c49981c00e7714c5194187ba168e250d6a5f04c4a27128912c24eea9" exitCode=0
Oct 30 03:57:35.574712 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:57:35.574718    4461 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7f9654778bpjznv" event={"ID":"d83b10c6-b612-447e-967d-f170f2a6170d","Type":"ContainerDied","Data":"07ca7664c49981c00e7714c5194187ba168e250d6a5f04c4a27128912c24eea9"}
Oct 30 03:57:36.581531 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:57:36.581488    4461 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7f9654778bpjznv" event={"ID":"d83b10c6-b612-447e-967d-f170f2a6170d","Type":"ContainerStarted","Data":"45f4ae7dbafa6f3bb247e692028e7aba4494fc71273485e5b85794cbeebcf391"}
Oct 30 03:57:36.657438 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:57:36.657376    4461 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7f9654778bpjznv" podStartSLOduration=3.537003636 podStartE2EDuration="5.657356997s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.120353361s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:57:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:57:36.643869397 +0000 UTC m=+3724.693674594" watchObservedRunningTime="2026-10-30 03:57:36.657356997 +0000 UTC m=+3724.707162171"
Oct 30 03:57:37.712074 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:57:37.712033    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:57:42.009280 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:57:42.009237    4461 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7f9654778bpjznv"
Oct 30 03:57:42.009737 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:57:42.009322    4461 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7f9654778bpjznv"
Oct 30 03:57:42.010454 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:57:42.010413    4461 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7f9654778bpjznv" podUID="d83b10c6-b612-447e-967d-f170f2a6170d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8001/health\": dial tcp 10.133.0.46:8001: connect: connection refused"
Oct 30 03:57:42.021573 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:57:42.021553    4461 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7f9654778bpjznv"
Oct 30 03:57:48.572924 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:57:48.572835    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:57:49.026320 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:57:49.026287    4461 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6c5444664c-cm9zb"]
Oct 30 03:57:49.073737 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:57:49.073679    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:57:49.149391 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:57:49.149356    4461 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6c5444664c-cm9zb" podUID="43ff4be2-222a-44f2-8ceb-de4b4d97adaa" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 03:57:49.246248 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:57:49.246213    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:57:52.009472 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:57:52.009415    4461 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7f9654778bpjznv" podUID="d83b10c6-b612-447e-967d-f170f2a6170d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8001/health\": dial tcp 10.133.0.46:8001: connect: connection refused"
Oct 30 03:57:54.508947 ip-10-0-143-156 kubenswrapper[4461]: E1030 03:57:54.508082    4461 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2k85b" containerName="extractor"
Oct 30 03:57:54.549693 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:57:54.549656    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:57:55.485915 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:57:55.485881    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:57:59.148976 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:57:59.148934    4461 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6c5444664c-cm9zb" podUID="43ff4be2-222a-44f2-8ceb-de4b4d97adaa" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 03:58:01.725111 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:58:01.725060    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:58:02.009626 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:58:02.009527    4461 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7f9654778bpjznv" podUID="d83b10c6-b612-447e-967d-f170f2a6170d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8001/health\": dial tcp 10.133.0.46:8001: connect: connection refused"
Oct 30 03:58:04.049513 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:58:04.049445    4461 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6c5444664c-cm9zb" podUID="43ff4be2-222a-44f2-8ceb-de4b4d97adaa" containerName="main" containerID="cri-o://ef9e48d98ffb8c476d91f8c0ded72501b47f63c03010213900d043e8a5898def" gracePeriod=45
Oct 30 03:58:09.149791 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:58:09.149755    4461 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6c5444664c-cm9zb" podUID="43ff4be2-222a-44f2-8ceb-de4b4d97adaa" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 03:58:09.150297 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:58:09.149859    4461 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6c5444664c-cm9zb"
Oct 30 03:58:12.009546 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:58:12.009502    4461 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7f9654778bpjznv" podUID="d83b10c6-b612-447e-967d-f170f2a6170d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8001/health\": dial tcp 10.133.0.46:8001: connect: connection refused"
Oct 30 03:58:16.274461 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:58:16.274419    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:58:16.889200 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:58:16.889164    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:58:17.066604 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:58:17.066570    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:58:18.237156 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:58:18.237067    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:58:19.149000 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:58:19.148961    4461 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6c5444664c-cm9zb" podUID="43ff4be2-222a-44f2-8ceb-de4b4d97adaa" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 03:58:22.009058 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:58:22.008989    4461 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7f9654778bpjznv" podUID="d83b10c6-b612-447e-967d-f170f2a6170d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8001/health\": dial tcp 10.133.0.46:8001: connect: connection refused"
Oct 30 03:58:24.504298 ip-10-0-143-156 kubenswrapper[4461]: E1030 03:58:24.504254    4461 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2k85b" containerName="extractor"
Oct 30 03:58:28.417361 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:58:28.417322    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:58:29.149356 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:58:29.149315    4461 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6c5444664c-cm9zb" podUID="43ff4be2-222a-44f2-8ceb-de4b4d97adaa" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 03:58:32.009928 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:58:32.009885    4461 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7f9654778bpjznv" podUID="d83b10c6-b612-447e-967d-f170f2a6170d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8001/health\": dial tcp 10.133.0.46:8001: connect: connection refused"
Oct 30 03:58:34.289447 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:58:34.289423    4461 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6c5444664c-cm9zb"
Oct 30 03:58:34.302965 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:58:34.302936    4461 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/43ff4be2-222a-44f2-8ceb-de4b4d97adaa-kube-api-access-ms7dd\" (UniqueName: \"kubernetes.io/projected/43ff4be2-222a-44f2-8ceb-de4b4d97adaa-kube-api-access-ms7dd\") pod \"43ff4be2-222a-44f2-8ceb-de4b4d97adaa\" (UID: \"43ff4be2-222a-44f2-8ceb-de4b4d97adaa\") "
Oct 30 03:58:34.303116 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:58:34.302984    4461 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/43ff4be2-222a-44f2-8ceb-de4b4d97adaa-tls-certs\" (UniqueName: \"kubernetes.io/secret/43ff4be2-222a-44f2-8ceb-de4b4d97adaa-tls-certs\") pod \"43ff4be2-222a-44f2-8ceb-de4b4d97adaa\" (UID: \"43ff4be2-222a-44f2-8ceb-de4b4d97adaa\") "
Oct 30 03:58:34.305001 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:58:34.304971    4461 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/43ff4be2-222a-44f2-8ceb-de4b4d97adaa-kube-api-access-ms7dd" pod "43ff4be2-222a-44f2-8ceb-de4b4d97adaa" (UID: "43ff4be2-222a-44f2-8ceb-de4b4d97adaa"). InnerVolumeSpecName "kube-api-access-ms7dd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:58:34.305113 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:58:34.305008    4461 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/43ff4be2-222a-44f2-8ceb-de4b4d97adaa-tls-certs" pod "43ff4be2-222a-44f2-8ceb-de4b4d97adaa" (UID: "43ff4be2-222a-44f2-8ceb-de4b4d97adaa"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:58:34.404118 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:58:34.404022    4461 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ms7dd\" (UniqueName: \"kubernetes.io/projected/43ff4be2-222a-44f2-8ceb-de4b4d97adaa-kube-api-access-ms7dd\") on node \"ip-10-0-143-156.ec2.internal\" DevicePath \"\""
Oct 30 03:58:34.404118 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:58:34.404048    4461 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/43ff4be2-222a-44f2-8ceb-de4b4d97adaa-tls-certs\") on node \"ip-10-0-143-156.ec2.internal\" DevicePath \"\""
Oct 30 03:58:34.828720 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:58:34.828689    4461 generic.go:356] "Generic (PLEG): container finished" podID="43ff4be2-222a-44f2-8ceb-de4b4d97adaa" containerID="ef9e48d98ffb8c476d91f8c0ded72501b47f63c03010213900d043e8a5898def" exitCode=0
Oct 30 03:58:34.828931 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:58:34.828741    4461 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6c5444664c-cm9zb" event={"ID":"43ff4be2-222a-44f2-8ceb-de4b4d97adaa","Type":"ContainerDied","Data":"ef9e48d98ffb8c476d91f8c0ded72501b47f63c03010213900d043e8a5898def"}
Oct 30 03:58:34.828931 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:58:34.828744    4461 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6c5444664c-cm9zb"
Oct 30 03:58:34.828931 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:58:34.828788    4461 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6c5444664c-cm9zb" event={"ID":"43ff4be2-222a-44f2-8ceb-de4b4d97adaa","Type":"ContainerDied","Data":"95505f259f816074ef6def98181f7551d275f7e61878f8797532e13ce195c1a8"}
Oct 30 03:58:34.828931 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:58:34.828812    4461 scope.go:122] "RemoveContainer" containerID="ef9e48d98ffb8c476d91f8c0ded72501b47f63c03010213900d043e8a5898def"
Oct 30 03:58:34.838322 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:58:34.838306    4461 scope.go:122] "RemoveContainer" containerID="ef9e48d98ffb8c476d91f8c0ded72501b47f63c03010213900d043e8a5898def"
Oct 30 03:58:34.838602 ip-10-0-143-156 kubenswrapper[4461]: E1030 03:58:34.838585    4461 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ef9e48d98ffb8c476d91f8c0ded72501b47f63c03010213900d043e8a5898def\": container with ID starting with ef9e48d98ffb8c476d91f8c0ded72501b47f63c03010213900d043e8a5898def not found: ID does not exist" containerID="ef9e48d98ffb8c476d91f8c0ded72501b47f63c03010213900d043e8a5898def"
Oct 30 03:58:34.838664 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:58:34.838617    4461 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ef9e48d98ffb8c476d91f8c0ded72501b47f63c03010213900d043e8a5898def"} err="failed to get container status \"ef9e48d98ffb8c476d91f8c0ded72501b47f63c03010213900d043e8a5898def\": rpc error: code = NotFound desc = could not find container \"ef9e48d98ffb8c476d91f8c0ded72501b47f63c03010213900d043e8a5898def\": container with ID starting with ef9e48d98ffb8c476d91f8c0ded72501b47f63c03010213900d043e8a5898def not found: ID does not exist"
Oct 30 03:58:34.882897 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:58:34.882869    4461 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6c5444664c-cm9zb"]
Oct 30 03:58:34.894544 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:58:34.894512    4461 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6c5444664c-cm9zb"]
Oct 30 03:58:36.114993 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:58:36.114955    4461 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="43ff4be2-222a-44f2-8ceb-de4b4d97adaa" path="/var/lib/kubelet/pods/43ff4be2-222a-44f2-8ceb-de4b4d97adaa/volumes"
Oct 30 03:58:38.741719 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:58:38.741680    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:58:38.919616 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:58:38.919585    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:58:39.156061 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:58:39.155986    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:58:42.009773 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:58:42.009728    4461 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7f9654778bpjznv" podUID="d83b10c6-b612-447e-967d-f170f2a6170d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8001/health\": dial tcp 10.133.0.46:8001: connect: connection refused"
Oct 30 03:58:44.208686 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:58:44.208651    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:58:46.283256 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:58:46.283218    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:58:48.690157 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:58:48.690116    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:58:52.009828 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:58:52.009773    4461 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7f9654778bpjznv" podUID="d83b10c6-b612-447e-967d-f170f2a6170d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8001/health\": dial tcp 10.133.0.46:8001: connect: connection refused"
Oct 30 03:58:52.523220 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:58:52.523185    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:58:52.692976 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:58:52.692944    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:58:54.503283 ip-10-0-143-156 kubenswrapper[4461]: E1030 03:58:54.503244    4461 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2k85b" containerName="extractor"
Oct 30 03:59:02.009481 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:59:02.009436    4461 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7f9654778bpjznv" podUID="d83b10c6-b612-447e-967d-f170f2a6170d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8001/health\": dial tcp 10.133.0.46:8001: connect: connection refused"
Oct 30 03:59:12.009460 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:59:12.009412    4461 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7f9654778bpjznv" podUID="d83b10c6-b612-447e-967d-f170f2a6170d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8001/health\": dial tcp 10.133.0.46:8001: connect: connection refused"
Oct 30 03:59:18.368130 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:59:18.368031    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:59:22.009799 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:59:22.009755    4461 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7f9654778bpjznv" podUID="d83b10c6-b612-447e-967d-f170f2a6170d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8001/health\": dial tcp 10.133.0.46:8001: connect: connection refused"
Oct 30 03:59:24.506318 ip-10-0-143-156 kubenswrapper[4461]: E1030 03:59:24.506273    4461 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2k85b" containerName="extractor"
Oct 30 03:59:28.557640 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:59:28.557594    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:59:32.024631 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:59:32.024589    4461 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7f9654778bpjznv"
Oct 30 03:59:32.036758 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:59:32.036733    4461 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7f9654778bpjznv"
Oct 30 03:59:42.054131 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:59:42.054080    4461 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7f9654778bpjznv"]
Oct 30 03:59:42.061648 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:59:42.061618    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:59:42.269672 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:59:42.269631    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:59:48.812341 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:59:48.812304    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 03:59:54.503534 ip-10-0-143-156 kubenswrapper[4461]: E1030 03:59:54.503475    4461 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2k85b" containerName="extractor"
Oct 30 03:59:57.076755 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:59:57.076678    4461 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7f9654778bpjznv" podUID="d83b10c6-b612-447e-967d-f170f2a6170d" containerName="main" containerID="cri-o://45f4ae7dbafa6f3bb247e692028e7aba4494fc71273485e5b85794cbeebcf391" gracePeriod=45
Oct 30 03:59:57.693282 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:59:57.693230    4461 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7f9654778bpjznv" podUID="d83b10c6-b612-447e-967d-f170f2a6170d" containerName="llm-d-routing-sidecar" containerID="cri-o://69ea986a4c5b9fe5bd1ace8b5ea90ada6ba6eaba651ecd8e7cac21078e495fd0" gracePeriod=45
Oct 30 03:59:57.930568 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:59:57.930543    4461 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7f9654778bpjznv"
Oct 30 03:59:58.031715 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:59:58.031684    4461 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d83b10c6-b612-447e-967d-f170f2a6170d-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d83b10c6-b612-447e-967d-f170f2a6170d-tmp-dir\") pod \"d83b10c6-b612-447e-967d-f170f2a6170d\" (UID: \"d83b10c6-b612-447e-967d-f170f2a6170d\") "
Oct 30 03:59:58.031895 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:59:58.031756    4461 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d83b10c6-b612-447e-967d-f170f2a6170d-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d83b10c6-b612-447e-967d-f170f2a6170d-model-cache\") pod \"d83b10c6-b612-447e-967d-f170f2a6170d\" (UID: \"d83b10c6-b612-447e-967d-f170f2a6170d\") "
Oct 30 03:59:58.031895 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:59:58.031782    4461 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d83b10c6-b612-447e-967d-f170f2a6170d-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d83b10c6-b612-447e-967d-f170f2a6170d-kserve-provision-location\") pod \"d83b10c6-b612-447e-967d-f170f2a6170d\" (UID: \"d83b10c6-b612-447e-967d-f170f2a6170d\") "
Oct 30 03:59:58.031895 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:59:58.031823    4461 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d83b10c6-b612-447e-967d-f170f2a6170d-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d83b10c6-b612-447e-967d-f170f2a6170d-cabundle-cert\") pod \"d83b10c6-b612-447e-967d-f170f2a6170d\" (UID: \"d83b10c6-b612-447e-967d-f170f2a6170d\") "
Oct 30 03:59:58.031895 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:59:58.031849    4461 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d83b10c6-b612-447e-967d-f170f2a6170d-tls-certs\" (UniqueName: \"kubernetes.io/secret/d83b10c6-b612-447e-967d-f170f2a6170d-tls-certs\") pod \"d83b10c6-b612-447e-967d-f170f2a6170d\" (UID: \"d83b10c6-b612-447e-967d-f170f2a6170d\") "
Oct 30 03:59:58.031895 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:59:58.031885    4461 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d83b10c6-b612-447e-967d-f170f2a6170d-kube-api-access-gg8fl\" (UniqueName: \"kubernetes.io/projected/d83b10c6-b612-447e-967d-f170f2a6170d-kube-api-access-gg8fl\") pod \"d83b10c6-b612-447e-967d-f170f2a6170d\" (UID: \"d83b10c6-b612-447e-967d-f170f2a6170d\") "
Oct 30 03:59:58.032198 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:59:58.031924    4461 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d83b10c6-b612-447e-967d-f170f2a6170d-dshm\" (UniqueName: \"kubernetes.io/empty-dir/d83b10c6-b612-447e-967d-f170f2a6170d-dshm\") pod \"d83b10c6-b612-447e-967d-f170f2a6170d\" (UID: \"d83b10c6-b612-447e-967d-f170f2a6170d\") "
Oct 30 03:59:58.032198 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:59:58.031957    4461 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d83b10c6-b612-447e-967d-f170f2a6170d-home\" (UniqueName: \"kubernetes.io/empty-dir/d83b10c6-b612-447e-967d-f170f2a6170d-home\") pod \"d83b10c6-b612-447e-967d-f170f2a6170d\" (UID: \"d83b10c6-b612-447e-967d-f170f2a6170d\") "
Oct 30 03:59:58.032198 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:59:58.032070    4461 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d83b10c6-b612-447e-967d-f170f2a6170d-model-cache" pod "d83b10c6-b612-447e-967d-f170f2a6170d" (UID: "d83b10c6-b612-447e-967d-f170f2a6170d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:59:58.032351 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:59:58.032243    4461 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d83b10c6-b612-447e-967d-f170f2a6170d-model-cache\") on node \"ip-10-0-143-156.ec2.internal\" DevicePath \"\""
Oct 30 03:59:58.032422 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:59:58.032395    4461 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d83b10c6-b612-447e-967d-f170f2a6170d-cabundle-cert" pod "d83b10c6-b612-447e-967d-f170f2a6170d" (UID: "d83b10c6-b612-447e-967d-f170f2a6170d"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:59:58.032745 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:59:58.032719    4461 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d83b10c6-b612-447e-967d-f170f2a6170d-home" pod "d83b10c6-b612-447e-967d-f170f2a6170d" (UID: "d83b10c6-b612-447e-967d-f170f2a6170d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:59:58.034366 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:59:58.034338    4461 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d83b10c6-b612-447e-967d-f170f2a6170d-tls-certs" pod "d83b10c6-b612-447e-967d-f170f2a6170d" (UID: "d83b10c6-b612-447e-967d-f170f2a6170d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:59:58.034366 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:59:58.034342    4461 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d83b10c6-b612-447e-967d-f170f2a6170d-dshm" pod "d83b10c6-b612-447e-967d-f170f2a6170d" (UID: "d83b10c6-b612-447e-967d-f170f2a6170d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:59:58.034504 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:59:58.034461    4461 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d83b10c6-b612-447e-967d-f170f2a6170d-kube-api-access-gg8fl" pod "d83b10c6-b612-447e-967d-f170f2a6170d" (UID: "d83b10c6-b612-447e-967d-f170f2a6170d"). InnerVolumeSpecName "kube-api-access-gg8fl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:59:58.050153 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:59:58.050118    4461 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d83b10c6-b612-447e-967d-f170f2a6170d-tmp-dir" pod "d83b10c6-b612-447e-967d-f170f2a6170d" (UID: "d83b10c6-b612-447e-967d-f170f2a6170d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:59:58.061031 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:59:58.061006    4461 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d83b10c6-b612-447e-967d-f170f2a6170d-kserve-provision-location" pod "d83b10c6-b612-447e-967d-f170f2a6170d" (UID: "d83b10c6-b612-447e-967d-f170f2a6170d"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:59:58.132836 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:59:58.132813    4461 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d83b10c6-b612-447e-967d-f170f2a6170d-kserve-provision-location\") on node \"ip-10-0-143-156.ec2.internal\" DevicePath \"\""
Oct 30 03:59:58.132836 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:59:58.132837    4461 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d83b10c6-b612-447e-967d-f170f2a6170d-cabundle-cert\") on node \"ip-10-0-143-156.ec2.internal\" DevicePath \"\""
Oct 30 03:59:58.133179 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:59:58.132847    4461 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d83b10c6-b612-447e-967d-f170f2a6170d-tls-certs\") on node \"ip-10-0-143-156.ec2.internal\" DevicePath \"\""
Oct 30 03:59:58.133179 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:59:58.132855    4461 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gg8fl\" (UniqueName: \"kubernetes.io/projected/d83b10c6-b612-447e-967d-f170f2a6170d-kube-api-access-gg8fl\") on node \"ip-10-0-143-156.ec2.internal\" DevicePath \"\""
Oct 30 03:59:58.133179 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:59:58.132864    4461 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d83b10c6-b612-447e-967d-f170f2a6170d-dshm\") on node \"ip-10-0-143-156.ec2.internal\" DevicePath \"\""
Oct 30 03:59:58.133179 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:59:58.132872    4461 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d83b10c6-b612-447e-967d-f170f2a6170d-home\") on node \"ip-10-0-143-156.ec2.internal\" DevicePath \"\""
Oct 30 03:59:58.133179 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:59:58.132896    4461 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d83b10c6-b612-447e-967d-f170f2a6170d-tmp-dir\") on node \"ip-10-0-143-156.ec2.internal\" DevicePath \"\""
Oct 30 03:59:58.168784 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:59:58.168750    4461 generic.go:356] "Generic (PLEG): container finished" podID="d83b10c6-b612-447e-967d-f170f2a6170d" containerID="45f4ae7dbafa6f3bb247e692028e7aba4494fc71273485e5b85794cbeebcf391" exitCode=0
Oct 30 03:59:58.168784 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:59:58.168780    4461 generic.go:356] "Generic (PLEG): container finished" podID="d83b10c6-b612-447e-967d-f170f2a6170d" containerID="69ea986a4c5b9fe5bd1ace8b5ea90ada6ba6eaba651ecd8e7cac21078e495fd0" exitCode=0
Oct 30 03:59:58.168924 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:59:58.168815    4461 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7f9654778bpjznv"
Oct 30 03:59:58.168924 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:59:58.168820    4461 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7f9654778bpjznv" event={"ID":"d83b10c6-b612-447e-967d-f170f2a6170d","Type":"ContainerDied","Data":"45f4ae7dbafa6f3bb247e692028e7aba4494fc71273485e5b85794cbeebcf391"}
Oct 30 03:59:58.168924 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:59:58.168875    4461 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7f9654778bpjznv" event={"ID":"d83b10c6-b612-447e-967d-f170f2a6170d","Type":"ContainerDied","Data":"69ea986a4c5b9fe5bd1ace8b5ea90ada6ba6eaba651ecd8e7cac21078e495fd0"}
Oct 30 03:59:58.168924 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:59:58.168901    4461 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7f9654778bpjznv" event={"ID":"d83b10c6-b612-447e-967d-f170f2a6170d","Type":"ContainerDied","Data":"2057d4ef0b73aacdba9240e8523f81b82199d8313b67d1a4751f7b552935e877"}
Oct 30 03:59:58.169065 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:59:58.168930    4461 scope.go:122] "RemoveContainer" containerID="45f4ae7dbafa6f3bb247e692028e7aba4494fc71273485e5b85794cbeebcf391"
Oct 30 03:59:58.177087 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:59:58.177066    4461 scope.go:122] "RemoveContainer" containerID="07ca7664c49981c00e7714c5194187ba168e250d6a5f04c4a27128912c24eea9"
Oct 30 03:59:58.184143 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:59:58.184126    4461 scope.go:122] "RemoveContainer" containerID="69ea986a4c5b9fe5bd1ace8b5ea90ada6ba6eaba651ecd8e7cac21078e495fd0"
Oct 30 03:59:58.193067 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:59:58.193045    4461 scope.go:122] "RemoveContainer" containerID="45f4ae7dbafa6f3bb247e692028e7aba4494fc71273485e5b85794cbeebcf391"
Oct 30 03:59:58.193561 ip-10-0-143-156 kubenswrapper[4461]: E1030 03:59:58.193537    4461 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"45f4ae7dbafa6f3bb247e692028e7aba4494fc71273485e5b85794cbeebcf391\": container with ID starting with 45f4ae7dbafa6f3bb247e692028e7aba4494fc71273485e5b85794cbeebcf391 not found: ID does not exist" containerID="45f4ae7dbafa6f3bb247e692028e7aba4494fc71273485e5b85794cbeebcf391"
Oct 30 03:59:58.193616 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:59:58.193571    4461 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"45f4ae7dbafa6f3bb247e692028e7aba4494fc71273485e5b85794cbeebcf391"} err="failed to get container status \"45f4ae7dbafa6f3bb247e692028e7aba4494fc71273485e5b85794cbeebcf391\": rpc error: code = NotFound desc = could not find container \"45f4ae7dbafa6f3bb247e692028e7aba4494fc71273485e5b85794cbeebcf391\": container with ID starting with 45f4ae7dbafa6f3bb247e692028e7aba4494fc71273485e5b85794cbeebcf391 not found: ID does not exist"
Oct 30 03:59:58.193616 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:59:58.193589    4461 scope.go:122] "RemoveContainer" containerID="07ca7664c49981c00e7714c5194187ba168e250d6a5f04c4a27128912c24eea9"
Oct 30 03:59:58.193842 ip-10-0-143-156 kubenswrapper[4461]: E1030 03:59:58.193823    4461 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"07ca7664c49981c00e7714c5194187ba168e250d6a5f04c4a27128912c24eea9\": container with ID starting with 07ca7664c49981c00e7714c5194187ba168e250d6a5f04c4a27128912c24eea9 not found: ID does not exist" containerID="07ca7664c49981c00e7714c5194187ba168e250d6a5f04c4a27128912c24eea9"
Oct 30 03:59:58.193911 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:59:58.193853    4461 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"07ca7664c49981c00e7714c5194187ba168e250d6a5f04c4a27128912c24eea9"} err="failed to get container status \"07ca7664c49981c00e7714c5194187ba168e250d6a5f04c4a27128912c24eea9\": rpc error: code = NotFound desc = could not find container \"07ca7664c49981c00e7714c5194187ba168e250d6a5f04c4a27128912c24eea9\": container with ID starting with 07ca7664c49981c00e7714c5194187ba168e250d6a5f04c4a27128912c24eea9 not found: ID does not exist"
Oct 30 03:59:58.193911 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:59:58.193875    4461 scope.go:122] "RemoveContainer" containerID="69ea986a4c5b9fe5bd1ace8b5ea90ada6ba6eaba651ecd8e7cac21078e495fd0"
Oct 30 03:59:58.194126 ip-10-0-143-156 kubenswrapper[4461]: E1030 03:59:58.194090    4461 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"69ea986a4c5b9fe5bd1ace8b5ea90ada6ba6eaba651ecd8e7cac21078e495fd0\": container with ID starting with 69ea986a4c5b9fe5bd1ace8b5ea90ada6ba6eaba651ecd8e7cac21078e495fd0 not found: ID does not exist" containerID="69ea986a4c5b9fe5bd1ace8b5ea90ada6ba6eaba651ecd8e7cac21078e495fd0"
Oct 30 03:59:58.194200 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:59:58.194134    4461 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"69ea986a4c5b9fe5bd1ace8b5ea90ada6ba6eaba651ecd8e7cac21078e495fd0"} err="failed to get container status \"69ea986a4c5b9fe5bd1ace8b5ea90ada6ba6eaba651ecd8e7cac21078e495fd0\": rpc error: code = NotFound desc = could not find container \"69ea986a4c5b9fe5bd1ace8b5ea90ada6ba6eaba651ecd8e7cac21078e495fd0\": container with ID starting with 69ea986a4c5b9fe5bd1ace8b5ea90ada6ba6eaba651ecd8e7cac21078e495fd0 not found: ID does not exist"
Oct 30 03:59:58.194200 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:59:58.194153    4461 scope.go:122] "RemoveContainer" containerID="45f4ae7dbafa6f3bb247e692028e7aba4494fc71273485e5b85794cbeebcf391"
Oct 30 03:59:58.194414 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:59:58.194388    4461 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"45f4ae7dbafa6f3bb247e692028e7aba4494fc71273485e5b85794cbeebcf391"} err="failed to get container status \"45f4ae7dbafa6f3bb247e692028e7aba4494fc71273485e5b85794cbeebcf391\": rpc error: code = NotFound desc = could not find container \"45f4ae7dbafa6f3bb247e692028e7aba4494fc71273485e5b85794cbeebcf391\": container with ID starting with 45f4ae7dbafa6f3bb247e692028e7aba4494fc71273485e5b85794cbeebcf391 not found: ID does not exist"
Oct 30 03:59:58.194414 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:59:58.194414    4461 scope.go:122] "RemoveContainer" containerID="07ca7664c49981c00e7714c5194187ba168e250d6a5f04c4a27128912c24eea9"
Oct 30 03:59:58.194657 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:59:58.194638    4461 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"07ca7664c49981c00e7714c5194187ba168e250d6a5f04c4a27128912c24eea9"} err="failed to get container status \"07ca7664c49981c00e7714c5194187ba168e250d6a5f04c4a27128912c24eea9\": rpc error: code = NotFound desc = could not find container \"07ca7664c49981c00e7714c5194187ba168e250d6a5f04c4a27128912c24eea9\": container with ID starting with 07ca7664c49981c00e7714c5194187ba168e250d6a5f04c4a27128912c24eea9 not found: ID does not exist"
Oct 30 03:59:58.194707 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:59:58.194657    4461 scope.go:122] "RemoveContainer" containerID="69ea986a4c5b9fe5bd1ace8b5ea90ada6ba6eaba651ecd8e7cac21078e495fd0"
Oct 30 03:59:58.194899 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:59:58.194879    4461 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"69ea986a4c5b9fe5bd1ace8b5ea90ada6ba6eaba651ecd8e7cac21078e495fd0"} err="failed to get container status \"69ea986a4c5b9fe5bd1ace8b5ea90ada6ba6eaba651ecd8e7cac21078e495fd0\": rpc error: code = NotFound desc = could not find container \"69ea986a4c5b9fe5bd1ace8b5ea90ada6ba6eaba651ecd8e7cac21078e495fd0\": container with ID starting with 69ea986a4c5b9fe5bd1ace8b5ea90ada6ba6eaba651ecd8e7cac21078e495fd0 not found: ID does not exist"
Oct 30 03:59:58.223228 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:59:58.223206    4461 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7f9654778bpjznv"]
Oct 30 03:59:58.227135 ip-10-0-143-156 kubenswrapper[4461]: I1030 03:59:58.227114    4461 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7f9654778bpjznv"]
Oct 30 04:00:00.114501 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:00:00.114468    4461 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d83b10c6-b612-447e-967d-f170f2a6170d" path="/var/lib/kubelet/pods/d83b10c6-b612-447e-967d-f170f2a6170d/volumes"
Oct 30 04:00:18.493868 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:00:18.493832    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 04:00:24.503185 ip-10-0-143-156 kubenswrapper[4461]: E1030 04:00:24.503142    4461 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2k85b" containerName="extractor"
Oct 30 04:00:28.676481 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:00:28.676446    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 04:00:32.280328 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:00:32.280295    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 04:00:32.465240 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:00:32.465200    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 04:00:35.502562 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:00:35.502525    4461 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-b97f49766-tf8fm"]
Oct 30 04:00:35.502941 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:00:35.502811    4461 cpu_manager.go:414] "containerMap: removing container" podUID="43ff4be2-222a-44f2-8ceb-de4b4d97adaa" containerName="main"
Oct 30 04:00:35.502941 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:00:35.502822    4461 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="43ff4be2-222a-44f2-8ceb-de4b4d97adaa" containerName="main"
Oct 30 04:00:35.502941 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:00:35.502830    4461 cpu_manager.go:414] "containerMap: removing container" podUID="d83b10c6-b612-447e-967d-f170f2a6170d" containerName="main"
Oct 30 04:00:35.502941 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:00:35.502835    4461 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d83b10c6-b612-447e-967d-f170f2a6170d" containerName="main"
Oct 30 04:00:35.502941 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:00:35.502851    4461 cpu_manager.go:414] "containerMap: removing container" podUID="d83b10c6-b612-447e-967d-f170f2a6170d" containerName="llm-d-routing-sidecar"
Oct 30 04:00:35.502941 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:00:35.502856    4461 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d83b10c6-b612-447e-967d-f170f2a6170d" containerName="llm-d-routing-sidecar"
Oct 30 04:00:35.502941 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:00:35.502865    4461 cpu_manager.go:414] "containerMap: removing container" podUID="d83b10c6-b612-447e-967d-f170f2a6170d" containerName="storage-initializer"
Oct 30 04:00:35.502941 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:00:35.502870    4461 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d83b10c6-b612-447e-967d-f170f2a6170d" containerName="storage-initializer"
Oct 30 04:00:35.502941 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:00:35.502923    4461 memory_manager.go:365] "RemoveStaleState removing state" podUID="43ff4be2-222a-44f2-8ceb-de4b4d97adaa" containerName="main"
Oct 30 04:00:35.502941 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:00:35.502932    4461 memory_manager.go:365] "RemoveStaleState removing state" podUID="d83b10c6-b612-447e-967d-f170f2a6170d" containerName="main"
Oct 30 04:00:35.502941 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:00:35.502940    4461 memory_manager.go:365] "RemoveStaleState removing state" podUID="d83b10c6-b612-447e-967d-f170f2a6170d" containerName="llm-d-routing-sidecar"
Oct 30 04:00:35.507008 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:00:35.506992    4461 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-b97f49766-tf8fm"
Oct 30 04:00:35.524850 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:00:35.524830    4461 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"custom-route-timeout-pd-test-kserve-self-signed-certs\""
Oct 30 04:00:35.527623 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:00:35.527590    4461 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"seaweedfs-s3-creds\""
Oct 30 04:00:35.527779 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:00:35.527758    4461 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"default-dockercfg-b4mk4\""
Oct 30 04:00:35.528001 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:00:35.527982    4461 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"openshift-service-ca.crt\""
Oct 30 04:00:35.537649 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:00:35.537625    4461 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"odh-kserve-custom-ca-bundle\""
Oct 30 04:00:35.537649 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:00:35.537629    4461 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"kube-root-ca.crt\""
Oct 30 04:00:35.541856 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:00:35.541833    4461 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-b97f49766-tf8fm"]
Oct 30 04:00:35.565505 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:00:35.565460    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 04:00:35.606228 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:00:35.606197    4461 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/76ef7227-645f-45df-9d1c-40b4b53614f9-model-cache\") pod \"custom-route-timeout-pd-test-kserve-b97f49766-tf8fm\" (UID: \"76ef7227-645f-45df-9d1c-40b4b53614f9\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-b97f49766-tf8fm"
Oct 30 04:00:35.606228 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:00:35.606232    4461 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9tfhd\" (UniqueName: \"kubernetes.io/projected/76ef7227-645f-45df-9d1c-40b4b53614f9-kube-api-access-9tfhd\") pod \"custom-route-timeout-pd-test-kserve-b97f49766-tf8fm\" (UID: \"76ef7227-645f-45df-9d1c-40b4b53614f9\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-b97f49766-tf8fm"
Oct 30 04:00:35.606463 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:00:35.606253    4461 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/76ef7227-645f-45df-9d1c-40b4b53614f9-home\") pod \"custom-route-timeout-pd-test-kserve-b97f49766-tf8fm\" (UID: \"76ef7227-645f-45df-9d1c-40b4b53614f9\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-b97f49766-tf8fm"
Oct 30 04:00:35.606463 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:00:35.606377    4461 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/76ef7227-645f-45df-9d1c-40b4b53614f9-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-b97f49766-tf8fm\" (UID: \"76ef7227-645f-45df-9d1c-40b4b53614f9\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-b97f49766-tf8fm"
Oct 30 04:00:35.606463 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:00:35.606437    4461 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/76ef7227-645f-45df-9d1c-40b4b53614f9-tmp-dir\") pod \"custom-route-timeout-pd-test-kserve-b97f49766-tf8fm\" (UID: \"76ef7227-645f-45df-9d1c-40b4b53614f9\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-b97f49766-tf8fm"
Oct 30 04:00:35.606463 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:00:35.606453    4461 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/76ef7227-645f-45df-9d1c-40b4b53614f9-dshm\") pod \"custom-route-timeout-pd-test-kserve-b97f49766-tf8fm\" (UID: \"76ef7227-645f-45df-9d1c-40b4b53614f9\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-b97f49766-tf8fm"
Oct 30 04:00:35.606654 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:00:35.606473    4461 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/76ef7227-645f-45df-9d1c-40b4b53614f9-cabundle-cert\") pod \"custom-route-timeout-pd-test-kserve-b97f49766-tf8fm\" (UID: \"76ef7227-645f-45df-9d1c-40b4b53614f9\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-b97f49766-tf8fm"
Oct 30 04:00:35.606654 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:00:35.606514    4461 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/76ef7227-645f-45df-9d1c-40b4b53614f9-kserve-provision-location\") pod \"custom-route-timeout-pd-test-kserve-b97f49766-tf8fm\" (UID: \"76ef7227-645f-45df-9d1c-40b4b53614f9\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-b97f49766-tf8fm"
Oct 30 04:00:35.612184 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:00:35.612161    4461 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"custom-route-timeout-pd-test-kserve-dockercfg-z29bq\""
Oct 30 04:00:35.707282 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:00:35.707244    4461 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/76ef7227-645f-45df-9d1c-40b4b53614f9-tmp-dir\") pod \"custom-route-timeout-pd-test-kserve-b97f49766-tf8fm\" (UID: \"76ef7227-645f-45df-9d1c-40b4b53614f9\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-b97f49766-tf8fm"
Oct 30 04:00:35.707282 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:00:35.707282    4461 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/76ef7227-645f-45df-9d1c-40b4b53614f9-dshm\") pod \"custom-route-timeout-pd-test-kserve-b97f49766-tf8fm\" (UID: \"76ef7227-645f-45df-9d1c-40b4b53614f9\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-b97f49766-tf8fm"
Oct 30 04:00:35.707545 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:00:35.707309    4461 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/76ef7227-645f-45df-9d1c-40b4b53614f9-cabundle-cert\") pod \"custom-route-timeout-pd-test-kserve-b97f49766-tf8fm\" (UID: \"76ef7227-645f-45df-9d1c-40b4b53614f9\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-b97f49766-tf8fm"
Oct 30 04:00:35.707545 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:00:35.707365    4461 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/76ef7227-645f-45df-9d1c-40b4b53614f9-kserve-provision-location\") pod \"custom-route-timeout-pd-test-kserve-b97f49766-tf8fm\" (UID: \"76ef7227-645f-45df-9d1c-40b4b53614f9\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-b97f49766-tf8fm"
Oct 30 04:00:35.707545 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:00:35.707390    4461 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/76ef7227-645f-45df-9d1c-40b4b53614f9-model-cache\") pod \"custom-route-timeout-pd-test-kserve-b97f49766-tf8fm\" (UID: \"76ef7227-645f-45df-9d1c-40b4b53614f9\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-b97f49766-tf8fm"
Oct 30 04:00:35.707545 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:00:35.707410    4461 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9tfhd\" (UniqueName: \"kubernetes.io/projected/76ef7227-645f-45df-9d1c-40b4b53614f9-kube-api-access-9tfhd\") pod \"custom-route-timeout-pd-test-kserve-b97f49766-tf8fm\" (UID: \"76ef7227-645f-45df-9d1c-40b4b53614f9\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-b97f49766-tf8fm"
Oct 30 04:00:35.707751 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:00:35.707537    4461 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/76ef7227-645f-45df-9d1c-40b4b53614f9-home\") pod \"custom-route-timeout-pd-test-kserve-b97f49766-tf8fm\" (UID: \"76ef7227-645f-45df-9d1c-40b4b53614f9\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-b97f49766-tf8fm"
Oct 30 04:00:35.707751 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:00:35.707641    4461 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/76ef7227-645f-45df-9d1c-40b4b53614f9-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-b97f49766-tf8fm\" (UID: \"76ef7227-645f-45df-9d1c-40b4b53614f9\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-b97f49766-tf8fm"
Oct 30 04:00:35.707751 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:00:35.707662    4461 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/76ef7227-645f-45df-9d1c-40b4b53614f9-tmp-dir\") pod \"custom-route-timeout-pd-test-kserve-b97f49766-tf8fm\" (UID: \"76ef7227-645f-45df-9d1c-40b4b53614f9\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-b97f49766-tf8fm"
Oct 30 04:00:35.707859 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:00:35.707845    4461 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/76ef7227-645f-45df-9d1c-40b4b53614f9-kserve-provision-location\") pod \"custom-route-timeout-pd-test-kserve-b97f49766-tf8fm\" (UID: \"76ef7227-645f-45df-9d1c-40b4b53614f9\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-b97f49766-tf8fm"
Oct 30 04:00:35.707937 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:00:35.707913    4461 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/76ef7227-645f-45df-9d1c-40b4b53614f9-model-cache\") pod \"custom-route-timeout-pd-test-kserve-b97f49766-tf8fm\" (UID: \"76ef7227-645f-45df-9d1c-40b4b53614f9\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-b97f49766-tf8fm"
Oct 30 04:00:35.707986 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:00:35.707949    4461 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/76ef7227-645f-45df-9d1c-40b4b53614f9-home\") pod \"custom-route-timeout-pd-test-kserve-b97f49766-tf8fm\" (UID: \"76ef7227-645f-45df-9d1c-40b4b53614f9\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-b97f49766-tf8fm"
Oct 30 04:00:35.708088 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:00:35.708065    4461 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/76ef7227-645f-45df-9d1c-40b4b53614f9-cabundle-cert\") pod \"custom-route-timeout-pd-test-kserve-b97f49766-tf8fm\" (UID: \"76ef7227-645f-45df-9d1c-40b4b53614f9\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-b97f49766-tf8fm"
Oct 30 04:00:35.709644 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:00:35.709619    4461 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/76ef7227-645f-45df-9d1c-40b4b53614f9-dshm\") pod \"custom-route-timeout-pd-test-kserve-b97f49766-tf8fm\" (UID: \"76ef7227-645f-45df-9d1c-40b4b53614f9\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-b97f49766-tf8fm"
Oct 30 04:00:35.710058 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:00:35.710038    4461 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/76ef7227-645f-45df-9d1c-40b4b53614f9-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-b97f49766-tf8fm\" (UID: \"76ef7227-645f-45df-9d1c-40b4b53614f9\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-b97f49766-tf8fm"
Oct 30 04:00:35.731204 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:00:35.731179    4461 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9tfhd\" (UniqueName: \"kubernetes.io/projected/76ef7227-645f-45df-9d1c-40b4b53614f9-kube-api-access-9tfhd\") pod \"custom-route-timeout-pd-test-kserve-b97f49766-tf8fm\" (UID: \"76ef7227-645f-45df-9d1c-40b4b53614f9\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-b97f49766-tf8fm"
Oct 30 04:00:35.755858 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:00:35.755787    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 04:00:35.823031 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:00:35.822995    4461 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-b97f49766-tf8fm"
Oct 30 04:00:36.033387 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:00:36.033305    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 04:00:36.156437 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:00:36.156407    4461 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-b97f49766-tf8fm"]
Oct 30 04:00:36.157457 ip-10-0-143-156 kubenswrapper[4461]: W1030 04:00:36.157431    4461 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod76ef7227_645f_45df_9d1c_40b4b53614f9.slice/crio-6295a85c5ec08080f5caf4c4be6aa053d9657d1a255b901a75d154a4faaf41ca WatchSource:0}: Error finding container 6295a85c5ec08080f5caf4c4be6aa053d9657d1a255b901a75d154a4faaf41ca: Status 404 returned error can't find the container with id 6295a85c5ec08080f5caf4c4be6aa053d9657d1a255b901a75d154a4faaf41ca
Oct 30 04:00:36.314140 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:00:36.314032    4461 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-b97f49766-tf8fm" event={"ID":"76ef7227-645f-45df-9d1c-40b4b53614f9","Type":"ContainerStarted","Data":"3a8efadc307f19d3cd06965032572e7d5e53a01caa71f5673729bd6418b85536"}
Oct 30 04:00:36.314140 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:00:36.314108    4461 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-b97f49766-tf8fm" event={"ID":"76ef7227-645f-45df-9d1c-40b4b53614f9","Type":"ContainerStarted","Data":"6295a85c5ec08080f5caf4c4be6aa053d9657d1a255b901a75d154a4faaf41ca"}
Oct 30 04:00:36.314374 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:00:36.314192    4461 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-b97f49766-tf8fm"
Oct 30 04:00:37.320144 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:00:37.320112    4461 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-b97f49766-tf8fm" event={"ID":"76ef7227-645f-45df-9d1c-40b4b53614f9","Type":"ContainerStarted","Data":"2e9b45fb90b04de27ea318cb92cc0a6de676cb225b68d20395002514793e113b"}
Oct 30 04:00:38.807745 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:00:38.807667    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 04:00:39.330330 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:00:39.330296    4461 generic.go:356] "Generic (PLEG): container finished" podID="76ef7227-645f-45df-9d1c-40b4b53614f9" containerID="2e9b45fb90b04de27ea318cb92cc0a6de676cb225b68d20395002514793e113b" exitCode=0
Oct 30 04:00:39.330533 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:00:39.330336    4461 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-b97f49766-tf8fm" event={"ID":"76ef7227-645f-45df-9d1c-40b4b53614f9","Type":"ContainerDied","Data":"2e9b45fb90b04de27ea318cb92cc0a6de676cb225b68d20395002514793e113b"}
Oct 30 04:00:39.648585 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:00:39.648490    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 04:00:40.337249 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:00:40.337212    4461 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-b97f49766-tf8fm" event={"ID":"76ef7227-645f-45df-9d1c-40b4b53614f9","Type":"ContainerStarted","Data":"55dd479290b849b28a07a2857058b905cf1ec6fe8251834850ad792133f9ba49"}
Oct 30 04:00:40.413331 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:00:40.413275    4461 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-b97f49766-tf8fm" podStartSLOduration=3.265713566 podStartE2EDuration="5.413258962s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.147545396s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:00:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:00:40.399675641 +0000 UTC m=+3908.449480817" watchObservedRunningTime="2026-10-30 04:00:40.413258962 +0000 UTC m=+3908.463064138"
Oct 30 04:00:45.823931 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:00:45.823843    4461 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-b97f49766-tf8fm"
Oct 30 04:00:45.823931 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:00:45.823910    4461 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-b97f49766-tf8fm"
Oct 30 04:00:45.825157 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:00:45.825123    4461 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-b97f49766-tf8fm" podUID="76ef7227-645f-45df-9d1c-40b4b53614f9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8001/health\": dial tcp 10.133.0.47:8001: connect: connection refused"
Oct 30 04:00:45.859690 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:00:45.859661    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 04:00:46.026658 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:00:46.026629    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 04:00:48.936426 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:00:48.936392    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 04:00:54.504784 ip-10-0-143-156 kubenswrapper[4461]: E1030 04:00:54.504744    4461 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2k85b" containerName="extractor"
Oct 30 04:00:55.823646 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:00:55.823601    4461 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-b97f49766-tf8fm" podUID="76ef7227-645f-45df-9d1c-40b4b53614f9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8001/health\": dial tcp 10.133.0.47:8001: connect: connection refused"
Oct 30 04:00:55.836389 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:00:55.836361    4461 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-b97f49766-tf8fm"
Oct 30 04:00:55.858424 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:00:55.858392    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 04:00:56.076463 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:00:56.076369    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 04:00:56.261343 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:00:56.261306    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 04:00:58.223650 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:00:58.223607    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 04:01:02.454408 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:01:02.451937    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 04:01:02.659237 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:01:02.659196    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 04:01:05.824352 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:01:05.824304    4461 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-b97f49766-tf8fm" podUID="76ef7227-645f-45df-9d1c-40b4b53614f9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8001/health\": dial tcp 10.133.0.47:8001: connect: connection refused"
Oct 30 04:01:15.824082 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:01:15.824028    4461 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-b97f49766-tf8fm" podUID="76ef7227-645f-45df-9d1c-40b4b53614f9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8001/health\": dial tcp 10.133.0.47:8001: connect: connection refused"
Oct 30 04:01:18.632869 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:01:18.632828    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 04:01:24.504447 ip-10-0-143-156 kubenswrapper[4461]: E1030 04:01:24.504396    4461 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2k85b" containerName="extractor"
Oct 30 04:01:25.823809 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:01:25.823761    4461 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-b97f49766-tf8fm" podUID="76ef7227-645f-45df-9d1c-40b4b53614f9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8001/health\": dial tcp 10.133.0.47:8001: connect: connection refused"
Oct 30 04:01:26.916633 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:01:26.916602    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 04:01:28.835889 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:01:28.835848    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 04:01:35.824118 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:01:35.824045    4461 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-b97f49766-tf8fm" podUID="76ef7227-645f-45df-9d1c-40b4b53614f9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8001/health\": dial tcp 10.133.0.47:8001: connect: connection refused"
Oct 30 04:01:45.823324 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:01:45.823284    4461 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-b97f49766-tf8fm" podUID="76ef7227-645f-45df-9d1c-40b4b53614f9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8001/health\": dial tcp 10.133.0.47:8001: connect: connection refused"
Oct 30 04:01:49.092024 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:01:49.091980    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 04:01:54.505702 ip-10-0-143-156 kubenswrapper[4461]: E1030 04:01:54.505651    4461 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2k85b" containerName="extractor"
Oct 30 04:01:55.823919 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:01:55.823863    4461 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-b97f49766-tf8fm" podUID="76ef7227-645f-45df-9d1c-40b4b53614f9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8001/health\": dial tcp 10.133.0.47:8001: connect: connection refused"
Oct 30 04:02:05.824229 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:02:05.824170    4461 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-b97f49766-tf8fm" podUID="76ef7227-645f-45df-9d1c-40b4b53614f9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8001/health\": dial tcp 10.133.0.47:8001: connect: connection refused"
Oct 30 04:02:15.823763 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:02:15.823649    4461 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-b97f49766-tf8fm" podUID="76ef7227-645f-45df-9d1c-40b4b53614f9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8001/health\": dial tcp 10.133.0.47:8001: connect: connection refused"
Oct 30 04:02:18.729007 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:02:18.728964    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 04:02:24.504770 ip-10-0-143-156 kubenswrapper[4461]: E1030 04:02:24.504723    4461 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2k85b" containerName="extractor"
Oct 30 04:02:25.823693 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:02:25.823645    4461 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-b97f49766-tf8fm" podUID="76ef7227-645f-45df-9d1c-40b4b53614f9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8001/health\": dial tcp 10.133.0.47:8001: connect: connection refused"
Oct 30 04:02:28.911404 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:02:28.911368    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 04:02:35.839067 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:02:35.839036    4461 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-b97f49766-tf8fm"
Oct 30 04:02:35.850679 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:02:35.850655    4461 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-b97f49766-tf8fm"
Oct 30 04:02:49.190383 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:02:49.190337    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 04:02:54.515521 ip-10-0-143-156 kubenswrapper[4461]: E1030 04:02:54.515480    4461 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2k85b" containerName="extractor"
Oct 30 04:02:58.696954 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:02:58.696914    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 04:02:58.707967 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:02:58.707939    4461 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-b97f49766-tf8fm"]
Oct 30 04:02:58.871649 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:02:58.871618    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 04:03:10.340136 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:03:10.340085    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 04:03:13.731961 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:03:13.731918    4461 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-b97f49766-tf8fm" podUID="76ef7227-645f-45df-9d1c-40b4b53614f9" containerName="main" containerID="cri-o://55dd479290b849b28a07a2857058b905cf1ec6fe8251834850ad792133f9ba49" gracePeriod=45
Oct 30 04:03:14.341943 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:03:14.341877    4461 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-b97f49766-tf8fm" podUID="76ef7227-645f-45df-9d1c-40b4b53614f9" containerName="llm-d-routing-sidecar" containerID="cri-o://3a8efadc307f19d3cd06965032572e7d5e53a01caa71f5673729bd6418b85536" gracePeriod=45
Oct 30 04:03:14.575866 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:03:14.575842    4461 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-b97f49766-tf8fm"
Oct 30 04:03:14.732992 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:03:14.732961    4461 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/76ef7227-645f-45df-9d1c-40b4b53614f9-kube-api-access-9tfhd\" (UniqueName: \"kubernetes.io/projected/76ef7227-645f-45df-9d1c-40b4b53614f9-kube-api-access-9tfhd\") pod \"76ef7227-645f-45df-9d1c-40b4b53614f9\" (UID: \"76ef7227-645f-45df-9d1c-40b4b53614f9\") "
Oct 30 04:03:14.732992 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:03:14.732998    4461 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/76ef7227-645f-45df-9d1c-40b4b53614f9-dshm\" (UniqueName: \"kubernetes.io/empty-dir/76ef7227-645f-45df-9d1c-40b4b53614f9-dshm\") pod \"76ef7227-645f-45df-9d1c-40b4b53614f9\" (UID: \"76ef7227-645f-45df-9d1c-40b4b53614f9\") "
Oct 30 04:03:14.733576 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:03:14.733043    4461 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/76ef7227-645f-45df-9d1c-40b4b53614f9-tls-certs\" (UniqueName: \"kubernetes.io/secret/76ef7227-645f-45df-9d1c-40b4b53614f9-tls-certs\") pod \"76ef7227-645f-45df-9d1c-40b4b53614f9\" (UID: \"76ef7227-645f-45df-9d1c-40b4b53614f9\") "
Oct 30 04:03:14.733576 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:03:14.733215    4461 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/76ef7227-645f-45df-9d1c-40b4b53614f9-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/76ef7227-645f-45df-9d1c-40b4b53614f9-cabundle-cert\") pod \"76ef7227-645f-45df-9d1c-40b4b53614f9\" (UID: \"76ef7227-645f-45df-9d1c-40b4b53614f9\") "
Oct 30 04:03:14.733576 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:03:14.733264    4461 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/76ef7227-645f-45df-9d1c-40b4b53614f9-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/76ef7227-645f-45df-9d1c-40b4b53614f9-model-cache\") pod \"76ef7227-645f-45df-9d1c-40b4b53614f9\" (UID: \"76ef7227-645f-45df-9d1c-40b4b53614f9\") "
Oct 30 04:03:14.733576 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:03:14.733303    4461 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/76ef7227-645f-45df-9d1c-40b4b53614f9-home\" (UniqueName: \"kubernetes.io/empty-dir/76ef7227-645f-45df-9d1c-40b4b53614f9-home\") pod \"76ef7227-645f-45df-9d1c-40b4b53614f9\" (UID: \"76ef7227-645f-45df-9d1c-40b4b53614f9\") "
Oct 30 04:03:14.733576 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:03:14.733334    4461 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/76ef7227-645f-45df-9d1c-40b4b53614f9-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/76ef7227-645f-45df-9d1c-40b4b53614f9-tmp-dir\") pod \"76ef7227-645f-45df-9d1c-40b4b53614f9\" (UID: \"76ef7227-645f-45df-9d1c-40b4b53614f9\") "
Oct 30 04:03:14.733576 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:03:14.733363    4461 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/76ef7227-645f-45df-9d1c-40b4b53614f9-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/76ef7227-645f-45df-9d1c-40b4b53614f9-kserve-provision-location\") pod \"76ef7227-645f-45df-9d1c-40b4b53614f9\" (UID: \"76ef7227-645f-45df-9d1c-40b4b53614f9\") "
Oct 30 04:03:14.733887 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:03:14.733616    4461 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/76ef7227-645f-45df-9d1c-40b4b53614f9-cabundle-cert" pod "76ef7227-645f-45df-9d1c-40b4b53614f9" (UID: "76ef7227-645f-45df-9d1c-40b4b53614f9"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:03:14.733955 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:03:14.733924    4461 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76ef7227-645f-45df-9d1c-40b4b53614f9-model-cache" pod "76ef7227-645f-45df-9d1c-40b4b53614f9" (UID: "76ef7227-645f-45df-9d1c-40b4b53614f9"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:03:14.734016 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:03:14.733988    4461 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76ef7227-645f-45df-9d1c-40b4b53614f9-home" pod "76ef7227-645f-45df-9d1c-40b4b53614f9" (UID: "76ef7227-645f-45df-9d1c-40b4b53614f9"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:03:14.735417 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:03:14.735388    4461 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76ef7227-645f-45df-9d1c-40b4b53614f9-dshm" pod "76ef7227-645f-45df-9d1c-40b4b53614f9" (UID: "76ef7227-645f-45df-9d1c-40b4b53614f9"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:03:14.735645 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:03:14.735610    4461 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/76ef7227-645f-45df-9d1c-40b4b53614f9-tls-certs" pod "76ef7227-645f-45df-9d1c-40b4b53614f9" (UID: "76ef7227-645f-45df-9d1c-40b4b53614f9"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:03:14.735753 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:03:14.735650    4461 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/76ef7227-645f-45df-9d1c-40b4b53614f9-kube-api-access-9tfhd" pod "76ef7227-645f-45df-9d1c-40b4b53614f9" (UID: "76ef7227-645f-45df-9d1c-40b4b53614f9"). InnerVolumeSpecName "kube-api-access-9tfhd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:03:14.746342 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:03:14.746307    4461 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76ef7227-645f-45df-9d1c-40b4b53614f9-tmp-dir" pod "76ef7227-645f-45df-9d1c-40b4b53614f9" (UID: "76ef7227-645f-45df-9d1c-40b4b53614f9"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:03:14.757795 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:03:14.757738    4461 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76ef7227-645f-45df-9d1c-40b4b53614f9-kserve-provision-location" pod "76ef7227-645f-45df-9d1c-40b4b53614f9" (UID: "76ef7227-645f-45df-9d1c-40b4b53614f9"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:03:14.834330 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:03:14.834303    4461 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/76ef7227-645f-45df-9d1c-40b4b53614f9-cabundle-cert\") on node \"ip-10-0-143-156.ec2.internal\" DevicePath \"\""
Oct 30 04:03:14.834330 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:03:14.834326    4461 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/76ef7227-645f-45df-9d1c-40b4b53614f9-model-cache\") on node \"ip-10-0-143-156.ec2.internal\" DevicePath \"\""
Oct 30 04:03:14.834330 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:03:14.834335    4461 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/76ef7227-645f-45df-9d1c-40b4b53614f9-home\") on node \"ip-10-0-143-156.ec2.internal\" DevicePath \"\""
Oct 30 04:03:14.834496 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:03:14.834343    4461 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/76ef7227-645f-45df-9d1c-40b4b53614f9-tmp-dir\") on node \"ip-10-0-143-156.ec2.internal\" DevicePath \"\""
Oct 30 04:03:14.834496 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:03:14.834352    4461 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/76ef7227-645f-45df-9d1c-40b4b53614f9-kserve-provision-location\") on node \"ip-10-0-143-156.ec2.internal\" DevicePath \"\""
Oct 30 04:03:14.834496 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:03:14.834360    4461 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9tfhd\" (UniqueName: \"kubernetes.io/projected/76ef7227-645f-45df-9d1c-40b4b53614f9-kube-api-access-9tfhd\") on node \"ip-10-0-143-156.ec2.internal\" DevicePath \"\""
Oct 30 04:03:14.834496 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:03:14.834368    4461 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/76ef7227-645f-45df-9d1c-40b4b53614f9-dshm\") on node \"ip-10-0-143-156.ec2.internal\" DevicePath \"\""
Oct 30 04:03:14.834496 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:03:14.834377    4461 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/76ef7227-645f-45df-9d1c-40b4b53614f9-tls-certs\") on node \"ip-10-0-143-156.ec2.internal\" DevicePath \"\""
Oct 30 04:03:14.978223 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:03:14.978189    4461 generic.go:356] "Generic (PLEG): container finished" podID="76ef7227-645f-45df-9d1c-40b4b53614f9" containerID="55dd479290b849b28a07a2857058b905cf1ec6fe8251834850ad792133f9ba49" exitCode=0
Oct 30 04:03:14.978223 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:03:14.978220    4461 generic.go:356] "Generic (PLEG): container finished" podID="76ef7227-645f-45df-9d1c-40b4b53614f9" containerID="3a8efadc307f19d3cd06965032572e7d5e53a01caa71f5673729bd6418b85536" exitCode=0
Oct 30 04:03:14.978420 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:03:14.978266    4461 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-b97f49766-tf8fm"
Oct 30 04:03:14.978420 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:03:14.978274    4461 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-b97f49766-tf8fm" event={"ID":"76ef7227-645f-45df-9d1c-40b4b53614f9","Type":"ContainerDied","Data":"55dd479290b849b28a07a2857058b905cf1ec6fe8251834850ad792133f9ba49"}
Oct 30 04:03:14.978420 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:03:14.978377    4461 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-b97f49766-tf8fm" event={"ID":"76ef7227-645f-45df-9d1c-40b4b53614f9","Type":"ContainerDied","Data":"3a8efadc307f19d3cd06965032572e7d5e53a01caa71f5673729bd6418b85536"}
Oct 30 04:03:14.978420 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:03:14.978407    4461 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-b97f49766-tf8fm" event={"ID":"76ef7227-645f-45df-9d1c-40b4b53614f9","Type":"ContainerDied","Data":"6295a85c5ec08080f5caf4c4be6aa053d9657d1a255b901a75d154a4faaf41ca"}
Oct 30 04:03:14.978597 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:03:14.978436    4461 scope.go:122] "RemoveContainer" containerID="55dd479290b849b28a07a2857058b905cf1ec6fe8251834850ad792133f9ba49"
Oct 30 04:03:14.987315 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:03:14.987298    4461 scope.go:122] "RemoveContainer" containerID="2e9b45fb90b04de27ea318cb92cc0a6de676cb225b68d20395002514793e113b"
Oct 30 04:03:14.994487 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:03:14.994471    4461 scope.go:122] "RemoveContainer" containerID="3a8efadc307f19d3cd06965032572e7d5e53a01caa71f5673729bd6418b85536"
Oct 30 04:03:15.001746 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:03:15.001723    4461 scope.go:122] "RemoveContainer" containerID="55dd479290b849b28a07a2857058b905cf1ec6fe8251834850ad792133f9ba49"
Oct 30 04:03:15.002002 ip-10-0-143-156 kubenswrapper[4461]: E1030 04:03:15.001984    4461 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"55dd479290b849b28a07a2857058b905cf1ec6fe8251834850ad792133f9ba49\": container with ID starting with 55dd479290b849b28a07a2857058b905cf1ec6fe8251834850ad792133f9ba49 not found: ID does not exist" containerID="55dd479290b849b28a07a2857058b905cf1ec6fe8251834850ad792133f9ba49"
Oct 30 04:03:15.002053 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:03:15.002011    4461 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"55dd479290b849b28a07a2857058b905cf1ec6fe8251834850ad792133f9ba49"} err="failed to get container status \"55dd479290b849b28a07a2857058b905cf1ec6fe8251834850ad792133f9ba49\": rpc error: code = NotFound desc = could not find container \"55dd479290b849b28a07a2857058b905cf1ec6fe8251834850ad792133f9ba49\": container with ID starting with 55dd479290b849b28a07a2857058b905cf1ec6fe8251834850ad792133f9ba49 not found: ID does not exist"
Oct 30 04:03:15.002053 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:03:15.002029    4461 scope.go:122] "RemoveContainer" containerID="2e9b45fb90b04de27ea318cb92cc0a6de676cb225b68d20395002514793e113b"
Oct 30 04:03:15.002276 ip-10-0-143-156 kubenswrapper[4461]: E1030 04:03:15.002256    4461 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2e9b45fb90b04de27ea318cb92cc0a6de676cb225b68d20395002514793e113b\": container with ID starting with 2e9b45fb90b04de27ea318cb92cc0a6de676cb225b68d20395002514793e113b not found: ID does not exist" containerID="2e9b45fb90b04de27ea318cb92cc0a6de676cb225b68d20395002514793e113b"
Oct 30 04:03:15.002354 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:03:15.002284    4461 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2e9b45fb90b04de27ea318cb92cc0a6de676cb225b68d20395002514793e113b"} err="failed to get container status \"2e9b45fb90b04de27ea318cb92cc0a6de676cb225b68d20395002514793e113b\": rpc error: code = NotFound desc = could not find container \"2e9b45fb90b04de27ea318cb92cc0a6de676cb225b68d20395002514793e113b\": container with ID starting with 2e9b45fb90b04de27ea318cb92cc0a6de676cb225b68d20395002514793e113b not found: ID does not exist"
Oct 30 04:03:15.002354 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:03:15.002307    4461 scope.go:122] "RemoveContainer" containerID="3a8efadc307f19d3cd06965032572e7d5e53a01caa71f5673729bd6418b85536"
Oct 30 04:03:15.002583 ip-10-0-143-156 kubenswrapper[4461]: E1030 04:03:15.002565    4461 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3a8efadc307f19d3cd06965032572e7d5e53a01caa71f5673729bd6418b85536\": container with ID starting with 3a8efadc307f19d3cd06965032572e7d5e53a01caa71f5673729bd6418b85536 not found: ID does not exist" containerID="3a8efadc307f19d3cd06965032572e7d5e53a01caa71f5673729bd6418b85536"
Oct 30 04:03:15.002627 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:03:15.002589    4461 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3a8efadc307f19d3cd06965032572e7d5e53a01caa71f5673729bd6418b85536"} err="failed to get container status \"3a8efadc307f19d3cd06965032572e7d5e53a01caa71f5673729bd6418b85536\": rpc error: code = NotFound desc = could not find container \"3a8efadc307f19d3cd06965032572e7d5e53a01caa71f5673729bd6418b85536\": container with ID starting with 3a8efadc307f19d3cd06965032572e7d5e53a01caa71f5673729bd6418b85536 not found: ID does not exist"
Oct 30 04:03:15.002627 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:03:15.002604    4461 scope.go:122] "RemoveContainer" containerID="55dd479290b849b28a07a2857058b905cf1ec6fe8251834850ad792133f9ba49"
Oct 30 04:03:15.002844 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:03:15.002825    4461 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"55dd479290b849b28a07a2857058b905cf1ec6fe8251834850ad792133f9ba49"} err="failed to get container status \"55dd479290b849b28a07a2857058b905cf1ec6fe8251834850ad792133f9ba49\": rpc error: code = NotFound desc = could not find container \"55dd479290b849b28a07a2857058b905cf1ec6fe8251834850ad792133f9ba49\": container with ID starting with 55dd479290b849b28a07a2857058b905cf1ec6fe8251834850ad792133f9ba49 not found: ID does not exist"
Oct 30 04:03:15.002890 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:03:15.002845    4461 scope.go:122] "RemoveContainer" containerID="2e9b45fb90b04de27ea318cb92cc0a6de676cb225b68d20395002514793e113b"
Oct 30 04:03:15.003063 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:03:15.003045    4461 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2e9b45fb90b04de27ea318cb92cc0a6de676cb225b68d20395002514793e113b"} err="failed to get container status \"2e9b45fb90b04de27ea318cb92cc0a6de676cb225b68d20395002514793e113b\": rpc error: code = NotFound desc = could not find container \"2e9b45fb90b04de27ea318cb92cc0a6de676cb225b68d20395002514793e113b\": container with ID starting with 2e9b45fb90b04de27ea318cb92cc0a6de676cb225b68d20395002514793e113b not found: ID does not exist"
Oct 30 04:03:15.003136 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:03:15.003064    4461 scope.go:122] "RemoveContainer" containerID="3a8efadc307f19d3cd06965032572e7d5e53a01caa71f5673729bd6418b85536"
Oct 30 04:03:15.003282 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:03:15.003265    4461 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3a8efadc307f19d3cd06965032572e7d5e53a01caa71f5673729bd6418b85536"} err="failed to get container status \"3a8efadc307f19d3cd06965032572e7d5e53a01caa71f5673729bd6418b85536\": rpc error: code = NotFound desc = could not find container \"3a8efadc307f19d3cd06965032572e7d5e53a01caa71f5673729bd6418b85536\": container with ID starting with 3a8efadc307f19d3cd06965032572e7d5e53a01caa71f5673729bd6418b85536 not found: ID does not exist"
Oct 30 04:03:15.031282 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:03:15.031237    4461 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-b97f49766-tf8fm"]
Oct 30 04:03:15.043821 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:03:15.043804    4461 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-b97f49766-tf8fm"]
Oct 30 04:03:16.114460 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:03:16.114431    4461 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="76ef7227-645f-45df-9d1c-40b4b53614f9" path="/var/lib/kubelet/pods/76ef7227-645f-45df-9d1c-40b4b53614f9/volumes"
Oct 30 04:03:18.834530 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:03:18.834486    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 04:03:21.256767 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:03:21.256725    4461 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-h4qh8"]
Oct 30 04:03:21.257283 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:03:21.257018    4461 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-h4qh8" podUID="f83d1219-fd7f-47f2-af01-40fb3a4cd2a0" containerName="istio-proxy" containerID="cri-o://100b56d0a786b9a8beeea46f62ad206f2fb7f3e5426ce3abfffe84093f0fd11a" gracePeriod=30
Oct 30 04:03:23.756556 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:03:23.756523    4461 patch_prober.go:29] interesting pod/openshift-ai-inference-openshift-default-5b4f8cc59b-h4qh8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 04:03:23.756940 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:03:23.756586    4461 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-h4qh8" podUID="f83d1219-fd7f-47f2-af01-40fb3a4cd2a0" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 04:03:24.503802 ip-10-0-143-156 kubenswrapper[4461]: E1030 04:03:24.503766    4461 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2k85b" containerName="extractor"
Oct 30 04:03:26.491349 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:03:26.491322    4461 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-h4qh8"
Oct 30 04:03:26.525079 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:03:26.525026    4461 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f83d1219-fd7f-47f2-af01-40fb3a4cd2a0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f83d1219-fd7f-47f2-af01-40fb3a4cd2a0-istio-data\") pod \"f83d1219-fd7f-47f2-af01-40fb3a4cd2a0\" (UID: \"f83d1219-fd7f-47f2-af01-40fb3a4cd2a0\") "
Oct 30 04:03:26.525243 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:03:26.525082    4461 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f83d1219-fd7f-47f2-af01-40fb3a4cd2a0-istio-token\" (UniqueName: \"kubernetes.io/projected/f83d1219-fd7f-47f2-af01-40fb3a4cd2a0-istio-token\") pod \"f83d1219-fd7f-47f2-af01-40fb3a4cd2a0\" (UID: \"f83d1219-fd7f-47f2-af01-40fb3a4cd2a0\") "
Oct 30 04:03:26.525243 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:03:26.525130    4461 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f83d1219-fd7f-47f2-af01-40fb3a4cd2a0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f83d1219-fd7f-47f2-af01-40fb3a4cd2a0-workload-socket\") pod \"f83d1219-fd7f-47f2-af01-40fb3a4cd2a0\" (UID: \"f83d1219-fd7f-47f2-af01-40fb3a4cd2a0\") "
Oct 30 04:03:26.525243 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:03:26.525163    4461 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f83d1219-fd7f-47f2-af01-40fb3a4cd2a0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f83d1219-fd7f-47f2-af01-40fb3a4cd2a0-credential-socket\") pod \"f83d1219-fd7f-47f2-af01-40fb3a4cd2a0\" (UID: \"f83d1219-fd7f-47f2-af01-40fb3a4cd2a0\") "
Oct 30 04:03:26.525243 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:03:26.525189    4461 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f83d1219-fd7f-47f2-af01-40fb3a4cd2a0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f83d1219-fd7f-47f2-af01-40fb3a4cd2a0-workload-certs\") pod \"f83d1219-fd7f-47f2-af01-40fb3a4cd2a0\" (UID: \"f83d1219-fd7f-47f2-af01-40fb3a4cd2a0\") "
Oct 30 04:03:26.525243 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:03:26.525223    4461 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f83d1219-fd7f-47f2-af01-40fb3a4cd2a0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f83d1219-fd7f-47f2-af01-40fb3a4cd2a0-istiod-ca-cert\") pod \"f83d1219-fd7f-47f2-af01-40fb3a4cd2a0\" (UID: \"f83d1219-fd7f-47f2-af01-40fb3a4cd2a0\") "
Oct 30 04:03:26.525489 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:03:26.525257    4461 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f83d1219-fd7f-47f2-af01-40fb3a4cd2a0-kube-api-access-klgb7\" (UniqueName: \"kubernetes.io/projected/f83d1219-fd7f-47f2-af01-40fb3a4cd2a0-kube-api-access-klgb7\") pod \"f83d1219-fd7f-47f2-af01-40fb3a4cd2a0\" (UID: \"f83d1219-fd7f-47f2-af01-40fb3a4cd2a0\") "
Oct 30 04:03:26.525489 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:03:26.525300    4461 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f83d1219-fd7f-47f2-af01-40fb3a4cd2a0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f83d1219-fd7f-47f2-af01-40fb3a4cd2a0-istio-podinfo\") pod \"f83d1219-fd7f-47f2-af01-40fb3a4cd2a0\" (UID: \"f83d1219-fd7f-47f2-af01-40fb3a4cd2a0\") "
Oct 30 04:03:26.525489 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:03:26.525327    4461 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f83d1219-fd7f-47f2-af01-40fb3a4cd2a0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f83d1219-fd7f-47f2-af01-40fb3a4cd2a0-istio-envoy\") pod \"f83d1219-fd7f-47f2-af01-40fb3a4cd2a0\" (UID: \"f83d1219-fd7f-47f2-af01-40fb3a4cd2a0\") "
Oct 30 04:03:26.525489 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:03:26.525330    4461 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f83d1219-fd7f-47f2-af01-40fb3a4cd2a0-istio-data" pod "f83d1219-fd7f-47f2-af01-40fb3a4cd2a0" (UID: "f83d1219-fd7f-47f2-af01-40fb3a4cd2a0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:03:26.525489 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:03:26.525360    4461 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f83d1219-fd7f-47f2-af01-40fb3a4cd2a0-workload-socket" pod "f83d1219-fd7f-47f2-af01-40fb3a4cd2a0" (UID: "f83d1219-fd7f-47f2-af01-40fb3a4cd2a0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:03:26.525748 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:03:26.525550    4461 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f83d1219-fd7f-47f2-af01-40fb3a4cd2a0-istio-data\") on node \"ip-10-0-143-156.ec2.internal\" DevicePath \"\""
Oct 30 04:03:26.525748 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:03:26.525569    4461 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f83d1219-fd7f-47f2-af01-40fb3a4cd2a0-workload-socket\") on node \"ip-10-0-143-156.ec2.internal\" DevicePath \"\""
Oct 30 04:03:26.525748 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:03:26.525632    4461 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f83d1219-fd7f-47f2-af01-40fb3a4cd2a0-credential-socket" pod "f83d1219-fd7f-47f2-af01-40fb3a4cd2a0" (UID: "f83d1219-fd7f-47f2-af01-40fb3a4cd2a0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:03:26.525748 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:03:26.525642    4461 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f83d1219-fd7f-47f2-af01-40fb3a4cd2a0-workload-certs" pod "f83d1219-fd7f-47f2-af01-40fb3a4cd2a0" (UID: "f83d1219-fd7f-47f2-af01-40fb3a4cd2a0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:03:26.525748 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:03:26.525691    4461 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f83d1219-fd7f-47f2-af01-40fb3a4cd2a0-istiod-ca-cert" pod "f83d1219-fd7f-47f2-af01-40fb3a4cd2a0" (UID: "f83d1219-fd7f-47f2-af01-40fb3a4cd2a0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:03:26.527298 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:03:26.527277    4461 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f83d1219-fd7f-47f2-af01-40fb3a4cd2a0-istio-token" pod "f83d1219-fd7f-47f2-af01-40fb3a4cd2a0" (UID: "f83d1219-fd7f-47f2-af01-40fb3a4cd2a0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:03:26.527418 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:03:26.527324    4461 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f83d1219-fd7f-47f2-af01-40fb3a4cd2a0-istio-podinfo" pod "f83d1219-fd7f-47f2-af01-40fb3a4cd2a0" (UID: "f83d1219-fd7f-47f2-af01-40fb3a4cd2a0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 04:03:26.527790 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:03:26.527766    4461 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f83d1219-fd7f-47f2-af01-40fb3a4cd2a0-istio-envoy" pod "f83d1219-fd7f-47f2-af01-40fb3a4cd2a0" (UID: "f83d1219-fd7f-47f2-af01-40fb3a4cd2a0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:03:26.527790 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:03:26.527775    4461 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f83d1219-fd7f-47f2-af01-40fb3a4cd2a0-kube-api-access-klgb7" pod "f83d1219-fd7f-47f2-af01-40fb3a4cd2a0" (UID: "f83d1219-fd7f-47f2-af01-40fb3a4cd2a0"). InnerVolumeSpecName "kube-api-access-klgb7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:03:26.626910 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:03:26.626875    4461 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f83d1219-fd7f-47f2-af01-40fb3a4cd2a0-istiod-ca-cert\") on node \"ip-10-0-143-156.ec2.internal\" DevicePath \"\""
Oct 30 04:03:26.626910 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:03:26.626904    4461 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-klgb7\" (UniqueName: \"kubernetes.io/projected/f83d1219-fd7f-47f2-af01-40fb3a4cd2a0-kube-api-access-klgb7\") on node \"ip-10-0-143-156.ec2.internal\" DevicePath \"\""
Oct 30 04:03:26.626910 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:03:26.626915    4461 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f83d1219-fd7f-47f2-af01-40fb3a4cd2a0-istio-podinfo\") on node \"ip-10-0-143-156.ec2.internal\" DevicePath \"\""
Oct 30 04:03:26.627168 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:03:26.626925    4461 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f83d1219-fd7f-47f2-af01-40fb3a4cd2a0-istio-envoy\") on node \"ip-10-0-143-156.ec2.internal\" DevicePath \"\""
Oct 30 04:03:26.627168 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:03:26.626934    4461 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f83d1219-fd7f-47f2-af01-40fb3a4cd2a0-istio-token\") on node \"ip-10-0-143-156.ec2.internal\" DevicePath \"\""
Oct 30 04:03:26.627168 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:03:26.626942    4461 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f83d1219-fd7f-47f2-af01-40fb3a4cd2a0-credential-socket\") on node \"ip-10-0-143-156.ec2.internal\" DevicePath \"\""
Oct 30 04:03:26.627168 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:03:26.626950    4461 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f83d1219-fd7f-47f2-af01-40fb3a4cd2a0-workload-certs\") on node \"ip-10-0-143-156.ec2.internal\" DevicePath \"\""
Oct 30 04:03:27.027374 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:03:27.027341    4461 generic.go:356] "Generic (PLEG): container finished" podID="f83d1219-fd7f-47f2-af01-40fb3a4cd2a0" containerID="100b56d0a786b9a8beeea46f62ad206f2fb7f3e5426ce3abfffe84093f0fd11a" exitCode=0
Oct 30 04:03:27.027522 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:03:27.027382    4461 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-h4qh8"
Oct 30 04:03:27.027522 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:03:27.027394    4461 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-h4qh8" event={"ID":"f83d1219-fd7f-47f2-af01-40fb3a4cd2a0","Type":"ContainerDied","Data":"100b56d0a786b9a8beeea46f62ad206f2fb7f3e5426ce3abfffe84093f0fd11a"}
Oct 30 04:03:27.027522 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:03:27.027466    4461 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-h4qh8" event={"ID":"f83d1219-fd7f-47f2-af01-40fb3a4cd2a0","Type":"ContainerDied","Data":"f359120081b80ce0e3fd9c157d3046362f8961e9e9ae37f4366d36a110e6cc66"}
Oct 30 04:03:27.027522 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:03:27.027490    4461 scope.go:122] "RemoveContainer" containerID="100b56d0a786b9a8beeea46f62ad206f2fb7f3e5426ce3abfffe84093f0fd11a"
Oct 30 04:03:27.036303 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:03:27.036286    4461 scope.go:122] "RemoveContainer" containerID="100b56d0a786b9a8beeea46f62ad206f2fb7f3e5426ce3abfffe84093f0fd11a"
Oct 30 04:03:27.036584 ip-10-0-143-156 kubenswrapper[4461]: E1030 04:03:27.036559    4461 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"100b56d0a786b9a8beeea46f62ad206f2fb7f3e5426ce3abfffe84093f0fd11a\": container with ID starting with 100b56d0a786b9a8beeea46f62ad206f2fb7f3e5426ce3abfffe84093f0fd11a not found: ID does not exist" containerID="100b56d0a786b9a8beeea46f62ad206f2fb7f3e5426ce3abfffe84093f0fd11a"
Oct 30 04:03:27.036669 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:03:27.036589    4461 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"100b56d0a786b9a8beeea46f62ad206f2fb7f3e5426ce3abfffe84093f0fd11a"} err="failed to get container status \"100b56d0a786b9a8beeea46f62ad206f2fb7f3e5426ce3abfffe84093f0fd11a\": rpc error: code = NotFound desc = could not find container \"100b56d0a786b9a8beeea46f62ad206f2fb7f3e5426ce3abfffe84093f0fd11a\": container with ID starting with 100b56d0a786b9a8beeea46f62ad206f2fb7f3e5426ce3abfffe84093f0fd11a not found: ID does not exist"
Oct 30 04:03:27.079580 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:03:27.079546    4461 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-h4qh8"]
Oct 30 04:03:27.092236 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:03:27.092214    4461 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-h4qh8"]
Oct 30 04:03:28.114138 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:03:28.114087    4461 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f83d1219-fd7f-47f2-af01-40fb3a4cd2a0" path="/var/lib/kubelet/pods/f83d1219-fd7f-47f2-af01-40fb3a4cd2a0/volumes"
Oct 30 04:03:29.045561 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:03:29.045526    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 04:03:29.561724 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:03:29.561693    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 04:03:30.739485 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:03:30.739450    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 04:03:30.937518 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:03:30.937484    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 04:03:31.190357 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:03:31.190267    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 04:03:31.453593 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:03:31.453516    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 04:03:31.644242 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:03:31.644213    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 04:03:33.901258 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:03:33.901227    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 04:03:34.066771 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:03:34.066724    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 04:03:34.509807 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:03:34.509769    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 04:03:37.358948 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:03:37.358908    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 04:03:37.535967 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:03:37.535932    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 04:03:48.902087 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:03:48.901984    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 04:03:49.081630 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:03:49.081591    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 04:03:49.323676 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:03:49.323642    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 04:03:49.495380 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:03:49.495343    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 04:03:52.638815 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:03:52.638778    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 04:03:54.503822 ip-10-0-143-156 kubenswrapper[4461]: E1030 04:03:54.503791    4461 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2k85b" containerName="extractor"
Oct 30 04:03:56.021338 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:03:56.021261    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 04:04:02.087343 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:04:02.087304    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 04:04:02.264637 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:04:02.264605    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 04:04:04.023360 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:04:04.023321    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 04:04:05.508194 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:04:05.508161    4461 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-555dc7c784-rt68c"]
Oct 30 04:04:05.508555 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:04:05.508454    4461 cpu_manager.go:414] "containerMap: removing container" podUID="f83d1219-fd7f-47f2-af01-40fb3a4cd2a0" containerName="istio-proxy"
Oct 30 04:04:05.508555 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:04:05.508465    4461 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f83d1219-fd7f-47f2-af01-40fb3a4cd2a0" containerName="istio-proxy"
Oct 30 04:04:05.508555 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:04:05.508476    4461 cpu_manager.go:414] "containerMap: removing container" podUID="76ef7227-645f-45df-9d1c-40b4b53614f9" containerName="main"
Oct 30 04:04:05.508555 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:04:05.508483    4461 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="76ef7227-645f-45df-9d1c-40b4b53614f9" containerName="main"
Oct 30 04:04:05.508555 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:04:05.508497    4461 cpu_manager.go:414] "containerMap: removing container" podUID="76ef7227-645f-45df-9d1c-40b4b53614f9" containerName="llm-d-routing-sidecar"
Oct 30 04:04:05.508555 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:04:05.508505    4461 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="76ef7227-645f-45df-9d1c-40b4b53614f9" containerName="llm-d-routing-sidecar"
Oct 30 04:04:05.508555 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:04:05.508516    4461 cpu_manager.go:414] "containerMap: removing container" podUID="76ef7227-645f-45df-9d1c-40b4b53614f9" containerName="storage-initializer"
Oct 30 04:04:05.508555 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:04:05.508521    4461 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="76ef7227-645f-45df-9d1c-40b4b53614f9" containerName="storage-initializer"
Oct 30 04:04:05.508859 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:04:05.508577    4461 memory_manager.go:365] "RemoveStaleState removing state" podUID="f83d1219-fd7f-47f2-af01-40fb3a4cd2a0" containerName="istio-proxy"
Oct 30 04:04:05.508859 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:04:05.508587    4461 memory_manager.go:365] "RemoveStaleState removing state" podUID="76ef7227-645f-45df-9d1c-40b4b53614f9" containerName="main"
Oct 30 04:04:05.508859 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:04:05.508595    4461 memory_manager.go:365] "RemoveStaleState removing state" podUID="76ef7227-645f-45df-9d1c-40b4b53614f9" containerName="llm-d-routing-sidecar"
Oct 30 04:04:05.510977 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:04:05.510960    4461 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-555dc7c784-rt68c"
Oct 30 04:04:05.527195 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:04:05.527168    4461 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"openshift-service-ca.crt\""
Oct 30 04:04:05.527195 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:04:05.527184    4461 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"seaweedfs-s3-creds\""
Oct 30 04:04:05.538949 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:04:05.538919    4461 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"kube-root-ca.crt\""
Oct 30 04:04:05.538949 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:04:05.538944    4461 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"odh-kserve-custom-ca-bundle\""
Oct 30 04:04:05.539105 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:04:05.538968    4461 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"router-with-refs-pd-test-kserve-self-signed-certs\""
Oct 30 04:04:05.539105 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:04:05.539024    4461 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"router-with-refs-pd-test-kserve-dockercfg-kzknk\""
Oct 30 04:04:05.539199 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:04:05.539105    4461 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"default-dockercfg-jmw65\""
Oct 30 04:04:05.541528 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:04:05.541509    4461 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-555dc7c784-rt68c"]
Oct 30 04:04:05.657994 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:04:05.657964    4461 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e2533e8e-70c7-4738-aaae-1bfd94190af0-cabundle-cert\") pod \"router-with-refs-pd-test-kserve-555dc7c784-rt68c\" (UID: \"e2533e8e-70c7-4738-aaae-1bfd94190af0\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-555dc7c784-rt68c"
Oct 30 04:04:05.657994 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:04:05.657998    4461 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e2533e8e-70c7-4738-aaae-1bfd94190af0-home\") pod \"router-with-refs-pd-test-kserve-555dc7c784-rt68c\" (UID: \"e2533e8e-70c7-4738-aaae-1bfd94190af0\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-555dc7c784-rt68c"
Oct 30 04:04:05.658218 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:04:05.658030    4461 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e2533e8e-70c7-4738-aaae-1bfd94190af0-tls-certs\") pod \"router-with-refs-pd-test-kserve-555dc7c784-rt68c\" (UID: \"e2533e8e-70c7-4738-aaae-1bfd94190af0\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-555dc7c784-rt68c"
Oct 30 04:04:05.658218 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:04:05.658162    4461 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e2533e8e-70c7-4738-aaae-1bfd94190af0-tmp-dir\") pod \"router-with-refs-pd-test-kserve-555dc7c784-rt68c\" (UID: \"e2533e8e-70c7-4738-aaae-1bfd94190af0\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-555dc7c784-rt68c"
Oct 30 04:04:05.658218 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:04:05.658195    4461 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e2533e8e-70c7-4738-aaae-1bfd94190af0-kserve-provision-location\") pod \"router-with-refs-pd-test-kserve-555dc7c784-rt68c\" (UID: \"e2533e8e-70c7-4738-aaae-1bfd94190af0\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-555dc7c784-rt68c"
Oct 30 04:04:05.658319 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:04:05.658219    4461 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e2533e8e-70c7-4738-aaae-1bfd94190af0-dshm\") pod \"router-with-refs-pd-test-kserve-555dc7c784-rt68c\" (UID: \"e2533e8e-70c7-4738-aaae-1bfd94190af0\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-555dc7c784-rt68c"
Oct 30 04:04:05.658359 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:04:05.658339    4461 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6g2ml\" (UniqueName: \"kubernetes.io/projected/e2533e8e-70c7-4738-aaae-1bfd94190af0-kube-api-access-6g2ml\") pod \"router-with-refs-pd-test-kserve-555dc7c784-rt68c\" (UID: \"e2533e8e-70c7-4738-aaae-1bfd94190af0\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-555dc7c784-rt68c"
Oct 30 04:04:05.658400 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:04:05.658370    4461 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e2533e8e-70c7-4738-aaae-1bfd94190af0-model-cache\") pod \"router-with-refs-pd-test-kserve-555dc7c784-rt68c\" (UID: \"e2533e8e-70c7-4738-aaae-1bfd94190af0\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-555dc7c784-rt68c"
Oct 30 04:04:05.726302 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:04:05.726266    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 04:04:05.759503 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:04:05.759447    4461 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6g2ml\" (UniqueName: \"kubernetes.io/projected/e2533e8e-70c7-4738-aaae-1bfd94190af0-kube-api-access-6g2ml\") pod \"router-with-refs-pd-test-kserve-555dc7c784-rt68c\" (UID: \"e2533e8e-70c7-4738-aaae-1bfd94190af0\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-555dc7c784-rt68c"
Oct 30 04:04:05.759503 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:04:05.759483    4461 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e2533e8e-70c7-4738-aaae-1bfd94190af0-model-cache\") pod \"router-with-refs-pd-test-kserve-555dc7c784-rt68c\" (UID: \"e2533e8e-70c7-4738-aaae-1bfd94190af0\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-555dc7c784-rt68c"
Oct 30 04:04:05.759649 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:04:05.759531    4461 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e2533e8e-70c7-4738-aaae-1bfd94190af0-cabundle-cert\") pod \"router-with-refs-pd-test-kserve-555dc7c784-rt68c\" (UID: \"e2533e8e-70c7-4738-aaae-1bfd94190af0\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-555dc7c784-rt68c"
Oct 30 04:04:05.759649 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:04:05.759560    4461 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e2533e8e-70c7-4738-aaae-1bfd94190af0-home\") pod \"router-with-refs-pd-test-kserve-555dc7c784-rt68c\" (UID: \"e2533e8e-70c7-4738-aaae-1bfd94190af0\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-555dc7c784-rt68c"
Oct 30 04:04:05.759730 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:04:05.759710    4461 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e2533e8e-70c7-4738-aaae-1bfd94190af0-tls-certs\") pod \"router-with-refs-pd-test-kserve-555dc7c784-rt68c\" (UID: \"e2533e8e-70c7-4738-aaae-1bfd94190af0\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-555dc7c784-rt68c"
Oct 30 04:04:05.759773 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:04:05.759761    4461 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e2533e8e-70c7-4738-aaae-1bfd94190af0-tmp-dir\") pod \"router-with-refs-pd-test-kserve-555dc7c784-rt68c\" (UID: \"e2533e8e-70c7-4738-aaae-1bfd94190af0\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-555dc7c784-rt68c"
Oct 30 04:04:05.759828 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:04:05.759802    4461 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e2533e8e-70c7-4738-aaae-1bfd94190af0-kserve-provision-location\") pod \"router-with-refs-pd-test-kserve-555dc7c784-rt68c\" (UID: \"e2533e8e-70c7-4738-aaae-1bfd94190af0\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-555dc7c784-rt68c"
Oct 30 04:04:05.759892 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:04:05.759838    4461 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e2533e8e-70c7-4738-aaae-1bfd94190af0-dshm\") pod \"router-with-refs-pd-test-kserve-555dc7c784-rt68c\" (UID: \"e2533e8e-70c7-4738-aaae-1bfd94190af0\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-555dc7c784-rt68c"
Oct 30 04:04:05.760013 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:04:05.759993    4461 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e2533e8e-70c7-4738-aaae-1bfd94190af0-home\") pod \"router-with-refs-pd-test-kserve-555dc7c784-rt68c\" (UID: \"e2533e8e-70c7-4738-aaae-1bfd94190af0\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-555dc7c784-rt68c"
Oct 30 04:04:05.760087 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:04:05.760066    4461 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e2533e8e-70c7-4738-aaae-1bfd94190af0-model-cache\") pod \"router-with-refs-pd-test-kserve-555dc7c784-rt68c\" (UID: \"e2533e8e-70c7-4738-aaae-1bfd94190af0\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-555dc7c784-rt68c"
Oct 30 04:04:05.760172 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:04:05.760138    4461 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e2533e8e-70c7-4738-aaae-1bfd94190af0-tmp-dir\") pod \"router-with-refs-pd-test-kserve-555dc7c784-rt68c\" (UID: \"e2533e8e-70c7-4738-aaae-1bfd94190af0\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-555dc7c784-rt68c"
Oct 30 04:04:05.760288 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:04:05.760264    4461 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e2533e8e-70c7-4738-aaae-1bfd94190af0-kserve-provision-location\") pod \"router-with-refs-pd-test-kserve-555dc7c784-rt68c\" (UID: \"e2533e8e-70c7-4738-aaae-1bfd94190af0\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-555dc7c784-rt68c"
Oct 30 04:04:05.760356 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:04:05.760338    4461 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e2533e8e-70c7-4738-aaae-1bfd94190af0-cabundle-cert\") pod \"router-with-refs-pd-test-kserve-555dc7c784-rt68c\" (UID: \"e2533e8e-70c7-4738-aaae-1bfd94190af0\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-555dc7c784-rt68c"
Oct 30 04:04:05.762073 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:04:05.762050    4461 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e2533e8e-70c7-4738-aaae-1bfd94190af0-dshm\") pod \"router-with-refs-pd-test-kserve-555dc7c784-rt68c\" (UID: \"e2533e8e-70c7-4738-aaae-1bfd94190af0\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-555dc7c784-rt68c"
Oct 30 04:04:05.762299 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:04:05.762282    4461 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e2533e8e-70c7-4738-aaae-1bfd94190af0-tls-certs\") pod \"router-with-refs-pd-test-kserve-555dc7c784-rt68c\" (UID: \"e2533e8e-70c7-4738-aaae-1bfd94190af0\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-555dc7c784-rt68c"
Oct 30 04:04:05.784484 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:04:05.784455    4461 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6g2ml\" (UniqueName: \"kubernetes.io/projected/e2533e8e-70c7-4738-aaae-1bfd94190af0-kube-api-access-6g2ml\") pod \"router-with-refs-pd-test-kserve-555dc7c784-rt68c\" (UID: \"e2533e8e-70c7-4738-aaae-1bfd94190af0\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-555dc7c784-rt68c"
Oct 30 04:04:05.823327 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:04:05.823305    4461 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-555dc7c784-rt68c"
Oct 30 04:04:06.147638 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:04:06.147610    4461 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-555dc7c784-rt68c"]
Oct 30 04:04:06.148915 ip-10-0-143-156 kubenswrapper[4461]: W1030 04:04:06.148888    4461 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode2533e8e_70c7_4738_aaae_1bfd94190af0.slice/crio-a29822eb0803d33be56babd370ccc4613b59306f6c4cbf82d91ee8d49efd09d6 WatchSource:0}: Error finding container a29822eb0803d33be56babd370ccc4613b59306f6c4cbf82d91ee8d49efd09d6: Status 404 returned error can't find the container with id a29822eb0803d33be56babd370ccc4613b59306f6c4cbf82d91ee8d49efd09d6
Oct 30 04:04:06.163797 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:04:06.163765    4461 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-555dc7c784-rt68c" event={"ID":"e2533e8e-70c7-4738-aaae-1bfd94190af0","Type":"ContainerStarted","Data":"a29822eb0803d33be56babd370ccc4613b59306f6c4cbf82d91ee8d49efd09d6"}
Oct 30 04:04:06.221220 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:04:06.221193    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 04:04:06.423166 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:04:06.423064    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 04:04:07.169136 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:04:07.169085    4461 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-555dc7c784-rt68c" event={"ID":"e2533e8e-70c7-4738-aaae-1bfd94190af0","Type":"ContainerStarted","Data":"10958607c49c2127d6e5d828bafcdaea063f772f75f4b99bf47f72c3b9bf01d7"}
Oct 30 04:04:07.169589 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:04:07.169209    4461 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-555dc7c784-rt68c"
Oct 30 04:04:08.177545 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:04:08.177502    4461 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-555dc7c784-rt68c" event={"ID":"e2533e8e-70c7-4738-aaae-1bfd94190af0","Type":"ContainerStarted","Data":"3dd86c14c3b731c0226b07a5150bf50f7d091fa1662122784b60caff09ff4d40"}
Oct 30 04:04:10.188013 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:04:10.187976    4461 generic.go:356] "Generic (PLEG): container finished" podID="e2533e8e-70c7-4738-aaae-1bfd94190af0" containerID="3dd86c14c3b731c0226b07a5150bf50f7d091fa1662122784b60caff09ff4d40" exitCode=0
Oct 30 04:04:10.188449 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:04:10.188048    4461 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-555dc7c784-rt68c" event={"ID":"e2533e8e-70c7-4738-aaae-1bfd94190af0","Type":"ContainerDied","Data":"3dd86c14c3b731c0226b07a5150bf50f7d091fa1662122784b60caff09ff4d40"}
Oct 30 04:04:11.195207 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:04:11.195168    4461 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-555dc7c784-rt68c" event={"ID":"e2533e8e-70c7-4738-aaae-1bfd94190af0","Type":"ContainerStarted","Data":"d2f99dbb242e9e21550ebbb4eecac4b6f264d6182be15bbdd176ac5b34dbaeb0"}
Oct 30 04:04:11.271466 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:04:11.271412    4461 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-555dc7c784-rt68c" podStartSLOduration=4.153018777 podStartE2EDuration="6.271398325s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.118379548s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:04:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:04:11.258325642 +0000 UTC m=+4119.308130838" watchObservedRunningTime="2026-10-30 04:04:11.271398325 +0000 UTC m=+4119.321203498"
Oct 30 04:04:15.824483 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:04:15.824435    4461 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-555dc7c784-rt68c"
Oct 30 04:04:15.824870 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:04:15.824528    4461 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-555dc7c784-rt68c"
Oct 30 04:04:15.825505 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:04:15.825478    4461 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-555dc7c784-rt68c" podUID="e2533e8e-70c7-4738-aaae-1bfd94190af0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8001/health\": dial tcp 10.133.0.48:8001: connect: connection refused"
Oct 30 04:04:18.968581 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:04:18.968546    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 04:04:24.506811 ip-10-0-143-156 kubenswrapper[4461]: E1030 04:04:24.506775    4461 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2k85b" containerName="extractor"
Oct 30 04:04:25.824687 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:04:25.824643    4461 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-555dc7c784-rt68c" podUID="e2533e8e-70c7-4738-aaae-1bfd94190af0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8001/health\": dial tcp 10.133.0.48:8001: connect: connection refused"
Oct 30 04:04:25.855051 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:04:25.855024    4461 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-555dc7c784-rt68c"
Oct 30 04:04:29.166670 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:04:29.166620    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 04:04:35.824191 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:04:35.824148    4461 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-555dc7c784-rt68c" podUID="e2533e8e-70c7-4738-aaae-1bfd94190af0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8001/health\": dial tcp 10.133.0.48:8001: connect: connection refused"
Oct 30 04:04:45.824315 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:04:45.824271    4461 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-555dc7c784-rt68c" podUID="e2533e8e-70c7-4738-aaae-1bfd94190af0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8001/health\": dial tcp 10.133.0.48:8001: connect: connection refused"
Oct 30 04:04:49.444069 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:04:49.444008    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 04:04:51.549323 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:04:51.549243    4461 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-nfxr8"]
Oct 30 04:04:51.553195 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:04:51.553168    4461 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-nfxr8"
Oct 30 04:04:51.568008 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:04:51.567984    4461 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-jtcmr"]
Oct 30 04:04:51.571181 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:04:51.571165    4461 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-jtcmr"
Oct 30 04:04:51.579543 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:04:51.579518    4461 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Oct 30 04:04:51.579543 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:04:51.579531    4461 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Oct 30 04:04:51.579729 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:04:51.579555    4461 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Oct 30 04:04:51.579729 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:04:51.579664    4461 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"openshift-ai-inference-openshift-default-dockercfg-pzw79\""
Oct 30 04:04:51.583813 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:04:51.583792    4461 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-nfxr8"]
Oct 30 04:04:51.609819 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:04:51.609783    4461 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-jtcmr"]
Oct 30 04:04:51.648506 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:04:51.648476    4461 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1dacbabe-8d44-4ebc-be3d-fb89d9f70742-workload-socket\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-nfxr8\" (UID: \"1dacbabe-8d44-4ebc-be3d-fb89d9f70742\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-nfxr8"
Oct 30 04:04:51.648642 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:04:51.648511    4461 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1dacbabe-8d44-4ebc-be3d-fb89d9f70742-istio-envoy\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-nfxr8\" (UID: \"1dacbabe-8d44-4ebc-be3d-fb89d9f70742\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-nfxr8"
Oct 30 04:04:51.648642 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:04:51.648615    4461 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1dacbabe-8d44-4ebc-be3d-fb89d9f70742-credential-socket\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-nfxr8\" (UID: \"1dacbabe-8d44-4ebc-be3d-fb89d9f70742\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-nfxr8"
Oct 30 04:04:51.648719 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:04:51.648663    4461 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1dacbabe-8d44-4ebc-be3d-fb89d9f70742-workload-certs\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-nfxr8\" (UID: \"1dacbabe-8d44-4ebc-be3d-fb89d9f70742\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-nfxr8"
Oct 30 04:04:51.648719 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:04:51.648696    4461 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1dacbabe-8d44-4ebc-be3d-fb89d9f70742-istio-token\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-nfxr8\" (UID: \"1dacbabe-8d44-4ebc-be3d-fb89d9f70742\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-nfxr8"
Oct 30 04:04:51.648786 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:04:51.648738    4461 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1dacbabe-8d44-4ebc-be3d-fb89d9f70742-istio-podinfo\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-nfxr8\" (UID: \"1dacbabe-8d44-4ebc-be3d-fb89d9f70742\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-nfxr8"
Oct 30 04:04:51.648786 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:04:51.648761    4461 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fmc94\" (UniqueName: \"kubernetes.io/projected/1dacbabe-8d44-4ebc-be3d-fb89d9f70742-kube-api-access-fmc94\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-nfxr8\" (UID: \"1dacbabe-8d44-4ebc-be3d-fb89d9f70742\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-nfxr8"
Oct 30 04:04:51.648849 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:04:51.648790    4461 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1dacbabe-8d44-4ebc-be3d-fb89d9f70742-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-nfxr8\" (UID: \"1dacbabe-8d44-4ebc-be3d-fb89d9f70742\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-nfxr8"
Oct 30 04:04:51.648849 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:04:51.648809    4461 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1dacbabe-8d44-4ebc-be3d-fb89d9f70742-istio-data\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-nfxr8\" (UID: \"1dacbabe-8d44-4ebc-be3d-fb89d9f70742\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-nfxr8"
Oct 30 04:04:51.749295 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:04:51.749262    4461 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1dacbabe-8d44-4ebc-be3d-fb89d9f70742-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-nfxr8\" (UID: \"1dacbabe-8d44-4ebc-be3d-fb89d9f70742\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-nfxr8"
Oct 30 04:04:51.749488 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:04:51.749310    4461 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1dacbabe-8d44-4ebc-be3d-fb89d9f70742-istio-data\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-nfxr8\" (UID: \"1dacbabe-8d44-4ebc-be3d-fb89d9f70742\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-nfxr8"
Oct 30 04:04:51.749488 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:04:51.749364    4461 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aedb2e20-2e07-42b1-b3c2-27223dc8d177-istio-data\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-jtcmr\" (UID: \"aedb2e20-2e07-42b1-b3c2-27223dc8d177\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-jtcmr"
Oct 30 04:04:51.749488 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:04:51.749400    4461 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aedb2e20-2e07-42b1-b3c2-27223dc8d177-credential-socket\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-jtcmr\" (UID: \"aedb2e20-2e07-42b1-b3c2-27223dc8d177\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-jtcmr"
Oct 30 04:04:51.749488 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:04:51.749472    4461 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aedb2e20-2e07-42b1-b3c2-27223dc8d177-workload-certs\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-jtcmr\" (UID: \"aedb2e20-2e07-42b1-b3c2-27223dc8d177\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-jtcmr"
Oct 30 04:04:51.749709 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:04:51.749531    4461 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1dacbabe-8d44-4ebc-be3d-fb89d9f70742-workload-socket\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-nfxr8\" (UID: \"1dacbabe-8d44-4ebc-be3d-fb89d9f70742\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-nfxr8"
Oct 30 04:04:51.749709 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:04:51.749557    4461 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1dacbabe-8d44-4ebc-be3d-fb89d9f70742-istio-envoy\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-nfxr8\" (UID: \"1dacbabe-8d44-4ebc-be3d-fb89d9f70742\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-nfxr8"
Oct 30 04:04:51.749709 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:04:51.749591    4461 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1dacbabe-8d44-4ebc-be3d-fb89d9f70742-credential-socket\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-nfxr8\" (UID: \"1dacbabe-8d44-4ebc-be3d-fb89d9f70742\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-nfxr8"
Oct 30 04:04:51.749709 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:04:51.749607    4461 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aedb2e20-2e07-42b1-b3c2-27223dc8d177-istio-envoy\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-jtcmr\" (UID: \"aedb2e20-2e07-42b1-b3c2-27223dc8d177\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-jtcmr"
Oct 30 04:04:51.749709 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:04:51.749623    4461 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aedb2e20-2e07-42b1-b3c2-27223dc8d177-istio-podinfo\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-jtcmr\" (UID: \"aedb2e20-2e07-42b1-b3c2-27223dc8d177\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-jtcmr"
Oct 30 04:04:51.749709 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:04:51.749656    4461 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aedb2e20-2e07-42b1-b3c2-27223dc8d177-istio-token\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-jtcmr\" (UID: \"aedb2e20-2e07-42b1-b3c2-27223dc8d177\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-jtcmr"
Oct 30 04:04:51.750025 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:04:51.749742    4461 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1dacbabe-8d44-4ebc-be3d-fb89d9f70742-workload-certs\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-nfxr8\" (UID: \"1dacbabe-8d44-4ebc-be3d-fb89d9f70742\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-nfxr8"
Oct 30 04:04:51.750025 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:04:51.749809    4461 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1dacbabe-8d44-4ebc-be3d-fb89d9f70742-istio-data\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-nfxr8\" (UID: \"1dacbabe-8d44-4ebc-be3d-fb89d9f70742\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-nfxr8"
Oct 30 04:04:51.750025 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:04:51.749858    4461 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1dacbabe-8d44-4ebc-be3d-fb89d9f70742-workload-socket\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-nfxr8\" (UID: \"1dacbabe-8d44-4ebc-be3d-fb89d9f70742\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-nfxr8"
Oct 30 04:04:51.750025 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:04:51.749863    4461 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1dacbabe-8d44-4ebc-be3d-fb89d9f70742-istio-token\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-nfxr8\" (UID: \"1dacbabe-8d44-4ebc-be3d-fb89d9f70742\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-nfxr8"
Oct 30 04:04:51.750025 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:04:51.749977    4461 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1dacbabe-8d44-4ebc-be3d-fb89d9f70742-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-nfxr8\" (UID: \"1dacbabe-8d44-4ebc-be3d-fb89d9f70742\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-nfxr8"
Oct 30 04:04:51.750025 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:04:51.749987    4461 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aedb2e20-2e07-42b1-b3c2-27223dc8d177-workload-socket\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-jtcmr\" (UID: \"aedb2e20-2e07-42b1-b3c2-27223dc8d177\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-jtcmr"
Oct 30 04:04:51.750025 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:04:51.749979    4461 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1dacbabe-8d44-4ebc-be3d-fb89d9f70742-credential-socket\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-nfxr8\" (UID: \"1dacbabe-8d44-4ebc-be3d-fb89d9f70742\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-nfxr8"
Oct 30 04:04:51.750329 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:04:51.750050    4461 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1dacbabe-8d44-4ebc-be3d-fb89d9f70742-istio-podinfo\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-nfxr8\" (UID: \"1dacbabe-8d44-4ebc-be3d-fb89d9f70742\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-nfxr8"
Oct 30 04:04:51.750329 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:04:51.750063    4461 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1dacbabe-8d44-4ebc-be3d-fb89d9f70742-workload-certs\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-nfxr8\" (UID: \"1dacbabe-8d44-4ebc-be3d-fb89d9f70742\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-nfxr8"
Oct 30 04:04:51.750329 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:04:51.750089    4461 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tvhxv\" (UniqueName: \"kubernetes.io/projected/aedb2e20-2e07-42b1-b3c2-27223dc8d177-kube-api-access-tvhxv\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-jtcmr\" (UID: \"aedb2e20-2e07-42b1-b3c2-27223dc8d177\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-jtcmr"
Oct 30 04:04:51.750329 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:04:51.750152    4461 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fmc94\" (UniqueName: \"kubernetes.io/projected/1dacbabe-8d44-4ebc-be3d-fb89d9f70742-kube-api-access-fmc94\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-nfxr8\" (UID: \"1dacbabe-8d44-4ebc-be3d-fb89d9f70742\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-nfxr8"
Oct 30 04:04:51.750329 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:04:51.750191    4461 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aedb2e20-2e07-42b1-b3c2-27223dc8d177-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-jtcmr\" (UID: \"aedb2e20-2e07-42b1-b3c2-27223dc8d177\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-jtcmr"
Oct 30 04:04:51.751902 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:04:51.751877    4461 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1dacbabe-8d44-4ebc-be3d-fb89d9f70742-istio-envoy\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-nfxr8\" (UID: \"1dacbabe-8d44-4ebc-be3d-fb89d9f70742\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-nfxr8"
Oct 30 04:04:51.752372 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:04:51.752343    4461 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1dacbabe-8d44-4ebc-be3d-fb89d9f70742-istio-podinfo\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-nfxr8\" (UID: \"1dacbabe-8d44-4ebc-be3d-fb89d9f70742\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-nfxr8"
Oct 30 04:04:51.773948 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:04:51.773912    4461 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1dacbabe-8d44-4ebc-be3d-fb89d9f70742-istio-token\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-nfxr8\" (UID: \"1dacbabe-8d44-4ebc-be3d-fb89d9f70742\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-nfxr8"
Oct 30 04:04:51.774046 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:04:51.773980    4461 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fmc94\" (UniqueName: \"kubernetes.io/projected/1dacbabe-8d44-4ebc-be3d-fb89d9f70742-kube-api-access-fmc94\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-nfxr8\" (UID: \"1dacbabe-8d44-4ebc-be3d-fb89d9f70742\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-nfxr8"
Oct 30 04:04:51.851551 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:04:51.851472    4461 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aedb2e20-2e07-42b1-b3c2-27223dc8d177-istio-data\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-jtcmr\" (UID: \"aedb2e20-2e07-42b1-b3c2-27223dc8d177\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-jtcmr"
Oct 30 04:04:51.851551 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:04:51.851513    4461 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aedb2e20-2e07-42b1-b3c2-27223dc8d177-credential-socket\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-jtcmr\" (UID: \"aedb2e20-2e07-42b1-b3c2-27223dc8d177\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-jtcmr"
Oct 30 04:04:51.851551 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:04:51.851532    4461 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aedb2e20-2e07-42b1-b3c2-27223dc8d177-workload-certs\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-jtcmr\" (UID: \"aedb2e20-2e07-42b1-b3c2-27223dc8d177\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-jtcmr"
Oct 30 04:04:51.851848 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:04:51.851572    4461 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aedb2e20-2e07-42b1-b3c2-27223dc8d177-istio-envoy\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-jtcmr\" (UID: \"aedb2e20-2e07-42b1-b3c2-27223dc8d177\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-jtcmr"
Oct 30 04:04:51.851848 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:04:51.851635    4461 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aedb2e20-2e07-42b1-b3c2-27223dc8d177-istio-podinfo\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-jtcmr\" (UID: \"aedb2e20-2e07-42b1-b3c2-27223dc8d177\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-jtcmr"
Oct 30 04:04:51.851848 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:04:51.851672    4461 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aedb2e20-2e07-42b1-b3c2-27223dc8d177-istio-token\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-jtcmr\" (UID: \"aedb2e20-2e07-42b1-b3c2-27223dc8d177\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-jtcmr"
Oct 30 04:04:51.851848 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:04:51.851699    4461 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aedb2e20-2e07-42b1-b3c2-27223dc8d177-workload-socket\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-jtcmr\" (UID: \"aedb2e20-2e07-42b1-b3c2-27223dc8d177\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-jtcmr"
Oct 30 04:04:51.851848 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:04:51.851738    4461 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tvhxv\" (UniqueName: \"kubernetes.io/projected/aedb2e20-2e07-42b1-b3c2-27223dc8d177-kube-api-access-tvhxv\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-jtcmr\" (UID: \"aedb2e20-2e07-42b1-b3c2-27223dc8d177\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-jtcmr"
Oct 30 04:04:51.851848 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:04:51.851762    4461 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aedb2e20-2e07-42b1-b3c2-27223dc8d177-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-jtcmr\" (UID: \"aedb2e20-2e07-42b1-b3c2-27223dc8d177\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-jtcmr"
Oct 30 04:04:51.852186 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:04:51.851965    4461 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aedb2e20-2e07-42b1-b3c2-27223dc8d177-credential-socket\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-jtcmr\" (UID: \"aedb2e20-2e07-42b1-b3c2-27223dc8d177\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-jtcmr"
Oct 30 04:04:51.852186 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:04:51.851986    4461 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aedb2e20-2e07-42b1-b3c2-27223dc8d177-workload-certs\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-jtcmr\" (UID: \"aedb2e20-2e07-42b1-b3c2-27223dc8d177\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-jtcmr"
Oct 30 04:04:51.852186 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:04:51.852062    4461 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aedb2e20-2e07-42b1-b3c2-27223dc8d177-istio-data\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-jtcmr\" (UID: \"aedb2e20-2e07-42b1-b3c2-27223dc8d177\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-jtcmr"
Oct 30 04:04:51.852588 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:04:51.852566    4461 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aedb2e20-2e07-42b1-b3c2-27223dc8d177-workload-socket\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-jtcmr\" (UID: \"aedb2e20-2e07-42b1-b3c2-27223dc8d177\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-jtcmr"
Oct 30 04:04:51.853063 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:04:51.853040    4461 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aedb2e20-2e07-42b1-b3c2-27223dc8d177-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-jtcmr\" (UID: \"aedb2e20-2e07-42b1-b3c2-27223dc8d177\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-jtcmr"
Oct 30 04:04:51.853873 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:04:51.853851    4461 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aedb2e20-2e07-42b1-b3c2-27223dc8d177-istio-envoy\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-jtcmr\" (UID: \"aedb2e20-2e07-42b1-b3c2-27223dc8d177\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-jtcmr"
Oct 30 04:04:51.854185 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:04:51.854169    4461 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aedb2e20-2e07-42b1-b3c2-27223dc8d177-istio-podinfo\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-jtcmr\" (UID: \"aedb2e20-2e07-42b1-b3c2-27223dc8d177\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-jtcmr"
Oct 30 04:04:51.866131 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:04:51.866109    4461 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-nfxr8"
Oct 30 04:04:51.877084 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:04:51.877062    4461 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tvhxv\" (UniqueName: \"kubernetes.io/projected/aedb2e20-2e07-42b1-b3c2-27223dc8d177-kube-api-access-tvhxv\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-jtcmr\" (UID: \"aedb2e20-2e07-42b1-b3c2-27223dc8d177\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-jtcmr"
Oct 30 04:04:51.877874 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:04:51.877852    4461 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aedb2e20-2e07-42b1-b3c2-27223dc8d177-istio-token\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-jtcmr\" (UID: \"aedb2e20-2e07-42b1-b3c2-27223dc8d177\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-jtcmr"
Oct 30 04:04:51.884783 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:04:51.884761    4461 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-jtcmr"
Oct 30 04:04:52.196828 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:04:52.196797    4461 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-nfxr8"]
Oct 30 04:04:52.196973 ip-10-0-143-156 kubenswrapper[4461]: W1030 04:04:52.196933    4461 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1dacbabe_8d44_4ebc_be3d_fb89d9f70742.slice/crio-b808b3d4b6529ce29eee2762a32f091cec90ae701954632954386122e462252e WatchSource:0}: Error finding container b808b3d4b6529ce29eee2762a32f091cec90ae701954632954386122e462252e: Status 404 returned error can't find the container with id b808b3d4b6529ce29eee2762a32f091cec90ae701954632954386122e462252e
Oct 30 04:04:52.199015 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:04:52.198980    4461 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 04:04:52.199128 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:04:52.199043    4461 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 04:04:52.199128 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:04:52.199069    4461 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 04:04:52.210841 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:04:52.210813    4461 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-jtcmr"]
Oct 30 04:04:52.211208 ip-10-0-143-156 kubenswrapper[4461]: W1030 04:04:52.211183    4461 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaedb2e20_2e07_42b1_b3c2_27223dc8d177.slice/crio-71274a6479e834c8926d3aea3526e0af0732e1100149de3d04e8e0a56a7a889a WatchSource:0}: Error finding container 71274a6479e834c8926d3aea3526e0af0732e1100149de3d04e8e0a56a7a889a: Status 404 returned error can't find the container with id 71274a6479e834c8926d3aea3526e0af0732e1100149de3d04e8e0a56a7a889a
Oct 30 04:04:52.213565 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:04:52.213530    4461 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 04:04:52.213673 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:04:52.213608    4461 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 04:04:52.213673 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:04:52.213650    4461 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 04:04:52.371144 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:04:52.371088    4461 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-jtcmr" event={"ID":"aedb2e20-2e07-42b1-b3c2-27223dc8d177","Type":"ContainerStarted","Data":"efda88805e2304ccd46923be30dbb7d41ea73c74a999c6b9069f3b56afcbff0d"}
Oct 30 04:04:52.373251 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:04:52.373212    4461 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-jtcmr" event={"ID":"aedb2e20-2e07-42b1-b3c2-27223dc8d177","Type":"ContainerStarted","Data":"71274a6479e834c8926d3aea3526e0af0732e1100149de3d04e8e0a56a7a889a"}
Oct 30 04:04:52.375027 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:04:52.374995    4461 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-nfxr8" event={"ID":"1dacbabe-8d44-4ebc-be3d-fb89d9f70742","Type":"ContainerStarted","Data":"7e75ec4cfc7f6817eb5ddad8c41febdfe278337b1cd941ee5b4ea8fbf2cc7b37"}
Oct 30 04:04:52.375231 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:04:52.375210    4461 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-nfxr8" event={"ID":"1dacbabe-8d44-4ebc-be3d-fb89d9f70742","Type":"ContainerStarted","Data":"b808b3d4b6529ce29eee2762a32f091cec90ae701954632954386122e462252e"}
Oct 30 04:04:52.424410 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:04:52.424343    4461 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-jtcmr" podStartSLOduration=1.424328574 podStartE2EDuration="1.424328574s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:04:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:04:52.423300042 +0000 UTC m=+4160.473105218" watchObservedRunningTime="2026-10-30 04:04:52.424328574 +0000 UTC m=+4160.474133743"
Oct 30 04:04:52.469238 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:04:52.469189    4461 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-nfxr8" podStartSLOduration=1.4691734379999999 podStartE2EDuration="1.469173438s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:04:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:04:52.468666262 +0000 UTC m=+4160.518471442" watchObservedRunningTime="2026-10-30 04:04:52.469173438 +0000 UTC m=+4160.518978614"
Oct 30 04:04:52.867812 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:04:52.867666    4461 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-nfxr8"
Oct 30 04:04:52.885284 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:04:52.885229    4461 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-jtcmr"
Oct 30 04:04:53.870382 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:04:53.870345    4461 patch_prober.go:29] interesting pod/openshift-ai-inference-openshift-default-5b4f8cc59b-nfxr8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" start-of-body=
Oct 30 04:04:53.870989 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:04:53.870958    4461 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-nfxr8" podUID="1dacbabe-8d44-4ebc-be3d-fb89d9f70742" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused"
Oct 30 04:04:53.890122 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:04:53.887298    4461 patch_prober.go:29] interesting pod/openshift-ai-inference-openshift-default-5b4f8cc59b-jtcmr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" start-of-body=
Oct 30 04:04:53.890122 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:04:53.887360    4461 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-jtcmr" podUID="aedb2e20-2e07-42b1-b3c2-27223dc8d177" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused"
Oct 30 04:04:54.539806 ip-10-0-143-156 kubenswrapper[4461]: E1030 04:04:54.539774    4461 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2k85b" containerName="extractor"
Oct 30 04:04:54.868180 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:04:54.867616    4461 patch_prober.go:29] interesting pod/openshift-ai-inference-openshift-default-5b4f8cc59b-nfxr8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" start-of-body=
Oct 30 04:04:54.868180 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:04:54.867684    4461 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-nfxr8" podUID="1dacbabe-8d44-4ebc-be3d-fb89d9f70742" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused"
Oct 30 04:04:54.888120 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:04:54.887500    4461 patch_prober.go:29] interesting pod/openshift-ai-inference-openshift-default-5b4f8cc59b-jtcmr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" start-of-body=
Oct 30 04:04:54.888120 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:04:54.887561    4461 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-jtcmr" podUID="aedb2e20-2e07-42b1-b3c2-27223dc8d177" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused"
Oct 30 04:04:55.823790 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:04:55.823741    4461 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-555dc7c784-rt68c" podUID="e2533e8e-70c7-4738-aaae-1bfd94190af0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8001/health\": dial tcp 10.133.0.48:8001: connect: connection refused"
Oct 30 04:04:55.874122 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:04:55.872517    4461 patch_prober.go:29] interesting pod/openshift-ai-inference-openshift-default-5b4f8cc59b-nfxr8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" start-of-body=
Oct 30 04:04:55.874122 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:04:55.872584    4461 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-nfxr8" podUID="1dacbabe-8d44-4ebc-be3d-fb89d9f70742" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused"
Oct 30 04:04:55.888121 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:04:55.886511    4461 patch_prober.go:29] interesting pod/openshift-ai-inference-openshift-default-5b4f8cc59b-jtcmr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" start-of-body=
Oct 30 04:04:55.888121 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:04:55.886577    4461 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-jtcmr" podUID="aedb2e20-2e07-42b1-b3c2-27223dc8d177" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused"
Oct 30 04:04:56.866552 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:04:56.866511    4461 patch_prober.go:29] interesting pod/openshift-ai-inference-openshift-default-5b4f8cc59b-nfxr8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" start-of-body=
Oct 30 04:04:56.866854 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:04:56.866827    4461 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-nfxr8" podUID="1dacbabe-8d44-4ebc-be3d-fb89d9f70742" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused"
Oct 30 04:04:56.888976 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:04:56.888938    4461 patch_prober.go:29] interesting pod/openshift-ai-inference-openshift-default-5b4f8cc59b-jtcmr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" start-of-body=
Oct 30 04:04:56.889607 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:04:56.889577    4461 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-jtcmr" podUID="aedb2e20-2e07-42b1-b3c2-27223dc8d177" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused"
Oct 30 04:04:57.867530 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:04:57.867491    4461 patch_prober.go:29] interesting pod/openshift-ai-inference-openshift-default-5b4f8cc59b-nfxr8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" start-of-body=
Oct 30 04:04:57.867831 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:04:57.867798    4461 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-nfxr8" podUID="1dacbabe-8d44-4ebc-be3d-fb89d9f70742" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused"
Oct 30 04:04:57.885474 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:04:57.885437    4461 patch_prober.go:29] interesting pod/openshift-ai-inference-openshift-default-5b4f8cc59b-jtcmr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" start-of-body=
Oct 30 04:04:57.885700 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:04:57.885673    4461 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-jtcmr" podUID="aedb2e20-2e07-42b1-b3c2-27223dc8d177" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused"
Oct 30 04:04:58.870531 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:04:58.870500    4461 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-nfxr8"
Oct 30 04:04:58.870959 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:04:58.870798    4461 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-nfxr8"
Oct 30 04:04:58.871445 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:04:58.871427    4461 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-nfxr8"
Oct 30 04:04:58.889566 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:04:58.889529    4461 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-jtcmr"
Oct 30 04:04:58.889827 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:04:58.889811    4461 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-jtcmr"
Oct 30 04:04:58.890526 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:04:58.890504    4461 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-jtcmr"
Oct 30 04:05:05.824560 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:05:05.824520    4461 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-555dc7c784-rt68c" podUID="e2533e8e-70c7-4738-aaae-1bfd94190af0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8001/health\": dial tcp 10.133.0.48:8001: connect: connection refused"
Oct 30 04:05:15.824140 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:05:15.824015    4461 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-555dc7c784-rt68c" podUID="e2533e8e-70c7-4738-aaae-1bfd94190af0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8001/health\": dial tcp 10.133.0.48:8001: connect: connection refused"
Oct 30 04:05:19.103913 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:05:19.103879    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 04:05:23.118312 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:05:23.118270    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 04:05:24.503989 ip-10-0-143-156 kubenswrapper[4461]: E1030 04:05:24.503952    4461 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2k85b" containerName="extractor"
Oct 30 04:05:25.824538 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:05:25.824496    4461 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-555dc7c784-rt68c" podUID="e2533e8e-70c7-4738-aaae-1bfd94190af0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8001/health\": dial tcp 10.133.0.48:8001: connect: connection refused"
Oct 30 04:05:29.279462 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:05:29.279422    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 04:05:35.824068 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:05:35.824015    4461 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-555dc7c784-rt68c" podUID="e2533e8e-70c7-4738-aaae-1bfd94190af0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8001/health\": dial tcp 10.133.0.48:8001: connect: connection refused"
Oct 30 04:05:45.824603 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:05:45.824557    4461 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-555dc7c784-rt68c" podUID="e2533e8e-70c7-4738-aaae-1bfd94190af0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8001/health\": dial tcp 10.133.0.48:8001: connect: connection refused"
Oct 30 04:05:49.597541 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:05:49.597502    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 04:05:54.504600 ip-10-0-143-156 kubenswrapper[4461]: E1030 04:05:54.504562    4461 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2k85b" containerName="extractor"
Oct 30 04:05:55.824413 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:05:55.824359    4461 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-555dc7c784-rt68c" podUID="e2533e8e-70c7-4738-aaae-1bfd94190af0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8001/health\": dial tcp 10.133.0.48:8001: connect: connection refused"
Oct 30 04:06:05.838594 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:06:05.838561    4461 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-555dc7c784-rt68c"
Oct 30 04:06:05.850670 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:06:05.850648    4461 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-555dc7c784-rt68c"
Oct 30 04:06:19.240347 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:06:19.240314    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 04:06:24.503463 ip-10-0-143-156 kubenswrapper[4461]: E1030 04:06:24.503427    4461 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2k85b" containerName="extractor"
Oct 30 04:06:29.416662 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:06:29.416623    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 04:06:35.358359 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:06:35.358326    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 04:06:35.528008 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:06:35.527977    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 04:06:39.678033 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:06:39.677998    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 04:06:50.697522 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:06:50.697435    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 04:06:54.503487 ip-10-0-143-156 kubenswrapper[4461]: E1030 04:06:54.503444    4461 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2k85b" containerName="extractor"
Oct 30 04:06:56.419563 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:06:56.419533    4461 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-555dc7c784-rt68c"]
Oct 30 04:06:56.440176 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:06:56.440140    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 04:06:56.639433 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:06:56.639399    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 04:06:59.994910 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:06:59.994874    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 04:07:00.164555 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:07:00.164518    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 04:07:00.335517 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:07:00.335438    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 04:07:00.505283 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:07:00.505255    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 04:07:00.751752 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:07:00.751717    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 04:07:00.922002 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:07:00.921960    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 04:07:02.984220 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:07:02.984187    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 04:07:03.152001 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:07:03.151967    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 04:07:03.827799 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:07:03.827758    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 04:07:05.227569 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:07:05.227531    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 04:07:05.408825 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:07:05.408789    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 04:07:06.898131 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:07:06.898078    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 04:07:11.446020 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:07:11.445984    4461 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-555dc7c784-rt68c" podUID="e2533e8e-70c7-4738-aaae-1bfd94190af0" containerName="main" containerID="cri-o://d2f99dbb242e9e21550ebbb4eecac4b6f264d6182be15bbdd176ac5b34dbaeb0" gracePeriod=45
Oct 30 04:07:12.062612 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:07:12.062583    4461 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-555dc7c784-rt68c" podUID="e2533e8e-70c7-4738-aaae-1bfd94190af0" containerName="llm-d-routing-sidecar" containerID="cri-o://10958607c49c2127d6e5d828bafcdaea063f772f75f4b99bf47f72c3b9bf01d7" gracePeriod=45
Oct 30 04:07:12.298083 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:07:12.298057    4461 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-555dc7c784-rt68c"
Oct 30 04:07:12.445671 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:07:12.445633    4461 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e2533e8e-70c7-4738-aaae-1bfd94190af0-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e2533e8e-70c7-4738-aaae-1bfd94190af0-model-cache\") pod \"e2533e8e-70c7-4738-aaae-1bfd94190af0\" (UID: \"e2533e8e-70c7-4738-aaae-1bfd94190af0\") "
Oct 30 04:07:12.445838 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:07:12.445723    4461 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e2533e8e-70c7-4738-aaae-1bfd94190af0-kube-api-access-6g2ml\" (UniqueName: \"kubernetes.io/projected/e2533e8e-70c7-4738-aaae-1bfd94190af0-kube-api-access-6g2ml\") pod \"e2533e8e-70c7-4738-aaae-1bfd94190af0\" (UID: \"e2533e8e-70c7-4738-aaae-1bfd94190af0\") "
Oct 30 04:07:12.445838 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:07:12.445747    4461 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e2533e8e-70c7-4738-aaae-1bfd94190af0-home\" (UniqueName: \"kubernetes.io/empty-dir/e2533e8e-70c7-4738-aaae-1bfd94190af0-home\") pod \"e2533e8e-70c7-4738-aaae-1bfd94190af0\" (UID: \"e2533e8e-70c7-4738-aaae-1bfd94190af0\") "
Oct 30 04:07:12.445838 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:07:12.445782    4461 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e2533e8e-70c7-4738-aaae-1bfd94190af0-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e2533e8e-70c7-4738-aaae-1bfd94190af0-tmp-dir\") pod \"e2533e8e-70c7-4738-aaae-1bfd94190af0\" (UID: \"e2533e8e-70c7-4738-aaae-1bfd94190af0\") "
Oct 30 04:07:12.445838 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:07:12.445803    4461 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e2533e8e-70c7-4738-aaae-1bfd94190af0-dshm\" (UniqueName: \"kubernetes.io/empty-dir/e2533e8e-70c7-4738-aaae-1bfd94190af0-dshm\") pod \"e2533e8e-70c7-4738-aaae-1bfd94190af0\" (UID: \"e2533e8e-70c7-4738-aaae-1bfd94190af0\") "
Oct 30 04:07:12.445838 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:07:12.445821    4461 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e2533e8e-70c7-4738-aaae-1bfd94190af0-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e2533e8e-70c7-4738-aaae-1bfd94190af0-cabundle-cert\") pod \"e2533e8e-70c7-4738-aaae-1bfd94190af0\" (UID: \"e2533e8e-70c7-4738-aaae-1bfd94190af0\") "
Oct 30 04:07:12.446430 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:07:12.445895    4461 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e2533e8e-70c7-4738-aaae-1bfd94190af0-tls-certs\" (UniqueName: \"kubernetes.io/secret/e2533e8e-70c7-4738-aaae-1bfd94190af0-tls-certs\") pod \"e2533e8e-70c7-4738-aaae-1bfd94190af0\" (UID: \"e2533e8e-70c7-4738-aaae-1bfd94190af0\") "
Oct 30 04:07:12.446430 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:07:12.445933    4461 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e2533e8e-70c7-4738-aaae-1bfd94190af0-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e2533e8e-70c7-4738-aaae-1bfd94190af0-kserve-provision-location\") pod \"e2533e8e-70c7-4738-aaae-1bfd94190af0\" (UID: \"e2533e8e-70c7-4738-aaae-1bfd94190af0\") "
Oct 30 04:07:12.446430 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:07:12.445960    4461 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e2533e8e-70c7-4738-aaae-1bfd94190af0-model-cache" pod "e2533e8e-70c7-4738-aaae-1bfd94190af0" (UID: "e2533e8e-70c7-4738-aaae-1bfd94190af0"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:07:12.446430 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:07:12.446226    4461 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e2533e8e-70c7-4738-aaae-1bfd94190af0-model-cache\") on node \"ip-10-0-143-156.ec2.internal\" DevicePath \"\""
Oct 30 04:07:12.446430 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:07:12.446324    4461 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e2533e8e-70c7-4738-aaae-1bfd94190af0-cabundle-cert" pod "e2533e8e-70c7-4738-aaae-1bfd94190af0" (UID: "e2533e8e-70c7-4738-aaae-1bfd94190af0"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:07:12.446823 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:07:12.446797    4461 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e2533e8e-70c7-4738-aaae-1bfd94190af0-home" pod "e2533e8e-70c7-4738-aaae-1bfd94190af0" (UID: "e2533e8e-70c7-4738-aaae-1bfd94190af0"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:07:12.448434 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:07:12.448398    4461 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e2533e8e-70c7-4738-aaae-1bfd94190af0-dshm" pod "e2533e8e-70c7-4738-aaae-1bfd94190af0" (UID: "e2533e8e-70c7-4738-aaae-1bfd94190af0"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:07:12.448434 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:07:12.448417    4461 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e2533e8e-70c7-4738-aaae-1bfd94190af0-kube-api-access-6g2ml" pod "e2533e8e-70c7-4738-aaae-1bfd94190af0" (UID: "e2533e8e-70c7-4738-aaae-1bfd94190af0"). InnerVolumeSpecName "kube-api-access-6g2ml". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:07:12.448568 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:07:12.448492    4461 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e2533e8e-70c7-4738-aaae-1bfd94190af0-tls-certs" pod "e2533e8e-70c7-4738-aaae-1bfd94190af0" (UID: "e2533e8e-70c7-4738-aaae-1bfd94190af0"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:07:12.458689 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:07:12.458652    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 04:07:12.458910 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:07:12.458887    4461 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e2533e8e-70c7-4738-aaae-1bfd94190af0-tmp-dir" pod "e2533e8e-70c7-4738-aaae-1bfd94190af0" (UID: "e2533e8e-70c7-4738-aaae-1bfd94190af0"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:07:12.471592 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:07:12.471563    4461 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e2533e8e-70c7-4738-aaae-1bfd94190af0-kserve-provision-location" pod "e2533e8e-70c7-4738-aaae-1bfd94190af0" (UID: "e2533e8e-70c7-4738-aaae-1bfd94190af0"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:07:12.547350 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:07:12.547267    4461 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6g2ml\" (UniqueName: \"kubernetes.io/projected/e2533e8e-70c7-4738-aaae-1bfd94190af0-kube-api-access-6g2ml\") on node \"ip-10-0-143-156.ec2.internal\" DevicePath \"\""
Oct 30 04:07:12.547350 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:07:12.547295    4461 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e2533e8e-70c7-4738-aaae-1bfd94190af0-home\") on node \"ip-10-0-143-156.ec2.internal\" DevicePath \"\""
Oct 30 04:07:12.547350 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:07:12.547308    4461 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e2533e8e-70c7-4738-aaae-1bfd94190af0-tmp-dir\") on node \"ip-10-0-143-156.ec2.internal\" DevicePath \"\""
Oct 30 04:07:12.547350 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:07:12.547319    4461 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e2533e8e-70c7-4738-aaae-1bfd94190af0-dshm\") on node \"ip-10-0-143-156.ec2.internal\" DevicePath \"\""
Oct 30 04:07:12.547350 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:07:12.547329    4461 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e2533e8e-70c7-4738-aaae-1bfd94190af0-cabundle-cert\") on node \"ip-10-0-143-156.ec2.internal\" DevicePath \"\""
Oct 30 04:07:12.547350 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:07:12.547340    4461 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e2533e8e-70c7-4738-aaae-1bfd94190af0-tls-certs\") on node \"ip-10-0-143-156.ec2.internal\" DevicePath \"\""
Oct 30 04:07:12.547350 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:07:12.547352    4461 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e2533e8e-70c7-4738-aaae-1bfd94190af0-kserve-provision-location\") on node \"ip-10-0-143-156.ec2.internal\" DevicePath \"\""
Oct 30 04:07:12.622440 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:07:12.622405    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 04:07:12.942517 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:07:12.942434    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 04:07:12.970470 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:07:12.970439    4461 generic.go:356] "Generic (PLEG): container finished" podID="e2533e8e-70c7-4738-aaae-1bfd94190af0" containerID="d2f99dbb242e9e21550ebbb4eecac4b6f264d6182be15bbdd176ac5b34dbaeb0" exitCode=0
Oct 30 04:07:12.970470 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:07:12.970465    4461 generic.go:356] "Generic (PLEG): container finished" podID="e2533e8e-70c7-4738-aaae-1bfd94190af0" containerID="10958607c49c2127d6e5d828bafcdaea063f772f75f4b99bf47f72c3b9bf01d7" exitCode=0
Oct 30 04:07:12.970668 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:07:12.970495    4461 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-555dc7c784-rt68c"
Oct 30 04:07:12.970668 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:07:12.970526    4461 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-555dc7c784-rt68c" event={"ID":"e2533e8e-70c7-4738-aaae-1bfd94190af0","Type":"ContainerDied","Data":"d2f99dbb242e9e21550ebbb4eecac4b6f264d6182be15bbdd176ac5b34dbaeb0"}
Oct 30 04:07:12.970668 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:07:12.970585    4461 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-555dc7c784-rt68c" event={"ID":"e2533e8e-70c7-4738-aaae-1bfd94190af0","Type":"ContainerDied","Data":"10958607c49c2127d6e5d828bafcdaea063f772f75f4b99bf47f72c3b9bf01d7"}
Oct 30 04:07:12.970668 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:07:12.970612    4461 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-555dc7c784-rt68c" event={"ID":"e2533e8e-70c7-4738-aaae-1bfd94190af0","Type":"ContainerDied","Data":"a29822eb0803d33be56babd370ccc4613b59306f6c4cbf82d91ee8d49efd09d6"}
Oct 30 04:07:12.970668 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:07:12.970642    4461 scope.go:122] "RemoveContainer" containerID="d2f99dbb242e9e21550ebbb4eecac4b6f264d6182be15bbdd176ac5b34dbaeb0"
Oct 30 04:07:12.978934 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:07:12.978919    4461 scope.go:122] "RemoveContainer" containerID="3dd86c14c3b731c0226b07a5150bf50f7d091fa1662122784b60caff09ff4d40"
Oct 30 04:07:12.986128 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:07:12.986110    4461 scope.go:122] "RemoveContainer" containerID="10958607c49c2127d6e5d828bafcdaea063f772f75f4b99bf47f72c3b9bf01d7"
Oct 30 04:07:12.992881 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:07:12.992862    4461 scope.go:122] "RemoveContainer" containerID="d2f99dbb242e9e21550ebbb4eecac4b6f264d6182be15bbdd176ac5b34dbaeb0"
Oct 30 04:07:12.993166 ip-10-0-143-156 kubenswrapper[4461]: E1030 04:07:12.993127    4461 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d2f99dbb242e9e21550ebbb4eecac4b6f264d6182be15bbdd176ac5b34dbaeb0\": container with ID starting with d2f99dbb242e9e21550ebbb4eecac4b6f264d6182be15bbdd176ac5b34dbaeb0 not found: ID does not exist" containerID="d2f99dbb242e9e21550ebbb4eecac4b6f264d6182be15bbdd176ac5b34dbaeb0"
Oct 30 04:07:12.993255 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:07:12.993178    4461 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d2f99dbb242e9e21550ebbb4eecac4b6f264d6182be15bbdd176ac5b34dbaeb0"} err="failed to get container status \"d2f99dbb242e9e21550ebbb4eecac4b6f264d6182be15bbdd176ac5b34dbaeb0\": rpc error: code = NotFound desc = could not find container \"d2f99dbb242e9e21550ebbb4eecac4b6f264d6182be15bbdd176ac5b34dbaeb0\": container with ID starting with d2f99dbb242e9e21550ebbb4eecac4b6f264d6182be15bbdd176ac5b34dbaeb0 not found: ID does not exist"
Oct 30 04:07:12.993255 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:07:12.993204    4461 scope.go:122] "RemoveContainer" containerID="3dd86c14c3b731c0226b07a5150bf50f7d091fa1662122784b60caff09ff4d40"
Oct 30 04:07:12.993461 ip-10-0-143-156 kubenswrapper[4461]: E1030 04:07:12.993442    4461 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3dd86c14c3b731c0226b07a5150bf50f7d091fa1662122784b60caff09ff4d40\": container with ID starting with 3dd86c14c3b731c0226b07a5150bf50f7d091fa1662122784b60caff09ff4d40 not found: ID does not exist" containerID="3dd86c14c3b731c0226b07a5150bf50f7d091fa1662122784b60caff09ff4d40"
Oct 30 04:07:12.993507 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:07:12.993469    4461 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3dd86c14c3b731c0226b07a5150bf50f7d091fa1662122784b60caff09ff4d40"} err="failed to get container status \"3dd86c14c3b731c0226b07a5150bf50f7d091fa1662122784b60caff09ff4d40\": rpc error: code = NotFound desc = could not find container \"3dd86c14c3b731c0226b07a5150bf50f7d091fa1662122784b60caff09ff4d40\": container with ID starting with 3dd86c14c3b731c0226b07a5150bf50f7d091fa1662122784b60caff09ff4d40 not found: ID does not exist"
Oct 30 04:07:12.993507 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:07:12.993487    4461 scope.go:122] "RemoveContainer" containerID="10958607c49c2127d6e5d828bafcdaea063f772f75f4b99bf47f72c3b9bf01d7"
Oct 30 04:07:12.993713 ip-10-0-143-156 kubenswrapper[4461]: E1030 04:07:12.993695    4461 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"10958607c49c2127d6e5d828bafcdaea063f772f75f4b99bf47f72c3b9bf01d7\": container with ID starting with 10958607c49c2127d6e5d828bafcdaea063f772f75f4b99bf47f72c3b9bf01d7 not found: ID does not exist" containerID="10958607c49c2127d6e5d828bafcdaea063f772f75f4b99bf47f72c3b9bf01d7"
Oct 30 04:07:12.993780 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:07:12.993723    4461 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"10958607c49c2127d6e5d828bafcdaea063f772f75f4b99bf47f72c3b9bf01d7"} err="failed to get container status \"10958607c49c2127d6e5d828bafcdaea063f772f75f4b99bf47f72c3b9bf01d7\": rpc error: code = NotFound desc = could not find container \"10958607c49c2127d6e5d828bafcdaea063f772f75f4b99bf47f72c3b9bf01d7\": container with ID starting with 10958607c49c2127d6e5d828bafcdaea063f772f75f4b99bf47f72c3b9bf01d7 not found: ID does not exist"
Oct 30 04:07:12.993780 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:07:12.993742    4461 scope.go:122] "RemoveContainer" containerID="d2f99dbb242e9e21550ebbb4eecac4b6f264d6182be15bbdd176ac5b34dbaeb0"
Oct 30 04:07:12.993966 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:07:12.993950    4461 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d2f99dbb242e9e21550ebbb4eecac4b6f264d6182be15bbdd176ac5b34dbaeb0"} err="failed to get container status \"d2f99dbb242e9e21550ebbb4eecac4b6f264d6182be15bbdd176ac5b34dbaeb0\": rpc error: code = NotFound desc = could not find container \"d2f99dbb242e9e21550ebbb4eecac4b6f264d6182be15bbdd176ac5b34dbaeb0\": container with ID starting with d2f99dbb242e9e21550ebbb4eecac4b6f264d6182be15bbdd176ac5b34dbaeb0 not found: ID does not exist"
Oct 30 04:07:12.994017 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:07:12.993966    4461 scope.go:122] "RemoveContainer" containerID="3dd86c14c3b731c0226b07a5150bf50f7d091fa1662122784b60caff09ff4d40"
Oct 30 04:07:12.994207 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:07:12.994184    4461 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3dd86c14c3b731c0226b07a5150bf50f7d091fa1662122784b60caff09ff4d40"} err="failed to get container status \"3dd86c14c3b731c0226b07a5150bf50f7d091fa1662122784b60caff09ff4d40\": rpc error: code = NotFound desc = could not find container \"3dd86c14c3b731c0226b07a5150bf50f7d091fa1662122784b60caff09ff4d40\": container with ID starting with 3dd86c14c3b731c0226b07a5150bf50f7d091fa1662122784b60caff09ff4d40 not found: ID does not exist"
Oct 30 04:07:12.994207 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:07:12.994207    4461 scope.go:122] "RemoveContainer" containerID="10958607c49c2127d6e5d828bafcdaea063f772f75f4b99bf47f72c3b9bf01d7"
Oct 30 04:07:12.994462 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:07:12.994445    4461 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"10958607c49c2127d6e5d828bafcdaea063f772f75f4b99bf47f72c3b9bf01d7"} err="failed to get container status \"10958607c49c2127d6e5d828bafcdaea063f772f75f4b99bf47f72c3b9bf01d7\": rpc error: code = NotFound desc = could not find container \"10958607c49c2127d6e5d828bafcdaea063f772f75f4b99bf47f72c3b9bf01d7\": container with ID starting with 10958607c49c2127d6e5d828bafcdaea063f772f75f4b99bf47f72c3b9bf01d7 not found: ID does not exist"
Oct 30 04:07:13.036929 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:07:13.036899    4461 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-555dc7c784-rt68c"]
Oct 30 04:07:13.054706 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:07:13.049831    4461 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-555dc7c784-rt68c"]
Oct 30 04:07:13.110587 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:07:13.110553    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 04:07:14.115806 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:07:14.115772    4461 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e2533e8e-70c7-4738-aaae-1bfd94190af0" path="/var/lib/kubelet/pods/e2533e8e-70c7-4738-aaae-1bfd94190af0/volumes"
Oct 30 04:07:19.328408 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:07:19.328367    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 04:07:24.503239 ip-10-0-143-156 kubenswrapper[4461]: E1030 04:07:24.503204    4461 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2k85b" containerName="extractor"
Oct 30 04:07:26.636993 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:07:26.636953    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 04:07:29.612714 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:07:29.612672    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 04:07:37.015730 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:07:37.015691    4461 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-v4p2m"]
Oct 30 04:07:37.016210 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:07:37.016033    4461 cpu_manager.go:414] "containerMap: removing container" podUID="e2533e8e-70c7-4738-aaae-1bfd94190af0" containerName="storage-initializer"
Oct 30 04:07:37.016210 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:07:37.016046    4461 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e2533e8e-70c7-4738-aaae-1bfd94190af0" containerName="storage-initializer"
Oct 30 04:07:37.016210 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:07:37.016058    4461 cpu_manager.go:414] "containerMap: removing container" podUID="e2533e8e-70c7-4738-aaae-1bfd94190af0" containerName="llm-d-routing-sidecar"
Oct 30 04:07:37.016210 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:07:37.016063    4461 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e2533e8e-70c7-4738-aaae-1bfd94190af0" containerName="llm-d-routing-sidecar"
Oct 30 04:07:37.016210 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:07:37.016090    4461 cpu_manager.go:414] "containerMap: removing container" podUID="e2533e8e-70c7-4738-aaae-1bfd94190af0" containerName="main"
Oct 30 04:07:37.016210 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:07:37.016111    4461 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e2533e8e-70c7-4738-aaae-1bfd94190af0" containerName="main"
Oct 30 04:07:37.016210 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:07:37.016164    4461 memory_manager.go:365] "RemoveStaleState removing state" podUID="e2533e8e-70c7-4738-aaae-1bfd94190af0" containerName="main"
Oct 30 04:07:37.016210 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:07:37.016173    4461 memory_manager.go:365] "RemoveStaleState removing state" podUID="e2533e8e-70c7-4738-aaae-1bfd94190af0" containerName="llm-d-routing-sidecar"
Oct 30 04:07:37.020758 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:07:37.020740    4461 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-v4p2m"
Oct 30 04:07:37.051340 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:07:37.051311    4461 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-v4p2m"]
Oct 30 04:07:37.056219 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:07:37.056192    4461 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-rh6p2"]
Oct 30 04:07:37.059611 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:07:37.059580    4461 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-rh6p2"
Oct 30 04:07:37.075385 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:07:37.075357    4461 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7c55427d-9f1e-4b24-9741-3ff0cfe751e7-istio-podinfo\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-v4p2m\" (UID: \"7c55427d-9f1e-4b24-9741-3ff0cfe751e7\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-v4p2m"
Oct 30 04:07:37.075497 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:07:37.075412    4461 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c55427d-9f1e-4b24-9741-3ff0cfe751e7-credential-socket\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-v4p2m\" (UID: \"7c55427d-9f1e-4b24-9741-3ff0cfe751e7\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-v4p2m"
Oct 30 04:07:37.075542 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:07:37.075490    4461 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7c55427d-9f1e-4b24-9741-3ff0cfe751e7-istio-envoy\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-v4p2m\" (UID: \"7c55427d-9f1e-4b24-9741-3ff0cfe751e7\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-v4p2m"
Oct 30 04:07:37.075542 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:07:37.075527    4461 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7c55427d-9f1e-4b24-9741-3ff0cfe751e7-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-v4p2m\" (UID: \"7c55427d-9f1e-4b24-9741-3ff0cfe751e7\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-v4p2m"
Oct 30 04:07:37.075648 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:07:37.075547    4461 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fbvb7\" (UniqueName: \"kubernetes.io/projected/7c55427d-9f1e-4b24-9741-3ff0cfe751e7-kube-api-access-fbvb7\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-v4p2m\" (UID: \"7c55427d-9f1e-4b24-9741-3ff0cfe751e7\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-v4p2m"
Oct 30 04:07:37.075705 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:07:37.075669    4461 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7c55427d-9f1e-4b24-9741-3ff0cfe751e7-istio-data\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-v4p2m\" (UID: \"7c55427d-9f1e-4b24-9741-3ff0cfe751e7\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-v4p2m"
Oct 30 04:07:37.075705 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:07:37.075699    4461 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7c55427d-9f1e-4b24-9741-3ff0cfe751e7-workload-certs\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-v4p2m\" (UID: \"7c55427d-9f1e-4b24-9741-3ff0cfe751e7\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-v4p2m"
Oct 30 04:07:37.075806 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:07:37.075718    4461 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7c55427d-9f1e-4b24-9741-3ff0cfe751e7-istio-token\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-v4p2m\" (UID: \"7c55427d-9f1e-4b24-9741-3ff0cfe751e7\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-v4p2m"
Oct 30 04:07:37.075851 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:07:37.075826    4461 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c55427d-9f1e-4b24-9741-3ff0cfe751e7-workload-socket\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-v4p2m\" (UID: \"7c55427d-9f1e-4b24-9741-3ff0cfe751e7\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-v4p2m"
Oct 30 04:07:37.083993 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:07:37.083965    4461 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-rh6p2"]
Oct 30 04:07:37.176511 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:07:37.176475    4461 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7c55427d-9f1e-4b24-9741-3ff0cfe751e7-istio-data\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-v4p2m\" (UID: \"7c55427d-9f1e-4b24-9741-3ff0cfe751e7\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-v4p2m"
Oct 30 04:07:37.176511 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:07:37.176520    4461 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7c55427d-9f1e-4b24-9741-3ff0cfe751e7-workload-certs\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-v4p2m\" (UID: \"7c55427d-9f1e-4b24-9741-3ff0cfe751e7\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-v4p2m"
Oct 30 04:07:37.176792 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:07:37.176550    4461 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7c55427d-9f1e-4b24-9741-3ff0cfe751e7-istio-token\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-v4p2m\" (UID: \"7c55427d-9f1e-4b24-9741-3ff0cfe751e7\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-v4p2m"
Oct 30 04:07:37.176792 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:07:37.176576    4461 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-662gk\" (UniqueName: \"kubernetes.io/projected/19942a3a-cdfa-4618-83b3-f828b5cbe33e-kube-api-access-662gk\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-rh6p2\" (UID: \"19942a3a-cdfa-4618-83b3-f828b5cbe33e\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-rh6p2"
Oct 30 04:07:37.176792 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:07:37.176618    4461 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c55427d-9f1e-4b24-9741-3ff0cfe751e7-workload-socket\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-v4p2m\" (UID: \"7c55427d-9f1e-4b24-9741-3ff0cfe751e7\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-v4p2m"
Oct 30 04:07:37.176792 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:07:37.176652    4461 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7c55427d-9f1e-4b24-9741-3ff0cfe751e7-istio-podinfo\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-v4p2m\" (UID: \"7c55427d-9f1e-4b24-9741-3ff0cfe751e7\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-v4p2m"
Oct 30 04:07:37.176792 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:07:37.176722    4461 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/19942a3a-cdfa-4618-83b3-f828b5cbe33e-istio-data\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-rh6p2\" (UID: \"19942a3a-cdfa-4618-83b3-f828b5cbe33e\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-rh6p2"
Oct 30 04:07:37.177083 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:07:37.176929    4461 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/19942a3a-cdfa-4618-83b3-f828b5cbe33e-workload-certs\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-rh6p2\" (UID: \"19942a3a-cdfa-4618-83b3-f828b5cbe33e\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-rh6p2"
Oct 30 04:07:37.177083 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:07:37.176933    4461 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7c55427d-9f1e-4b24-9741-3ff0cfe751e7-istio-data\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-v4p2m\" (UID: \"7c55427d-9f1e-4b24-9741-3ff0cfe751e7\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-v4p2m"
Oct 30 04:07:37.177083 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:07:37.176984    4461 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/19942a3a-cdfa-4618-83b3-f828b5cbe33e-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-rh6p2\" (UID: \"19942a3a-cdfa-4618-83b3-f828b5cbe33e\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-rh6p2"
Oct 30 04:07:37.177083 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:07:37.176989    4461 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7c55427d-9f1e-4b24-9741-3ff0cfe751e7-workload-certs\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-v4p2m\" (UID: \"7c55427d-9f1e-4b24-9741-3ff0cfe751e7\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-v4p2m"
Oct 30 04:07:37.177432 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:07:37.177118    4461 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c55427d-9f1e-4b24-9741-3ff0cfe751e7-credential-socket\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-v4p2m\" (UID: \"7c55427d-9f1e-4b24-9741-3ff0cfe751e7\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-v4p2m"
Oct 30 04:07:37.177432 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:07:37.177137    4461 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c55427d-9f1e-4b24-9741-3ff0cfe751e7-workload-socket\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-v4p2m\" (UID: \"7c55427d-9f1e-4b24-9741-3ff0cfe751e7\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-v4p2m"
Oct 30 04:07:37.177432 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:07:37.177165    4461 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7c55427d-9f1e-4b24-9741-3ff0cfe751e7-istio-envoy\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-v4p2m\" (UID: \"7c55427d-9f1e-4b24-9741-3ff0cfe751e7\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-v4p2m"
Oct 30 04:07:37.177432 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:07:37.177191    4461 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/19942a3a-cdfa-4618-83b3-f828b5cbe33e-credential-socket\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-rh6p2\" (UID: \"19942a3a-cdfa-4618-83b3-f828b5cbe33e\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-rh6p2"
Oct 30 04:07:37.177432 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:07:37.177216    4461 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/19942a3a-cdfa-4618-83b3-f828b5cbe33e-istio-token\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-rh6p2\" (UID: \"19942a3a-cdfa-4618-83b3-f828b5cbe33e\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-rh6p2"
Oct 30 04:07:37.177432 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:07:37.177239    4461 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/19942a3a-cdfa-4618-83b3-f828b5cbe33e-workload-socket\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-rh6p2\" (UID: \"19942a3a-cdfa-4618-83b3-f828b5cbe33e\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-rh6p2"
Oct 30 04:07:37.177432 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:07:37.177398    4461 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7c55427d-9f1e-4b24-9741-3ff0cfe751e7-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-v4p2m\" (UID: \"7c55427d-9f1e-4b24-9741-3ff0cfe751e7\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-v4p2m"
Oct 30 04:07:37.177432 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:07:37.177406    4461 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c55427d-9f1e-4b24-9741-3ff0cfe751e7-credential-socket\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-v4p2m\" (UID: \"7c55427d-9f1e-4b24-9741-3ff0cfe751e7\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-v4p2m"
Oct 30 04:07:37.177793 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:07:37.177440    4461 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fbvb7\" (UniqueName: \"kubernetes.io/projected/7c55427d-9f1e-4b24-9741-3ff0cfe751e7-kube-api-access-fbvb7\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-v4p2m\" (UID: \"7c55427d-9f1e-4b24-9741-3ff0cfe751e7\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-v4p2m"
Oct 30 04:07:37.177793 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:07:37.177511    4461 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/19942a3a-cdfa-4618-83b3-f828b5cbe33e-istio-envoy\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-rh6p2\" (UID: \"19942a3a-cdfa-4618-83b3-f828b5cbe33e\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-rh6p2"
Oct 30 04:07:37.177793 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:07:37.177537    4461 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/19942a3a-cdfa-4618-83b3-f828b5cbe33e-istio-podinfo\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-rh6p2\" (UID: \"19942a3a-cdfa-4618-83b3-f828b5cbe33e\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-rh6p2"
Oct 30 04:07:37.177956 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:07:37.177936    4461 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7c55427d-9f1e-4b24-9741-3ff0cfe751e7-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-v4p2m\" (UID: \"7c55427d-9f1e-4b24-9741-3ff0cfe751e7\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-v4p2m"
Oct 30 04:07:37.179334 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:07:37.179312    4461 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7c55427d-9f1e-4b24-9741-3ff0cfe751e7-istio-envoy\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-v4p2m\" (UID: \"7c55427d-9f1e-4b24-9741-3ff0cfe751e7\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-v4p2m"
Oct 30 04:07:37.179461 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:07:37.179443    4461 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7c55427d-9f1e-4b24-9741-3ff0cfe751e7-istio-podinfo\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-v4p2m\" (UID: \"7c55427d-9f1e-4b24-9741-3ff0cfe751e7\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-v4p2m"
Oct 30 04:07:37.200084 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:07:37.200060    4461 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7c55427d-9f1e-4b24-9741-3ff0cfe751e7-istio-token\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-v4p2m\" (UID: \"7c55427d-9f1e-4b24-9741-3ff0cfe751e7\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-v4p2m"
Oct 30 04:07:37.200406 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:07:37.200388    4461 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fbvb7\" (UniqueName: \"kubernetes.io/projected/7c55427d-9f1e-4b24-9741-3ff0cfe751e7-kube-api-access-fbvb7\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-v4p2m\" (UID: \"7c55427d-9f1e-4b24-9741-3ff0cfe751e7\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-v4p2m"
Oct 30 04:07:37.278855 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:07:37.278775    4461 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-662gk\" (UniqueName: \"kubernetes.io/projected/19942a3a-cdfa-4618-83b3-f828b5cbe33e-kube-api-access-662gk\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-rh6p2\" (UID: \"19942a3a-cdfa-4618-83b3-f828b5cbe33e\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-rh6p2"
Oct 30 04:07:37.278855 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:07:37.278821    4461 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/19942a3a-cdfa-4618-83b3-f828b5cbe33e-istio-data\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-rh6p2\" (UID: \"19942a3a-cdfa-4618-83b3-f828b5cbe33e\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-rh6p2"
Oct 30 04:07:37.278855 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:07:37.278842    4461 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/19942a3a-cdfa-4618-83b3-f828b5cbe33e-workload-certs\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-rh6p2\" (UID: \"19942a3a-cdfa-4618-83b3-f828b5cbe33e\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-rh6p2"
Oct 30 04:07:37.279145 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:07:37.278954    4461 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/19942a3a-cdfa-4618-83b3-f828b5cbe33e-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-rh6p2\" (UID: \"19942a3a-cdfa-4618-83b3-f828b5cbe33e\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-rh6p2"
Oct 30 04:07:37.279145 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:07:37.279006    4461 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/19942a3a-cdfa-4618-83b3-f828b5cbe33e-credential-socket\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-rh6p2\" (UID: \"19942a3a-cdfa-4618-83b3-f828b5cbe33e\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-rh6p2"
Oct 30 04:07:37.279145 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:07:37.279031    4461 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/19942a3a-cdfa-4618-83b3-f828b5cbe33e-istio-token\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-rh6p2\" (UID: \"19942a3a-cdfa-4618-83b3-f828b5cbe33e\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-rh6p2"
Oct 30 04:07:37.279145 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:07:37.279056    4461 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/19942a3a-cdfa-4618-83b3-f828b5cbe33e-workload-socket\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-rh6p2\" (UID: \"19942a3a-cdfa-4618-83b3-f828b5cbe33e\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-rh6p2"
Oct 30 04:07:37.279145 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:07:37.279118    4461 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/19942a3a-cdfa-4618-83b3-f828b5cbe33e-istio-envoy\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-rh6p2\" (UID: \"19942a3a-cdfa-4618-83b3-f828b5cbe33e\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-rh6p2"
Oct 30 04:07:37.279392 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:07:37.279177    4461 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/19942a3a-cdfa-4618-83b3-f828b5cbe33e-istio-podinfo\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-rh6p2\" (UID: \"19942a3a-cdfa-4618-83b3-f828b5cbe33e\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-rh6p2"
Oct 30 04:07:37.279392 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:07:37.279194    4461 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/19942a3a-cdfa-4618-83b3-f828b5cbe33e-workload-certs\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-rh6p2\" (UID: \"19942a3a-cdfa-4618-83b3-f828b5cbe33e\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-rh6p2"
Oct 30 04:07:37.279497 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:07:37.279389    4461 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/19942a3a-cdfa-4618-83b3-f828b5cbe33e-istio-data\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-rh6p2\" (UID: \"19942a3a-cdfa-4618-83b3-f828b5cbe33e\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-rh6p2"
Oct 30 04:07:37.279497 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:07:37.279426    4461 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/19942a3a-cdfa-4618-83b3-f828b5cbe33e-workload-socket\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-rh6p2\" (UID: \"19942a3a-cdfa-4618-83b3-f828b5cbe33e\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-rh6p2"
Oct 30 04:07:37.279497 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:07:37.279449    4461 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/19942a3a-cdfa-4618-83b3-f828b5cbe33e-credential-socket\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-rh6p2\" (UID: \"19942a3a-cdfa-4618-83b3-f828b5cbe33e\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-rh6p2"
Oct 30 04:07:37.279668 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:07:37.279643    4461 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/19942a3a-cdfa-4618-83b3-f828b5cbe33e-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-rh6p2\" (UID: \"19942a3a-cdfa-4618-83b3-f828b5cbe33e\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-rh6p2"
Oct 30 04:07:37.281237 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:07:37.281217    4461 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/19942a3a-cdfa-4618-83b3-f828b5cbe33e-istio-envoy\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-rh6p2\" (UID: \"19942a3a-cdfa-4618-83b3-f828b5cbe33e\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-rh6p2"
Oct 30 04:07:37.281506 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:07:37.281488    4461 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/19942a3a-cdfa-4618-83b3-f828b5cbe33e-istio-podinfo\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-rh6p2\" (UID: \"19942a3a-cdfa-4618-83b3-f828b5cbe33e\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-rh6p2"
Oct 30 04:07:37.302857 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:07:37.302820    4461 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/19942a3a-cdfa-4618-83b3-f828b5cbe33e-istio-token\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-rh6p2\" (UID: \"19942a3a-cdfa-4618-83b3-f828b5cbe33e\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-rh6p2"
Oct 30 04:07:37.302985 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:07:37.302936    4461 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-662gk\" (UniqueName: \"kubernetes.io/projected/19942a3a-cdfa-4618-83b3-f828b5cbe33e-kube-api-access-662gk\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-rh6p2\" (UID: \"19942a3a-cdfa-4618-83b3-f828b5cbe33e\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-rh6p2"
Oct 30 04:07:37.333332 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:07:37.333295    4461 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-v4p2m"
Oct 30 04:07:37.372811 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:07:37.372778    4461 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-rh6p2"
Oct 30 04:07:37.645911 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:07:37.645889    4461 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-v4p2m"]
Oct 30 04:07:37.647402 ip-10-0-143-156 kubenswrapper[4461]: W1030 04:07:37.647376    4461 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c55427d_9f1e_4b24_9741_3ff0cfe751e7.slice/crio-5bc3cd36a3a9b3a3d6ea08adc25be559d6f022ffbc4fa816a58aef0213f785e0 WatchSource:0}: Error finding container 5bc3cd36a3a9b3a3d6ea08adc25be559d6f022ffbc4fa816a58aef0213f785e0: Status 404 returned error can't find the container with id 5bc3cd36a3a9b3a3d6ea08adc25be559d6f022ffbc4fa816a58aef0213f785e0
Oct 30 04:07:37.650961 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:07:37.650288    4461 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 04:07:37.650961 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:07:37.650375    4461 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 04:07:37.650961 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:07:37.650414    4461 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 04:07:37.684218 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:07:37.684193    4461 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-rh6p2"]
Oct 30 04:07:37.684658 ip-10-0-143-156 kubenswrapper[4461]: W1030 04:07:37.684638    4461 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod19942a3a_cdfa_4618_83b3_f828b5cbe33e.slice/crio-bc35bc00cb97cf6f82221f5765564b13c65bda6552a00ba546b6b68d4ba789df WatchSource:0}: Error finding container bc35bc00cb97cf6f82221f5765564b13c65bda6552a00ba546b6b68d4ba789df: Status 404 returned error can't find the container with id bc35bc00cb97cf6f82221f5765564b13c65bda6552a00ba546b6b68d4ba789df
Oct 30 04:07:37.687385 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:07:37.687303    4461 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 04:07:37.687385 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:07:37.687371    4461 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 04:07:37.687498 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:07:37.687408    4461 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 04:07:38.069179 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:07:38.069133    4461 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-rh6p2" event={"ID":"19942a3a-cdfa-4618-83b3-f828b5cbe33e","Type":"ContainerStarted","Data":"c01d0ed4e0a04302e11ad4948c310b7a78840708bfce0145303f3972da480ff9"}
Oct 30 04:07:38.069646 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:07:38.069205    4461 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-rh6p2" event={"ID":"19942a3a-cdfa-4618-83b3-f828b5cbe33e","Type":"ContainerStarted","Data":"bc35bc00cb97cf6f82221f5765564b13c65bda6552a00ba546b6b68d4ba789df"}
Oct 30 04:07:38.070884 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:07:38.070857    4461 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-v4p2m" event={"ID":"7c55427d-9f1e-4b24-9741-3ff0cfe751e7","Type":"ContainerStarted","Data":"a26c32ebcc68eacfe09420438374d0a20cb6f19f19151fadbd2cfa3ebc793af2"}
Oct 30 04:07:38.071004 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:07:38.070907    4461 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-v4p2m" event={"ID":"7c55427d-9f1e-4b24-9741-3ff0cfe751e7","Type":"ContainerStarted","Data":"5bc3cd36a3a9b3a3d6ea08adc25be559d6f022ffbc4fa816a58aef0213f785e0"}
Oct 30 04:07:38.116510 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:07:38.116453    4461 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-rh6p2" podStartSLOduration=1.116433708 podStartE2EDuration="1.116433708s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:07:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:07:38.115415211 +0000 UTC m=+4326.165220415" watchObservedRunningTime="2026-10-30 04:07:38.116433708 +0000 UTC m=+4326.166238884"
Oct 30 04:07:38.169122 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:07:38.168574    4461 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-v4p2m" podStartSLOduration=2.168553325 podStartE2EDuration="2.168553325s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:07:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:07:38.166645877 +0000 UTC m=+4326.216451170" watchObservedRunningTime="2026-10-30 04:07:38.168553325 +0000 UTC m=+4326.218358538"
Oct 30 04:07:38.333586 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:07:38.333494    4461 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-v4p2m"
Oct 30 04:07:38.334772 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:07:38.334750    4461 patch_prober.go:29] interesting pod/openshift-ai-inference-openshift-default-5b4f8cc59b-v4p2m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" start-of-body=
Oct 30 04:07:38.334840 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:07:38.334812    4461 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-v4p2m" podUID="7c55427d-9f1e-4b24-9741-3ff0cfe751e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused"
Oct 30 04:07:38.385122 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:07:38.379873    4461 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-rh6p2"
Oct 30 04:07:38.385122 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:07:38.380062    4461 patch_prober.go:29] interesting pod/openshift-ai-inference-openshift-default-5b4f8cc59b-rh6p2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" start-of-body=
Oct 30 04:07:38.385122 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:07:38.380130    4461 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-rh6p2" podUID="19942a3a-cdfa-4618-83b3-f828b5cbe33e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused"
Oct 30 04:07:39.334008 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:07:39.333970    4461 patch_prober.go:29] interesting pod/openshift-ai-inference-openshift-default-5b4f8cc59b-v4p2m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" start-of-body=
Oct 30 04:07:39.334485 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:07:39.334022    4461 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-v4p2m" podUID="7c55427d-9f1e-4b24-9741-3ff0cfe751e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused"
Oct 30 04:07:39.381728 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:07:39.381694    4461 patch_prober.go:29] interesting pod/openshift-ai-inference-openshift-default-5b4f8cc59b-rh6p2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" start-of-body=
Oct 30 04:07:39.382024 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:07:39.381998    4461 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-rh6p2" podUID="19942a3a-cdfa-4618-83b3-f828b5cbe33e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused"
Oct 30 04:07:40.334415 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:07:40.334379    4461 patch_prober.go:29] interesting pod/openshift-ai-inference-openshift-default-5b4f8cc59b-v4p2m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" start-of-body=
Oct 30 04:07:40.334810 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:07:40.334441    4461 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-v4p2m" podUID="7c55427d-9f1e-4b24-9741-3ff0cfe751e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused"
Oct 30 04:07:40.377551 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:07:40.377514    4461 patch_prober.go:29] interesting pod/openshift-ai-inference-openshift-default-5b4f8cc59b-rh6p2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" start-of-body=
Oct 30 04:07:40.377829 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:07:40.377798    4461 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-rh6p2" podUID="19942a3a-cdfa-4618-83b3-f828b5cbe33e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused"
Oct 30 04:07:41.333890 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:07:41.333854    4461 patch_prober.go:29] interesting pod/openshift-ai-inference-openshift-default-5b4f8cc59b-v4p2m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" start-of-body=
Oct 30 04:07:41.334047 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:07:41.333912    4461 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-v4p2m" podUID="7c55427d-9f1e-4b24-9741-3ff0cfe751e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused"
Oct 30 04:07:41.375135 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:07:41.374772    4461 patch_prober.go:29] interesting pod/openshift-ai-inference-openshift-default-5b4f8cc59b-rh6p2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" start-of-body=
Oct 30 04:07:41.375135 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:07:41.374831    4461 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-rh6p2" podUID="19942a3a-cdfa-4618-83b3-f828b5cbe33e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused"
Oct 30 04:07:42.334661 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:07:42.334625    4461 patch_prober.go:29] interesting pod/openshift-ai-inference-openshift-default-5b4f8cc59b-v4p2m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" start-of-body=
Oct 30 04:07:42.334835 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:07:42.334674    4461 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-v4p2m" podUID="7c55427d-9f1e-4b24-9741-3ff0cfe751e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused"
Oct 30 04:07:42.383774 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:07:42.383738    4461 patch_prober.go:29] interesting pod/openshift-ai-inference-openshift-default-5b4f8cc59b-rh6p2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" start-of-body=
Oct 30 04:07:42.384412 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:07:42.384380    4461 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-rh6p2" podUID="19942a3a-cdfa-4618-83b3-f828b5cbe33e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused"
Oct 30 04:07:43.334559 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:07:43.334527    4461 patch_prober.go:29] interesting pod/openshift-ai-inference-openshift-default-5b4f8cc59b-v4p2m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" start-of-body=
Oct 30 04:07:43.334840 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:07:43.334575    4461 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-v4p2m" podUID="7c55427d-9f1e-4b24-9741-3ff0cfe751e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused"
Oct 30 04:07:43.375702 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:07:43.375664    4461 patch_prober.go:29] interesting pod/openshift-ai-inference-openshift-default-5b4f8cc59b-rh6p2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" start-of-body=
Oct 30 04:07:43.375980 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:07:43.375950    4461 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-rh6p2" podUID="19942a3a-cdfa-4618-83b3-f828b5cbe33e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused"
Oct 30 04:07:44.337712 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:07:44.337678    4461 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-v4p2m"
Oct 30 04:07:44.338082 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:07:44.337950    4461 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-v4p2m"
Oct 30 04:07:44.338698 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:07:44.338682    4461 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-v4p2m"
Oct 30 04:07:44.377278 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:07:44.377246    4461 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-rh6p2"
Oct 30 04:07:44.377536 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:07:44.377519    4461 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-rh6p2"
Oct 30 04:07:44.378276 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:07:44.378255    4461 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-rh6p2"
Oct 30 04:07:46.705601 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:07:46.705562    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 04:07:46.877698 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:07:46.877651    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 04:07:49.879607 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:07:49.879556    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 04:07:51.746648 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:07:51.746617    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 04:07:54.131501 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:07:54.131473    4461 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7bf6cbddc7jq7tr"]
Oct 30 04:07:54.134636 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:07:54.134619    4461 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7bf6cbddc7jq7tr"
Oct 30 04:07:54.150136 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:07:54.150086    4461 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-fac1c4ae\"/\"odh-kserve-custom-ca-bundle\""
Oct 30 04:07:54.150136 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:07:54.150123    4461 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-fac1c4ae\"/\"kube-root-ca.crt\""
Oct 30 04:07:54.162745 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:07:54.162625    4461 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-fac1c4ae\"/\"seaweedfs-s3-creds\""
Oct 30 04:07:54.162745 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:07:54.162638    4461 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-fac1c4ae\"/\"default-dockercfg-n99sg\""
Oct 30 04:07:54.162745 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:07:54.162692    4461 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-fac1c4ae\"/\"llmisvd10b93c8eba12f87c3a350a5cae2ee0c-kserve-self-signed-certs\""
Oct 30 04:07:54.162745 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:07:54.162701    4461 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-fac1c4ae\"/\"openshift-service-ca.crt\""
Oct 30 04:07:54.163050 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:07:54.162872    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 04:07:54.165289 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:07:54.165264    4461 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7bf6cbddc7jq7tr"]
Oct 30 04:07:54.251227 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:07:54.251184    4461 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/66a2bf81-b5e1-47a7-b3c2-0acca369b832-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-7bf6cbddc7jq7tr\" (UID: \"66a2bf81-b5e1-47a7-b3c2-0acca369b832\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7bf6cbddc7jq7tr"
Oct 30 04:07:54.251227 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:07:54.251223    4461 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/66a2bf81-b5e1-47a7-b3c2-0acca369b832-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-7bf6cbddc7jq7tr\" (UID: \"66a2bf81-b5e1-47a7-b3c2-0acca369b832\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7bf6cbddc7jq7tr"
Oct 30 04:07:54.251467 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:07:54.251242    4461 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9x27q\" (UniqueName: \"kubernetes.io/projected/66a2bf81-b5e1-47a7-b3c2-0acca369b832-kube-api-access-9x27q\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-7bf6cbddc7jq7tr\" (UID: \"66a2bf81-b5e1-47a7-b3c2-0acca369b832\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7bf6cbddc7jq7tr"
Oct 30 04:07:54.251467 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:07:54.251375    4461 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/66a2bf81-b5e1-47a7-b3c2-0acca369b832-dshm\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-7bf6cbddc7jq7tr\" (UID: \"66a2bf81-b5e1-47a7-b3c2-0acca369b832\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7bf6cbddc7jq7tr"
Oct 30 04:07:54.251467 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:07:54.251416    4461 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/66a2bf81-b5e1-47a7-b3c2-0acca369b832-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-7bf6cbddc7jq7tr\" (UID: \"66a2bf81-b5e1-47a7-b3c2-0acca369b832\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7bf6cbddc7jq7tr"
Oct 30 04:07:54.251467 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:07:54.251435    4461 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/66a2bf81-b5e1-47a7-b3c2-0acca369b832-home\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-7bf6cbddc7jq7tr\" (UID: \"66a2bf81-b5e1-47a7-b3c2-0acca369b832\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7bf6cbddc7jq7tr"
Oct 30 04:07:54.251639 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:07:54.251517    4461 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/66a2bf81-b5e1-47a7-b3c2-0acca369b832-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-7bf6cbddc7jq7tr\" (UID: \"66a2bf81-b5e1-47a7-b3c2-0acca369b832\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7bf6cbddc7jq7tr"
Oct 30 04:07:54.251639 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:07:54.251544    4461 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/66a2bf81-b5e1-47a7-b3c2-0acca369b832-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-7bf6cbddc7jq7tr\" (UID: \"66a2bf81-b5e1-47a7-b3c2-0acca369b832\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7bf6cbddc7jq7tr"
Oct 30 04:07:54.352830 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:07:54.352795    4461 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/66a2bf81-b5e1-47a7-b3c2-0acca369b832-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-7bf6cbddc7jq7tr\" (UID: \"66a2bf81-b5e1-47a7-b3c2-0acca369b832\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7bf6cbddc7jq7tr"
Oct 30 04:07:54.353005 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:07:54.352841    4461 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/66a2bf81-b5e1-47a7-b3c2-0acca369b832-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-7bf6cbddc7jq7tr\" (UID: \"66a2bf81-b5e1-47a7-b3c2-0acca369b832\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7bf6cbddc7jq7tr"
Oct 30 04:07:54.353005 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:07:54.352877    4461 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/66a2bf81-b5e1-47a7-b3c2-0acca369b832-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-7bf6cbddc7jq7tr\" (UID: \"66a2bf81-b5e1-47a7-b3c2-0acca369b832\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7bf6cbddc7jq7tr"
Oct 30 04:07:54.353005 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:07:54.352893    4461 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/66a2bf81-b5e1-47a7-b3c2-0acca369b832-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-7bf6cbddc7jq7tr\" (UID: \"66a2bf81-b5e1-47a7-b3c2-0acca369b832\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7bf6cbddc7jq7tr"
Oct 30 04:07:54.353005 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:07:54.352909    4461 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9x27q\" (UniqueName: \"kubernetes.io/projected/66a2bf81-b5e1-47a7-b3c2-0acca369b832-kube-api-access-9x27q\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-7bf6cbddc7jq7tr\" (UID: \"66a2bf81-b5e1-47a7-b3c2-0acca369b832\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7bf6cbddc7jq7tr"
Oct 30 04:07:54.353005 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:07:54.352936    4461 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/66a2bf81-b5e1-47a7-b3c2-0acca369b832-dshm\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-7bf6cbddc7jq7tr\" (UID: \"66a2bf81-b5e1-47a7-b3c2-0acca369b832\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7bf6cbddc7jq7tr"
Oct 30 04:07:54.353005 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:07:54.352971    4461 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/66a2bf81-b5e1-47a7-b3c2-0acca369b832-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-7bf6cbddc7jq7tr\" (UID: \"66a2bf81-b5e1-47a7-b3c2-0acca369b832\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7bf6cbddc7jq7tr"
Oct 30 04:07:54.353005 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:07:54.352996    4461 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/66a2bf81-b5e1-47a7-b3c2-0acca369b832-home\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-7bf6cbddc7jq7tr\" (UID: \"66a2bf81-b5e1-47a7-b3c2-0acca369b832\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7bf6cbddc7jq7tr"
Oct 30 04:07:54.353377 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:07:54.353281    4461 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/66a2bf81-b5e1-47a7-b3c2-0acca369b832-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-7bf6cbddc7jq7tr\" (UID: \"66a2bf81-b5e1-47a7-b3c2-0acca369b832\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7bf6cbddc7jq7tr"
Oct 30 04:07:54.353377 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:07:54.353333    4461 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/66a2bf81-b5e1-47a7-b3c2-0acca369b832-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-7bf6cbddc7jq7tr\" (UID: \"66a2bf81-b5e1-47a7-b3c2-0acca369b832\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7bf6cbddc7jq7tr"
Oct 30 04:07:54.353493 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:07:54.353417    4461 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/66a2bf81-b5e1-47a7-b3c2-0acca369b832-home\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-7bf6cbddc7jq7tr\" (UID: \"66a2bf81-b5e1-47a7-b3c2-0acca369b832\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7bf6cbddc7jq7tr"
Oct 30 04:07:54.353669 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:07:54.353638    4461 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/66a2bf81-b5e1-47a7-b3c2-0acca369b832-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-7bf6cbddc7jq7tr\" (UID: \"66a2bf81-b5e1-47a7-b3c2-0acca369b832\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7bf6cbddc7jq7tr"
Oct 30 04:07:54.353712 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:07:54.353699    4461 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/66a2bf81-b5e1-47a7-b3c2-0acca369b832-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-7bf6cbddc7jq7tr\" (UID: \"66a2bf81-b5e1-47a7-b3c2-0acca369b832\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7bf6cbddc7jq7tr"
Oct 30 04:07:54.355305 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:07:54.355281    4461 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/66a2bf81-b5e1-47a7-b3c2-0acca369b832-dshm\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-7bf6cbddc7jq7tr\" (UID: \"66a2bf81-b5e1-47a7-b3c2-0acca369b832\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7bf6cbddc7jq7tr"
Oct 30 04:07:54.355438 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:07:54.355420    4461 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/66a2bf81-b5e1-47a7-b3c2-0acca369b832-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-7bf6cbddc7jq7tr\" (UID: \"66a2bf81-b5e1-47a7-b3c2-0acca369b832\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7bf6cbddc7jq7tr"
Oct 30 04:07:54.376695 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:07:54.376672    4461 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9x27q\" (UniqueName: \"kubernetes.io/projected/66a2bf81-b5e1-47a7-b3c2-0acca369b832-kube-api-access-9x27q\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-7bf6cbddc7jq7tr\" (UID: \"66a2bf81-b5e1-47a7-b3c2-0acca369b832\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7bf6cbddc7jq7tr"
Oct 30 04:07:54.449399 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:07:54.449343    4461 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7bf6cbddc7jq7tr"
Oct 30 04:07:54.508069 ip-10-0-143-156 kubenswrapper[4461]: E1030 04:07:54.508037    4461 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2k85b" containerName="extractor"
Oct 30 04:07:54.774292 ip-10-0-143-156 kubenswrapper[4461]: W1030 04:07:54.774247    4461 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod66a2bf81_b5e1_47a7_b3c2_0acca369b832.slice/crio-974d9fddd5d1ddcfeb220f5cc11ca0e5f75aeb17d4315cc502442d3dda5bdc5b WatchSource:0}: Error finding container 974d9fddd5d1ddcfeb220f5cc11ca0e5f75aeb17d4315cc502442d3dda5bdc5b: Status 404 returned error can't find the container with id 974d9fddd5d1ddcfeb220f5cc11ca0e5f75aeb17d4315cc502442d3dda5bdc5b
Oct 30 04:07:54.774426 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:07:54.774287    4461 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7bf6cbddc7jq7tr"]
Oct 30 04:07:55.169817 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:07:55.169737    4461 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7bf6cbddc7jq7tr" event={"ID":"66a2bf81-b5e1-47a7-b3c2-0acca369b832","Type":"ContainerStarted","Data":"715ccfebeef606591b6da3090592709d9c03fc9145dcabaa26a04da5e396a7e6"}
Oct 30 04:07:55.169817 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:07:55.169798    4461 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7bf6cbddc7jq7tr" event={"ID":"66a2bf81-b5e1-47a7-b3c2-0acca369b832","Type":"ContainerStarted","Data":"974d9fddd5d1ddcfeb220f5cc11ca0e5f75aeb17d4315cc502442d3dda5bdc5b"}
Oct 30 04:07:57.180021 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:07:57.179946    4461 generic.go:356] "Generic (PLEG): container finished" podID="66a2bf81-b5e1-47a7-b3c2-0acca369b832" containerID="715ccfebeef606591b6da3090592709d9c03fc9145dcabaa26a04da5e396a7e6" exitCode=0
Oct 30 04:07:57.180389 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:07:57.180021    4461 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7bf6cbddc7jq7tr" event={"ID":"66a2bf81-b5e1-47a7-b3c2-0acca369b832","Type":"ContainerDied","Data":"715ccfebeef606591b6da3090592709d9c03fc9145dcabaa26a04da5e396a7e6"}
Oct 30 04:07:58.187603 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:07:58.187556    4461 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7bf6cbddc7jq7tr" event={"ID":"66a2bf81-b5e1-47a7-b3c2-0acca369b832","Type":"ContainerStarted","Data":"852f08bee46811381989f94c9a2ed2f7f45d65c726d71cbe302a2d4e44f0b54b"}
Oct 30 04:07:58.248145 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:07:58.248080    4461 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7bf6cbddc7jq7tr" podStartSLOduration=2.011716977 podStartE2EDuration="4.248065529s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.236348552s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:07:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:07:58.234723535 +0000 UTC m=+4346.284528712" watchObservedRunningTime="2026-10-30 04:07:58.248065529 +0000 UTC m=+4346.297870706"
Oct 30 04:07:58.450235 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:07:58.450133    4461 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7bf6cbddc7jq7tr"
Oct 30 04:08:04.449842 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:08:04.449801    4461 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7bf6cbddc7jq7tr"
Oct 30 04:08:04.451658 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:08:04.451623    4461 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7bf6cbddc7jq7tr" podUID="66a2bf81-b5e1-47a7-b3c2-0acca369b832" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/health\": dial tcp 10.133.0.53:8000: connect: connection refused"
Oct 30 04:08:14.224250 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:08:14.224209    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 04:08:14.450883 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:08:14.450794    4461 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7bf6cbddc7jq7tr" podUID="66a2bf81-b5e1-47a7-b3c2-0acca369b832" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/health\": dial tcp 10.133.0.53:8000: connect: connection refused"
Oct 30 04:08:15.091804 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:08:15.091760    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 04:08:19.513038 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:08:19.513005    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 04:08:21.481740 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:08:21.481696    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 04:08:24.450223 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:08:24.450173    4461 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7bf6cbddc7jq7tr" podUID="66a2bf81-b5e1-47a7-b3c2-0acca369b832" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/health\": dial tcp 10.133.0.53:8000: connect: connection refused"
Oct 30 04:08:24.504585 ip-10-0-143-156 kubenswrapper[4461]: E1030 04:08:24.504551    4461 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2k85b" containerName="extractor"
Oct 30 04:08:29.729996 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:08:29.729958    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 04:08:30.668970 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:08:30.668937    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 04:08:34.449845 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:08:34.449805    4461 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7bf6cbddc7jq7tr" podUID="66a2bf81-b5e1-47a7-b3c2-0acca369b832" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/health\": dial tcp 10.133.0.53:8000: connect: connection refused"
Oct 30 04:08:36.965401 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:08:36.965356    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 04:08:37.150855 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:08:37.150823    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 04:08:44.450500 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:08:44.450457    4461 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7bf6cbddc7jq7tr" podUID="66a2bf81-b5e1-47a7-b3c2-0acca369b832" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/health\": dial tcp 10.133.0.53:8000: connect: connection refused"
Oct 30 04:08:49.942182 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:08:49.942145    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 04:08:51.036720 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:08:51.036675    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 04:08:54.450403 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:08:54.450362    4461 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7bf6cbddc7jq7tr" podUID="66a2bf81-b5e1-47a7-b3c2-0acca369b832" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/health\": dial tcp 10.133.0.53:8000: connect: connection refused"
Oct 30 04:08:54.504522 ip-10-0-143-156 kubenswrapper[4461]: E1030 04:08:54.504480    4461 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2k85b" containerName="extractor"
Oct 30 04:09:04.450371 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:09:04.450311    4461 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7bf6cbddc7jq7tr" podUID="66a2bf81-b5e1-47a7-b3c2-0acca369b832" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/health\": dial tcp 10.133.0.53:8000: connect: connection refused"
Oct 30 04:09:14.449747 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:09:14.449702    4461 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7bf6cbddc7jq7tr" podUID="66a2bf81-b5e1-47a7-b3c2-0acca369b832" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/health\": dial tcp 10.133.0.53:8000: connect: connection refused"
Oct 30 04:09:19.605901 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:09:19.605866    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 04:09:24.450595 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:09:24.450553    4461 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7bf6cbddc7jq7tr" podUID="66a2bf81-b5e1-47a7-b3c2-0acca369b832" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/health\": dial tcp 10.133.0.53:8000: connect: connection refused"
Oct 30 04:09:24.504972 ip-10-0-143-156 kubenswrapper[4461]: E1030 04:09:24.504935    4461 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2k85b" containerName="extractor"
Oct 30 04:09:25.698393 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:09:25.698357    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 04:09:27.659571 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:09:27.659532    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 04:09:29.844558 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:09:29.844504    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 04:09:34.450137 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:09:34.450072    4461 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7bf6cbddc7jq7tr" podUID="66a2bf81-b5e1-47a7-b3c2-0acca369b832" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/health\": dial tcp 10.133.0.53:8000: connect: connection refused"
Oct 30 04:09:44.450758 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:09:44.450660    4461 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7bf6cbddc7jq7tr" podUID="66a2bf81-b5e1-47a7-b3c2-0acca369b832" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/health\": dial tcp 10.133.0.53:8000: connect: connection refused"
Oct 30 04:09:48.080723 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:09:48.080676    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 04:09:48.284071 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:09:48.284033    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 04:09:48.865868 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:09:48.865836    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 04:09:49.218948 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:09:49.218913    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 04:09:50.065072 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:09:50.065042    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 04:09:50.241936 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:09:50.241901    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 04:09:53.197841 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:09:53.197799    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 04:09:54.299788 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:09:54.299752    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 04:09:54.460510 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:09:54.460479    4461 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7bf6cbddc7jq7tr"
Oct 30 04:09:54.468292 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:09:54.468272    4461 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7bf6cbddc7jq7tr"
Oct 30 04:09:54.493929 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:09:54.493904    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 04:09:54.504009 ip-10-0-143-156 kubenswrapper[4461]: E1030 04:09:54.503981    4461 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2k85b" containerName="extractor"
Oct 30 04:09:54.666896 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:09:54.666820    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 04:09:55.564459 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:09:55.564386    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 04:09:55.732772 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:09:55.732727    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 04:09:59.887310 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:09:59.887277    4461 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7bf6cbddc7jq7tr"]
Oct 30 04:09:59.977178 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:09:59.977086    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 04:10:01.101575 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:10:01.101543    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 04:10:01.310436 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:10:01.310397    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 04:10:08.787352 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:10:08.787314    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 04:10:14.909622 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:10:14.909560    4461 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7bf6cbddc7jq7tr" podUID="66a2bf81-b5e1-47a7-b3c2-0acca369b832" containerName="main" containerID="cri-o://852f08bee46811381989f94c9a2ed2f7f45d65c726d71cbe302a2d4e44f0b54b" gracePeriod=45
Oct 30 04:10:15.461152 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:10:15.461108    4461 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7bf6cbddc7jq7tr" podUID="66a2bf81-b5e1-47a7-b3c2-0acca369b832" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/health\": dial tcp 10.133.0.53:8000: connect: connection refused"
Oct 30 04:10:15.657293 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:10:15.657269    4461 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7bf6cbddc7jq7tr"
Oct 30 04:10:15.742361 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:10:15.742328    4461 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/66a2bf81-b5e1-47a7-b3c2-0acca369b832-dshm\" (UniqueName: \"kubernetes.io/empty-dir/66a2bf81-b5e1-47a7-b3c2-0acca369b832-dshm\") pod \"66a2bf81-b5e1-47a7-b3c2-0acca369b832\" (UID: \"66a2bf81-b5e1-47a7-b3c2-0acca369b832\") "
Oct 30 04:10:15.742525 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:10:15.742418    4461 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/66a2bf81-b5e1-47a7-b3c2-0acca369b832-home\" (UniqueName: \"kubernetes.io/empty-dir/66a2bf81-b5e1-47a7-b3c2-0acca369b832-home\") pod \"66a2bf81-b5e1-47a7-b3c2-0acca369b832\" (UID: \"66a2bf81-b5e1-47a7-b3c2-0acca369b832\") "
Oct 30 04:10:15.742525 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:10:15.742436    4461 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/66a2bf81-b5e1-47a7-b3c2-0acca369b832-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/66a2bf81-b5e1-47a7-b3c2-0acca369b832-model-cache\") pod \"66a2bf81-b5e1-47a7-b3c2-0acca369b832\" (UID: \"66a2bf81-b5e1-47a7-b3c2-0acca369b832\") "
Oct 30 04:10:15.742525 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:10:15.742469    4461 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/66a2bf81-b5e1-47a7-b3c2-0acca369b832-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/66a2bf81-b5e1-47a7-b3c2-0acca369b832-tmp-dir\") pod \"66a2bf81-b5e1-47a7-b3c2-0acca369b832\" (UID: \"66a2bf81-b5e1-47a7-b3c2-0acca369b832\") "
Oct 30 04:10:15.742525 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:10:15.742490    4461 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/66a2bf81-b5e1-47a7-b3c2-0acca369b832-tls-certs\" (UniqueName: \"kubernetes.io/secret/66a2bf81-b5e1-47a7-b3c2-0acca369b832-tls-certs\") pod \"66a2bf81-b5e1-47a7-b3c2-0acca369b832\" (UID: \"66a2bf81-b5e1-47a7-b3c2-0acca369b832\") "
Oct 30 04:10:15.742525 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:10:15.742511    4461 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/66a2bf81-b5e1-47a7-b3c2-0acca369b832-kube-api-access-9x27q\" (UniqueName: \"kubernetes.io/projected/66a2bf81-b5e1-47a7-b3c2-0acca369b832-kube-api-access-9x27q\") pod \"66a2bf81-b5e1-47a7-b3c2-0acca369b832\" (UID: \"66a2bf81-b5e1-47a7-b3c2-0acca369b832\") "
Oct 30 04:10:15.742794 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:10:15.742576    4461 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/66a2bf81-b5e1-47a7-b3c2-0acca369b832-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/66a2bf81-b5e1-47a7-b3c2-0acca369b832-cabundle-cert\") pod \"66a2bf81-b5e1-47a7-b3c2-0acca369b832\" (UID: \"66a2bf81-b5e1-47a7-b3c2-0acca369b832\") "
Oct 30 04:10:15.742794 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:10:15.742608    4461 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/66a2bf81-b5e1-47a7-b3c2-0acca369b832-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/66a2bf81-b5e1-47a7-b3c2-0acca369b832-kserve-provision-location\") pod \"66a2bf81-b5e1-47a7-b3c2-0acca369b832\" (UID: \"66a2bf81-b5e1-47a7-b3c2-0acca369b832\") "
Oct 30 04:10:15.742909 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:10:15.742783    4461 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66a2bf81-b5e1-47a7-b3c2-0acca369b832-model-cache" pod "66a2bf81-b5e1-47a7-b3c2-0acca369b832" (UID: "66a2bf81-b5e1-47a7-b3c2-0acca369b832"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:10:15.742909 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:10:15.742898    4461 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/66a2bf81-b5e1-47a7-b3c2-0acca369b832-model-cache\") on node \"ip-10-0-143-156.ec2.internal\" DevicePath \"\""
Oct 30 04:10:15.743235 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:10:15.743172    4461 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/66a2bf81-b5e1-47a7-b3c2-0acca369b832-cabundle-cert" pod "66a2bf81-b5e1-47a7-b3c2-0acca369b832" (UID: "66a2bf81-b5e1-47a7-b3c2-0acca369b832"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:10:15.743371 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:10:15.743281    4461 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66a2bf81-b5e1-47a7-b3c2-0acca369b832-home" pod "66a2bf81-b5e1-47a7-b3c2-0acca369b832" (UID: "66a2bf81-b5e1-47a7-b3c2-0acca369b832"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:10:15.745142 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:10:15.745117    4461 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66a2bf81-b5e1-47a7-b3c2-0acca369b832-dshm" pod "66a2bf81-b5e1-47a7-b3c2-0acca369b832" (UID: "66a2bf81-b5e1-47a7-b3c2-0acca369b832"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:10:15.745281 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:10:15.745225    4461 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/66a2bf81-b5e1-47a7-b3c2-0acca369b832-tls-certs" pod "66a2bf81-b5e1-47a7-b3c2-0acca369b832" (UID: "66a2bf81-b5e1-47a7-b3c2-0acca369b832"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:10:15.745348 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:10:15.745317    4461 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/66a2bf81-b5e1-47a7-b3c2-0acca369b832-kube-api-access-9x27q" pod "66a2bf81-b5e1-47a7-b3c2-0acca369b832" (UID: "66a2bf81-b5e1-47a7-b3c2-0acca369b832"). InnerVolumeSpecName "kube-api-access-9x27q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:10:15.761339 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:10:15.761312    4461 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66a2bf81-b5e1-47a7-b3c2-0acca369b832-tmp-dir" pod "66a2bf81-b5e1-47a7-b3c2-0acca369b832" (UID: "66a2bf81-b5e1-47a7-b3c2-0acca369b832"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:10:15.766584 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:10:15.766559    4461 generic.go:356] "Generic (PLEG): container finished" podID="66a2bf81-b5e1-47a7-b3c2-0acca369b832" containerID="852f08bee46811381989f94c9a2ed2f7f45d65c726d71cbe302a2d4e44f0b54b" exitCode=0
Oct 30 04:10:15.766682 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:10:15.766617    4461 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7bf6cbddc7jq7tr"
Oct 30 04:10:15.766682 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:10:15.766640    4461 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7bf6cbddc7jq7tr" event={"ID":"66a2bf81-b5e1-47a7-b3c2-0acca369b832","Type":"ContainerDied","Data":"852f08bee46811381989f94c9a2ed2f7f45d65c726d71cbe302a2d4e44f0b54b"}
Oct 30 04:10:15.766757 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:10:15.766708    4461 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7bf6cbddc7jq7tr" event={"ID":"66a2bf81-b5e1-47a7-b3c2-0acca369b832","Type":"ContainerDied","Data":"974d9fddd5d1ddcfeb220f5cc11ca0e5f75aeb17d4315cc502442d3dda5bdc5b"}
Oct 30 04:10:15.766757 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:10:15.766736    4461 scope.go:122] "RemoveContainer" containerID="852f08bee46811381989f94c9a2ed2f7f45d65c726d71cbe302a2d4e44f0b54b"
Oct 30 04:10:15.772535 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:10:15.772515    4461 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66a2bf81-b5e1-47a7-b3c2-0acca369b832-kserve-provision-location" pod "66a2bf81-b5e1-47a7-b3c2-0acca369b832" (UID: "66a2bf81-b5e1-47a7-b3c2-0acca369b832"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:10:15.774660 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:10:15.774644    4461 scope.go:122] "RemoveContainer" containerID="715ccfebeef606591b6da3090592709d9c03fc9145dcabaa26a04da5e396a7e6"
Oct 30 04:10:15.782275 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:10:15.782252    4461 scope.go:122] "RemoveContainer" containerID="852f08bee46811381989f94c9a2ed2f7f45d65c726d71cbe302a2d4e44f0b54b"
Oct 30 04:10:15.782522 ip-10-0-143-156 kubenswrapper[4461]: E1030 04:10:15.782503    4461 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"852f08bee46811381989f94c9a2ed2f7f45d65c726d71cbe302a2d4e44f0b54b\": container with ID starting with 852f08bee46811381989f94c9a2ed2f7f45d65c726d71cbe302a2d4e44f0b54b not found: ID does not exist" containerID="852f08bee46811381989f94c9a2ed2f7f45d65c726d71cbe302a2d4e44f0b54b"
Oct 30 04:10:15.782607 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:10:15.782531    4461 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"852f08bee46811381989f94c9a2ed2f7f45d65c726d71cbe302a2d4e44f0b54b"} err="failed to get container status \"852f08bee46811381989f94c9a2ed2f7f45d65c726d71cbe302a2d4e44f0b54b\": rpc error: code = NotFound desc = could not find container \"852f08bee46811381989f94c9a2ed2f7f45d65c726d71cbe302a2d4e44f0b54b\": container with ID starting with 852f08bee46811381989f94c9a2ed2f7f45d65c726d71cbe302a2d4e44f0b54b not found: ID does not exist"
Oct 30 04:10:15.782607 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:10:15.782548    4461 scope.go:122] "RemoveContainer" containerID="715ccfebeef606591b6da3090592709d9c03fc9145dcabaa26a04da5e396a7e6"
Oct 30 04:10:15.782785 ip-10-0-143-156 kubenswrapper[4461]: E1030 04:10:15.782768    4461 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"715ccfebeef606591b6da3090592709d9c03fc9145dcabaa26a04da5e396a7e6\": container with ID starting with 715ccfebeef606591b6da3090592709d9c03fc9145dcabaa26a04da5e396a7e6 not found: ID does not exist" containerID="715ccfebeef606591b6da3090592709d9c03fc9145dcabaa26a04da5e396a7e6"
Oct 30 04:10:15.782830 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:10:15.782791    4461 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"715ccfebeef606591b6da3090592709d9c03fc9145dcabaa26a04da5e396a7e6"} err="failed to get container status \"715ccfebeef606591b6da3090592709d9c03fc9145dcabaa26a04da5e396a7e6\": rpc error: code = NotFound desc = could not find container \"715ccfebeef606591b6da3090592709d9c03fc9145dcabaa26a04da5e396a7e6\": container with ID starting with 715ccfebeef606591b6da3090592709d9c03fc9145dcabaa26a04da5e396a7e6 not found: ID does not exist"
Oct 30 04:10:15.843614 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:10:15.843584    4461 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/66a2bf81-b5e1-47a7-b3c2-0acca369b832-home\") on node \"ip-10-0-143-156.ec2.internal\" DevicePath \"\""
Oct 30 04:10:15.843614 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:10:15.843612    4461 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/66a2bf81-b5e1-47a7-b3c2-0acca369b832-tmp-dir\") on node \"ip-10-0-143-156.ec2.internal\" DevicePath \"\""
Oct 30 04:10:15.843754 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:10:15.843623    4461 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/66a2bf81-b5e1-47a7-b3c2-0acca369b832-tls-certs\") on node \"ip-10-0-143-156.ec2.internal\" DevicePath \"\""
Oct 30 04:10:15.843754 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:10:15.843633    4461 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9x27q\" (UniqueName: \"kubernetes.io/projected/66a2bf81-b5e1-47a7-b3c2-0acca369b832-kube-api-access-9x27q\") on node \"ip-10-0-143-156.ec2.internal\" DevicePath \"\""
Oct 30 04:10:15.843754 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:10:15.843642    4461 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/66a2bf81-b5e1-47a7-b3c2-0acca369b832-cabundle-cert\") on node \"ip-10-0-143-156.ec2.internal\" DevicePath \"\""
Oct 30 04:10:15.843754 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:10:15.843651    4461 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/66a2bf81-b5e1-47a7-b3c2-0acca369b832-kserve-provision-location\") on node \"ip-10-0-143-156.ec2.internal\" DevicePath \"\""
Oct 30 04:10:15.843754 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:10:15.843659    4461 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/66a2bf81-b5e1-47a7-b3c2-0acca369b832-dshm\") on node \"ip-10-0-143-156.ec2.internal\" DevicePath \"\""
Oct 30 04:10:16.115479 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:10:16.115450    4461 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7bf6cbddc7jq7tr"]
Oct 30 04:10:16.128008 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:10:16.127976    4461 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7bf6cbddc7jq7tr"]
Oct 30 04:10:18.114264 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:10:18.114223    4461 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="66a2bf81-b5e1-47a7-b3c2-0acca369b832" path="/var/lib/kubelet/pods/66a2bf81-b5e1-47a7-b3c2-0acca369b832/volumes"
Oct 30 04:10:19.706917 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:10:19.706821    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 04:10:20.107599 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:10:20.107543    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 04:10:20.295221 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:10:20.295188    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 04:10:24.504073 ip-10-0-143-156 kubenswrapper[4461]: E1030 04:10:24.504030    4461 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2k85b" containerName="extractor"
Oct 30 04:10:26.626317 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:10:26.626280    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 04:10:29.964915 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:10:29.964856    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 04:10:31.883372 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:10:31.883328    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 04:10:32.252151 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:10:32.252110    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 04:10:32.425718 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:10:32.425687    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 04:10:32.439335 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:10:32.439309    4461 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"]
Oct 30 04:10:32.439650 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:10:32.439638    4461 cpu_manager.go:414] "containerMap: removing container" podUID="66a2bf81-b5e1-47a7-b3c2-0acca369b832" containerName="main"
Oct 30 04:10:32.439695 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:10:32.439651    4461 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="66a2bf81-b5e1-47a7-b3c2-0acca369b832" containerName="main"
Oct 30 04:10:32.439695 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:10:32.439672    4461 cpu_manager.go:414] "containerMap: removing container" podUID="66a2bf81-b5e1-47a7-b3c2-0acca369b832" containerName="storage-initializer"
Oct 30 04:10:32.439695 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:10:32.439677    4461 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="66a2bf81-b5e1-47a7-b3c2-0acca369b832" containerName="storage-initializer"
Oct 30 04:10:32.439796 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:10:32.439728    4461 memory_manager.go:365] "RemoveStaleState removing state" podUID="66a2bf81-b5e1-47a7-b3c2-0acca369b832" containerName="main"
Oct 30 04:10:32.441792 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:10:32.441771    4461 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Oct 30 04:10:32.478249 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:10:32.478224    4461 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"]
Oct 30 04:10:32.490271 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:10:32.490251    4461 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"seaweedfs-s3-creds\""
Oct 30 04:10:32.490498 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:10:32.490476    4461 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"llmisvc-model-fb-opt-125m-route-dc21cb-57cf7736-dockercfg-8cr9j\""
Oct 30 04:10:32.490549 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:10:32.490476    4461 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"llmisv32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-self-signed-certs\""
Oct 30 04:10:32.490549 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:10:32.490481    4461 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"default-dockercfg-b9z9s\""
Oct 30 04:10:32.490955 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:10:32.490940    4461 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"odh-kserve-custom-ca-bundle\""
Oct 30 04:10:32.491083 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:10:32.491064    4461 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fc64f170-9e8b-4ad8-8604-60849e50a5f2-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"fc64f170-9e8b-4ad8-8604-60849e50a5f2\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Oct 30 04:10:32.491136 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:10:32.491083    4461 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"openshift-service-ca.crt\""
Oct 30 04:10:32.491136 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:10:32.491106    4461 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kr6hb\" (UniqueName: \"kubernetes.io/projected/fc64f170-9e8b-4ad8-8604-60849e50a5f2-kube-api-access-kr6hb\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"fc64f170-9e8b-4ad8-8604-60849e50a5f2\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Oct 30 04:10:32.491254 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:10:32.491183    4461 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"kube-root-ca.crt\""
Oct 30 04:10:32.491254 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:10:32.491223    4461 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fc64f170-9e8b-4ad8-8604-60849e50a5f2-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"fc64f170-9e8b-4ad8-8604-60849e50a5f2\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Oct 30 04:10:32.491254 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:10:32.491242    4461 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fc64f170-9e8b-4ad8-8604-60849e50a5f2-home\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"fc64f170-9e8b-4ad8-8604-60849e50a5f2\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Oct 30 04:10:32.491407 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:10:32.491258    4461 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fc64f170-9e8b-4ad8-8604-60849e50a5f2-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"fc64f170-9e8b-4ad8-8604-60849e50a5f2\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Oct 30 04:10:32.491407 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:10:32.491347    4461 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fc64f170-9e8b-4ad8-8604-60849e50a5f2-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"fc64f170-9e8b-4ad8-8604-60849e50a5f2\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Oct 30 04:10:32.491407 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:10:32.491371    4461 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fc64f170-9e8b-4ad8-8604-60849e50a5f2-dshm\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"fc64f170-9e8b-4ad8-8604-60849e50a5f2\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Oct 30 04:10:32.491407 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:10:32.491393    4461 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fc64f170-9e8b-4ad8-8604-60849e50a5f2-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"fc64f170-9e8b-4ad8-8604-60849e50a5f2\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Oct 30 04:10:32.592165 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:10:32.592055    4461 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fc64f170-9e8b-4ad8-8604-60849e50a5f2-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"fc64f170-9e8b-4ad8-8604-60849e50a5f2\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Oct 30 04:10:32.592165 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:10:32.592128    4461 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fc64f170-9e8b-4ad8-8604-60849e50a5f2-home\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"fc64f170-9e8b-4ad8-8604-60849e50a5f2\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Oct 30 04:10:32.592165 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:10:32.592153    4461 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fc64f170-9e8b-4ad8-8604-60849e50a5f2-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"fc64f170-9e8b-4ad8-8604-60849e50a5f2\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Oct 30 04:10:32.592466 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:10:32.592199    4461 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fc64f170-9e8b-4ad8-8604-60849e50a5f2-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"fc64f170-9e8b-4ad8-8604-60849e50a5f2\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Oct 30 04:10:32.592466 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:10:32.592227    4461 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fc64f170-9e8b-4ad8-8604-60849e50a5f2-dshm\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"fc64f170-9e8b-4ad8-8604-60849e50a5f2\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Oct 30 04:10:32.592466 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:10:32.592258    4461 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fc64f170-9e8b-4ad8-8604-60849e50a5f2-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"fc64f170-9e8b-4ad8-8604-60849e50a5f2\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Oct 30 04:10:32.592466 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:10:32.592347    4461 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fc64f170-9e8b-4ad8-8604-60849e50a5f2-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"fc64f170-9e8b-4ad8-8604-60849e50a5f2\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Oct 30 04:10:32.592466 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:10:32.592377    4461 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kr6hb\" (UniqueName: \"kubernetes.io/projected/fc64f170-9e8b-4ad8-8604-60849e50a5f2-kube-api-access-kr6hb\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"fc64f170-9e8b-4ad8-8604-60849e50a5f2\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Oct 30 04:10:32.592720 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:10:32.592581    4461 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fc64f170-9e8b-4ad8-8604-60849e50a5f2-home\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"fc64f170-9e8b-4ad8-8604-60849e50a5f2\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Oct 30 04:10:32.592720 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:10:32.592622    4461 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fc64f170-9e8b-4ad8-8604-60849e50a5f2-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"fc64f170-9e8b-4ad8-8604-60849e50a5f2\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Oct 30 04:10:32.592813 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:10:32.592715    4461 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fc64f170-9e8b-4ad8-8604-60849e50a5f2-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"fc64f170-9e8b-4ad8-8604-60849e50a5f2\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Oct 30 04:10:32.592813 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:10:32.592732    4461 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fc64f170-9e8b-4ad8-8604-60849e50a5f2-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"fc64f170-9e8b-4ad8-8604-60849e50a5f2\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Oct 30 04:10:32.593069 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:10:32.593051    4461 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fc64f170-9e8b-4ad8-8604-60849e50a5f2-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"fc64f170-9e8b-4ad8-8604-60849e50a5f2\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Oct 30 04:10:32.594431 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:10:32.594409    4461 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fc64f170-9e8b-4ad8-8604-60849e50a5f2-dshm\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"fc64f170-9e8b-4ad8-8604-60849e50a5f2\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Oct 30 04:10:32.594535 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:10:32.594520    4461 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fc64f170-9e8b-4ad8-8604-60849e50a5f2-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"fc64f170-9e8b-4ad8-8604-60849e50a5f2\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Oct 30 04:10:32.602695 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:10:32.602672    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 04:10:32.615662 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:10:32.615636    4461 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kr6hb\" (UniqueName: \"kubernetes.io/projected/fc64f170-9e8b-4ad8-8604-60849e50a5f2-kube-api-access-kr6hb\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"fc64f170-9e8b-4ad8-8604-60849e50a5f2\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Oct 30 04:10:32.756852 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:10:32.756824    4461 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Oct 30 04:10:33.067985 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:10:33.067954    4461 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"]
Oct 30 04:10:33.068712 ip-10-0-143-156 kubenswrapper[4461]: W1030 04:10:33.068684    4461 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc64f170_9e8b_4ad8_8604_60849e50a5f2.slice/crio-920adc7da8be88ef67a5136d8b2f2958d82988fd4ddf7a328a55cea458062f36 WatchSource:0}: Error finding container 920adc7da8be88ef67a5136d8b2f2958d82988fd4ddf7a328a55cea458062f36: Status 404 returned error can't find the container with id 920adc7da8be88ef67a5136d8b2f2958d82988fd4ddf7a328a55cea458062f36
Oct 30 04:10:33.717668 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:10:33.717635    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 04:10:33.839714 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:10:33.839679    4461 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" event={"ID":"fc64f170-9e8b-4ad8-8604-60849e50a5f2","Type":"ContainerStarted","Data":"dc158230868e5ff2e377d9a72256434644161e08a140f5001fb4826e3d0ddcb2"}
Oct 30 04:10:33.839880 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:10:33.839731    4461 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" event={"ID":"fc64f170-9e8b-4ad8-8604-60849e50a5f2","Type":"ContainerStarted","Data":"920adc7da8be88ef67a5136d8b2f2958d82988fd4ddf7a328a55cea458062f36"}
Oct 30 04:10:33.894787 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:10:33.894729    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 04:10:35.848035 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:10:35.848006    4461 generic.go:356] "Generic (PLEG): container finished" podID="fc64f170-9e8b-4ad8-8604-60849e50a5f2" containerID="dc158230868e5ff2e377d9a72256434644161e08a140f5001fb4826e3d0ddcb2" exitCode=0
Oct 30 04:10:35.848448 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:10:35.848078    4461 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" event={"ID":"fc64f170-9e8b-4ad8-8604-60849e50a5f2","Type":"ContainerDied","Data":"dc158230868e5ff2e377d9a72256434644161e08a140f5001fb4826e3d0ddcb2"}
Oct 30 04:10:36.854312 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:10:36.854281    4461 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" event={"ID":"fc64f170-9e8b-4ad8-8604-60849e50a5f2","Type":"ContainerStarted","Data":"05d20f812ea9305fa946b8f47f4d3684493db9a578c670ed15c53401f451e15c"}
Oct 30 04:10:36.929493 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:10:36.929442    4461 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podStartSLOduration=2.659836867 podStartE2EDuration="4.929427847s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.26959098s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:10:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:10:36.91556142 +0000 UTC m=+4504.965366598" watchObservedRunningTime="2026-10-30 04:10:36.929427847 +0000 UTC m=+4504.979233046"
Oct 30 04:10:37.757908 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:10:37.757873    4461 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Oct 30 04:10:38.730740 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:10:38.730700    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 04:10:42.757165 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:10:42.757127    4461 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Oct 30 04:10:42.758600 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:10:42.758575    4461 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="fc64f170-9e8b-4ad8-8604-60849e50a5f2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/health\": dial tcp 10.133.0.54:8000: connect: connection refused"
Oct 30 04:10:48.922224 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:10:48.922188    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 04:10:49.102786 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:10:49.102735    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 04:10:49.341459 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:10:49.341422    4461 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-test-model-based-routing-only-fae765e6/model-routing-only-gateway-openshift-default-7d46dbd96b-dzr9v"]
Oct 30 04:10:49.344900 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:10:49.344874    4461 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-model-based-routing-only-fae765e6/model-routing-only-gateway-openshift-default-7d46dbd96b-dzr9v"
Oct 30 04:10:49.360342 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:10:49.360312    4461 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-model-based-routing-only-fae765e6\"/\"model-routing-only-gateway-openshift-default-dockercfg-2jq9v\""
Oct 30 04:10:49.360498 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:10:49.360379    4461 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-model-based-routing-only-fae765e6\"/\"openshift-service-ca.crt\""
Oct 30 04:10:49.364254 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:10:49.364230    4461 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-model-based-routing-only-fae765e6\"/\"kube-root-ca.crt\""
Oct 30 04:10:49.364254 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:10:49.364235    4461 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-model-based-routing-only-fae765e6\"/\"openshift-gw-ca-root-cert\""
Oct 30 04:10:49.374349 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:10:49.374319    4461 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-model-based-routing-only-fae765e6/model-routing-only-gateway-openshift-default-7d46dbd96b-dzr9v"]
Oct 30 04:10:49.465824 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:10:49.465792    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 04:10:49.471527 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:10:49.471466    4461 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/107fadeb-7edf-49eb-8e72-8b9c7a12dbda-workload-socket\") pod \"model-routing-only-gateway-openshift-default-7d46dbd96b-dzr9v\" (UID: \"107fadeb-7edf-49eb-8e72-8b9c7a12dbda\") " pod="e2e-test-model-based-routing-only-fae765e6/model-routing-only-gateway-openshift-default-7d46dbd96b-dzr9v"
Oct 30 04:10:49.471527 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:10:49.471526    4461 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/107fadeb-7edf-49eb-8e72-8b9c7a12dbda-istiod-ca-cert\") pod \"model-routing-only-gateway-openshift-default-7d46dbd96b-dzr9v\" (UID: \"107fadeb-7edf-49eb-8e72-8b9c7a12dbda\") " pod="e2e-test-model-based-routing-only-fae765e6/model-routing-only-gateway-openshift-default-7d46dbd96b-dzr9v"
Oct 30 04:10:49.471958 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:10:49.471582    4461 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/107fadeb-7edf-49eb-8e72-8b9c7a12dbda-istio-data\") pod \"model-routing-only-gateway-openshift-default-7d46dbd96b-dzr9v\" (UID: \"107fadeb-7edf-49eb-8e72-8b9c7a12dbda\") " pod="e2e-test-model-based-routing-only-fae765e6/model-routing-only-gateway-openshift-default-7d46dbd96b-dzr9v"
Oct 30 04:10:49.471958 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:10:49.471617    4461 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/107fadeb-7edf-49eb-8e72-8b9c7a12dbda-workload-certs\") pod \"model-routing-only-gateway-openshift-default-7d46dbd96b-dzr9v\" (UID: \"107fadeb-7edf-49eb-8e72-8b9c7a12dbda\") " pod="e2e-test-model-based-routing-only-fae765e6/model-routing-only-gateway-openshift-default-7d46dbd96b-dzr9v"
Oct 30 04:10:49.471958 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:10:49.471669    4461 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/107fadeb-7edf-49eb-8e72-8b9c7a12dbda-istio-envoy\") pod \"model-routing-only-gateway-openshift-default-7d46dbd96b-dzr9v\" (UID: \"107fadeb-7edf-49eb-8e72-8b9c7a12dbda\") " pod="e2e-test-model-based-routing-only-fae765e6/model-routing-only-gateway-openshift-default-7d46dbd96b-dzr9v"
Oct 30 04:10:49.471958 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:10:49.471702    4461 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hrk54\" (UniqueName: \"kubernetes.io/projected/107fadeb-7edf-49eb-8e72-8b9c7a12dbda-kube-api-access-hrk54\") pod \"model-routing-only-gateway-openshift-default-7d46dbd96b-dzr9v\" (UID: \"107fadeb-7edf-49eb-8e72-8b9c7a12dbda\") " pod="e2e-test-model-based-routing-only-fae765e6/model-routing-only-gateway-openshift-default-7d46dbd96b-dzr9v"
Oct 30 04:10:49.473162 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:10:49.473041    4461 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/107fadeb-7edf-49eb-8e72-8b9c7a12dbda-istio-podinfo\") pod \"model-routing-only-gateway-openshift-default-7d46dbd96b-dzr9v\" (UID: \"107fadeb-7edf-49eb-8e72-8b9c7a12dbda\") " pod="e2e-test-model-based-routing-only-fae765e6/model-routing-only-gateway-openshift-default-7d46dbd96b-dzr9v"
Oct 30 04:10:49.473162 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:10:49.473133    4461 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/107fadeb-7edf-49eb-8e72-8b9c7a12dbda-istio-token\") pod \"model-routing-only-gateway-openshift-default-7d46dbd96b-dzr9v\" (UID: \"107fadeb-7edf-49eb-8e72-8b9c7a12dbda\") " pod="e2e-test-model-based-routing-only-fae765e6/model-routing-only-gateway-openshift-default-7d46dbd96b-dzr9v"
Oct 30 04:10:49.473395 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:10:49.473182    4461 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/107fadeb-7edf-49eb-8e72-8b9c7a12dbda-credential-socket\") pod \"model-routing-only-gateway-openshift-default-7d46dbd96b-dzr9v\" (UID: \"107fadeb-7edf-49eb-8e72-8b9c7a12dbda\") " pod="e2e-test-model-based-routing-only-fae765e6/model-routing-only-gateway-openshift-default-7d46dbd96b-dzr9v"
Oct 30 04:10:49.574260 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:10:49.574223    4461 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/107fadeb-7edf-49eb-8e72-8b9c7a12dbda-workload-socket\") pod \"model-routing-only-gateway-openshift-default-7d46dbd96b-dzr9v\" (UID: \"107fadeb-7edf-49eb-8e72-8b9c7a12dbda\") " pod="e2e-test-model-based-routing-only-fae765e6/model-routing-only-gateway-openshift-default-7d46dbd96b-dzr9v"
Oct 30 04:10:49.574448 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:10:49.574267    4461 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/107fadeb-7edf-49eb-8e72-8b9c7a12dbda-istiod-ca-cert\") pod \"model-routing-only-gateway-openshift-default-7d46dbd96b-dzr9v\" (UID: \"107fadeb-7edf-49eb-8e72-8b9c7a12dbda\") " pod="e2e-test-model-based-routing-only-fae765e6/model-routing-only-gateway-openshift-default-7d46dbd96b-dzr9v"
Oct 30 04:10:49.574448 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:10:49.574319    4461 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/107fadeb-7edf-49eb-8e72-8b9c7a12dbda-istio-data\") pod \"model-routing-only-gateway-openshift-default-7d46dbd96b-dzr9v\" (UID: \"107fadeb-7edf-49eb-8e72-8b9c7a12dbda\") " pod="e2e-test-model-based-routing-only-fae765e6/model-routing-only-gateway-openshift-default-7d46dbd96b-dzr9v"
Oct 30 04:10:49.574448 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:10:49.574344    4461 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/107fadeb-7edf-49eb-8e72-8b9c7a12dbda-workload-certs\") pod \"model-routing-only-gateway-openshift-default-7d46dbd96b-dzr9v\" (UID: \"107fadeb-7edf-49eb-8e72-8b9c7a12dbda\") " pod="e2e-test-model-based-routing-only-fae765e6/model-routing-only-gateway-openshift-default-7d46dbd96b-dzr9v"
Oct 30 04:10:49.574448 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:10:49.574393    4461 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/107fadeb-7edf-49eb-8e72-8b9c7a12dbda-istio-envoy\") pod \"model-routing-only-gateway-openshift-default-7d46dbd96b-dzr9v\" (UID: \"107fadeb-7edf-49eb-8e72-8b9c7a12dbda\") " pod="e2e-test-model-based-routing-only-fae765e6/model-routing-only-gateway-openshift-default-7d46dbd96b-dzr9v"
Oct 30 04:10:49.574448 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:10:49.574421    4461 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hrk54\" (UniqueName: \"kubernetes.io/projected/107fadeb-7edf-49eb-8e72-8b9c7a12dbda-kube-api-access-hrk54\") pod \"model-routing-only-gateway-openshift-default-7d46dbd96b-dzr9v\" (UID: \"107fadeb-7edf-49eb-8e72-8b9c7a12dbda\") " pod="e2e-test-model-based-routing-only-fae765e6/model-routing-only-gateway-openshift-default-7d46dbd96b-dzr9v"
Oct 30 04:10:49.574448 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:10:49.574449    4461 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/107fadeb-7edf-49eb-8e72-8b9c7a12dbda-istio-podinfo\") pod \"model-routing-only-gateway-openshift-default-7d46dbd96b-dzr9v\" (UID: \"107fadeb-7edf-49eb-8e72-8b9c7a12dbda\") " pod="e2e-test-model-based-routing-only-fae765e6/model-routing-only-gateway-openshift-default-7d46dbd96b-dzr9v"
Oct 30 04:10:49.574782 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:10:49.574481    4461 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/107fadeb-7edf-49eb-8e72-8b9c7a12dbda-istio-token\") pod \"model-routing-only-gateway-openshift-default-7d46dbd96b-dzr9v\" (UID: \"107fadeb-7edf-49eb-8e72-8b9c7a12dbda\") " pod="e2e-test-model-based-routing-only-fae765e6/model-routing-only-gateway-openshift-default-7d46dbd96b-dzr9v"
Oct 30 04:10:49.574782 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:10:49.574524    4461 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/107fadeb-7edf-49eb-8e72-8b9c7a12dbda-credential-socket\") pod \"model-routing-only-gateway-openshift-default-7d46dbd96b-dzr9v\" (UID: \"107fadeb-7edf-49eb-8e72-8b9c7a12dbda\") " pod="e2e-test-model-based-routing-only-fae765e6/model-routing-only-gateway-openshift-default-7d46dbd96b-dzr9v"
Oct 30 04:10:49.574782 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:10:49.574725    4461 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/107fadeb-7edf-49eb-8e72-8b9c7a12dbda-workload-socket\") pod \"model-routing-only-gateway-openshift-default-7d46dbd96b-dzr9v\" (UID: \"107fadeb-7edf-49eb-8e72-8b9c7a12dbda\") " pod="e2e-test-model-based-routing-only-fae765e6/model-routing-only-gateway-openshift-default-7d46dbd96b-dzr9v"
Oct 30 04:10:49.574782 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:10:49.574771    4461 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/107fadeb-7edf-49eb-8e72-8b9c7a12dbda-istio-data\") pod \"model-routing-only-gateway-openshift-default-7d46dbd96b-dzr9v\" (UID: \"107fadeb-7edf-49eb-8e72-8b9c7a12dbda\") " pod="e2e-test-model-based-routing-only-fae765e6/model-routing-only-gateway-openshift-default-7d46dbd96b-dzr9v"
Oct 30 04:10:49.574987 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:10:49.574936    4461 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/107fadeb-7edf-49eb-8e72-8b9c7a12dbda-workload-certs\") pod \"model-routing-only-gateway-openshift-default-7d46dbd96b-dzr9v\" (UID: \"107fadeb-7edf-49eb-8e72-8b9c7a12dbda\") " pod="e2e-test-model-based-routing-only-fae765e6/model-routing-only-gateway-openshift-default-7d46dbd96b-dzr9v"
Oct 30 04:10:49.575180 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:10:49.575154    4461 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/107fadeb-7edf-49eb-8e72-8b9c7a12dbda-credential-socket\") pod \"model-routing-only-gateway-openshift-default-7d46dbd96b-dzr9v\" (UID: \"107fadeb-7edf-49eb-8e72-8b9c7a12dbda\") " pod="e2e-test-model-based-routing-only-fae765e6/model-routing-only-gateway-openshift-default-7d46dbd96b-dzr9v"
Oct 30 04:10:49.575344 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:10:49.575324    4461 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/107fadeb-7edf-49eb-8e72-8b9c7a12dbda-istiod-ca-cert\") pod \"model-routing-only-gateway-openshift-default-7d46dbd96b-dzr9v\" (UID: \"107fadeb-7edf-49eb-8e72-8b9c7a12dbda\") " pod="e2e-test-model-based-routing-only-fae765e6/model-routing-only-gateway-openshift-default-7d46dbd96b-dzr9v"
Oct 30 04:10:49.577028 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:10:49.576995    4461 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/107fadeb-7edf-49eb-8e72-8b9c7a12dbda-istio-envoy\") pod \"model-routing-only-gateway-openshift-default-7d46dbd96b-dzr9v\" (UID: \"107fadeb-7edf-49eb-8e72-8b9c7a12dbda\") " pod="e2e-test-model-based-routing-only-fae765e6/model-routing-only-gateway-openshift-default-7d46dbd96b-dzr9v"
Oct 30 04:10:49.577215 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:10:49.577195    4461 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/107fadeb-7edf-49eb-8e72-8b9c7a12dbda-istio-podinfo\") pod \"model-routing-only-gateway-openshift-default-7d46dbd96b-dzr9v\" (UID: \"107fadeb-7edf-49eb-8e72-8b9c7a12dbda\") " pod="e2e-test-model-based-routing-only-fae765e6/model-routing-only-gateway-openshift-default-7d46dbd96b-dzr9v"
Oct 30 04:10:49.598871 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:10:49.598804    4461 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/107fadeb-7edf-49eb-8e72-8b9c7a12dbda-istio-token\") pod \"model-routing-only-gateway-openshift-default-7d46dbd96b-dzr9v\" (UID: \"107fadeb-7edf-49eb-8e72-8b9c7a12dbda\") " pod="e2e-test-model-based-routing-only-fae765e6/model-routing-only-gateway-openshift-default-7d46dbd96b-dzr9v"
Oct 30 04:10:49.599063 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:10:49.599038    4461 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hrk54\" (UniqueName: \"kubernetes.io/projected/107fadeb-7edf-49eb-8e72-8b9c7a12dbda-kube-api-access-hrk54\") pod \"model-routing-only-gateway-openshift-default-7d46dbd96b-dzr9v\" (UID: \"107fadeb-7edf-49eb-8e72-8b9c7a12dbda\") " pod="e2e-test-model-based-routing-only-fae765e6/model-routing-only-gateway-openshift-default-7d46dbd96b-dzr9v"
Oct 30 04:10:49.638013 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:10:49.637960    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 04:10:49.663444 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:10:49.663412    4461 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-model-based-routing-only-fae765e6/model-routing-only-gateway-openshift-default-7d46dbd96b-dzr9v"
Oct 30 04:10:49.999202 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:10:49.999177    4461 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-model-based-routing-only-fae765e6/model-routing-only-gateway-openshift-default-7d46dbd96b-dzr9v"]
Oct 30 04:10:50.000399 ip-10-0-143-156 kubenswrapper[4461]: W1030 04:10:50.000367    4461 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod107fadeb_7edf_49eb_8e72_8b9c7a12dbda.slice/crio-18772f3b27c870198f84eac27a65a54add0367b4b72d1a6d0a8be2417011fc40 WatchSource:0}: Error finding container 18772f3b27c870198f84eac27a65a54add0367b4b72d1a6d0a8be2417011fc40: Status 404 returned error can't find the container with id 18772f3b27c870198f84eac27a65a54add0367b4b72d1a6d0a8be2417011fc40
Oct 30 04:10:50.002599 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:10:50.002570    4461 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 04:10:50.002672 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:10:50.002654    4461 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 04:10:50.002710 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:10:50.002682    4461 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 04:10:50.195655 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:10:50.195604    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 04:10:50.915803 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:10:50.915769    4461 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-model-based-routing-only-fae765e6/model-routing-only-gateway-openshift-default-7d46dbd96b-dzr9v" event={"ID":"107fadeb-7edf-49eb-8e72-8b9c7a12dbda","Type":"ContainerStarted","Data":"416bbd177ed2100a62b4c12f6b8b35cf43e4d7fd2910866803bae303428d8a74"}
Oct 30 04:10:50.915993 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:10:50.915821    4461 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-model-based-routing-only-fae765e6/model-routing-only-gateway-openshift-default-7d46dbd96b-dzr9v" event={"ID":"107fadeb-7edf-49eb-8e72-8b9c7a12dbda","Type":"ContainerStarted","Data":"18772f3b27c870198f84eac27a65a54add0367b4b72d1a6d0a8be2417011fc40"}
Oct 30 04:10:51.663976 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:10:51.663945    4461 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-model-based-routing-only-fae765e6/model-routing-only-gateway-openshift-default-7d46dbd96b-dzr9v"
Oct 30 04:10:51.669194 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:10:51.669166    4461 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-model-based-routing-only-fae765e6/model-routing-only-gateway-openshift-default-7d46dbd96b-dzr9v"
Oct 30 04:10:51.728862 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:10:51.728810    4461 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-test-model-based-routing-only-fae765e6/model-routing-only-gateway-openshift-default-7d46dbd96b-dzr9v" podStartSLOduration=2.728796274 podStartE2EDuration="2.728796274s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:10:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:10:50.986654024 +0000 UTC m=+4519.036459219" watchObservedRunningTime="2026-10-30 04:10:51.728796274 +0000 UTC m=+4519.778601450"
Oct 30 04:10:51.921109 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:10:51.921019    4461 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-model-based-routing-only-fae765e6/model-routing-only-gateway-openshift-default-7d46dbd96b-dzr9v"
Oct 30 04:10:51.922223 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:10:51.922198    4461 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-model-based-routing-only-fae765e6/model-routing-only-gateway-openshift-default-7d46dbd96b-dzr9v"
Oct 30 04:10:52.436373 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:10:52.436330    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 04:10:52.757822 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:10:52.757786    4461 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="fc64f170-9e8b-4ad8-8604-60849e50a5f2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/health\": dial tcp 10.133.0.54:8000: connect: connection refused"
Oct 30 04:10:53.020240 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:10:53.020155    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 04:10:54.107501 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:10:54.107467    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 04:10:54.504682 ip-10-0-143-156 kubenswrapper[4461]: E1030 04:10:54.504649    4461 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2k85b" containerName="extractor"
Oct 30 04:11:02.758113 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:11:02.758064    4461 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="fc64f170-9e8b-4ad8-8604-60849e50a5f2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/health\": dial tcp 10.133.0.54:8000: connect: connection refused"
Oct 30 04:11:03.972249 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:11:03.972213    4461 generic.go:356] "Generic (PLEG): container finished" podID="107fadeb-7edf-49eb-8e72-8b9c7a12dbda" containerID="416bbd177ed2100a62b4c12f6b8b35cf43e4d7fd2910866803bae303428d8a74" exitCode=0
Oct 30 04:11:03.978274 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:11:03.972290    4461 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-model-based-routing-only-fae765e6/model-routing-only-gateway-openshift-default-7d46dbd96b-dzr9v" event={"ID":"107fadeb-7edf-49eb-8e72-8b9c7a12dbda","Type":"ContainerDied","Data":"416bbd177ed2100a62b4c12f6b8b35cf43e4d7fd2910866803bae303428d8a74"}
Oct 30 04:11:03.978274 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:11:03.973177    4461 scope.go:122] "RemoveContainer" containerID="416bbd177ed2100a62b4c12f6b8b35cf43e4d7fd2910866803bae303428d8a74"
Oct 30 04:11:03.979003 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:11:03.978953    4461 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 04:11:03.979165 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:11:03.979138    4461 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 04:11:03.979275 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:11:03.979254    4461 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 04:11:04.987041 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:11:04.986981    4461 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-model-based-routing-only-fae765e6/model-routing-only-gateway-openshift-default-7d46dbd96b-dzr9v" event={"ID":"107fadeb-7edf-49eb-8e72-8b9c7a12dbda","Type":"ContainerStarted","Data":"f4e7a3eb6920cdd950e07f708713dcada5a3588e426a2f3cd52cebfe2e80a605"}
Oct 30 04:11:05.666123 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:11:05.665470    4461 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-model-based-routing-only-fae765e6/model-routing-only-gateway-openshift-default-7d46dbd96b-dzr9v"
Oct 30 04:11:05.667616 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:11:05.667301    4461 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-model-based-routing-only-fae765e6/model-routing-only-gateway-openshift-default-7d46dbd96b-dzr9v" podUID="107fadeb-7edf-49eb-8e72-8b9c7a12dbda" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused"
Oct 30 04:11:06.668122 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:11:06.665522    4461 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-model-based-routing-only-fae765e6/model-routing-only-gateway-openshift-default-7d46dbd96b-dzr9v" podUID="107fadeb-7edf-49eb-8e72-8b9c7a12dbda" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused"
Oct 30 04:11:06.921602 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:11:06.921509    4461 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-model-based-routing-only-fae765e6/model-routing-only-gateway-openshift-default-7d46dbd96b-dzr9v"
Oct 30 04:11:07.665563 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:11:07.665517    4461 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-model-based-routing-only-fae765e6/model-routing-only-gateway-openshift-default-7d46dbd96b-dzr9v" podUID="107fadeb-7edf-49eb-8e72-8b9c7a12dbda" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused"
Oct 30 04:11:08.672120 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:11:08.665553    4461 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-model-based-routing-only-fae765e6/model-routing-only-gateway-openshift-default-7d46dbd96b-dzr9v" podUID="107fadeb-7edf-49eb-8e72-8b9c7a12dbda" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused"
Oct 30 04:11:09.665548 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:11:09.665503    4461 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-model-based-routing-only-fae765e6/model-routing-only-gateway-openshift-default-7d46dbd96b-dzr9v" podUID="107fadeb-7edf-49eb-8e72-8b9c7a12dbda" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused"
Oct 30 04:11:10.668231 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:11:10.668196    4461 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-model-based-routing-only-fae765e6/model-routing-only-gateway-openshift-default-7d46dbd96b-dzr9v"
Oct 30 04:11:10.669197 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:11:10.669177    4461 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-model-based-routing-only-fae765e6/model-routing-only-gateway-openshift-default-7d46dbd96b-dzr9v"
Oct 30 04:11:12.758167 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:11:12.758120    4461 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="fc64f170-9e8b-4ad8-8604-60849e50a5f2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/health\": dial tcp 10.133.0.54:8000: connect: connection refused"
Oct 30 04:11:15.556900 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:11:15.556824    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 04:11:16.519895 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:11:16.519858    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 04:11:17.205587 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:11:17.205555    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 04:11:19.883805 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:11:19.883753    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 04:11:22.757791 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:11:22.757737    4461 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="fc64f170-9e8b-4ad8-8604-60849e50a5f2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/health\": dial tcp 10.133.0.54:8000: connect: connection refused"
Oct 30 04:11:23.230035 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:11:23.230000    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 04:11:24.504133 ip-10-0-143-156 kubenswrapper[4461]: E1030 04:11:24.504070    4461 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2k85b" containerName="extractor"
Oct 30 04:11:27.037705 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:11:27.037662    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 04:11:29.380308 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:11:29.380259    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 04:11:29.544828 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:11:29.544791    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 04:11:30.083620 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:11:30.083572    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 04:11:32.758055 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:11:32.758010    4461 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="fc64f170-9e8b-4ad8-8604-60849e50a5f2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/health\": dial tcp 10.133.0.54:8000: connect: connection refused"
Oct 30 04:11:42.757590 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:11:42.757535    4461 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="fc64f170-9e8b-4ad8-8604-60849e50a5f2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/health\": dial tcp 10.133.0.54:8000: connect: connection refused"
Oct 30 04:11:50.307989 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:11:50.307945    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 04:11:52.757755 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:11:52.757705    4461 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="fc64f170-9e8b-4ad8-8604-60849e50a5f2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/health\": dial tcp 10.133.0.54:8000: connect: connection refused"
Oct 30 04:11:54.504474 ip-10-0-143-156 kubenswrapper[4461]: E1030 04:11:54.504431    4461 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2k85b" containerName="extractor"
Oct 30 04:12:02.757819 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:12:02.757774    4461 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="fc64f170-9e8b-4ad8-8604-60849e50a5f2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/health\": dial tcp 10.133.0.54:8000: connect: connection refused"
Oct 30 04:12:08.481123 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:12:08.481068    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 04:12:08.652794 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:12:08.652756    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 04:12:12.767636 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:12:12.767593    4461 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Oct 30 04:12:12.775082 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:12:12.775057    4461 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Oct 30 04:12:13.206012 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:12:13.205934    4461 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-prefill-7885b78964-7wn6r"]
Oct 30 04:12:13.209120 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:12:13.209083    4461 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-prefill-7885b78964-7wn6r"
Oct 30 04:12:13.237610 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:12:13.237579    4461 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-prefill-7885b78964-7wn6r"]
Oct 30 04:12:13.244652 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:12:13.244629    4461 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-tls-resources-e4cad0c3\"/\"seaweedfs-s3-creds\""
Oct 30 04:12:13.244652 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:12:13.244649    4461 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-tls-resources-e4cad0c3\"/\"default-dockercfg-qshzj\""
Oct 30 04:12:13.244810 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:12:13.244743    4461 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-tls-resources-e4cad0c3\"/\"tls-verification-test-kserve-self-signed-certs\""
Oct 30 04:12:13.298243 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:12:13.298211    4461 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-tls-resources-e4cad0c3\"/\"odh-kserve-custom-ca-bundle\""
Oct 30 04:12:13.298396 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:12:13.298251    4461 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-tls-resources-e4cad0c3\"/\"openshift-service-ca.crt\""
Oct 30 04:12:13.298396 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:12:13.298315    4461 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-tls-resources-e4cad0c3\"/\"kube-root-ca.crt\""
Oct 30 04:12:13.322904 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:12:13.322879    4461 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c9f22c83-71df-4dd5-8eba-8c059009324c-home\") pod \"tls-verification-test-kserve-prefill-7885b78964-7wn6r\" (UID: \"c9f22c83-71df-4dd5-8eba-8c059009324c\") " pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-prefill-7885b78964-7wn6r"
Oct 30 04:12:13.323028 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:12:13.322914    4461 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rx2dg\" (UniqueName: \"kubernetes.io/projected/c9f22c83-71df-4dd5-8eba-8c059009324c-kube-api-access-rx2dg\") pod \"tls-verification-test-kserve-prefill-7885b78964-7wn6r\" (UID: \"c9f22c83-71df-4dd5-8eba-8c059009324c\") " pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-prefill-7885b78964-7wn6r"
Oct 30 04:12:13.323028 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:12:13.322936    4461 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c9f22c83-71df-4dd5-8eba-8c059009324c-model-cache\") pod \"tls-verification-test-kserve-prefill-7885b78964-7wn6r\" (UID: \"c9f22c83-71df-4dd5-8eba-8c059009324c\") " pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-prefill-7885b78964-7wn6r"
Oct 30 04:12:13.323129 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:12:13.323076    4461 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c9f22c83-71df-4dd5-8eba-8c059009324c-tmp-dir\") pod \"tls-verification-test-kserve-prefill-7885b78964-7wn6r\" (UID: \"c9f22c83-71df-4dd5-8eba-8c059009324c\") " pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-prefill-7885b78964-7wn6r"
Oct 30 04:12:13.323169 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:12:13.323126    4461 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c9f22c83-71df-4dd5-8eba-8c059009324c-cabundle-cert\") pod \"tls-verification-test-kserve-prefill-7885b78964-7wn6r\" (UID: \"c9f22c83-71df-4dd5-8eba-8c059009324c\") " pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-prefill-7885b78964-7wn6r"
Oct 30 04:12:13.323169 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:12:13.323148    4461 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c9f22c83-71df-4dd5-8eba-8c059009324c-kserve-provision-location\") pod \"tls-verification-test-kserve-prefill-7885b78964-7wn6r\" (UID: \"c9f22c83-71df-4dd5-8eba-8c059009324c\") " pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-prefill-7885b78964-7wn6r"
Oct 30 04:12:13.323232 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:12:13.323197    4461 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c9f22c83-71df-4dd5-8eba-8c059009324c-dshm\") pod \"tls-verification-test-kserve-prefill-7885b78964-7wn6r\" (UID: \"c9f22c83-71df-4dd5-8eba-8c059009324c\") " pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-prefill-7885b78964-7wn6r"
Oct 30 04:12:13.323278 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:12:13.323267    4461 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c9f22c83-71df-4dd5-8eba-8c059009324c-tls-certs\") pod \"tls-verification-test-kserve-prefill-7885b78964-7wn6r\" (UID: \"c9f22c83-71df-4dd5-8eba-8c059009324c\") " pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-prefill-7885b78964-7wn6r"
Oct 30 04:12:13.341653 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:12:13.341628    4461 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-model-based-routing-only-fae765e6/model-routing-only-gateway-openshift-default-7d46dbd96b-dzr9v"]
Oct 30 04:12:13.341825 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:12:13.341806    4461 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-test-model-based-routing-only-fae765e6/model-routing-only-gateway-openshift-default-7d46dbd96b-dzr9v" podUID="107fadeb-7edf-49eb-8e72-8b9c7a12dbda" containerName="istio-proxy" containerID="cri-o://f4e7a3eb6920cdd950e07f708713dcada5a3588e426a2f3cd52cebfe2e80a605" gracePeriod=30
Oct 30 04:12:13.384033 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:12:13.383998    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 04:12:13.423947 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:12:13.423919    4461 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c9f22c83-71df-4dd5-8eba-8c059009324c-tls-certs\") pod \"tls-verification-test-kserve-prefill-7885b78964-7wn6r\" (UID: \"c9f22c83-71df-4dd5-8eba-8c059009324c\") " pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-prefill-7885b78964-7wn6r"
Oct 30 04:12:13.424130 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:12:13.423954    4461 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c9f22c83-71df-4dd5-8eba-8c059009324c-home\") pod \"tls-verification-test-kserve-prefill-7885b78964-7wn6r\" (UID: \"c9f22c83-71df-4dd5-8eba-8c059009324c\") " pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-prefill-7885b78964-7wn6r"
Oct 30 04:12:13.424130 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:12:13.423995    4461 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rx2dg\" (UniqueName: \"kubernetes.io/projected/c9f22c83-71df-4dd5-8eba-8c059009324c-kube-api-access-rx2dg\") pod \"tls-verification-test-kserve-prefill-7885b78964-7wn6r\" (UID: \"c9f22c83-71df-4dd5-8eba-8c059009324c\") " pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-prefill-7885b78964-7wn6r"
Oct 30 04:12:13.424130 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:12:13.424024    4461 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c9f22c83-71df-4dd5-8eba-8c059009324c-model-cache\") pod \"tls-verification-test-kserve-prefill-7885b78964-7wn6r\" (UID: \"c9f22c83-71df-4dd5-8eba-8c059009324c\") " pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-prefill-7885b78964-7wn6r"
Oct 30 04:12:13.424130 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:12:13.424088    4461 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c9f22c83-71df-4dd5-8eba-8c059009324c-tmp-dir\") pod \"tls-verification-test-kserve-prefill-7885b78964-7wn6r\" (UID: \"c9f22c83-71df-4dd5-8eba-8c059009324c\") " pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-prefill-7885b78964-7wn6r"
Oct 30 04:12:13.424358 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:12:13.424150    4461 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c9f22c83-71df-4dd5-8eba-8c059009324c-cabundle-cert\") pod \"tls-verification-test-kserve-prefill-7885b78964-7wn6r\" (UID: \"c9f22c83-71df-4dd5-8eba-8c059009324c\") " pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-prefill-7885b78964-7wn6r"
Oct 30 04:12:13.424358 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:12:13.424176    4461 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c9f22c83-71df-4dd5-8eba-8c059009324c-kserve-provision-location\") pod \"tls-verification-test-kserve-prefill-7885b78964-7wn6r\" (UID: \"c9f22c83-71df-4dd5-8eba-8c059009324c\") " pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-prefill-7885b78964-7wn6r"
Oct 30 04:12:13.424358 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:12:13.424215    4461 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c9f22c83-71df-4dd5-8eba-8c059009324c-dshm\") pod \"tls-verification-test-kserve-prefill-7885b78964-7wn6r\" (UID: \"c9f22c83-71df-4dd5-8eba-8c059009324c\") " pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-prefill-7885b78964-7wn6r"
Oct 30 04:12:13.424511 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:12:13.424486    4461 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c9f22c83-71df-4dd5-8eba-8c059009324c-model-cache\") pod \"tls-verification-test-kserve-prefill-7885b78964-7wn6r\" (UID: \"c9f22c83-71df-4dd5-8eba-8c059009324c\") " pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-prefill-7885b78964-7wn6r"
Oct 30 04:12:13.424632 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:12:13.424605    4461 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c9f22c83-71df-4dd5-8eba-8c059009324c-home\") pod \"tls-verification-test-kserve-prefill-7885b78964-7wn6r\" (UID: \"c9f22c83-71df-4dd5-8eba-8c059009324c\") " pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-prefill-7885b78964-7wn6r"
Oct 30 04:12:13.424823 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:12:13.424797    4461 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c9f22c83-71df-4dd5-8eba-8c059009324c-kserve-provision-location\") pod \"tls-verification-test-kserve-prefill-7885b78964-7wn6r\" (UID: \"c9f22c83-71df-4dd5-8eba-8c059009324c\") " pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-prefill-7885b78964-7wn6r"
Oct 30 04:12:13.424959 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:12:13.424933    4461 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c9f22c83-71df-4dd5-8eba-8c059009324c-cabundle-cert\") pod \"tls-verification-test-kserve-prefill-7885b78964-7wn6r\" (UID: \"c9f22c83-71df-4dd5-8eba-8c059009324c\") " pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-prefill-7885b78964-7wn6r"
Oct 30 04:12:13.425045 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:12:13.424936    4461 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c9f22c83-71df-4dd5-8eba-8c059009324c-tmp-dir\") pod \"tls-verification-test-kserve-prefill-7885b78964-7wn6r\" (UID: \"c9f22c83-71df-4dd5-8eba-8c059009324c\") " pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-prefill-7885b78964-7wn6r"
Oct 30 04:12:13.426722 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:12:13.426700    4461 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c9f22c83-71df-4dd5-8eba-8c059009324c-tls-certs\") pod \"tls-verification-test-kserve-prefill-7885b78964-7wn6r\" (UID: \"c9f22c83-71df-4dd5-8eba-8c059009324c\") " pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-prefill-7885b78964-7wn6r"
Oct 30 04:12:13.426800 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:12:13.426712    4461 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c9f22c83-71df-4dd5-8eba-8c059009324c-dshm\") pod \"tls-verification-test-kserve-prefill-7885b78964-7wn6r\" (UID: \"c9f22c83-71df-4dd5-8eba-8c059009324c\") " pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-prefill-7885b78964-7wn6r"
Oct 30 04:12:13.447569 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:12:13.447542    4461 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rx2dg\" (UniqueName: \"kubernetes.io/projected/c9f22c83-71df-4dd5-8eba-8c059009324c-kube-api-access-rx2dg\") pod \"tls-verification-test-kserve-prefill-7885b78964-7wn6r\" (UID: \"c9f22c83-71df-4dd5-8eba-8c059009324c\") " pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-prefill-7885b78964-7wn6r"
Oct 30 04:12:13.525061 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:12:13.524988    4461 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-prefill-7885b78964-7wn6r"
Oct 30 04:12:13.622195 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:12:13.622165    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 04:12:13.801054 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:12:13.800972    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 04:12:13.848608 ip-10-0-143-156 kubenswrapper[4461]: W1030 04:12:13.848580    4461 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc9f22c83_71df_4dd5_8eba_8c059009324c.slice/crio-5da21b72e5d40194465ea9d7231c9bb22c303f88b2ec36069a2a361da018565d WatchSource:0}: Error finding container 5da21b72e5d40194465ea9d7231c9bb22c303f88b2ec36069a2a361da018565d: Status 404 returned error can't find the container with id 5da21b72e5d40194465ea9d7231c9bb22c303f88b2ec36069a2a361da018565d
Oct 30 04:12:13.852562 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:12:13.852540    4461 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-prefill-7885b78964-7wn6r"]
Oct 30 04:12:14.282186 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:12:14.282151    4461 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-prefill-7885b78964-7wn6r" event={"ID":"c9f22c83-71df-4dd5-8eba-8c059009324c","Type":"ContainerStarted","Data":"8780f09da0fb87ecba5a9121120b03659ea947cae46b800b132a984df1a7a1de"}
Oct 30 04:12:14.282351 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:12:14.282206    4461 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-prefill-7885b78964-7wn6r" event={"ID":"c9f22c83-71df-4dd5-8eba-8c059009324c","Type":"ContainerStarted","Data":"5da21b72e5d40194465ea9d7231c9bb22c303f88b2ec36069a2a361da018565d"}
Oct 30 04:12:17.298478 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:12:17.298446    4461 generic.go:356] "Generic (PLEG): container finished" podID="c9f22c83-71df-4dd5-8eba-8c059009324c" containerID="8780f09da0fb87ecba5a9121120b03659ea947cae46b800b132a984df1a7a1de" exitCode=0
Oct 30 04:12:17.298938 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:12:17.298489    4461 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-prefill-7885b78964-7wn6r" event={"ID":"c9f22c83-71df-4dd5-8eba-8c059009324c","Type":"ContainerDied","Data":"8780f09da0fb87ecba5a9121120b03659ea947cae46b800b132a984df1a7a1de"}
Oct 30 04:12:18.304983 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:12:18.304942    4461 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-prefill-7885b78964-7wn6r" event={"ID":"c9f22c83-71df-4dd5-8eba-8c059009324c","Type":"ContainerStarted","Data":"cc9dbc2165dfa82d386d825ad370ca30fb535fc087ada35099faf8c1fc75cb12"}
Oct 30 04:12:18.628470 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:12:18.628434    4461 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-model-based-routing-only-fae765e6/model-routing-only-gateway-openshift-default-7d46dbd96b-dzr9v"
Oct 30 04:12:18.677484 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:12:18.676209    4461 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-prefill-7885b78964-7wn6r" podStartSLOduration=3.230973872 podStartE2EDuration="5.676185458s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.445211586s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:12:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:12:18.381453324 +0000 UTC m=+4606.431258541" watchObservedRunningTime="2026-10-30 04:12:18.676185458 +0000 UTC m=+4606.725990645"
Oct 30 04:12:18.690025 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:12:18.688365    4461 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/107fadeb-7edf-49eb-8e72-8b9c7a12dbda-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/107fadeb-7edf-49eb-8e72-8b9c7a12dbda-credential-socket\") pod \"107fadeb-7edf-49eb-8e72-8b9c7a12dbda\" (UID: \"107fadeb-7edf-49eb-8e72-8b9c7a12dbda\") "
Oct 30 04:12:18.690025 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:12:18.688440    4461 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/107fadeb-7edf-49eb-8e72-8b9c7a12dbda-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/107fadeb-7edf-49eb-8e72-8b9c7a12dbda-istio-data\") pod \"107fadeb-7edf-49eb-8e72-8b9c7a12dbda\" (UID: \"107fadeb-7edf-49eb-8e72-8b9c7a12dbda\") "
Oct 30 04:12:18.690025 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:12:18.688503    4461 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/107fadeb-7edf-49eb-8e72-8b9c7a12dbda-istio-token\" (UniqueName: \"kubernetes.io/projected/107fadeb-7edf-49eb-8e72-8b9c7a12dbda-istio-token\") pod \"107fadeb-7edf-49eb-8e72-8b9c7a12dbda\" (UID: \"107fadeb-7edf-49eb-8e72-8b9c7a12dbda\") "
Oct 30 04:12:18.690025 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:12:18.688541    4461 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/107fadeb-7edf-49eb-8e72-8b9c7a12dbda-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/107fadeb-7edf-49eb-8e72-8b9c7a12dbda-istio-envoy\") pod \"107fadeb-7edf-49eb-8e72-8b9c7a12dbda\" (UID: \"107fadeb-7edf-49eb-8e72-8b9c7a12dbda\") "
Oct 30 04:12:18.690025 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:12:18.688617    4461 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/107fadeb-7edf-49eb-8e72-8b9c7a12dbda-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/107fadeb-7edf-49eb-8e72-8b9c7a12dbda-istiod-ca-cert\") pod \"107fadeb-7edf-49eb-8e72-8b9c7a12dbda\" (UID: \"107fadeb-7edf-49eb-8e72-8b9c7a12dbda\") "
Oct 30 04:12:18.690025 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:12:18.688663    4461 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/107fadeb-7edf-49eb-8e72-8b9c7a12dbda-kube-api-access-hrk54\" (UniqueName: \"kubernetes.io/projected/107fadeb-7edf-49eb-8e72-8b9c7a12dbda-kube-api-access-hrk54\") pod \"107fadeb-7edf-49eb-8e72-8b9c7a12dbda\" (UID: \"107fadeb-7edf-49eb-8e72-8b9c7a12dbda\") "
Oct 30 04:12:18.690025 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:12:18.688696    4461 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/107fadeb-7edf-49eb-8e72-8b9c7a12dbda-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/107fadeb-7edf-49eb-8e72-8b9c7a12dbda-istio-podinfo\") pod \"107fadeb-7edf-49eb-8e72-8b9c7a12dbda\" (UID: \"107fadeb-7edf-49eb-8e72-8b9c7a12dbda\") "
Oct 30 04:12:18.690025 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:12:18.688735    4461 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/107fadeb-7edf-49eb-8e72-8b9c7a12dbda-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/107fadeb-7edf-49eb-8e72-8b9c7a12dbda-workload-certs\") pod \"107fadeb-7edf-49eb-8e72-8b9c7a12dbda\" (UID: \"107fadeb-7edf-49eb-8e72-8b9c7a12dbda\") "
Oct 30 04:12:18.690025 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:12:18.688776    4461 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/107fadeb-7edf-49eb-8e72-8b9c7a12dbda-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/107fadeb-7edf-49eb-8e72-8b9c7a12dbda-workload-socket\") pod \"107fadeb-7edf-49eb-8e72-8b9c7a12dbda\" (UID: \"107fadeb-7edf-49eb-8e72-8b9c7a12dbda\") "
Oct 30 04:12:18.690997 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:12:18.690731    4461 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/107fadeb-7edf-49eb-8e72-8b9c7a12dbda-workload-socket" pod "107fadeb-7edf-49eb-8e72-8b9c7a12dbda" (UID: "107fadeb-7edf-49eb-8e72-8b9c7a12dbda"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:12:18.691356 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:12:18.691158    4461 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/107fadeb-7edf-49eb-8e72-8b9c7a12dbda-credential-socket" pod "107fadeb-7edf-49eb-8e72-8b9c7a12dbda" (UID: "107fadeb-7edf-49eb-8e72-8b9c7a12dbda"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:12:18.691851 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:12:18.691499    4461 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/107fadeb-7edf-49eb-8e72-8b9c7a12dbda-istio-data" pod "107fadeb-7edf-49eb-8e72-8b9c7a12dbda" (UID: "107fadeb-7edf-49eb-8e72-8b9c7a12dbda"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:12:18.693572 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:12:18.693515    4461 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/107fadeb-7edf-49eb-8e72-8b9c7a12dbda-workload-certs" pod "107fadeb-7edf-49eb-8e72-8b9c7a12dbda" (UID: "107fadeb-7edf-49eb-8e72-8b9c7a12dbda"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:12:18.695151 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:12:18.693853    4461 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/107fadeb-7edf-49eb-8e72-8b9c7a12dbda-istiod-ca-cert" pod "107fadeb-7edf-49eb-8e72-8b9c7a12dbda" (UID: "107fadeb-7edf-49eb-8e72-8b9c7a12dbda"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:12:18.699341 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:12:18.699286    4461 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/107fadeb-7edf-49eb-8e72-8b9c7a12dbda-istio-podinfo" pod "107fadeb-7edf-49eb-8e72-8b9c7a12dbda" (UID: "107fadeb-7edf-49eb-8e72-8b9c7a12dbda"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 04:12:18.718638 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:12:18.718570    4461 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/107fadeb-7edf-49eb-8e72-8b9c7a12dbda-kube-api-access-hrk54" pod "107fadeb-7edf-49eb-8e72-8b9c7a12dbda" (UID: "107fadeb-7edf-49eb-8e72-8b9c7a12dbda"). InnerVolumeSpecName "kube-api-access-hrk54". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:12:18.718868 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:12:18.718839    4461 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/107fadeb-7edf-49eb-8e72-8b9c7a12dbda-istio-envoy" pod "107fadeb-7edf-49eb-8e72-8b9c7a12dbda" (UID: "107fadeb-7edf-49eb-8e72-8b9c7a12dbda"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:12:18.719330 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:12:18.719296    4461 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/107fadeb-7edf-49eb-8e72-8b9c7a12dbda-istio-token" pod "107fadeb-7edf-49eb-8e72-8b9c7a12dbda" (UID: "107fadeb-7edf-49eb-8e72-8b9c7a12dbda"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:12:18.790759 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:12:18.790719    4461 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/107fadeb-7edf-49eb-8e72-8b9c7a12dbda-istio-data\") on node \"ip-10-0-143-156.ec2.internal\" DevicePath \"\""
Oct 30 04:12:18.790759 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:12:18.790761    4461 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/107fadeb-7edf-49eb-8e72-8b9c7a12dbda-istio-token\") on node \"ip-10-0-143-156.ec2.internal\" DevicePath \"\""
Oct 30 04:12:18.790990 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:12:18.790776    4461 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/107fadeb-7edf-49eb-8e72-8b9c7a12dbda-istio-envoy\") on node \"ip-10-0-143-156.ec2.internal\" DevicePath \"\""
Oct 30 04:12:18.790990 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:12:18.790792    4461 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/107fadeb-7edf-49eb-8e72-8b9c7a12dbda-istiod-ca-cert\") on node \"ip-10-0-143-156.ec2.internal\" DevicePath \"\""
Oct 30 04:12:18.790990 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:12:18.790808    4461 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hrk54\" (UniqueName: \"kubernetes.io/projected/107fadeb-7edf-49eb-8e72-8b9c7a12dbda-kube-api-access-hrk54\") on node \"ip-10-0-143-156.ec2.internal\" DevicePath \"\""
Oct 30 04:12:18.790990 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:12:18.790824    4461 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/107fadeb-7edf-49eb-8e72-8b9c7a12dbda-istio-podinfo\") on node \"ip-10-0-143-156.ec2.internal\" DevicePath \"\""
Oct 30 04:12:18.790990 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:12:18.790839    4461 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/107fadeb-7edf-49eb-8e72-8b9c7a12dbda-workload-certs\") on node \"ip-10-0-143-156.ec2.internal\" DevicePath \"\""
Oct 30 04:12:18.790990 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:12:18.790853    4461 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/107fadeb-7edf-49eb-8e72-8b9c7a12dbda-workload-socket\") on node \"ip-10-0-143-156.ec2.internal\" DevicePath \"\""
Oct 30 04:12:18.790990 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:12:18.790869    4461 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/107fadeb-7edf-49eb-8e72-8b9c7a12dbda-credential-socket\") on node \"ip-10-0-143-156.ec2.internal\" DevicePath \"\""
Oct 30 04:12:19.312080 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:12:19.312046    4461 generic.go:356] "Generic (PLEG): container finished" podID="107fadeb-7edf-49eb-8e72-8b9c7a12dbda" containerID="f4e7a3eb6920cdd950e07f708713dcada5a3588e426a2f3cd52cebfe2e80a605" exitCode=0
Oct 30 04:12:19.312531 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:12:19.312106    4461 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-model-based-routing-only-fae765e6/model-routing-only-gateway-openshift-default-7d46dbd96b-dzr9v"
Oct 30 04:12:19.312531 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:12:19.312136    4461 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-model-based-routing-only-fae765e6/model-routing-only-gateway-openshift-default-7d46dbd96b-dzr9v" event={"ID":"107fadeb-7edf-49eb-8e72-8b9c7a12dbda","Type":"ContainerDied","Data":"f4e7a3eb6920cdd950e07f708713dcada5a3588e426a2f3cd52cebfe2e80a605"}
Oct 30 04:12:19.312531 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:12:19.312196    4461 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-model-based-routing-only-fae765e6/model-routing-only-gateway-openshift-default-7d46dbd96b-dzr9v" event={"ID":"107fadeb-7edf-49eb-8e72-8b9c7a12dbda","Type":"ContainerDied","Data":"18772f3b27c870198f84eac27a65a54add0367b4b72d1a6d0a8be2417011fc40"}
Oct 30 04:12:19.312531 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:12:19.312219    4461 scope.go:122] "RemoveContainer" containerID="f4e7a3eb6920cdd950e07f708713dcada5a3588e426a2f3cd52cebfe2e80a605"
Oct 30 04:12:19.320836 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:12:19.320810    4461 scope.go:122] "RemoveContainer" containerID="416bbd177ed2100a62b4c12f6b8b35cf43e4d7fd2910866803bae303428d8a74"
Oct 30 04:12:19.328986 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:12:19.328960    4461 scope.go:122] "RemoveContainer" containerID="f4e7a3eb6920cdd950e07f708713dcada5a3588e426a2f3cd52cebfe2e80a605"
Oct 30 04:12:19.329297 ip-10-0-143-156 kubenswrapper[4461]: E1030 04:12:19.329274    4461 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f4e7a3eb6920cdd950e07f708713dcada5a3588e426a2f3cd52cebfe2e80a605\": container with ID starting with f4e7a3eb6920cdd950e07f708713dcada5a3588e426a2f3cd52cebfe2e80a605 not found: ID does not exist" containerID="f4e7a3eb6920cdd950e07f708713dcada5a3588e426a2f3cd52cebfe2e80a605"
Oct 30 04:12:19.329407 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:12:19.329304    4461 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f4e7a3eb6920cdd950e07f708713dcada5a3588e426a2f3cd52cebfe2e80a605"} err="failed to get container status \"f4e7a3eb6920cdd950e07f708713dcada5a3588e426a2f3cd52cebfe2e80a605\": rpc error: code = NotFound desc = could not find container \"f4e7a3eb6920cdd950e07f708713dcada5a3588e426a2f3cd52cebfe2e80a605\": container with ID starting with f4e7a3eb6920cdd950e07f708713dcada5a3588e426a2f3cd52cebfe2e80a605 not found: ID does not exist"
Oct 30 04:12:19.329407 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:12:19.329321    4461 scope.go:122] "RemoveContainer" containerID="416bbd177ed2100a62b4c12f6b8b35cf43e4d7fd2910866803bae303428d8a74"
Oct 30 04:12:19.329579 ip-10-0-143-156 kubenswrapper[4461]: E1030 04:12:19.329561    4461 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"416bbd177ed2100a62b4c12f6b8b35cf43e4d7fd2910866803bae303428d8a74\": container with ID starting with 416bbd177ed2100a62b4c12f6b8b35cf43e4d7fd2910866803bae303428d8a74 not found: ID does not exist" containerID="416bbd177ed2100a62b4c12f6b8b35cf43e4d7fd2910866803bae303428d8a74"
Oct 30 04:12:19.329646 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:12:19.329586    4461 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"416bbd177ed2100a62b4c12f6b8b35cf43e4d7fd2910866803bae303428d8a74"} err="failed to get container status \"416bbd177ed2100a62b4c12f6b8b35cf43e4d7fd2910866803bae303428d8a74\": rpc error: code = NotFound desc = could not find container \"416bbd177ed2100a62b4c12f6b8b35cf43e4d7fd2910866803bae303428d8a74\": container with ID starting with 416bbd177ed2100a62b4c12f6b8b35cf43e4d7fd2910866803bae303428d8a74 not found: ID does not exist"
Oct 30 04:12:19.364322 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:12:19.364289    4461 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-model-based-routing-only-fae765e6/model-routing-only-gateway-openshift-default-7d46dbd96b-dzr9v"]
Oct 30 04:12:19.377030 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:12:19.377000    4461 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-test-model-based-routing-only-fae765e6/model-routing-only-gateway-openshift-default-7d46dbd96b-dzr9v"]
Oct 30 04:12:19.739589 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:12:19.739550    4461 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"]
Oct 30 04:12:19.773680 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:12:19.773649    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 04:12:19.963907 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:12:19.963867    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 04:12:20.115463 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:12:20.115377    4461 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="107fadeb-7edf-49eb-8e72-8b9c7a12dbda" path="/var/lib/kubelet/pods/107fadeb-7edf-49eb-8e72-8b9c7a12dbda/volumes"
Oct 30 04:12:21.073630 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:12:21.073598    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 04:12:23.525283 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:12:23.525247    4461 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-prefill-7885b78964-7wn6r"
Oct 30 04:12:23.525775 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:12:23.525326    4461 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-prefill-7885b78964-7wn6r"
Oct 30 04:12:23.526690 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:12:23.526662    4461 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-prefill-7885b78964-7wn6r" podUID="c9f22c83-71df-4dd5-8eba-8c059009324c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/health\": dial tcp 10.133.0.56:8000: connect: connection refused"
Oct 30 04:12:24.504944 ip-10-0-143-156 kubenswrapper[4461]: E1030 04:12:24.504898    4461 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2k85b" containerName="extractor"
Oct 30 04:12:30.207862 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:12:30.207824    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 04:12:33.525711 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:12:33.525669    4461 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-prefill-7885b78964-7wn6r" podUID="c9f22c83-71df-4dd5-8eba-8c059009324c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/health\": dial tcp 10.133.0.56:8000: connect: connection refused"
Oct 30 04:12:34.721508 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:12:34.721472    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 04:12:34.762603 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:12:34.762551    4461 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="fc64f170-9e8b-4ad8-8604-60849e50a5f2" containerName="main" containerID="cri-o://05d20f812ea9305fa946b8f47f4d3684493db9a578c670ed15c53401f451e15c" gracePeriod=45
Oct 30 04:12:34.888914 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:12:34.888872    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 04:12:35.382889 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:12:35.382857    4461 generic.go:356] "Generic (PLEG): container finished" podID="fc64f170-9e8b-4ad8-8604-60849e50a5f2" containerID="05d20f812ea9305fa946b8f47f4d3684493db9a578c670ed15c53401f451e15c" exitCode=0
Oct 30 04:12:35.383035 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:12:35.382906    4461 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" event={"ID":"fc64f170-9e8b-4ad8-8604-60849e50a5f2","Type":"ContainerDied","Data":"05d20f812ea9305fa946b8f47f4d3684493db9a578c670ed15c53401f451e15c"}
Oct 30 04:12:35.514412 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:12:35.514383    4461 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Oct 30 04:12:35.551508 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:12:35.551418    4461 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fc64f170-9e8b-4ad8-8604-60849e50a5f2-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fc64f170-9e8b-4ad8-8604-60849e50a5f2-kserve-provision-location\") pod \"fc64f170-9e8b-4ad8-8604-60849e50a5f2\" (UID: \"fc64f170-9e8b-4ad8-8604-60849e50a5f2\") "
Oct 30 04:12:35.551508 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:12:35.551468    4461 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fc64f170-9e8b-4ad8-8604-60849e50a5f2-tls-certs\" (UniqueName: \"kubernetes.io/secret/fc64f170-9e8b-4ad8-8604-60849e50a5f2-tls-certs\") pod \"fc64f170-9e8b-4ad8-8604-60849e50a5f2\" (UID: \"fc64f170-9e8b-4ad8-8604-60849e50a5f2\") "
Oct 30 04:12:35.551508 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:12:35.551503    4461 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fc64f170-9e8b-4ad8-8604-60849e50a5f2-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fc64f170-9e8b-4ad8-8604-60849e50a5f2-model-cache\") pod \"fc64f170-9e8b-4ad8-8604-60849e50a5f2\" (UID: \"fc64f170-9e8b-4ad8-8604-60849e50a5f2\") "
Oct 30 04:12:35.551816 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:12:35.551539    4461 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fc64f170-9e8b-4ad8-8604-60849e50a5f2-home\" (UniqueName: \"kubernetes.io/empty-dir/fc64f170-9e8b-4ad8-8604-60849e50a5f2-home\") pod \"fc64f170-9e8b-4ad8-8604-60849e50a5f2\" (UID: \"fc64f170-9e8b-4ad8-8604-60849e50a5f2\") "
Oct 30 04:12:35.551816 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:12:35.551582    4461 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fc64f170-9e8b-4ad8-8604-60849e50a5f2-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fc64f170-9e8b-4ad8-8604-60849e50a5f2-cabundle-cert\") pod \"fc64f170-9e8b-4ad8-8604-60849e50a5f2\" (UID: \"fc64f170-9e8b-4ad8-8604-60849e50a5f2\") "
Oct 30 04:12:35.551816 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:12:35.551666    4461 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fc64f170-9e8b-4ad8-8604-60849e50a5f2-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fc64f170-9e8b-4ad8-8604-60849e50a5f2-tmp-dir\") pod \"fc64f170-9e8b-4ad8-8604-60849e50a5f2\" (UID: \"fc64f170-9e8b-4ad8-8604-60849e50a5f2\") "
Oct 30 04:12:35.551816 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:12:35.551699    4461 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fc64f170-9e8b-4ad8-8604-60849e50a5f2-kube-api-access-kr6hb\" (UniqueName: \"kubernetes.io/projected/fc64f170-9e8b-4ad8-8604-60849e50a5f2-kube-api-access-kr6hb\") pod \"fc64f170-9e8b-4ad8-8604-60849e50a5f2\" (UID: \"fc64f170-9e8b-4ad8-8604-60849e50a5f2\") "
Oct 30 04:12:35.551816 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:12:35.551736    4461 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fc64f170-9e8b-4ad8-8604-60849e50a5f2-dshm\" (UniqueName: \"kubernetes.io/empty-dir/fc64f170-9e8b-4ad8-8604-60849e50a5f2-dshm\") pod \"fc64f170-9e8b-4ad8-8604-60849e50a5f2\" (UID: \"fc64f170-9e8b-4ad8-8604-60849e50a5f2\") "
Oct 30 04:12:35.552211 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:12:35.552181    4461 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fc64f170-9e8b-4ad8-8604-60849e50a5f2-cabundle-cert" pod "fc64f170-9e8b-4ad8-8604-60849e50a5f2" (UID: "fc64f170-9e8b-4ad8-8604-60849e50a5f2"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:12:35.552441 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:12:35.552423    4461 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc64f170-9e8b-4ad8-8604-60849e50a5f2-model-cache" pod "fc64f170-9e8b-4ad8-8604-60849e50a5f2" (UID: "fc64f170-9e8b-4ad8-8604-60849e50a5f2"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:12:35.553339 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:12:35.553053    4461 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc64f170-9e8b-4ad8-8604-60849e50a5f2-home" pod "fc64f170-9e8b-4ad8-8604-60849e50a5f2" (UID: "fc64f170-9e8b-4ad8-8604-60849e50a5f2"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:12:35.556378 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:12:35.556339    4461 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc64f170-9e8b-4ad8-8604-60849e50a5f2-dshm" pod "fc64f170-9e8b-4ad8-8604-60849e50a5f2" (UID: "fc64f170-9e8b-4ad8-8604-60849e50a5f2"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:12:35.560407 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:12:35.560379    4461 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fc64f170-9e8b-4ad8-8604-60849e50a5f2-tls-certs" pod "fc64f170-9e8b-4ad8-8604-60849e50a5f2" (UID: "fc64f170-9e8b-4ad8-8604-60849e50a5f2"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:12:35.560835 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:12:35.560791    4461 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fc64f170-9e8b-4ad8-8604-60849e50a5f2-kube-api-access-kr6hb" pod "fc64f170-9e8b-4ad8-8604-60849e50a5f2" (UID: "fc64f170-9e8b-4ad8-8604-60849e50a5f2"). InnerVolumeSpecName "kube-api-access-kr6hb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:12:35.574220 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:12:35.574188    4461 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc64f170-9e8b-4ad8-8604-60849e50a5f2-tmp-dir" pod "fc64f170-9e8b-4ad8-8604-60849e50a5f2" (UID: "fc64f170-9e8b-4ad8-8604-60849e50a5f2"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:12:35.583798 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:12:35.583762    4461 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc64f170-9e8b-4ad8-8604-60849e50a5f2-kserve-provision-location" pod "fc64f170-9e8b-4ad8-8604-60849e50a5f2" (UID: "fc64f170-9e8b-4ad8-8604-60849e50a5f2"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:12:35.652568 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:12:35.652534    4461 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fc64f170-9e8b-4ad8-8604-60849e50a5f2-cabundle-cert\") on node \"ip-10-0-143-156.ec2.internal\" DevicePath \"\""
Oct 30 04:12:35.652743 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:12:35.652583    4461 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fc64f170-9e8b-4ad8-8604-60849e50a5f2-tmp-dir\") on node \"ip-10-0-143-156.ec2.internal\" DevicePath \"\""
Oct 30 04:12:35.652743 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:12:35.652592    4461 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kr6hb\" (UniqueName: \"kubernetes.io/projected/fc64f170-9e8b-4ad8-8604-60849e50a5f2-kube-api-access-kr6hb\") on node \"ip-10-0-143-156.ec2.internal\" DevicePath \"\""
Oct 30 04:12:35.652743 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:12:35.652602    4461 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fc64f170-9e8b-4ad8-8604-60849e50a5f2-dshm\") on node \"ip-10-0-143-156.ec2.internal\" DevicePath \"\""
Oct 30 04:12:35.652743 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:12:35.652614    4461 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fc64f170-9e8b-4ad8-8604-60849e50a5f2-kserve-provision-location\") on node \"ip-10-0-143-156.ec2.internal\" DevicePath \"\""
Oct 30 04:12:35.652743 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:12:35.652624    4461 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fc64f170-9e8b-4ad8-8604-60849e50a5f2-tls-certs\") on node \"ip-10-0-143-156.ec2.internal\" DevicePath \"\""
Oct 30 04:12:35.652743 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:12:35.652633    4461 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fc64f170-9e8b-4ad8-8604-60849e50a5f2-model-cache\") on node \"ip-10-0-143-156.ec2.internal\" DevicePath \"\""
Oct 30 04:12:35.652743 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:12:35.652640    4461 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fc64f170-9e8b-4ad8-8604-60849e50a5f2-home\") on node \"ip-10-0-143-156.ec2.internal\" DevicePath \"\""
Oct 30 04:12:36.388593 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:12:36.388512    4461 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Oct 30 04:12:36.389061 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:12:36.388520    4461 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" event={"ID":"fc64f170-9e8b-4ad8-8604-60849e50a5f2","Type":"ContainerDied","Data":"920adc7da8be88ef67a5136d8b2f2958d82988fd4ddf7a328a55cea458062f36"}
Oct 30 04:12:36.389061 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:12:36.388661    4461 scope.go:122] "RemoveContainer" containerID="05d20f812ea9305fa946b8f47f4d3684493db9a578c670ed15c53401f451e15c"
Oct 30 04:12:36.399733 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:12:36.399690    4461 scope.go:122] "RemoveContainer" containerID="dc158230868e5ff2e377d9a72256434644161e08a140f5001fb4826e3d0ddcb2"
Oct 30 04:12:36.454599 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:12:36.454567    4461 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"]
Oct 30 04:12:36.467062 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:12:36.467027    4461 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"]
Oct 30 04:12:36.634763 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:12:36.634722    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 04:12:38.115133 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:12:38.115077    4461 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fc64f170-9e8b-4ad8-8604-60849e50a5f2" path="/var/lib/kubelet/pods/fc64f170-9e8b-4ad8-8604-60849e50a5f2/volumes"
Oct 30 04:12:42.571650 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:12:42.571612    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 04:12:43.526143 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:12:43.526087    4461 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-prefill-7885b78964-7wn6r" podUID="c9f22c83-71df-4dd5-8eba-8c059009324c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/health\": dial tcp 10.133.0.56:8000: connect: connection refused"
Oct 30 04:12:50.446721 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:12:50.446532    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 04:12:51.297116 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:12:51.297062    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 04:12:53.007628 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:12:53.007596    4461 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-rh6p2"]
Oct 30 04:12:53.008130 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:12:53.007846    4461 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-rh6p2" podUID="19942a3a-cdfa-4618-83b3-f828b5cbe33e" containerName="istio-proxy" containerID="cri-o://c01d0ed4e0a04302e11ad4948c310b7a78840708bfce0145303f3972da480ff9" gracePeriod=30
Oct 30 04:12:53.020949 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:12:53.020871    4461 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-jtcmr"]
Oct 30 04:12:53.021243 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:12:53.021131    4461 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-jtcmr" podUID="aedb2e20-2e07-42b1-b3c2-27223dc8d177" containerName="istio-proxy" containerID="cri-o://efda88805e2304ccd46923be30dbb7d41ea73c74a999c6b9069f3b56afcbff0d" gracePeriod=30
Oct 30 04:12:53.022194 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:12:53.022170    4461 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-v4p2m"]
Oct 30 04:12:53.022342 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:12:53.022323    4461 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-v4p2m" podUID="7c55427d-9f1e-4b24-9741-3ff0cfe751e7" containerName="istio-proxy" containerID="cri-o://a26c32ebcc68eacfe09420438374d0a20cb6f19f19151fadbd2cfa3ebc793af2" gracePeriod=30
Oct 30 04:12:53.032703 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:12:53.032674    4461 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-nfxr8"]
Oct 30 04:12:53.032870 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:12:53.032849    4461 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-nfxr8" podUID="1dacbabe-8d44-4ebc-be3d-fb89d9f70742" containerName="istio-proxy" containerID="cri-o://7e75ec4cfc7f6817eb5ddad8c41febdfe278337b1cd941ee5b4ea8fbf2cc7b37" gracePeriod=30
Oct 30 04:12:53.525394 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:12:53.525357    4461 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-prefill-7885b78964-7wn6r" podUID="c9f22c83-71df-4dd5-8eba-8c059009324c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/health\": dial tcp 10.133.0.56:8000: connect: connection refused"
Oct 30 04:12:54.504881 ip-10-0-143-156 kubenswrapper[4461]: E1030 04:12:54.504846    4461 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2k85b" containerName="extractor"
Oct 30 04:12:56.339330 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:12:56.339290    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 04:12:58.129344 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:12:58.129295    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 04:12:58.315582 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:12:58.315556    4461 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-v4p2m"
Oct 30 04:12:58.331765 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:12:58.331741    4461 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-jtcmr"
Oct 30 04:12:58.335830 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:12:58.335809    4461 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-nfxr8"
Oct 30 04:12:58.339175 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:12:58.339158    4461 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-rh6p2"
Oct 30 04:12:58.474929 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:12:58.474891    4461 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1dacbabe-8d44-4ebc-be3d-fb89d9f70742-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1dacbabe-8d44-4ebc-be3d-fb89d9f70742-workload-socket\") pod \"1dacbabe-8d44-4ebc-be3d-fb89d9f70742\" (UID: \"1dacbabe-8d44-4ebc-be3d-fb89d9f70742\") "
Oct 30 04:12:58.475134 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:12:58.474944    4461 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aedb2e20-2e07-42b1-b3c2-27223dc8d177-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aedb2e20-2e07-42b1-b3c2-27223dc8d177-istiod-ca-cert\") pod \"aedb2e20-2e07-42b1-b3c2-27223dc8d177\" (UID: \"aedb2e20-2e07-42b1-b3c2-27223dc8d177\") "
Oct 30 04:12:58.475134 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:12:58.474997    4461 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aedb2e20-2e07-42b1-b3c2-27223dc8d177-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aedb2e20-2e07-42b1-b3c2-27223dc8d177-istio-data\") pod \"aedb2e20-2e07-42b1-b3c2-27223dc8d177\" (UID: \"aedb2e20-2e07-42b1-b3c2-27223dc8d177\") "
Oct 30 04:12:58.475134 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:12:58.475023    4461 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/19942a3a-cdfa-4618-83b3-f828b5cbe33e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/19942a3a-cdfa-4618-83b3-f828b5cbe33e-istio-data\") pod \"19942a3a-cdfa-4618-83b3-f828b5cbe33e\" (UID: \"19942a3a-cdfa-4618-83b3-f828b5cbe33e\") "
Oct 30 04:12:58.475134 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:12:58.475059    4461 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aedb2e20-2e07-42b1-b3c2-27223dc8d177-istio-token\" (UniqueName: \"kubernetes.io/projected/aedb2e20-2e07-42b1-b3c2-27223dc8d177-istio-token\") pod \"aedb2e20-2e07-42b1-b3c2-27223dc8d177\" (UID: \"aedb2e20-2e07-42b1-b3c2-27223dc8d177\") "
Oct 30 04:12:58.475134 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:12:58.475086    4461 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/19942a3a-cdfa-4618-83b3-f828b5cbe33e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/19942a3a-cdfa-4618-83b3-f828b5cbe33e-workload-socket\") pod \"19942a3a-cdfa-4618-83b3-f828b5cbe33e\" (UID: \"19942a3a-cdfa-4618-83b3-f828b5cbe33e\") "
Oct 30 04:12:58.475440 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:12:58.475139    4461 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7c55427d-9f1e-4b24-9741-3ff0cfe751e7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7c55427d-9f1e-4b24-9741-3ff0cfe751e7-istio-podinfo\") pod \"7c55427d-9f1e-4b24-9741-3ff0cfe751e7\" (UID: \"7c55427d-9f1e-4b24-9741-3ff0cfe751e7\") "
Oct 30 04:12:58.475440 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:12:58.475168    4461 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/19942a3a-cdfa-4618-83b3-f828b5cbe33e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/19942a3a-cdfa-4618-83b3-f828b5cbe33e-istio-envoy\") pod \"19942a3a-cdfa-4618-83b3-f828b5cbe33e\" (UID: \"19942a3a-cdfa-4618-83b3-f828b5cbe33e\") "
Oct 30 04:12:58.475440 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:12:58.475200    4461 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1dacbabe-8d44-4ebc-be3d-fb89d9f70742-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1dacbabe-8d44-4ebc-be3d-fb89d9f70742-istiod-ca-cert\") pod \"1dacbabe-8d44-4ebc-be3d-fb89d9f70742\" (UID: \"1dacbabe-8d44-4ebc-be3d-fb89d9f70742\") "
Oct 30 04:12:58.475440 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:12:58.475227    4461 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1dacbabe-8d44-4ebc-be3d-fb89d9f70742-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1dacbabe-8d44-4ebc-be3d-fb89d9f70742-istio-data\") pod \"1dacbabe-8d44-4ebc-be3d-fb89d9f70742\" (UID: \"1dacbabe-8d44-4ebc-be3d-fb89d9f70742\") "
Oct 30 04:12:58.475440 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:12:58.475248    4461 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1dacbabe-8d44-4ebc-be3d-fb89d9f70742-workload-socket" pod "1dacbabe-8d44-4ebc-be3d-fb89d9f70742" (UID: "1dacbabe-8d44-4ebc-be3d-fb89d9f70742"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:12:58.475440 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:12:58.475257    4461 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aedb2e20-2e07-42b1-b3c2-27223dc8d177-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aedb2e20-2e07-42b1-b3c2-27223dc8d177-workload-certs\") pod \"aedb2e20-2e07-42b1-b3c2-27223dc8d177\" (UID: \"aedb2e20-2e07-42b1-b3c2-27223dc8d177\") "
Oct 30 04:12:58.475440 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:12:58.475321    4461 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19942a3a-cdfa-4618-83b3-f828b5cbe33e-istio-data" pod "19942a3a-cdfa-4618-83b3-f828b5cbe33e" (UID: "19942a3a-cdfa-4618-83b3-f828b5cbe33e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:12:58.475440 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:12:58.475325    4461 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/19942a3a-cdfa-4618-83b3-f828b5cbe33e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/19942a3a-cdfa-4618-83b3-f828b5cbe33e-istiod-ca-cert\") pod \"19942a3a-cdfa-4618-83b3-f828b5cbe33e\" (UID: \"19942a3a-cdfa-4618-83b3-f828b5cbe33e\") "
Oct 30 04:12:58.475440 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:12:58.475332    4461 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aedb2e20-2e07-42b1-b3c2-27223dc8d177-istio-data" pod "aedb2e20-2e07-42b1-b3c2-27223dc8d177" (UID: "aedb2e20-2e07-42b1-b3c2-27223dc8d177"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:12:58.475440 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:12:58.475349    4461 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aedb2e20-2e07-42b1-b3c2-27223dc8d177-istiod-ca-cert" pod "aedb2e20-2e07-42b1-b3c2-27223dc8d177" (UID: "aedb2e20-2e07-42b1-b3c2-27223dc8d177"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:12:58.475440 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:12:58.475366    4461 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/19942a3a-cdfa-4618-83b3-f828b5cbe33e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/19942a3a-cdfa-4618-83b3-f828b5cbe33e-credential-socket\") pod \"19942a3a-cdfa-4618-83b3-f828b5cbe33e\" (UID: \"19942a3a-cdfa-4618-83b3-f828b5cbe33e\") "
Oct 30 04:12:58.475440 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:12:58.475431    4461 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7c55427d-9f1e-4b24-9741-3ff0cfe751e7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c55427d-9f1e-4b24-9741-3ff0cfe751e7-credential-socket\") pod \"7c55427d-9f1e-4b24-9741-3ff0cfe751e7\" (UID: \"7c55427d-9f1e-4b24-9741-3ff0cfe751e7\") "
Oct 30 04:12:58.476166 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:12:58.475460    4461 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7c55427d-9f1e-4b24-9741-3ff0cfe751e7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7c55427d-9f1e-4b24-9741-3ff0cfe751e7-istiod-ca-cert\") pod \"7c55427d-9f1e-4b24-9741-3ff0cfe751e7\" (UID: \"7c55427d-9f1e-4b24-9741-3ff0cfe751e7\") "
Oct 30 04:12:58.476166 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:12:58.475478    4461 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aedb2e20-2e07-42b1-b3c2-27223dc8d177-workload-certs" pod "aedb2e20-2e07-42b1-b3c2-27223dc8d177" (UID: "aedb2e20-2e07-42b1-b3c2-27223dc8d177"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:12:58.476166 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:12:58.475501    4461 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aedb2e20-2e07-42b1-b3c2-27223dc8d177-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aedb2e20-2e07-42b1-b3c2-27223dc8d177-istio-envoy\") pod \"aedb2e20-2e07-42b1-b3c2-27223dc8d177\" (UID: \"aedb2e20-2e07-42b1-b3c2-27223dc8d177\") "
Oct 30 04:12:58.476166 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:12:58.475570    4461 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7c55427d-9f1e-4b24-9741-3ff0cfe751e7-istio-token\" (UniqueName: \"kubernetes.io/projected/7c55427d-9f1e-4b24-9741-3ff0cfe751e7-istio-token\") pod \"7c55427d-9f1e-4b24-9741-3ff0cfe751e7\" (UID: \"7c55427d-9f1e-4b24-9741-3ff0cfe751e7\") "
Oct 30 04:12:58.476166 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:12:58.475621    4461 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1dacbabe-8d44-4ebc-be3d-fb89d9f70742-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1dacbabe-8d44-4ebc-be3d-fb89d9f70742-credential-socket\") pod \"1dacbabe-8d44-4ebc-be3d-fb89d9f70742\" (UID: \"1dacbabe-8d44-4ebc-be3d-fb89d9f70742\") "
Oct 30 04:12:58.476166 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:12:58.475655    4461 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7c55427d-9f1e-4b24-9741-3ff0cfe751e7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7c55427d-9f1e-4b24-9741-3ff0cfe751e7-workload-certs\") pod \"7c55427d-9f1e-4b24-9741-3ff0cfe751e7\" (UID: \"7c55427d-9f1e-4b24-9741-3ff0cfe751e7\") "
Oct 30 04:12:58.476166 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:12:58.475684    4461 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/19942a3a-cdfa-4618-83b3-f828b5cbe33e-istio-token\" (UniqueName: \"kubernetes.io/projected/19942a3a-cdfa-4618-83b3-f828b5cbe33e-istio-token\") pod \"19942a3a-cdfa-4618-83b3-f828b5cbe33e\" (UID: \"19942a3a-cdfa-4618-83b3-f828b5cbe33e\") "
Oct 30 04:12:58.476166 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:12:58.475705    4461 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1dacbabe-8d44-4ebc-be3d-fb89d9f70742-istio-data" pod "1dacbabe-8d44-4ebc-be3d-fb89d9f70742" (UID: "1dacbabe-8d44-4ebc-be3d-fb89d9f70742"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:12:58.476166 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:12:58.475719    4461 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/19942a3a-cdfa-4618-83b3-f828b5cbe33e-kube-api-access-662gk\" (UniqueName: \"kubernetes.io/projected/19942a3a-cdfa-4618-83b3-f828b5cbe33e-kube-api-access-662gk\") pod \"19942a3a-cdfa-4618-83b3-f828b5cbe33e\" (UID: \"19942a3a-cdfa-4618-83b3-f828b5cbe33e\") "
Oct 30 04:12:58.476166 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:12:58.475788    4461 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1dacbabe-8d44-4ebc-be3d-fb89d9f70742-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1dacbabe-8d44-4ebc-be3d-fb89d9f70742-istio-podinfo\") pod \"1dacbabe-8d44-4ebc-be3d-fb89d9f70742\" (UID: \"1dacbabe-8d44-4ebc-be3d-fb89d9f70742\") "
Oct 30 04:12:58.476166 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:12:58.475788    4461 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/19942a3a-cdfa-4618-83b3-f828b5cbe33e-istiod-ca-cert" pod "19942a3a-cdfa-4618-83b3-f828b5cbe33e" (UID: "19942a3a-cdfa-4618-83b3-f828b5cbe33e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:12:58.476166 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:12:58.475820    4461 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aedb2e20-2e07-42b1-b3c2-27223dc8d177-kube-api-access-tvhxv\" (UniqueName: \"kubernetes.io/projected/aedb2e20-2e07-42b1-b3c2-27223dc8d177-kube-api-access-tvhxv\") pod \"aedb2e20-2e07-42b1-b3c2-27223dc8d177\" (UID: \"aedb2e20-2e07-42b1-b3c2-27223dc8d177\") "
Oct 30 04:12:58.476166 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:12:58.475851    4461 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/19942a3a-cdfa-4618-83b3-f828b5cbe33e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/19942a3a-cdfa-4618-83b3-f828b5cbe33e-istio-podinfo\") pod \"19942a3a-cdfa-4618-83b3-f828b5cbe33e\" (UID: \"19942a3a-cdfa-4618-83b3-f828b5cbe33e\") "
Oct 30 04:12:58.476166 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:12:58.475882    4461 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1dacbabe-8d44-4ebc-be3d-fb89d9f70742-kube-api-access-fmc94\" (UniqueName: \"kubernetes.io/projected/1dacbabe-8d44-4ebc-be3d-fb89d9f70742-kube-api-access-fmc94\") pod \"1dacbabe-8d44-4ebc-be3d-fb89d9f70742\" (UID: \"1dacbabe-8d44-4ebc-be3d-fb89d9f70742\") "
Oct 30 04:12:58.476166 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:12:58.475912    4461 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1dacbabe-8d44-4ebc-be3d-fb89d9f70742-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1dacbabe-8d44-4ebc-be3d-fb89d9f70742-workload-certs\") pod \"1dacbabe-8d44-4ebc-be3d-fb89d9f70742\" (UID: \"1dacbabe-8d44-4ebc-be3d-fb89d9f70742\") "
Oct 30 04:12:58.476166 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:12:58.475949    4461 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7c55427d-9f1e-4b24-9741-3ff0cfe751e7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7c55427d-9f1e-4b24-9741-3ff0cfe751e7-istio-envoy\") pod \"7c55427d-9f1e-4b24-9741-3ff0cfe751e7\" (UID: \"7c55427d-9f1e-4b24-9741-3ff0cfe751e7\") "
Oct 30 04:12:58.476166 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:12:58.475981    4461 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1dacbabe-8d44-4ebc-be3d-fb89d9f70742-istio-token\" (UniqueName: \"kubernetes.io/projected/1dacbabe-8d44-4ebc-be3d-fb89d9f70742-istio-token\") pod \"1dacbabe-8d44-4ebc-be3d-fb89d9f70742\" (UID: \"1dacbabe-8d44-4ebc-be3d-fb89d9f70742\") "
Oct 30 04:12:58.476166 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:12:58.476013    4461 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7c55427d-9f1e-4b24-9741-3ff0cfe751e7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c55427d-9f1e-4b24-9741-3ff0cfe751e7-workload-socket\") pod \"7c55427d-9f1e-4b24-9741-3ff0cfe751e7\" (UID: \"7c55427d-9f1e-4b24-9741-3ff0cfe751e7\") "
Oct 30 04:12:58.476166 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:12:58.476042    4461 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aedb2e20-2e07-42b1-b3c2-27223dc8d177-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aedb2e20-2e07-42b1-b3c2-27223dc8d177-workload-socket\") pod \"aedb2e20-2e07-42b1-b3c2-27223dc8d177\" (UID: \"aedb2e20-2e07-42b1-b3c2-27223dc8d177\") "
Oct 30 04:12:58.476166 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:12:58.476070    4461 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/19942a3a-cdfa-4618-83b3-f828b5cbe33e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/19942a3a-cdfa-4618-83b3-f828b5cbe33e-workload-certs\") pod \"19942a3a-cdfa-4618-83b3-f828b5cbe33e\" (UID: \"19942a3a-cdfa-4618-83b3-f828b5cbe33e\") "
Oct 30 04:12:58.476166 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:12:58.476080    4461 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19942a3a-cdfa-4618-83b3-f828b5cbe33e-workload-socket" pod "19942a3a-cdfa-4618-83b3-f828b5cbe33e" (UID: "19942a3a-cdfa-4618-83b3-f828b5cbe33e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:12:58.476166 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:12:58.476121    4461 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/aedb2e20-2e07-42b1-b3c2-27223dc8d177-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aedb2e20-2e07-42b1-b3c2-27223dc8d177-istio-podinfo\") pod \"aedb2e20-2e07-42b1-b3c2-27223dc8d177\" (UID: \"aedb2e20-2e07-42b1-b3c2-27223dc8d177\") "
Oct 30 04:12:58.476166 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:12:58.476158    4461 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1dacbabe-8d44-4ebc-be3d-fb89d9f70742-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1dacbabe-8d44-4ebc-be3d-fb89d9f70742-istio-envoy\") pod \"1dacbabe-8d44-4ebc-be3d-fb89d9f70742\" (UID: \"1dacbabe-8d44-4ebc-be3d-fb89d9f70742\") "
Oct 30 04:12:58.477382 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:12:58.476204    4461 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aedb2e20-2e07-42b1-b3c2-27223dc8d177-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aedb2e20-2e07-42b1-b3c2-27223dc8d177-credential-socket\") pod \"aedb2e20-2e07-42b1-b3c2-27223dc8d177\" (UID: \"aedb2e20-2e07-42b1-b3c2-27223dc8d177\") "
Oct 30 04:12:58.477382 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:12:58.476242    4461 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7c55427d-9f1e-4b24-9741-3ff0cfe751e7-kube-api-access-fbvb7\" (UniqueName: \"kubernetes.io/projected/7c55427d-9f1e-4b24-9741-3ff0cfe751e7-kube-api-access-fbvb7\") pod \"7c55427d-9f1e-4b24-9741-3ff0cfe751e7\" (UID: \"7c55427d-9f1e-4b24-9741-3ff0cfe751e7\") "
Oct 30 04:12:58.477382 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:12:58.476272    4461 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7c55427d-9f1e-4b24-9741-3ff0cfe751e7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7c55427d-9f1e-4b24-9741-3ff0cfe751e7-istio-data\") pod \"7c55427d-9f1e-4b24-9741-3ff0cfe751e7\" (UID: \"7c55427d-9f1e-4b24-9741-3ff0cfe751e7\") "
Oct 30 04:12:58.477382 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:12:58.476318    4461 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c55427d-9f1e-4b24-9741-3ff0cfe751e7-credential-socket" pod "7c55427d-9f1e-4b24-9741-3ff0cfe751e7" (UID: "7c55427d-9f1e-4b24-9741-3ff0cfe751e7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:12:58.477382 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:12:58.476393    4461 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1dacbabe-8d44-4ebc-be3d-fb89d9f70742-istiod-ca-cert" pod "1dacbabe-8d44-4ebc-be3d-fb89d9f70742" (UID: "1dacbabe-8d44-4ebc-be3d-fb89d9f70742"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:12:58.477382 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:12:58.476551    4461 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19942a3a-cdfa-4618-83b3-f828b5cbe33e-credential-socket" pod "19942a3a-cdfa-4618-83b3-f828b5cbe33e" (UID: "19942a3a-cdfa-4618-83b3-f828b5cbe33e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:12:58.477382 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:12:58.476565    4461 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c55427d-9f1e-4b24-9741-3ff0cfe751e7-istio-data" pod "7c55427d-9f1e-4b24-9741-3ff0cfe751e7" (UID: "7c55427d-9f1e-4b24-9741-3ff0cfe751e7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:12:58.477382 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:12:58.476757    4461 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1dacbabe-8d44-4ebc-be3d-fb89d9f70742-workload-certs" pod "1dacbabe-8d44-4ebc-be3d-fb89d9f70742" (UID: "1dacbabe-8d44-4ebc-be3d-fb89d9f70742"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:12:58.477382 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:12:58.476916    4461 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7c55427d-9f1e-4b24-9741-3ff0cfe751e7-istiod-ca-cert" pod "7c55427d-9f1e-4b24-9741-3ff0cfe751e7" (UID: "7c55427d-9f1e-4b24-9741-3ff0cfe751e7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:12:58.477382 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:12:58.476996    4461 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c55427d-9f1e-4b24-9741-3ff0cfe751e7-workload-certs" pod "7c55427d-9f1e-4b24-9741-3ff0cfe751e7" (UID: "7c55427d-9f1e-4b24-9741-3ff0cfe751e7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:12:58.477382 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:12:58.477306    4461 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1dacbabe-8d44-4ebc-be3d-fb89d9f70742-credential-socket" pod "1dacbabe-8d44-4ebc-be3d-fb89d9f70742" (UID: "1dacbabe-8d44-4ebc-be3d-fb89d9f70742"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:12:58.478673 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:12:58.477394    4461 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1dacbabe-8d44-4ebc-be3d-fb89d9f70742-credential-socket\") on node \"ip-10-0-143-156.ec2.internal\" DevicePath \"\""
Oct 30 04:12:58.478673 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:12:58.477414    4461 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7c55427d-9f1e-4b24-9741-3ff0cfe751e7-workload-certs\") on node \"ip-10-0-143-156.ec2.internal\" DevicePath \"\""
Oct 30 04:12:58.478673 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:12:58.477429    4461 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1dacbabe-8d44-4ebc-be3d-fb89d9f70742-workload-certs\") on node \"ip-10-0-143-156.ec2.internal\" DevicePath \"\""
Oct 30 04:12:58.478673 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:12:58.477442    4461 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7c55427d-9f1e-4b24-9741-3ff0cfe751e7-istio-data\") on node \"ip-10-0-143-156.ec2.internal\" DevicePath \"\""
Oct 30 04:12:58.478673 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:12:58.477454    4461 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1dacbabe-8d44-4ebc-be3d-fb89d9f70742-workload-socket\") on node \"ip-10-0-143-156.ec2.internal\" DevicePath \"\""
Oct 30 04:12:58.478673 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:12:58.477468    4461 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aedb2e20-2e07-42b1-b3c2-27223dc8d177-istiod-ca-cert\") on node \"ip-10-0-143-156.ec2.internal\" DevicePath \"\""
Oct 30 04:12:58.478673 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:12:58.477481    4461 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aedb2e20-2e07-42b1-b3c2-27223dc8d177-istio-data\") on node \"ip-10-0-143-156.ec2.internal\" DevicePath \"\""
Oct 30 04:12:58.478673 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:12:58.477495    4461 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/19942a3a-cdfa-4618-83b3-f828b5cbe33e-istio-data\") on node \"ip-10-0-143-156.ec2.internal\" DevicePath \"\""
Oct 30 04:12:58.478673 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:12:58.477494    4461 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19942a3a-cdfa-4618-83b3-f828b5cbe33e-workload-certs" pod "19942a3a-cdfa-4618-83b3-f828b5cbe33e" (UID: "19942a3a-cdfa-4618-83b3-f828b5cbe33e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:12:58.478673 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:12:58.477759    4461 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/19942a3a-cdfa-4618-83b3-f828b5cbe33e-workload-socket\") on node \"ip-10-0-143-156.ec2.internal\" DevicePath \"\""
Oct 30 04:12:58.478673 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:12:58.477783    4461 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1dacbabe-8d44-4ebc-be3d-fb89d9f70742-istiod-ca-cert\") on node \"ip-10-0-143-156.ec2.internal\" DevicePath \"\""
Oct 30 04:12:58.478673 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:12:58.477798    4461 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1dacbabe-8d44-4ebc-be3d-fb89d9f70742-istio-data\") on node \"ip-10-0-143-156.ec2.internal\" DevicePath \"\""
Oct 30 04:12:58.478673 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:12:58.477816    4461 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aedb2e20-2e07-42b1-b3c2-27223dc8d177-workload-certs\") on node \"ip-10-0-143-156.ec2.internal\" DevicePath \"\""
Oct 30 04:12:58.478673 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:12:58.477830    4461 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/19942a3a-cdfa-4618-83b3-f828b5cbe33e-istiod-ca-cert\") on node \"ip-10-0-143-156.ec2.internal\" DevicePath \"\""
Oct 30 04:12:58.478673 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:12:58.477842    4461 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/19942a3a-cdfa-4618-83b3-f828b5cbe33e-credential-socket\") on node \"ip-10-0-143-156.ec2.internal\" DevicePath \"\""
Oct 30 04:12:58.478673 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:12:58.477857    4461 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c55427d-9f1e-4b24-9741-3ff0cfe751e7-credential-socket\") on node \"ip-10-0-143-156.ec2.internal\" DevicePath \"\""
Oct 30 04:12:58.478673 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:12:58.477871    4461 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7c55427d-9f1e-4b24-9741-3ff0cfe751e7-istiod-ca-cert\") on node \"ip-10-0-143-156.ec2.internal\" DevicePath \"\""
Oct 30 04:12:58.478673 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:12:58.478060    4461 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c55427d-9f1e-4b24-9741-3ff0cfe751e7-workload-socket" pod "7c55427d-9f1e-4b24-9741-3ff0cfe751e7" (UID: "7c55427d-9f1e-4b24-9741-3ff0cfe751e7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:12:58.478673 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:12:58.478181    4461 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aedb2e20-2e07-42b1-b3c2-27223dc8d177-workload-socket" pod "aedb2e20-2e07-42b1-b3c2-27223dc8d177" (UID: "aedb2e20-2e07-42b1-b3c2-27223dc8d177"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:12:58.478673 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:12:58.478412    4461 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aedb2e20-2e07-42b1-b3c2-27223dc8d177-credential-socket" pod "aedb2e20-2e07-42b1-b3c2-27223dc8d177" (UID: "aedb2e20-2e07-42b1-b3c2-27223dc8d177"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:12:58.481270 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:12:58.480882    4461 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aedb2e20-2e07-42b1-b3c2-27223dc8d177-istio-token" pod "aedb2e20-2e07-42b1-b3c2-27223dc8d177" (UID: "aedb2e20-2e07-42b1-b3c2-27223dc8d177"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:12:58.481270 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:12:58.481114    4461 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7c55427d-9f1e-4b24-9741-3ff0cfe751e7-istio-token" pod "7c55427d-9f1e-4b24-9741-3ff0cfe751e7" (UID: "7c55427d-9f1e-4b24-9741-3ff0cfe751e7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:12:58.481441 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:12:58.481322    4461 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19942a3a-cdfa-4618-83b3-f828b5cbe33e-istio-envoy" pod "19942a3a-cdfa-4618-83b3-f828b5cbe33e" (UID: "19942a3a-cdfa-4618-83b3-f828b5cbe33e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:12:58.482057 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:12:58.482028    4461 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1dacbabe-8d44-4ebc-be3d-fb89d9f70742-istio-envoy" pod "1dacbabe-8d44-4ebc-be3d-fb89d9f70742" (UID: "1dacbabe-8d44-4ebc-be3d-fb89d9f70742"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:12:58.483276 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:12:58.483197    4461 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7c55427d-9f1e-4b24-9741-3ff0cfe751e7-istio-podinfo" pod "7c55427d-9f1e-4b24-9741-3ff0cfe751e7" (UID: "7c55427d-9f1e-4b24-9741-3ff0cfe751e7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 04:12:58.483557 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:12:58.483534    4461 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1dacbabe-8d44-4ebc-be3d-fb89d9f70742-istio-token" pod "1dacbabe-8d44-4ebc-be3d-fb89d9f70742" (UID: "1dacbabe-8d44-4ebc-be3d-fb89d9f70742"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:12:58.483656 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:12:58.483540    4461 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7c55427d-9f1e-4b24-9741-3ff0cfe751e7-kube-api-access-fbvb7" pod "7c55427d-9f1e-4b24-9741-3ff0cfe751e7" (UID: "7c55427d-9f1e-4b24-9741-3ff0cfe751e7"). InnerVolumeSpecName "kube-api-access-fbvb7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:12:58.483656 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:12:58.483530    4461 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c55427d-9f1e-4b24-9741-3ff0cfe751e7-istio-envoy" pod "7c55427d-9f1e-4b24-9741-3ff0cfe751e7" (UID: "7c55427d-9f1e-4b24-9741-3ff0cfe751e7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:12:58.483656 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:12:58.483582    4461 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/19942a3a-cdfa-4618-83b3-f828b5cbe33e-istio-podinfo" pod "19942a3a-cdfa-4618-83b3-f828b5cbe33e" (UID: "19942a3a-cdfa-4618-83b3-f828b5cbe33e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 04:12:58.483909 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:12:58.483877    4461 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1dacbabe-8d44-4ebc-be3d-fb89d9f70742-kube-api-access-fmc94" pod "1dacbabe-8d44-4ebc-be3d-fb89d9f70742" (UID: "1dacbabe-8d44-4ebc-be3d-fb89d9f70742"). InnerVolumeSpecName "kube-api-access-fmc94". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:12:58.483975 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:12:58.483956    4461 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aedb2e20-2e07-42b1-b3c2-27223dc8d177-kube-api-access-tvhxv" pod "aedb2e20-2e07-42b1-b3c2-27223dc8d177" (UID: "aedb2e20-2e07-42b1-b3c2-27223dc8d177"). InnerVolumeSpecName "kube-api-access-tvhxv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:12:58.484089 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:12:58.484063    4461 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1dacbabe-8d44-4ebc-be3d-fb89d9f70742-istio-podinfo" pod "1dacbabe-8d44-4ebc-be3d-fb89d9f70742" (UID: "1dacbabe-8d44-4ebc-be3d-fb89d9f70742"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 04:12:58.484324 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:12:58.484293    4461 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aedb2e20-2e07-42b1-b3c2-27223dc8d177-istio-envoy" pod "aedb2e20-2e07-42b1-b3c2-27223dc8d177" (UID: "aedb2e20-2e07-42b1-b3c2-27223dc8d177"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:12:58.485056 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:12:58.485037    4461 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/19942a3a-cdfa-4618-83b3-f828b5cbe33e-kube-api-access-662gk" pod "19942a3a-cdfa-4618-83b3-f828b5cbe33e" (UID: "19942a3a-cdfa-4618-83b3-f828b5cbe33e"). InnerVolumeSpecName "kube-api-access-662gk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:12:58.485299 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:12:58.485277    4461 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/19942a3a-cdfa-4618-83b3-f828b5cbe33e-istio-token" pod "19942a3a-cdfa-4618-83b3-f828b5cbe33e" (UID: "19942a3a-cdfa-4618-83b3-f828b5cbe33e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:12:58.485683 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:12:58.485662    4461 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/aedb2e20-2e07-42b1-b3c2-27223dc8d177-istio-podinfo" pod "aedb2e20-2e07-42b1-b3c2-27223dc8d177" (UID: "aedb2e20-2e07-42b1-b3c2-27223dc8d177"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 04:12:58.487285 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:12:58.487266    4461 generic.go:356] "Generic (PLEG): container finished" podID="7c55427d-9f1e-4b24-9741-3ff0cfe751e7" containerID="a26c32ebcc68eacfe09420438374d0a20cb6f19f19151fadbd2cfa3ebc793af2" exitCode=0
Oct 30 04:12:58.487377 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:12:58.487330    4461 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-v4p2m"
Oct 30 04:12:58.487377 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:12:58.487356    4461 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-v4p2m" event={"ID":"7c55427d-9f1e-4b24-9741-3ff0cfe751e7","Type":"ContainerDied","Data":"a26c32ebcc68eacfe09420438374d0a20cb6f19f19151fadbd2cfa3ebc793af2"}
Oct 30 04:12:58.487474 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:12:58.487406    4461 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-v4p2m" event={"ID":"7c55427d-9f1e-4b24-9741-3ff0cfe751e7","Type":"ContainerDied","Data":"5bc3cd36a3a9b3a3d6ea08adc25be559d6f022ffbc4fa816a58aef0213f785e0"}
Oct 30 04:12:58.487474 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:12:58.487431    4461 scope.go:122] "RemoveContainer" containerID="a26c32ebcc68eacfe09420438374d0a20cb6f19f19151fadbd2cfa3ebc793af2"
Oct 30 04:12:58.488903 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:12:58.488883    4461 generic.go:356] "Generic (PLEG): container finished" podID="aedb2e20-2e07-42b1-b3c2-27223dc8d177" containerID="efda88805e2304ccd46923be30dbb7d41ea73c74a999c6b9069f3b56afcbff0d" exitCode=0
Oct 30 04:12:58.488992 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:12:58.488953    4461 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-jtcmr" event={"ID":"aedb2e20-2e07-42b1-b3c2-27223dc8d177","Type":"ContainerDied","Data":"efda88805e2304ccd46923be30dbb7d41ea73c74a999c6b9069f3b56afcbff0d"}
Oct 30 04:12:58.489060 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:12:58.488994    4461 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-jtcmr" event={"ID":"aedb2e20-2e07-42b1-b3c2-27223dc8d177","Type":"ContainerDied","Data":"71274a6479e834c8926d3aea3526e0af0732e1100149de3d04e8e0a56a7a889a"}
Oct 30 04:12:58.489060 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:12:58.489045    4461 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-jtcmr"
Oct 30 04:12:58.490823 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:12:58.490799    4461 generic.go:356] "Generic (PLEG): container finished" podID="1dacbabe-8d44-4ebc-be3d-fb89d9f70742" containerID="7e75ec4cfc7f6817eb5ddad8c41febdfe278337b1cd941ee5b4ea8fbf2cc7b37" exitCode=0
Oct 30 04:12:58.490928 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:12:58.490861    4461 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-nfxr8"
Oct 30 04:12:58.490928 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:12:58.490914    4461 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-nfxr8" event={"ID":"1dacbabe-8d44-4ebc-be3d-fb89d9f70742","Type":"ContainerDied","Data":"7e75ec4cfc7f6817eb5ddad8c41febdfe278337b1cd941ee5b4ea8fbf2cc7b37"}
Oct 30 04:12:58.491052 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:12:58.490954    4461 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-nfxr8" event={"ID":"1dacbabe-8d44-4ebc-be3d-fb89d9f70742","Type":"ContainerDied","Data":"b808b3d4b6529ce29eee2762a32f091cec90ae701954632954386122e462252e"}
Oct 30 04:12:58.492840 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:12:58.492688    4461 generic.go:356] "Generic (PLEG): container finished" podID="19942a3a-cdfa-4618-83b3-f828b5cbe33e" containerID="c01d0ed4e0a04302e11ad4948c310b7a78840708bfce0145303f3972da480ff9" exitCode=0
Oct 30 04:12:58.492840 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:12:58.492765    4461 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-rh6p2" event={"ID":"19942a3a-cdfa-4618-83b3-f828b5cbe33e","Type":"ContainerDied","Data":"c01d0ed4e0a04302e11ad4948c310b7a78840708bfce0145303f3972da480ff9"}
Oct 30 04:12:58.492840 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:12:58.492798    4461 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-rh6p2" event={"ID":"19942a3a-cdfa-4618-83b3-f828b5cbe33e","Type":"ContainerDied","Data":"bc35bc00cb97cf6f82221f5765564b13c65bda6552a00ba546b6b68d4ba789df"}
Oct 30 04:12:58.493050 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:12:58.492840    4461 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-rh6p2"
Oct 30 04:12:58.502632 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:12:58.502595    4461 scope.go:122] "RemoveContainer" containerID="a26c32ebcc68eacfe09420438374d0a20cb6f19f19151fadbd2cfa3ebc793af2"
Oct 30 04:12:58.503999 ip-10-0-143-156 kubenswrapper[4461]: E1030 04:12:58.503948    4461 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a26c32ebcc68eacfe09420438374d0a20cb6f19f19151fadbd2cfa3ebc793af2\": container with ID starting with a26c32ebcc68eacfe09420438374d0a20cb6f19f19151fadbd2cfa3ebc793af2 not found: ID does not exist" containerID="a26c32ebcc68eacfe09420438374d0a20cb6f19f19151fadbd2cfa3ebc793af2"
Oct 30 04:12:58.504085 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:12:58.504011    4461 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a26c32ebcc68eacfe09420438374d0a20cb6f19f19151fadbd2cfa3ebc793af2"} err="failed to get container status \"a26c32ebcc68eacfe09420438374d0a20cb6f19f19151fadbd2cfa3ebc793af2\": rpc error: code = NotFound desc = could not find container \"a26c32ebcc68eacfe09420438374d0a20cb6f19f19151fadbd2cfa3ebc793af2\": container with ID starting with a26c32ebcc68eacfe09420438374d0a20cb6f19f19151fadbd2cfa3ebc793af2 not found: ID does not exist"
Oct 30 04:12:58.504196 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:12:58.504035    4461 scope.go:122] "RemoveContainer" containerID="efda88805e2304ccd46923be30dbb7d41ea73c74a999c6b9069f3b56afcbff0d"
Oct 30 04:12:58.521778 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:12:58.521741    4461 scope.go:122] "RemoveContainer" containerID="efda88805e2304ccd46923be30dbb7d41ea73c74a999c6b9069f3b56afcbff0d"
Oct 30 04:12:58.522152 ip-10-0-143-156 kubenswrapper[4461]: E1030 04:12:58.522131    4461 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"efda88805e2304ccd46923be30dbb7d41ea73c74a999c6b9069f3b56afcbff0d\": container with ID starting with efda88805e2304ccd46923be30dbb7d41ea73c74a999c6b9069f3b56afcbff0d not found: ID does not exist" containerID="efda88805e2304ccd46923be30dbb7d41ea73c74a999c6b9069f3b56afcbff0d"
Oct 30 04:12:58.522244 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:12:58.522165    4461 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"efda88805e2304ccd46923be30dbb7d41ea73c74a999c6b9069f3b56afcbff0d"} err="failed to get container status \"efda88805e2304ccd46923be30dbb7d41ea73c74a999c6b9069f3b56afcbff0d\": rpc error: code = NotFound desc = could not find container \"efda88805e2304ccd46923be30dbb7d41ea73c74a999c6b9069f3b56afcbff0d\": container with ID starting with efda88805e2304ccd46923be30dbb7d41ea73c74a999c6b9069f3b56afcbff0d not found: ID does not exist"
Oct 30 04:12:58.522244 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:12:58.522190    4461 scope.go:122] "RemoveContainer" containerID="7e75ec4cfc7f6817eb5ddad8c41febdfe278337b1cd941ee5b4ea8fbf2cc7b37"
Oct 30 04:12:58.530254 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:12:58.530233    4461 scope.go:122] "RemoveContainer" containerID="7e75ec4cfc7f6817eb5ddad8c41febdfe278337b1cd941ee5b4ea8fbf2cc7b37"
Oct 30 04:12:58.530725 ip-10-0-143-156 kubenswrapper[4461]: E1030 04:12:58.530684    4461 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7e75ec4cfc7f6817eb5ddad8c41febdfe278337b1cd941ee5b4ea8fbf2cc7b37\": container with ID starting with 7e75ec4cfc7f6817eb5ddad8c41febdfe278337b1cd941ee5b4ea8fbf2cc7b37 not found: ID does not exist" containerID="7e75ec4cfc7f6817eb5ddad8c41febdfe278337b1cd941ee5b4ea8fbf2cc7b37"
Oct 30 04:12:58.530816 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:12:58.530734    4461 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7e75ec4cfc7f6817eb5ddad8c41febdfe278337b1cd941ee5b4ea8fbf2cc7b37"} err="failed to get container status \"7e75ec4cfc7f6817eb5ddad8c41febdfe278337b1cd941ee5b4ea8fbf2cc7b37\": rpc error: code = NotFound desc = could not find container \"7e75ec4cfc7f6817eb5ddad8c41febdfe278337b1cd941ee5b4ea8fbf2cc7b37\": container with ID starting with 7e75ec4cfc7f6817eb5ddad8c41febdfe278337b1cd941ee5b4ea8fbf2cc7b37 not found: ID does not exist"
Oct 30 04:12:58.530816 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:12:58.530756    4461 scope.go:122] "RemoveContainer" containerID="c01d0ed4e0a04302e11ad4948c310b7a78840708bfce0145303f3972da480ff9"
Oct 30 04:12:58.539613 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:12:58.539580    4461 scope.go:122] "RemoveContainer" containerID="c01d0ed4e0a04302e11ad4948c310b7a78840708bfce0145303f3972da480ff9"
Oct 30 04:12:58.539917 ip-10-0-143-156 kubenswrapper[4461]: E1030 04:12:58.539889    4461 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c01d0ed4e0a04302e11ad4948c310b7a78840708bfce0145303f3972da480ff9\": container with ID starting with c01d0ed4e0a04302e11ad4948c310b7a78840708bfce0145303f3972da480ff9 not found: ID does not exist" containerID="c01d0ed4e0a04302e11ad4948c310b7a78840708bfce0145303f3972da480ff9"
Oct 30 04:12:58.539988 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:12:58.539926    4461 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c01d0ed4e0a04302e11ad4948c310b7a78840708bfce0145303f3972da480ff9"} err="failed to get container status \"c01d0ed4e0a04302e11ad4948c310b7a78840708bfce0145303f3972da480ff9\": rpc error: code = NotFound desc = could not find container \"c01d0ed4e0a04302e11ad4948c310b7a78840708bfce0145303f3972da480ff9\": container with ID starting with c01d0ed4e0a04302e11ad4948c310b7a78840708bfce0145303f3972da480ff9 not found: ID does not exist"
Oct 30 04:12:58.551479 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:12:58.551426    4461 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-nfxr8"]
Oct 30 04:12:58.557998 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:12:58.557976    4461 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-nfxr8"]
Oct 30 04:12:58.578409 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:12:58.578377    4461 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/19942a3a-cdfa-4618-83b3-f828b5cbe33e-istio-podinfo\") on node \"ip-10-0-143-156.ec2.internal\" DevicePath \"\""
Oct 30 04:12:58.578409 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:12:58.578409    4461 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fmc94\" (UniqueName: \"kubernetes.io/projected/1dacbabe-8d44-4ebc-be3d-fb89d9f70742-kube-api-access-fmc94\") on node \"ip-10-0-143-156.ec2.internal\" DevicePath \"\""
Oct 30 04:12:58.578586 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:12:58.578427    4461 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7c55427d-9f1e-4b24-9741-3ff0cfe751e7-istio-envoy\") on node \"ip-10-0-143-156.ec2.internal\" DevicePath \"\""
Oct 30 04:12:58.578586 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:12:58.578440    4461 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1dacbabe-8d44-4ebc-be3d-fb89d9f70742-istio-token\") on node \"ip-10-0-143-156.ec2.internal\" DevicePath \"\""
Oct 30 04:12:58.578586 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:12:58.578454    4461 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c55427d-9f1e-4b24-9741-3ff0cfe751e7-workload-socket\") on node \"ip-10-0-143-156.ec2.internal\" DevicePath \"\""
Oct 30 04:12:58.578586 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:12:58.578467    4461 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aedb2e20-2e07-42b1-b3c2-27223dc8d177-workload-socket\") on node \"ip-10-0-143-156.ec2.internal\" DevicePath \"\""
Oct 30 04:12:58.578586 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:12:58.578481    4461 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/19942a3a-cdfa-4618-83b3-f828b5cbe33e-workload-certs\") on node \"ip-10-0-143-156.ec2.internal\" DevicePath \"\""
Oct 30 04:12:58.578586 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:12:58.578494    4461 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aedb2e20-2e07-42b1-b3c2-27223dc8d177-istio-podinfo\") on node \"ip-10-0-143-156.ec2.internal\" DevicePath \"\""
Oct 30 04:12:58.578586 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:12:58.578507    4461 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1dacbabe-8d44-4ebc-be3d-fb89d9f70742-istio-envoy\") on node \"ip-10-0-143-156.ec2.internal\" DevicePath \"\""
Oct 30 04:12:58.578586 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:12:58.578520    4461 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aedb2e20-2e07-42b1-b3c2-27223dc8d177-credential-socket\") on node \"ip-10-0-143-156.ec2.internal\" DevicePath \"\""
Oct 30 04:12:58.578586 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:12:58.578532    4461 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fbvb7\" (UniqueName: \"kubernetes.io/projected/7c55427d-9f1e-4b24-9741-3ff0cfe751e7-kube-api-access-fbvb7\") on node \"ip-10-0-143-156.ec2.internal\" DevicePath \"\""
Oct 30 04:12:58.578586 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:12:58.578545    4461 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aedb2e20-2e07-42b1-b3c2-27223dc8d177-istio-token\") on node \"ip-10-0-143-156.ec2.internal\" DevicePath \"\""
Oct 30 04:12:58.578586 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:12:58.578559    4461 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7c55427d-9f1e-4b24-9741-3ff0cfe751e7-istio-podinfo\") on node \"ip-10-0-143-156.ec2.internal\" DevicePath \"\""
Oct 30 04:12:58.578586 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:12:58.578571    4461 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/19942a3a-cdfa-4618-83b3-f828b5cbe33e-istio-envoy\") on node \"ip-10-0-143-156.ec2.internal\" DevicePath \"\""
Oct 30 04:12:58.578586 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:12:58.578584    4461 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aedb2e20-2e07-42b1-b3c2-27223dc8d177-istio-envoy\") on node \"ip-10-0-143-156.ec2.internal\" DevicePath \"\""
Oct 30 04:12:58.578965 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:12:58.578595    4461 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7c55427d-9f1e-4b24-9741-3ff0cfe751e7-istio-token\") on node \"ip-10-0-143-156.ec2.internal\" DevicePath \"\""
Oct 30 04:12:58.578965 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:12:58.578607    4461 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/19942a3a-cdfa-4618-83b3-f828b5cbe33e-istio-token\") on node \"ip-10-0-143-156.ec2.internal\" DevicePath \"\""
Oct 30 04:12:58.578965 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:12:58.578620    4461 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-662gk\" (UniqueName: \"kubernetes.io/projected/19942a3a-cdfa-4618-83b3-f828b5cbe33e-kube-api-access-662gk\") on node \"ip-10-0-143-156.ec2.internal\" DevicePath \"\""
Oct 30 04:12:58.578965 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:12:58.578632    4461 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1dacbabe-8d44-4ebc-be3d-fb89d9f70742-istio-podinfo\") on node \"ip-10-0-143-156.ec2.internal\" DevicePath \"\""
Oct 30 04:12:58.578965 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:12:58.578645    4461 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tvhxv\" (UniqueName: \"kubernetes.io/projected/aedb2e20-2e07-42b1-b3c2-27223dc8d177-kube-api-access-tvhxv\") on node \"ip-10-0-143-156.ec2.internal\" DevicePath \"\""
Oct 30 04:12:58.594428 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:12:58.594394    4461 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-jtcmr"]
Oct 30 04:12:58.597796 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:12:58.597771    4461 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-jtcmr"]
Oct 30 04:12:58.634296 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:12:58.634266    4461 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-rh6p2"]
Oct 30 04:12:58.640931 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:12:58.640898    4461 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-rh6p2"]
Oct 30 04:12:58.675306 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:12:58.675274    4461 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-v4p2m"]
Oct 30 04:12:58.680295 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:12:58.680265    4461 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-v4p2m"]
Oct 30 04:13:00.115185 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:13:00.115149    4461 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="19942a3a-cdfa-4618-83b3-f828b5cbe33e" path="/var/lib/kubelet/pods/19942a3a-cdfa-4618-83b3-f828b5cbe33e/volumes"
Oct 30 04:13:00.115768 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:13:00.115748    4461 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1dacbabe-8d44-4ebc-be3d-fb89d9f70742" path="/var/lib/kubelet/pods/1dacbabe-8d44-4ebc-be3d-fb89d9f70742/volumes"
Oct 30 04:13:00.116227 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:13:00.116208    4461 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7c55427d-9f1e-4b24-9741-3ff0cfe751e7" path="/var/lib/kubelet/pods/7c55427d-9f1e-4b24-9741-3ff0cfe751e7/volumes"
Oct 30 04:13:00.116948 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:13:00.116928    4461 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aedb2e20-2e07-42b1-b3c2-27223dc8d177" path="/var/lib/kubelet/pods/aedb2e20-2e07-42b1-b3c2-27223dc8d177/volumes"
Oct 30 04:13:03.525786 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:13:03.525749    4461 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-prefill-7885b78964-7wn6r" podUID="c9f22c83-71df-4dd5-8eba-8c059009324c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/health\": dial tcp 10.133.0.56:8000: connect: connection refused"
Oct 30 04:13:09.951225 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:13:09.951186    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 04:13:10.121162 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:13:10.121127    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 04:13:13.526011 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:13:13.525967    4461 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-prefill-7885b78964-7wn6r" podUID="c9f22c83-71df-4dd5-8eba-8c059009324c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/health\": dial tcp 10.133.0.56:8000: connect: connection refused"
Oct 30 04:13:15.975611 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:13:15.975558    4461 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-564c47989f-fpb5f"]
Oct 30 04:13:15.976083 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:13:15.976022    4461 cpu_manager.go:414] "containerMap: removing container" podUID="fc64f170-9e8b-4ad8-8604-60849e50a5f2" containerName="main"
Oct 30 04:13:15.976083 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:13:15.976041    4461 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fc64f170-9e8b-4ad8-8604-60849e50a5f2" containerName="main"
Oct 30 04:13:15.976083 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:13:15.976056    4461 cpu_manager.go:414] "containerMap: removing container" podUID="107fadeb-7edf-49eb-8e72-8b9c7a12dbda" containerName="istio-proxy"
Oct 30 04:13:15.976083 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:13:15.976065    4461 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="107fadeb-7edf-49eb-8e72-8b9c7a12dbda" containerName="istio-proxy"
Oct 30 04:13:15.976083 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:13:15.976082    4461 cpu_manager.go:414] "containerMap: removing container" podUID="107fadeb-7edf-49eb-8e72-8b9c7a12dbda" containerName="istio-proxy"
Oct 30 04:13:15.976269 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:13:15.976091    4461 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="107fadeb-7edf-49eb-8e72-8b9c7a12dbda" containerName="istio-proxy"
Oct 30 04:13:15.976269 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:13:15.976138    4461 cpu_manager.go:414] "containerMap: removing container" podUID="fc64f170-9e8b-4ad8-8604-60849e50a5f2" containerName="storage-initializer"
Oct 30 04:13:15.976269 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:13:15.976146    4461 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fc64f170-9e8b-4ad8-8604-60849e50a5f2" containerName="storage-initializer"
Oct 30 04:13:15.976269 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:13:15.976158    4461 cpu_manager.go:414] "containerMap: removing container" podUID="7c55427d-9f1e-4b24-9741-3ff0cfe751e7" containerName="istio-proxy"
Oct 30 04:13:15.976269 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:13:15.976165    4461 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7c55427d-9f1e-4b24-9741-3ff0cfe751e7" containerName="istio-proxy"
Oct 30 04:13:15.976269 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:13:15.976179    4461 cpu_manager.go:414] "containerMap: removing container" podUID="aedb2e20-2e07-42b1-b3c2-27223dc8d177" containerName="istio-proxy"
Oct 30 04:13:15.976269 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:13:15.976186    4461 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aedb2e20-2e07-42b1-b3c2-27223dc8d177" containerName="istio-proxy"
Oct 30 04:13:15.976269 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:13:15.976202    4461 cpu_manager.go:414] "containerMap: removing container" podUID="19942a3a-cdfa-4618-83b3-f828b5cbe33e" containerName="istio-proxy"
Oct 30 04:13:15.976269 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:13:15.976209    4461 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="19942a3a-cdfa-4618-83b3-f828b5cbe33e" containerName="istio-proxy"
Oct 30 04:13:15.976269 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:13:15.976222    4461 cpu_manager.go:414] "containerMap: removing container" podUID="1dacbabe-8d44-4ebc-be3d-fb89d9f70742" containerName="istio-proxy"
Oct 30 04:13:15.976269 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:13:15.976230    4461 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1dacbabe-8d44-4ebc-be3d-fb89d9f70742" containerName="istio-proxy"
Oct 30 04:13:15.976580 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:13:15.976308    4461 memory_manager.go:365] "RemoveStaleState removing state" podUID="7c55427d-9f1e-4b24-9741-3ff0cfe751e7" containerName="istio-proxy"
Oct 30 04:13:15.976580 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:13:15.976320    4461 memory_manager.go:365] "RemoveStaleState removing state" podUID="aedb2e20-2e07-42b1-b3c2-27223dc8d177" containerName="istio-proxy"
Oct 30 04:13:15.976580 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:13:15.976330    4461 memory_manager.go:365] "RemoveStaleState removing state" podUID="19942a3a-cdfa-4618-83b3-f828b5cbe33e" containerName="istio-proxy"
Oct 30 04:13:15.976580 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:13:15.976340    4461 memory_manager.go:365] "RemoveStaleState removing state" podUID="1dacbabe-8d44-4ebc-be3d-fb89d9f70742" containerName="istio-proxy"
Oct 30 04:13:15.976580 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:13:15.976349    4461 memory_manager.go:365] "RemoveStaleState removing state" podUID="fc64f170-9e8b-4ad8-8604-60849e50a5f2" containerName="main"
Oct 30 04:13:15.976580 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:13:15.976357    4461 memory_manager.go:365] "RemoveStaleState removing state" podUID="107fadeb-7edf-49eb-8e72-8b9c7a12dbda" containerName="istio-proxy"
Oct 30 04:13:15.976580 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:13:15.976524    4461 memory_manager.go:365] "RemoveStaleState removing state" podUID="107fadeb-7edf-49eb-8e72-8b9c7a12dbda" containerName="istio-proxy"
Oct 30 04:13:15.978883 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:13:15.978863    4461 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-564c47989f-fpb5f"
Oct 30 04:13:15.996832 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:13:15.996797    4461 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"openshift-service-ca.crt\""
Oct 30 04:13:16.009147 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:13:16.009084    4461 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"default-dockercfg-vcvsf\""
Oct 30 04:13:16.009147 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:13:16.009091    4461 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"seaweedfs-s3-creds\""
Oct 30 04:13:16.009147 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:13:16.009109    4461 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"odh-kserve-custom-ca-bundle\""
Oct 30 04:13:16.009147 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:13:16.009091    4461 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"kube-root-ca.crt\""
Oct 30 04:13:16.009481 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:13:16.009207    4461 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"prestop-hook-test-kserve-self-signed-certs\""
Oct 30 04:13:16.010513 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:13:16.010491    4461 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-564c47989f-fpb5f"]
Oct 30 04:13:16.024581 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:13:16.024552    4461 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/70c0deb1-98a8-4bf0-ba9f-e4eb32612355-home\") pod \"prestop-hook-test-kserve-564c47989f-fpb5f\" (UID: \"70c0deb1-98a8-4bf0-ba9f-e4eb32612355\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-564c47989f-fpb5f"
Oct 30 04:13:16.024690 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:13:16.024641    4461 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/70c0deb1-98a8-4bf0-ba9f-e4eb32612355-cabundle-cert\") pod \"prestop-hook-test-kserve-564c47989f-fpb5f\" (UID: \"70c0deb1-98a8-4bf0-ba9f-e4eb32612355\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-564c47989f-fpb5f"
Oct 30 04:13:16.024728 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:13:16.024704    4461 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/70c0deb1-98a8-4bf0-ba9f-e4eb32612355-model-cache\") pod \"prestop-hook-test-kserve-564c47989f-fpb5f\" (UID: \"70c0deb1-98a8-4bf0-ba9f-e4eb32612355\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-564c47989f-fpb5f"
Oct 30 04:13:16.024821 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:13:16.024803    4461 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/70c0deb1-98a8-4bf0-ba9f-e4eb32612355-dshm\") pod \"prestop-hook-test-kserve-564c47989f-fpb5f\" (UID: \"70c0deb1-98a8-4bf0-ba9f-e4eb32612355\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-564c47989f-fpb5f"
Oct 30 04:13:16.024886 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:13:16.024860    4461 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/70c0deb1-98a8-4bf0-ba9f-e4eb32612355-tls-certs\") pod \"prestop-hook-test-kserve-564c47989f-fpb5f\" (UID: \"70c0deb1-98a8-4bf0-ba9f-e4eb32612355\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-564c47989f-fpb5f"
Oct 30 04:13:16.025010 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:13:16.024919    4461 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/70c0deb1-98a8-4bf0-ba9f-e4eb32612355-tmp-dir\") pod \"prestop-hook-test-kserve-564c47989f-fpb5f\" (UID: \"70c0deb1-98a8-4bf0-ba9f-e4eb32612355\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-564c47989f-fpb5f"
Oct 30 04:13:16.025010 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:13:16.024956    4461 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/70c0deb1-98a8-4bf0-ba9f-e4eb32612355-kserve-provision-location\") pod \"prestop-hook-test-kserve-564c47989f-fpb5f\" (UID: \"70c0deb1-98a8-4bf0-ba9f-e4eb32612355\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-564c47989f-fpb5f"
Oct 30 04:13:16.025010 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:13:16.024975    4461 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nnx5h\" (UniqueName: \"kubernetes.io/projected/70c0deb1-98a8-4bf0-ba9f-e4eb32612355-kube-api-access-nnx5h\") pod \"prestop-hook-test-kserve-564c47989f-fpb5f\" (UID: \"70c0deb1-98a8-4bf0-ba9f-e4eb32612355\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-564c47989f-fpb5f"
Oct 30 04:13:16.087160 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:13:16.087116    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 04:13:16.126164 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:13:16.126128    4461 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/70c0deb1-98a8-4bf0-ba9f-e4eb32612355-model-cache\") pod \"prestop-hook-test-kserve-564c47989f-fpb5f\" (UID: \"70c0deb1-98a8-4bf0-ba9f-e4eb32612355\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-564c47989f-fpb5f"
Oct 30 04:13:16.126164 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:13:16.126170    4461 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/70c0deb1-98a8-4bf0-ba9f-e4eb32612355-dshm\") pod \"prestop-hook-test-kserve-564c47989f-fpb5f\" (UID: \"70c0deb1-98a8-4bf0-ba9f-e4eb32612355\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-564c47989f-fpb5f"
Oct 30 04:13:16.126441 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:13:16.126304    4461 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/70c0deb1-98a8-4bf0-ba9f-e4eb32612355-tls-certs\") pod \"prestop-hook-test-kserve-564c47989f-fpb5f\" (UID: \"70c0deb1-98a8-4bf0-ba9f-e4eb32612355\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-564c47989f-fpb5f"
Oct 30 04:13:16.126441 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:13:16.126348    4461 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/70c0deb1-98a8-4bf0-ba9f-e4eb32612355-tmp-dir\") pod \"prestop-hook-test-kserve-564c47989f-fpb5f\" (UID: \"70c0deb1-98a8-4bf0-ba9f-e4eb32612355\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-564c47989f-fpb5f"
Oct 30 04:13:16.126441 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:13:16.126386    4461 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/70c0deb1-98a8-4bf0-ba9f-e4eb32612355-kserve-provision-location\") pod \"prestop-hook-test-kserve-564c47989f-fpb5f\" (UID: \"70c0deb1-98a8-4bf0-ba9f-e4eb32612355\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-564c47989f-fpb5f"
Oct 30 04:13:16.126649 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:13:16.126629    4461 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nnx5h\" (UniqueName: \"kubernetes.io/projected/70c0deb1-98a8-4bf0-ba9f-e4eb32612355-kube-api-access-nnx5h\") pod \"prestop-hook-test-kserve-564c47989f-fpb5f\" (UID: \"70c0deb1-98a8-4bf0-ba9f-e4eb32612355\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-564c47989f-fpb5f"
Oct 30 04:13:16.126725 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:13:16.126692    4461 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/70c0deb1-98a8-4bf0-ba9f-e4eb32612355-tmp-dir\") pod \"prestop-hook-test-kserve-564c47989f-fpb5f\" (UID: \"70c0deb1-98a8-4bf0-ba9f-e4eb32612355\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-564c47989f-fpb5f"
Oct 30 04:13:16.126777 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:13:16.126743    4461 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/70c0deb1-98a8-4bf0-ba9f-e4eb32612355-kserve-provision-location\") pod \"prestop-hook-test-kserve-564c47989f-fpb5f\" (UID: \"70c0deb1-98a8-4bf0-ba9f-e4eb32612355\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-564c47989f-fpb5f"
Oct 30 04:13:16.126777 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:13:16.126753    4461 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/70c0deb1-98a8-4bf0-ba9f-e4eb32612355-home\") pod \"prestop-hook-test-kserve-564c47989f-fpb5f\" (UID: \"70c0deb1-98a8-4bf0-ba9f-e4eb32612355\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-564c47989f-fpb5f"
Oct 30 04:13:16.126882 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:13:16.126782    4461 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/70c0deb1-98a8-4bf0-ba9f-e4eb32612355-cabundle-cert\") pod \"prestop-hook-test-kserve-564c47989f-fpb5f\" (UID: \"70c0deb1-98a8-4bf0-ba9f-e4eb32612355\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-564c47989f-fpb5f"
Oct 30 04:13:16.127071 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:13:16.127044    4461 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/70c0deb1-98a8-4bf0-ba9f-e4eb32612355-home\") pod \"prestop-hook-test-kserve-564c47989f-fpb5f\" (UID: \"70c0deb1-98a8-4bf0-ba9f-e4eb32612355\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-564c47989f-fpb5f"
Oct 30 04:13:16.127183 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:13:16.127158    4461 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/70c0deb1-98a8-4bf0-ba9f-e4eb32612355-model-cache\") pod \"prestop-hook-test-kserve-564c47989f-fpb5f\" (UID: \"70c0deb1-98a8-4bf0-ba9f-e4eb32612355\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-564c47989f-fpb5f"
Oct 30 04:13:16.127437 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:13:16.127415    4461 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/70c0deb1-98a8-4bf0-ba9f-e4eb32612355-cabundle-cert\") pod \"prestop-hook-test-kserve-564c47989f-fpb5f\" (UID: \"70c0deb1-98a8-4bf0-ba9f-e4eb32612355\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-564c47989f-fpb5f"
Oct 30 04:13:16.128525 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:13:16.128497    4461 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/70c0deb1-98a8-4bf0-ba9f-e4eb32612355-dshm\") pod \"prestop-hook-test-kserve-564c47989f-fpb5f\" (UID: \"70c0deb1-98a8-4bf0-ba9f-e4eb32612355\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-564c47989f-fpb5f"
Oct 30 04:13:16.128933 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:13:16.128909    4461 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/70c0deb1-98a8-4bf0-ba9f-e4eb32612355-tls-certs\") pod \"prestop-hook-test-kserve-564c47989f-fpb5f\" (UID: \"70c0deb1-98a8-4bf0-ba9f-e4eb32612355\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-564c47989f-fpb5f"
Oct 30 04:13:16.150267 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:13:16.150241    4461 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nnx5h\" (UniqueName: \"kubernetes.io/projected/70c0deb1-98a8-4bf0-ba9f-e4eb32612355-kube-api-access-nnx5h\") pod \"prestop-hook-test-kserve-564c47989f-fpb5f\" (UID: \"70c0deb1-98a8-4bf0-ba9f-e4eb32612355\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-564c47989f-fpb5f"
Oct 30 04:13:16.292437 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:13:16.292343    4461 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-564c47989f-fpb5f"
Oct 30 04:13:16.318883 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:13:16.318853    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 04:13:16.616337 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:13:16.616250    4461 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-564c47989f-fpb5f"]
Oct 30 04:13:16.628005 ip-10-0-143-156 kubenswrapper[4461]: W1030 04:13:16.627976    4461 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod70c0deb1_98a8_4bf0_ba9f_e4eb32612355.slice/crio-5294749cbde377b2eb59fe6ae8654c68bd9d48e5b0e3567c2a84a23a4934d895 WatchSource:0}: Error finding container 5294749cbde377b2eb59fe6ae8654c68bd9d48e5b0e3567c2a84a23a4934d895: Status 404 returned error can't find the container with id 5294749cbde377b2eb59fe6ae8654c68bd9d48e5b0e3567c2a84a23a4934d895
Oct 30 04:13:17.082049 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:13:17.082007    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 04:13:17.588751 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:13:17.588715    4461 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-564c47989f-fpb5f" event={"ID":"70c0deb1-98a8-4bf0-ba9f-e4eb32612355","Type":"ContainerStarted","Data":"d03770dc8a4c429064a7d6ebeba5b5cdcac9d0292a89f959a03f7f24dc89e341"}
Oct 30 04:13:17.588939 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:13:17.588779    4461 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-564c47989f-fpb5f" event={"ID":"70c0deb1-98a8-4bf0-ba9f-e4eb32612355","Type":"ContainerStarted","Data":"5294749cbde377b2eb59fe6ae8654c68bd9d48e5b0e3567c2a84a23a4934d895"}
Oct 30 04:13:19.600621 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:13:19.600584    4461 generic.go:356] "Generic (PLEG): container finished" podID="70c0deb1-98a8-4bf0-ba9f-e4eb32612355" containerID="d03770dc8a4c429064a7d6ebeba5b5cdcac9d0292a89f959a03f7f24dc89e341" exitCode=0
Oct 30 04:13:19.601015 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:13:19.600651    4461 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-564c47989f-fpb5f" event={"ID":"70c0deb1-98a8-4bf0-ba9f-e4eb32612355","Type":"ContainerDied","Data":"d03770dc8a4c429064a7d6ebeba5b5cdcac9d0292a89f959a03f7f24dc89e341"}
Oct 30 04:13:20.046118 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:13:20.046060    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 04:13:20.607908 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:13:20.607871    4461 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-564c47989f-fpb5f" event={"ID":"70c0deb1-98a8-4bf0-ba9f-e4eb32612355","Type":"ContainerStarted","Data":"43483e5cba90abddcb86e27c99f75edeffdb8ca6ad68dfdba1c72fafeb3680f6"}
Oct 30 04:13:20.681409 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:13:20.681346    4461 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-564c47989f-fpb5f" podStartSLOduration=3.444306597 podStartE2EDuration="5.681332866s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.237026269s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:13:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:13:20.668459743 +0000 UTC m=+4668.718264920" watchObservedRunningTime="2026-10-30 04:13:20.681332866 +0000 UTC m=+4668.731138042"
Oct 30 04:13:21.222689 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:13:21.222656    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 04:13:21.292885 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:13:21.292845    4461 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-564c47989f-fpb5f"
Oct 30 04:13:22.925005 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:13:22.924970    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 04:13:23.114615 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:13:23.114583    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 04:13:23.525479 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:13:23.525435    4461 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-prefill-7885b78964-7wn6r" podUID="c9f22c83-71df-4dd5-8eba-8c059009324c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/health\": dial tcp 10.133.0.56:8000: connect: connection refused"
Oct 30 04:13:24.506917 ip-10-0-143-156 kubenswrapper[4461]: E1030 04:13:24.506872    4461 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2k85b" containerName="extractor"
Oct 30 04:13:26.292576 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:13:26.292541    4461 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-564c47989f-fpb5f"
Oct 30 04:13:26.294402 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:13:26.294371    4461 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-564c47989f-fpb5f" podUID="70c0deb1-98a8-4bf0-ba9f-e4eb32612355" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/health\": dial tcp 10.133.0.57:8000: connect: connection refused"
Oct 30 04:13:30.324477 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:13:30.324433    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 04:13:33.526009 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:13:33.525969    4461 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-prefill-7885b78964-7wn6r" podUID="c9f22c83-71df-4dd5-8eba-8c059009324c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/health\": dial tcp 10.133.0.56:8000: connect: connection refused"
Oct 30 04:13:36.293502 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:13:36.293450    4461 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-564c47989f-fpb5f" podUID="70c0deb1-98a8-4bf0-ba9f-e4eb32612355" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/health\": dial tcp 10.133.0.57:8000: connect: connection refused"
Oct 30 04:13:43.526082 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:13:43.526036    4461 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-prefill-7885b78964-7wn6r" podUID="c9f22c83-71df-4dd5-8eba-8c059009324c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/health\": dial tcp 10.133.0.56:8000: connect: connection refused"
Oct 30 04:13:46.293632 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:13:46.293584    4461 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-564c47989f-fpb5f" podUID="70c0deb1-98a8-4bf0-ba9f-e4eb32612355" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/health\": dial tcp 10.133.0.57:8000: connect: connection refused"
Oct 30 04:13:50.574963 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:13:50.574926    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 04:13:53.428259 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:13:53.428220    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 04:13:53.526030 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:13:53.525986    4461 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-prefill-7885b78964-7wn6r" podUID="c9f22c83-71df-4dd5-8eba-8c059009324c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/health\": dial tcp 10.133.0.56:8000: connect: connection refused"
Oct 30 04:13:54.504839 ip-10-0-143-156 kubenswrapper[4461]: E1030 04:13:54.504800    4461 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2k85b" containerName="extractor"
Oct 30 04:13:56.293262 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:13:56.293217    4461 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-564c47989f-fpb5f" podUID="70c0deb1-98a8-4bf0-ba9f-e4eb32612355" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/health\": dial tcp 10.133.0.57:8000: connect: connection refused"
Oct 30 04:14:03.525497 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:14:03.525452    4461 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-prefill-7885b78964-7wn6r" podUID="c9f22c83-71df-4dd5-8eba-8c059009324c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/health\": dial tcp 10.133.0.56:8000: connect: connection refused"
Oct 30 04:14:06.292718 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:14:06.292673    4461 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-564c47989f-fpb5f" podUID="70c0deb1-98a8-4bf0-ba9f-e4eb32612355" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/health\": dial tcp 10.133.0.57:8000: connect: connection refused"
Oct 30 04:14:13.535486 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:14:13.535452    4461 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-prefill-7885b78964-7wn6r"
Oct 30 04:14:13.545500 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:14:13.545474    4461 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-prefill-7885b78964-7wn6r"
Oct 30 04:14:16.292761 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:14:16.292655    4461 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-564c47989f-fpb5f" podUID="70c0deb1-98a8-4bf0-ba9f-e4eb32612355" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/health\": dial tcp 10.133.0.57:8000: connect: connection refused"
Oct 30 04:14:20.187151 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:14:20.187115    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 04:14:24.503755 ip-10-0-143-156 kubenswrapper[4461]: E1030 04:14:24.503713    4461 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2k85b" containerName="extractor"
Oct 30 04:14:26.292742 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:14:26.292691    4461 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-564c47989f-fpb5f" podUID="70c0deb1-98a8-4bf0-ba9f-e4eb32612355" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/health\": dial tcp 10.133.0.57:8000: connect: connection refused"
Oct 30 04:14:30.460768 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:14:30.460738    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 04:14:36.292908 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:14:36.292847    4461 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-564c47989f-fpb5f" podUID="70c0deb1-98a8-4bf0-ba9f-e4eb32612355" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/health\": dial tcp 10.133.0.57:8000: connect: connection refused"
Oct 30 04:14:46.293574 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:14:46.293524    4461 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-564c47989f-fpb5f" podUID="70c0deb1-98a8-4bf0-ba9f-e4eb32612355" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/health\": dial tcp 10.133.0.57:8000: connect: connection refused"
Oct 30 04:14:50.699707 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:14:50.699662    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 04:14:54.506693 ip-10-0-143-156 kubenswrapper[4461]: E1030 04:14:54.506652    4461 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2k85b" containerName="extractor"
Oct 30 04:14:56.293771 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:14:56.293723    4461 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-564c47989f-fpb5f" podUID="70c0deb1-98a8-4bf0-ba9f-e4eb32612355" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/health\": dial tcp 10.133.0.57:8000: connect: connection refused"
Oct 30 04:15:06.292913 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:15:06.292863    4461 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-564c47989f-fpb5f" podUID="70c0deb1-98a8-4bf0-ba9f-e4eb32612355" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/health\": dial tcp 10.133.0.57:8000: connect: connection refused"
Oct 30 04:15:16.302694 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:15:16.302655    4461 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-564c47989f-fpb5f"
Oct 30 04:15:16.310437 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:15:16.310411    4461 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-564c47989f-fpb5f"
Oct 30 04:15:16.583276 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:15:16.583192    4461 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-564c47989f-fpb5f"]
Oct 30 04:15:17.935785 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:15:17.935748    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 04:15:18.110870 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:15:18.110831    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 04:15:20.302078 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:15:20.302044    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 04:15:24.503315 ip-10-0-143-156 kubenswrapper[4461]: E1030 04:15:24.503277    4461 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2k85b" containerName="extractor"
Oct 30 04:15:30.579922 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:15:30.579883    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 04:15:33.137492 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:15:33.137446    4461 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-564c47989f-fpb5f" podUID="70c0deb1-98a8-4bf0-ba9f-e4eb32612355" containerName="main" containerID="cri-o://43483e5cba90abddcb86e27c99f75edeffdb8ca6ad68dfdba1c72fafeb3680f6" gracePeriod=45
Oct 30 04:15:33.303984 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:15:33.303949    4461 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-564c47989f-fpb5f" podUID="70c0deb1-98a8-4bf0-ba9f-e4eb32612355" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/health\": dial tcp 10.133.0.57:8000: connect: connection refused"
Oct 30 04:15:33.872491 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:15:33.872463    4461 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-564c47989f-fpb5f"
Oct 30 04:15:33.947311 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:15:33.947206    4461 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/70c0deb1-98a8-4bf0-ba9f-e4eb32612355-home\" (UniqueName: \"kubernetes.io/empty-dir/70c0deb1-98a8-4bf0-ba9f-e4eb32612355-home\") pod \"70c0deb1-98a8-4bf0-ba9f-e4eb32612355\" (UID: \"70c0deb1-98a8-4bf0-ba9f-e4eb32612355\") "
Oct 30 04:15:33.947311 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:15:33.947264    4461 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/70c0deb1-98a8-4bf0-ba9f-e4eb32612355-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/70c0deb1-98a8-4bf0-ba9f-e4eb32612355-kserve-provision-location\") pod \"70c0deb1-98a8-4bf0-ba9f-e4eb32612355\" (UID: \"70c0deb1-98a8-4bf0-ba9f-e4eb32612355\") "
Oct 30 04:15:33.947311 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:15:33.947310    4461 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/70c0deb1-98a8-4bf0-ba9f-e4eb32612355-tls-certs\" (UniqueName: \"kubernetes.io/secret/70c0deb1-98a8-4bf0-ba9f-e4eb32612355-tls-certs\") pod \"70c0deb1-98a8-4bf0-ba9f-e4eb32612355\" (UID: \"70c0deb1-98a8-4bf0-ba9f-e4eb32612355\") "
Oct 30 04:15:33.947614 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:15:33.947353    4461 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/70c0deb1-98a8-4bf0-ba9f-e4eb32612355-kube-api-access-nnx5h\" (UniqueName: \"kubernetes.io/projected/70c0deb1-98a8-4bf0-ba9f-e4eb32612355-kube-api-access-nnx5h\") pod \"70c0deb1-98a8-4bf0-ba9f-e4eb32612355\" (UID: \"70c0deb1-98a8-4bf0-ba9f-e4eb32612355\") "
Oct 30 04:15:33.947614 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:15:33.947421    4461 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/70c0deb1-98a8-4bf0-ba9f-e4eb32612355-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/70c0deb1-98a8-4bf0-ba9f-e4eb32612355-cabundle-cert\") pod \"70c0deb1-98a8-4bf0-ba9f-e4eb32612355\" (UID: \"70c0deb1-98a8-4bf0-ba9f-e4eb32612355\") "
Oct 30 04:15:33.947614 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:15:33.947463    4461 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/70c0deb1-98a8-4bf0-ba9f-e4eb32612355-dshm\" (UniqueName: \"kubernetes.io/empty-dir/70c0deb1-98a8-4bf0-ba9f-e4eb32612355-dshm\") pod \"70c0deb1-98a8-4bf0-ba9f-e4eb32612355\" (UID: \"70c0deb1-98a8-4bf0-ba9f-e4eb32612355\") "
Oct 30 04:15:33.947614 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:15:33.947521    4461 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/70c0deb1-98a8-4bf0-ba9f-e4eb32612355-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/70c0deb1-98a8-4bf0-ba9f-e4eb32612355-tmp-dir\") pod \"70c0deb1-98a8-4bf0-ba9f-e4eb32612355\" (UID: \"70c0deb1-98a8-4bf0-ba9f-e4eb32612355\") "
Oct 30 04:15:33.947614 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:15:33.947570    4461 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/70c0deb1-98a8-4bf0-ba9f-e4eb32612355-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/70c0deb1-98a8-4bf0-ba9f-e4eb32612355-model-cache\") pod \"70c0deb1-98a8-4bf0-ba9f-e4eb32612355\" (UID: \"70c0deb1-98a8-4bf0-ba9f-e4eb32612355\") "
Oct 30 04:15:33.948114 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:15:33.948065    4461 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/70c0deb1-98a8-4bf0-ba9f-e4eb32612355-home" pod "70c0deb1-98a8-4bf0-ba9f-e4eb32612355" (UID: "70c0deb1-98a8-4bf0-ba9f-e4eb32612355"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:15:33.948245 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:15:33.948218    4461 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/70c0deb1-98a8-4bf0-ba9f-e4eb32612355-model-cache" pod "70c0deb1-98a8-4bf0-ba9f-e4eb32612355" (UID: "70c0deb1-98a8-4bf0-ba9f-e4eb32612355"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:15:33.948794 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:15:33.948759    4461 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/70c0deb1-98a8-4bf0-ba9f-e4eb32612355-cabundle-cert" pod "70c0deb1-98a8-4bf0-ba9f-e4eb32612355" (UID: "70c0deb1-98a8-4bf0-ba9f-e4eb32612355"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:15:33.950711 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:15:33.950673    4461 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/70c0deb1-98a8-4bf0-ba9f-e4eb32612355-dshm" pod "70c0deb1-98a8-4bf0-ba9f-e4eb32612355" (UID: "70c0deb1-98a8-4bf0-ba9f-e4eb32612355"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:15:33.951226 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:15:33.951200    4461 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/70c0deb1-98a8-4bf0-ba9f-e4eb32612355-tls-certs" pod "70c0deb1-98a8-4bf0-ba9f-e4eb32612355" (UID: "70c0deb1-98a8-4bf0-ba9f-e4eb32612355"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:15:33.952865 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:15:33.952809    4461 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/70c0deb1-98a8-4bf0-ba9f-e4eb32612355-kube-api-access-nnx5h" pod "70c0deb1-98a8-4bf0-ba9f-e4eb32612355" (UID: "70c0deb1-98a8-4bf0-ba9f-e4eb32612355"). InnerVolumeSpecName "kube-api-access-nnx5h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:15:33.963167 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:15:33.963139    4461 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/70c0deb1-98a8-4bf0-ba9f-e4eb32612355-tmp-dir" pod "70c0deb1-98a8-4bf0-ba9f-e4eb32612355" (UID: "70c0deb1-98a8-4bf0-ba9f-e4eb32612355"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:15:33.972614 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:15:33.972581    4461 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/70c0deb1-98a8-4bf0-ba9f-e4eb32612355-kserve-provision-location" pod "70c0deb1-98a8-4bf0-ba9f-e4eb32612355" (UID: "70c0deb1-98a8-4bf0-ba9f-e4eb32612355"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:15:34.048771 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:15:34.048735    4461 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/70c0deb1-98a8-4bf0-ba9f-e4eb32612355-tls-certs\") on node \"ip-10-0-143-156.ec2.internal\" DevicePath \"\""
Oct 30 04:15:34.048771 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:15:34.048766    4461 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nnx5h\" (UniqueName: \"kubernetes.io/projected/70c0deb1-98a8-4bf0-ba9f-e4eb32612355-kube-api-access-nnx5h\") on node \"ip-10-0-143-156.ec2.internal\" DevicePath \"\""
Oct 30 04:15:34.048771 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:15:34.048777    4461 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/70c0deb1-98a8-4bf0-ba9f-e4eb32612355-cabundle-cert\") on node \"ip-10-0-143-156.ec2.internal\" DevicePath \"\""
Oct 30 04:15:34.048998 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:15:34.048785    4461 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/70c0deb1-98a8-4bf0-ba9f-e4eb32612355-dshm\") on node \"ip-10-0-143-156.ec2.internal\" DevicePath \"\""
Oct 30 04:15:34.048998 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:15:34.048793    4461 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/70c0deb1-98a8-4bf0-ba9f-e4eb32612355-tmp-dir\") on node \"ip-10-0-143-156.ec2.internal\" DevicePath \"\""
Oct 30 04:15:34.048998 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:15:34.048801    4461 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/70c0deb1-98a8-4bf0-ba9f-e4eb32612355-model-cache\") on node \"ip-10-0-143-156.ec2.internal\" DevicePath \"\""
Oct 30 04:15:34.048998 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:15:34.048808    4461 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/70c0deb1-98a8-4bf0-ba9f-e4eb32612355-home\") on node \"ip-10-0-143-156.ec2.internal\" DevicePath \"\""
Oct 30 04:15:34.048998 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:15:34.048816    4461 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/70c0deb1-98a8-4bf0-ba9f-e4eb32612355-kserve-provision-location\") on node \"ip-10-0-143-156.ec2.internal\" DevicePath \"\""
Oct 30 04:15:34.174787 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:15:34.174755    4461 generic.go:356] "Generic (PLEG): container finished" podID="70c0deb1-98a8-4bf0-ba9f-e4eb32612355" containerID="43483e5cba90abddcb86e27c99f75edeffdb8ca6ad68dfdba1c72fafeb3680f6" exitCode=0
Oct 30 04:15:34.175199 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:15:34.174792    4461 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-564c47989f-fpb5f" event={"ID":"70c0deb1-98a8-4bf0-ba9f-e4eb32612355","Type":"ContainerDied","Data":"43483e5cba90abddcb86e27c99f75edeffdb8ca6ad68dfdba1c72fafeb3680f6"}
Oct 30 04:15:34.175199 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:15:34.174809    4461 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-564c47989f-fpb5f"
Oct 30 04:15:34.175199 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:15:34.174832    4461 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-564c47989f-fpb5f" event={"ID":"70c0deb1-98a8-4bf0-ba9f-e4eb32612355","Type":"ContainerDied","Data":"5294749cbde377b2eb59fe6ae8654c68bd9d48e5b0e3567c2a84a23a4934d895"}
Oct 30 04:15:34.175199 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:15:34.174852    4461 scope.go:122] "RemoveContainer" containerID="43483e5cba90abddcb86e27c99f75edeffdb8ca6ad68dfdba1c72fafeb3680f6"
Oct 30 04:15:34.182867 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:15:34.182848    4461 scope.go:122] "RemoveContainer" containerID="d03770dc8a4c429064a7d6ebeba5b5cdcac9d0292a89f959a03f7f24dc89e341"
Oct 30 04:15:34.190289 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:15:34.190265    4461 scope.go:122] "RemoveContainer" containerID="43483e5cba90abddcb86e27c99f75edeffdb8ca6ad68dfdba1c72fafeb3680f6"
Oct 30 04:15:34.190612 ip-10-0-143-156 kubenswrapper[4461]: E1030 04:15:34.190573    4461 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"43483e5cba90abddcb86e27c99f75edeffdb8ca6ad68dfdba1c72fafeb3680f6\": container with ID starting with 43483e5cba90abddcb86e27c99f75edeffdb8ca6ad68dfdba1c72fafeb3680f6 not found: ID does not exist" containerID="43483e5cba90abddcb86e27c99f75edeffdb8ca6ad68dfdba1c72fafeb3680f6"
Oct 30 04:15:34.190684 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:15:34.190618    4461 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"43483e5cba90abddcb86e27c99f75edeffdb8ca6ad68dfdba1c72fafeb3680f6"} err="failed to get container status \"43483e5cba90abddcb86e27c99f75edeffdb8ca6ad68dfdba1c72fafeb3680f6\": rpc error: code = NotFound desc = could not find container \"43483e5cba90abddcb86e27c99f75edeffdb8ca6ad68dfdba1c72fafeb3680f6\": container with ID starting with 43483e5cba90abddcb86e27c99f75edeffdb8ca6ad68dfdba1c72fafeb3680f6 not found: ID does not exist"
Oct 30 04:15:34.190684 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:15:34.190637    4461 scope.go:122] "RemoveContainer" containerID="d03770dc8a4c429064a7d6ebeba5b5cdcac9d0292a89f959a03f7f24dc89e341"
Oct 30 04:15:34.190890 ip-10-0-143-156 kubenswrapper[4461]: E1030 04:15:34.190871    4461 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d03770dc8a4c429064a7d6ebeba5b5cdcac9d0292a89f959a03f7f24dc89e341\": container with ID starting with d03770dc8a4c429064a7d6ebeba5b5cdcac9d0292a89f959a03f7f24dc89e341 not found: ID does not exist" containerID="d03770dc8a4c429064a7d6ebeba5b5cdcac9d0292a89f959a03f7f24dc89e341"
Oct 30 04:15:34.190947 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:15:34.190897    4461 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d03770dc8a4c429064a7d6ebeba5b5cdcac9d0292a89f959a03f7f24dc89e341"} err="failed to get container status \"d03770dc8a4c429064a7d6ebeba5b5cdcac9d0292a89f959a03f7f24dc89e341\": rpc error: code = NotFound desc = could not find container \"d03770dc8a4c429064a7d6ebeba5b5cdcac9d0292a89f959a03f7f24dc89e341\": container with ID starting with d03770dc8a4c429064a7d6ebeba5b5cdcac9d0292a89f959a03f7f24dc89e341 not found: ID does not exist"
Oct 30 04:15:34.233803 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:15:34.233773    4461 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-564c47989f-fpb5f"]
Oct 30 04:15:34.246707 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:15:34.246675    4461 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-564c47989f-fpb5f"]
Oct 30 04:15:35.433805 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:15:35.433769    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 04:15:35.604342 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:15:35.604313    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 04:15:36.114696 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:15:36.114666    4461 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="70c0deb1-98a8-4bf0-ba9f-e4eb32612355" path="/var/lib/kubelet/pods/70c0deb1-98a8-4bf0-ba9f-e4eb32612355/volumes"
Oct 30 04:15:50.816341 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:15:50.816265    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 04:15:54.503788 ip-10-0-143-156 kubenswrapper[4461]: E1030 04:15:54.503739    4461 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2k85b" containerName="extractor"
Oct 30 04:16:20.450031 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:16:20.449985    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 04:16:24.503621 ip-10-0-143-156 kubenswrapper[4461]: E1030 04:16:24.503573    4461 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2k85b" containerName="extractor"
Oct 30 04:16:30.719600 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:16:30.719558    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 04:16:37.699667 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:16:37.699628    4461 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-prefill-7885b78964-7wn6r"]
Oct 30 04:16:37.773771 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:16:37.773731    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 04:16:37.958391 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:16:37.958317    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 04:16:40.884337 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:16:40.884298    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 04:16:48.218721 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:16:48.218682    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 04:16:50.924990 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:16:50.924954    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 04:16:52.721940 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:16:52.721892    4461 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-prefill-7885b78964-7wn6r" podUID="c9f22c83-71df-4dd5-8eba-8c059009324c" containerName="main" containerID="cri-o://cc9dbc2165dfa82d386d825ad370ca30fb535fc087ada35099faf8c1fc75cb12" gracePeriod=45
Oct 30 04:16:53.464922 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:16:53.464896    4461 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-prefill-7885b78964-7wn6r"
Oct 30 04:16:53.467552 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:16:53.467526    4461 generic.go:356] "Generic (PLEG): container finished" podID="c9f22c83-71df-4dd5-8eba-8c059009324c" containerID="cc9dbc2165dfa82d386d825ad370ca30fb535fc087ada35099faf8c1fc75cb12" exitCode=0
Oct 30 04:16:53.467675 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:16:53.467559    4461 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-prefill-7885b78964-7wn6r" event={"ID":"c9f22c83-71df-4dd5-8eba-8c059009324c","Type":"ContainerDied","Data":"cc9dbc2165dfa82d386d825ad370ca30fb535fc087ada35099faf8c1fc75cb12"}
Oct 30 04:16:53.467675 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:16:53.467573    4461 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-prefill-7885b78964-7wn6r"
Oct 30 04:16:53.467675 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:16:53.467614    4461 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-prefill-7885b78964-7wn6r" event={"ID":"c9f22c83-71df-4dd5-8eba-8c059009324c","Type":"ContainerDied","Data":"5da21b72e5d40194465ea9d7231c9bb22c303f88b2ec36069a2a361da018565d"}
Oct 30 04:16:53.467675 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:16:53.467640    4461 scope.go:122] "RemoveContainer" containerID="cc9dbc2165dfa82d386d825ad370ca30fb535fc087ada35099faf8c1fc75cb12"
Oct 30 04:16:53.475083 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:16:53.475057    4461 scope.go:122] "RemoveContainer" containerID="8780f09da0fb87ecba5a9121120b03659ea947cae46b800b132a984df1a7a1de"
Oct 30 04:16:53.482302 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:16:53.482288    4461 scope.go:122] "RemoveContainer" containerID="cc9dbc2165dfa82d386d825ad370ca30fb535fc087ada35099faf8c1fc75cb12"
Oct 30 04:16:53.482546 ip-10-0-143-156 kubenswrapper[4461]: E1030 04:16:53.482527    4461 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cc9dbc2165dfa82d386d825ad370ca30fb535fc087ada35099faf8c1fc75cb12\": container with ID starting with cc9dbc2165dfa82d386d825ad370ca30fb535fc087ada35099faf8c1fc75cb12 not found: ID does not exist" containerID="cc9dbc2165dfa82d386d825ad370ca30fb535fc087ada35099faf8c1fc75cb12"
Oct 30 04:16:53.482593 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:16:53.482558    4461 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cc9dbc2165dfa82d386d825ad370ca30fb535fc087ada35099faf8c1fc75cb12"} err="failed to get container status \"cc9dbc2165dfa82d386d825ad370ca30fb535fc087ada35099faf8c1fc75cb12\": rpc error: code = NotFound desc = could not find container \"cc9dbc2165dfa82d386d825ad370ca30fb535fc087ada35099faf8c1fc75cb12\": container with ID starting with cc9dbc2165dfa82d386d825ad370ca30fb535fc087ada35099faf8c1fc75cb12 not found: ID does not exist"
Oct 30 04:16:53.482593 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:16:53.482576    4461 scope.go:122] "RemoveContainer" containerID="8780f09da0fb87ecba5a9121120b03659ea947cae46b800b132a984df1a7a1de"
Oct 30 04:16:53.482796 ip-10-0-143-156 kubenswrapper[4461]: E1030 04:16:53.482774    4461 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8780f09da0fb87ecba5a9121120b03659ea947cae46b800b132a984df1a7a1de\": container with ID starting with 8780f09da0fb87ecba5a9121120b03659ea947cae46b800b132a984df1a7a1de not found: ID does not exist" containerID="8780f09da0fb87ecba5a9121120b03659ea947cae46b800b132a984df1a7a1de"
Oct 30 04:16:53.482849 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:16:53.482803    4461 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8780f09da0fb87ecba5a9121120b03659ea947cae46b800b132a984df1a7a1de"} err="failed to get container status \"8780f09da0fb87ecba5a9121120b03659ea947cae46b800b132a984df1a7a1de\": rpc error: code = NotFound desc = could not find container \"8780f09da0fb87ecba5a9121120b03659ea947cae46b800b132a984df1a7a1de\": container with ID starting with 8780f09da0fb87ecba5a9121120b03659ea947cae46b800b132a984df1a7a1de not found: ID does not exist"
Oct 30 04:16:53.608360 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:16:53.608278    4461 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c9f22c83-71df-4dd5-8eba-8c059009324c-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c9f22c83-71df-4dd5-8eba-8c059009324c-model-cache\") pod \"c9f22c83-71df-4dd5-8eba-8c059009324c\" (UID: \"c9f22c83-71df-4dd5-8eba-8c059009324c\") "
Oct 30 04:16:53.608360 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:16:53.608316    4461 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c9f22c83-71df-4dd5-8eba-8c059009324c-dshm\" (UniqueName: \"kubernetes.io/empty-dir/c9f22c83-71df-4dd5-8eba-8c059009324c-dshm\") pod \"c9f22c83-71df-4dd5-8eba-8c059009324c\" (UID: \"c9f22c83-71df-4dd5-8eba-8c059009324c\") "
Oct 30 04:16:53.608360 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:16:53.608342    4461 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c9f22c83-71df-4dd5-8eba-8c059009324c-tls-certs\" (UniqueName: \"kubernetes.io/secret/c9f22c83-71df-4dd5-8eba-8c059009324c-tls-certs\") pod \"c9f22c83-71df-4dd5-8eba-8c059009324c\" (UID: \"c9f22c83-71df-4dd5-8eba-8c059009324c\") "
Oct 30 04:16:53.608648 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:16:53.608370    4461 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c9f22c83-71df-4dd5-8eba-8c059009324c-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c9f22c83-71df-4dd5-8eba-8c059009324c-cabundle-cert\") pod \"c9f22c83-71df-4dd5-8eba-8c059009324c\" (UID: \"c9f22c83-71df-4dd5-8eba-8c059009324c\") "
Oct 30 04:16:53.608648 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:16:53.608408    4461 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c9f22c83-71df-4dd5-8eba-8c059009324c-kube-api-access-rx2dg\" (UniqueName: \"kubernetes.io/projected/c9f22c83-71df-4dd5-8eba-8c059009324c-kube-api-access-rx2dg\") pod \"c9f22c83-71df-4dd5-8eba-8c059009324c\" (UID: \"c9f22c83-71df-4dd5-8eba-8c059009324c\") "
Oct 30 04:16:53.608648 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:16:53.608477    4461 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c9f22c83-71df-4dd5-8eba-8c059009324c-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c9f22c83-71df-4dd5-8eba-8c059009324c-kserve-provision-location\") pod \"c9f22c83-71df-4dd5-8eba-8c059009324c\" (UID: \"c9f22c83-71df-4dd5-8eba-8c059009324c\") "
Oct 30 04:16:53.608648 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:16:53.608535    4461 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c9f22c83-71df-4dd5-8eba-8c059009324c-home\" (UniqueName: \"kubernetes.io/empty-dir/c9f22c83-71df-4dd5-8eba-8c059009324c-home\") pod \"c9f22c83-71df-4dd5-8eba-8c059009324c\" (UID: \"c9f22c83-71df-4dd5-8eba-8c059009324c\") "
Oct 30 04:16:53.608648 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:16:53.608565    4461 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c9f22c83-71df-4dd5-8eba-8c059009324c-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c9f22c83-71df-4dd5-8eba-8c059009324c-tmp-dir\") pod \"c9f22c83-71df-4dd5-8eba-8c059009324c\" (UID: \"c9f22c83-71df-4dd5-8eba-8c059009324c\") "
Oct 30 04:16:53.608927 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:16:53.608661    4461 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c9f22c83-71df-4dd5-8eba-8c059009324c-model-cache" pod "c9f22c83-71df-4dd5-8eba-8c059009324c" (UID: "c9f22c83-71df-4dd5-8eba-8c059009324c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:16:53.608927 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:16:53.608862    4461 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c9f22c83-71df-4dd5-8eba-8c059009324c-model-cache\") on node \"ip-10-0-143-156.ec2.internal\" DevicePath \"\""
Oct 30 04:16:53.609244 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:16:53.609211    4461 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c9f22c83-71df-4dd5-8eba-8c059009324c-cabundle-cert" pod "c9f22c83-71df-4dd5-8eba-8c059009324c" (UID: "c9f22c83-71df-4dd5-8eba-8c059009324c"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:16:53.609365 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:16:53.609279    4461 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c9f22c83-71df-4dd5-8eba-8c059009324c-home" pod "c9f22c83-71df-4dd5-8eba-8c059009324c" (UID: "c9f22c83-71df-4dd5-8eba-8c059009324c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:16:53.610663 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:16:53.610639    4461 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c9f22c83-71df-4dd5-8eba-8c059009324c-dshm" pod "c9f22c83-71df-4dd5-8eba-8c059009324c" (UID: "c9f22c83-71df-4dd5-8eba-8c059009324c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:16:53.610894 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:16:53.610848    4461 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c9f22c83-71df-4dd5-8eba-8c059009324c-tls-certs" pod "c9f22c83-71df-4dd5-8eba-8c059009324c" (UID: "c9f22c83-71df-4dd5-8eba-8c059009324c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:16:53.610977 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:16:53.610931    4461 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c9f22c83-71df-4dd5-8eba-8c059009324c-kube-api-access-rx2dg" pod "c9f22c83-71df-4dd5-8eba-8c059009324c" (UID: "c9f22c83-71df-4dd5-8eba-8c059009324c"). InnerVolumeSpecName "kube-api-access-rx2dg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:16:53.628065 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:16:53.628040    4461 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c9f22c83-71df-4dd5-8eba-8c059009324c-tmp-dir" pod "c9f22c83-71df-4dd5-8eba-8c059009324c" (UID: "c9f22c83-71df-4dd5-8eba-8c059009324c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:16:53.629049 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:16:53.629027    4461 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c9f22c83-71df-4dd5-8eba-8c059009324c-kserve-provision-location" pod "c9f22c83-71df-4dd5-8eba-8c059009324c" (UID: "c9f22c83-71df-4dd5-8eba-8c059009324c"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:16:53.710133 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:16:53.710092    4461 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c9f22c83-71df-4dd5-8eba-8c059009324c-dshm\") on node \"ip-10-0-143-156.ec2.internal\" DevicePath \"\""
Oct 30 04:16:53.710133 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:16:53.710134    4461 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c9f22c83-71df-4dd5-8eba-8c059009324c-tls-certs\") on node \"ip-10-0-143-156.ec2.internal\" DevicePath \"\""
Oct 30 04:16:53.710287 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:16:53.710149    4461 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c9f22c83-71df-4dd5-8eba-8c059009324c-cabundle-cert\") on node \"ip-10-0-143-156.ec2.internal\" DevicePath \"\""
Oct 30 04:16:53.710287 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:16:53.710161    4461 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rx2dg\" (UniqueName: \"kubernetes.io/projected/c9f22c83-71df-4dd5-8eba-8c059009324c-kube-api-access-rx2dg\") on node \"ip-10-0-143-156.ec2.internal\" DevicePath \"\""
Oct 30 04:16:53.710287 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:16:53.710174    4461 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c9f22c83-71df-4dd5-8eba-8c059009324c-kserve-provision-location\") on node \"ip-10-0-143-156.ec2.internal\" DevicePath \"\""
Oct 30 04:16:53.710287 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:16:53.710186    4461 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c9f22c83-71df-4dd5-8eba-8c059009324c-home\") on node \"ip-10-0-143-156.ec2.internal\" DevicePath \"\""
Oct 30 04:16:53.710287 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:16:53.710197    4461 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c9f22c83-71df-4dd5-8eba-8c059009324c-tmp-dir\") on node \"ip-10-0-143-156.ec2.internal\" DevicePath \"\""
Oct 30 04:16:53.813552 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:16:53.813524    4461 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-prefill-7885b78964-7wn6r"]
Oct 30 04:16:53.817612 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:16:53.817588    4461 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-prefill-7885b78964-7wn6r"]
Oct 30 04:16:54.119112 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:16:54.119066    4461 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c9f22c83-71df-4dd5-8eba-8c059009324c" path="/var/lib/kubelet/pods/c9f22c83-71df-4dd5-8eba-8c059009324c/volumes"
Oct 30 04:16:54.301382 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:16:54.301349    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 04:16:54.503483 ip-10-0-143-156 kubenswrapper[4461]: E1030 04:16:54.503453    4461 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2k85b" containerName="extractor"
Oct 30 04:16:54.518195 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:16:54.518170    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 04:17:20.569538 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:17:20.569458    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 04:17:24.502733 ip-10-0-143-156 kubenswrapper[4461]: E1030 04:17:24.502696    4461 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2k85b" containerName="extractor"
Oct 30 04:17:27.925262 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:17:27.925220    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 04:17:28.133959 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:17:28.133926    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 04:17:30.798981 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:17:30.798941    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 04:17:34.533911 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:17:34.533873    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 04:17:35.972883 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:17:35.972853    4461 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-5d77dc844-lw2zh"]
Oct 30 04:17:35.973272 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:17:35.973202    4461 cpu_manager.go:414] "containerMap: removing container" podUID="c9f22c83-71df-4dd5-8eba-8c059009324c" containerName="main"
Oct 30 04:17:35.973272 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:17:35.973214    4461 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c9f22c83-71df-4dd5-8eba-8c059009324c" containerName="main"
Oct 30 04:17:35.973272 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:17:35.973226    4461 cpu_manager.go:414] "containerMap: removing container" podUID="70c0deb1-98a8-4bf0-ba9f-e4eb32612355" containerName="storage-initializer"
Oct 30 04:17:35.973272 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:17:35.973231    4461 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="70c0deb1-98a8-4bf0-ba9f-e4eb32612355" containerName="storage-initializer"
Oct 30 04:17:35.973272 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:17:35.973240    4461 cpu_manager.go:414] "containerMap: removing container" podUID="70c0deb1-98a8-4bf0-ba9f-e4eb32612355" containerName="main"
Oct 30 04:17:35.973272 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:17:35.973245    4461 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="70c0deb1-98a8-4bf0-ba9f-e4eb32612355" containerName="main"
Oct 30 04:17:35.973272 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:17:35.973259    4461 cpu_manager.go:414] "containerMap: removing container" podUID="c9f22c83-71df-4dd5-8eba-8c059009324c" containerName="storage-initializer"
Oct 30 04:17:35.973272 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:17:35.973265    4461 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c9f22c83-71df-4dd5-8eba-8c059009324c" containerName="storage-initializer"
Oct 30 04:17:35.973521 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:17:35.973306    4461 memory_manager.go:365] "RemoveStaleState removing state" podUID="70c0deb1-98a8-4bf0-ba9f-e4eb32612355" containerName="main"
Oct 30 04:17:35.973521 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:17:35.973319    4461 memory_manager.go:365] "RemoveStaleState removing state" podUID="c9f22c83-71df-4dd5-8eba-8c059009324c" containerName="main"
Oct 30 04:17:35.977167 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:17:35.977149    4461 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-5d77dc844-lw2zh"
Oct 30 04:17:35.999432 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:17:35.999395    4461 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-rolling-upgrade-coordination-a9229638\"/\"openshift-service-ca.crt\""
Oct 30 04:17:36.012255 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:17:36.012225    4461 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-rolling-upgrade-coordination-a9229638\"/\"kube-root-ca.crt\""
Oct 30 04:17:36.012408 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:17:36.012296    4461 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-rolling-upgrade-coordination-a9229638\"/\"default-dockercfg-sgm5x\""
Oct 30 04:17:36.012408 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:17:36.012346    4461 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-rolling-upgrade-coordination-a9229638\"/\"rolling-upgrade-test-kserve-self-signed-certs\""
Oct 30 04:17:36.013504 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:17:36.013488    4461 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-5d77dc844-lw2zh"]
Oct 30 04:17:36.052759 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:17:36.052721    4461 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/87a3b7a4-a34a-4e46-b39f-b57c67e0e8fc-model-cache\") pod \"rolling-upgrade-test-kserve-5d77dc844-lw2zh\" (UID: \"87a3b7a4-a34a-4e46-b39f-b57c67e0e8fc\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-5d77dc844-lw2zh"
Oct 30 04:17:36.052953 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:17:36.052775    4461 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/87a3b7a4-a34a-4e46-b39f-b57c67e0e8fc-tmp-dir\") pod \"rolling-upgrade-test-kserve-5d77dc844-lw2zh\" (UID: \"87a3b7a4-a34a-4e46-b39f-b57c67e0e8fc\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-5d77dc844-lw2zh"
Oct 30 04:17:36.052953 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:17:36.052856    4461 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/87a3b7a4-a34a-4e46-b39f-b57c67e0e8fc-home\") pod \"rolling-upgrade-test-kserve-5d77dc844-lw2zh\" (UID: \"87a3b7a4-a34a-4e46-b39f-b57c67e0e8fc\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-5d77dc844-lw2zh"
Oct 30 04:17:36.052953 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:17:36.052883    4461 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n5xjv\" (UniqueName: \"kubernetes.io/projected/87a3b7a4-a34a-4e46-b39f-b57c67e0e8fc-kube-api-access-n5xjv\") pod \"rolling-upgrade-test-kserve-5d77dc844-lw2zh\" (UID: \"87a3b7a4-a34a-4e46-b39f-b57c67e0e8fc\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-5d77dc844-lw2zh"
Oct 30 04:17:36.052953 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:17:36.052918    4461 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/87a3b7a4-a34a-4e46-b39f-b57c67e0e8fc-dshm\") pod \"rolling-upgrade-test-kserve-5d77dc844-lw2zh\" (UID: \"87a3b7a4-a34a-4e46-b39f-b57c67e0e8fc\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-5d77dc844-lw2zh"
Oct 30 04:17:36.053124 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:17:36.053022    4461 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/87a3b7a4-a34a-4e46-b39f-b57c67e0e8fc-tls-certs\") pod \"rolling-upgrade-test-kserve-5d77dc844-lw2zh\" (UID: \"87a3b7a4-a34a-4e46-b39f-b57c67e0e8fc\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-5d77dc844-lw2zh"
Oct 30 04:17:36.115470 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:17:36.115438    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 04:17:36.154192 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:17:36.154162    4461 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/87a3b7a4-a34a-4e46-b39f-b57c67e0e8fc-tls-certs\") pod \"rolling-upgrade-test-kserve-5d77dc844-lw2zh\" (UID: \"87a3b7a4-a34a-4e46-b39f-b57c67e0e8fc\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-5d77dc844-lw2zh"
Oct 30 04:17:36.154386 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:17:36.154204    4461 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/87a3b7a4-a34a-4e46-b39f-b57c67e0e8fc-model-cache\") pod \"rolling-upgrade-test-kserve-5d77dc844-lw2zh\" (UID: \"87a3b7a4-a34a-4e46-b39f-b57c67e0e8fc\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-5d77dc844-lw2zh"
Oct 30 04:17:36.154386 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:17:36.154242    4461 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/87a3b7a4-a34a-4e46-b39f-b57c67e0e8fc-tmp-dir\") pod \"rolling-upgrade-test-kserve-5d77dc844-lw2zh\" (UID: \"87a3b7a4-a34a-4e46-b39f-b57c67e0e8fc\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-5d77dc844-lw2zh"
Oct 30 04:17:36.154386 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:17:36.154277    4461 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/87a3b7a4-a34a-4e46-b39f-b57c67e0e8fc-home\") pod \"rolling-upgrade-test-kserve-5d77dc844-lw2zh\" (UID: \"87a3b7a4-a34a-4e46-b39f-b57c67e0e8fc\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-5d77dc844-lw2zh"
Oct 30 04:17:36.154386 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:17:36.154301    4461 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n5xjv\" (UniqueName: \"kubernetes.io/projected/87a3b7a4-a34a-4e46-b39f-b57c67e0e8fc-kube-api-access-n5xjv\") pod \"rolling-upgrade-test-kserve-5d77dc844-lw2zh\" (UID: \"87a3b7a4-a34a-4e46-b39f-b57c67e0e8fc\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-5d77dc844-lw2zh"
Oct 30 04:17:36.154386 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:17:36.154332    4461 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/87a3b7a4-a34a-4e46-b39f-b57c67e0e8fc-dshm\") pod \"rolling-upgrade-test-kserve-5d77dc844-lw2zh\" (UID: \"87a3b7a4-a34a-4e46-b39f-b57c67e0e8fc\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-5d77dc844-lw2zh"
Oct 30 04:17:36.154724 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:17:36.154700    4461 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/87a3b7a4-a34a-4e46-b39f-b57c67e0e8fc-model-cache\") pod \"rolling-upgrade-test-kserve-5d77dc844-lw2zh\" (UID: \"87a3b7a4-a34a-4e46-b39f-b57c67e0e8fc\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-5d77dc844-lw2zh"
Oct 30 04:17:36.154789 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:17:36.154719    4461 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/87a3b7a4-a34a-4e46-b39f-b57c67e0e8fc-tmp-dir\") pod \"rolling-upgrade-test-kserve-5d77dc844-lw2zh\" (UID: \"87a3b7a4-a34a-4e46-b39f-b57c67e0e8fc\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-5d77dc844-lw2zh"
Oct 30 04:17:36.154830 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:17:36.154790    4461 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/87a3b7a4-a34a-4e46-b39f-b57c67e0e8fc-home\") pod \"rolling-upgrade-test-kserve-5d77dc844-lw2zh\" (UID: \"87a3b7a4-a34a-4e46-b39f-b57c67e0e8fc\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-5d77dc844-lw2zh"
Oct 30 04:17:36.156523 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:17:36.156499    4461 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/87a3b7a4-a34a-4e46-b39f-b57c67e0e8fc-dshm\") pod \"rolling-upgrade-test-kserve-5d77dc844-lw2zh\" (UID: \"87a3b7a4-a34a-4e46-b39f-b57c67e0e8fc\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-5d77dc844-lw2zh"
Oct 30 04:17:36.156710 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:17:36.156693    4461 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/87a3b7a4-a34a-4e46-b39f-b57c67e0e8fc-tls-certs\") pod \"rolling-upgrade-test-kserve-5d77dc844-lw2zh\" (UID: \"87a3b7a4-a34a-4e46-b39f-b57c67e0e8fc\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-5d77dc844-lw2zh"
Oct 30 04:17:36.179849 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:17:36.179825    4461 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n5xjv\" (UniqueName: \"kubernetes.io/projected/87a3b7a4-a34a-4e46-b39f-b57c67e0e8fc-kube-api-access-n5xjv\") pod \"rolling-upgrade-test-kserve-5d77dc844-lw2zh\" (UID: \"87a3b7a4-a34a-4e46-b39f-b57c67e0e8fc\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-5d77dc844-lw2zh"
Oct 30 04:17:36.291425 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:17:36.291339    4461 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-5d77dc844-lw2zh"
Oct 30 04:17:36.389789 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:17:36.389759    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 04:17:36.588057 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:17:36.587985    4461 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-5d77dc844-lw2zh"]
Oct 30 04:17:36.591084 ip-10-0-143-156 kubenswrapper[4461]: W1030 04:17:36.591058    4461 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod87a3b7a4_a34a_4e46_b39f_b57c67e0e8fc.slice/crio-311bfac4642f6a877eb28eb28fc99c148dcf8ad503a3c4a7ec4575b2f79d0530 WatchSource:0}: Error finding container 311bfac4642f6a877eb28eb28fc99c148dcf8ad503a3c4a7ec4575b2f79d0530: Status 404 returned error can't find the container with id 311bfac4642f6a877eb28eb28fc99c148dcf8ad503a3c4a7ec4575b2f79d0530
Oct 30 04:17:36.619454 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:17:36.619430    4461 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-5d77dc844-lw2zh" event={"ID":"87a3b7a4-a34a-4e46-b39f-b57c67e0e8fc","Type":"ContainerStarted","Data":"311bfac4642f6a877eb28eb28fc99c148dcf8ad503a3c4a7ec4575b2f79d0530"}
Oct 30 04:17:37.624455 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:17:37.624420    4461 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-5d77dc844-lw2zh" event={"ID":"87a3b7a4-a34a-4e46-b39f-b57c67e0e8fc","Type":"ContainerStarted","Data":"fe0da9000fb4eea8b6db60888f83b6670b7198c384c96de418edee1ebcaa9466"}
Oct 30 04:17:37.667849 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:17:37.667802    4461 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-5d77dc844-lw2zh" podStartSLOduration=2.667788191 podStartE2EDuration="2.667788191s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:17:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:17:37.667073081 +0000 UTC m=+4925.716878256" watchObservedRunningTime="2026-10-30 04:17:37.667788191 +0000 UTC m=+4925.717593366"
Oct 30 04:17:38.292279 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:17:38.292250    4461 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-5d77dc844-lw2zh"
Oct 30 04:17:40.404329 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:17:40.404298    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 04:17:40.570313 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:17:40.570276    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 04:17:46.122185 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:17:46.122152    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 04:17:46.291641 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:17:46.291601    4461 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-5d77dc844-lw2zh"
Oct 30 04:17:46.304246 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:17:46.304216    4461 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-5d77dc844-lw2zh"
Oct 30 04:17:46.666466 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:17:46.666437    4461 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-5d77dc844-lw2zh"
Oct 30 04:17:51.054567 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:17:51.054535    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 04:17:54.020820 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:17:54.020779    4461 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-kqr6c"]
Oct 30 04:17:54.024304 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:17:54.024281    4461 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-kqr6c"
Oct 30 04:17:54.056697 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:17:54.056665    4461 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-kqr6c"]
Oct 30 04:17:54.109656 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:17:54.109619    4461 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc40c202-21a0-44ec-b556-9deab9ddae95-workload-socket\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-kqr6c\" (UID: \"bc40c202-21a0-44ec-b556-9deab9ddae95\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-kqr6c"
Oct 30 04:17:54.109656 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:17:54.109654    4461 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bc40c202-21a0-44ec-b556-9deab9ddae95-istio-podinfo\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-kqr6c\" (UID: \"bc40c202-21a0-44ec-b556-9deab9ddae95\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-kqr6c"
Oct 30 04:17:54.109848 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:17:54.109670    4461 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bc40c202-21a0-44ec-b556-9deab9ddae95-istio-envoy\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-kqr6c\" (UID: \"bc40c202-21a0-44ec-b556-9deab9ddae95\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-kqr6c"
Oct 30 04:17:54.109848 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:17:54.109801    4461 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bc40c202-21a0-44ec-b556-9deab9ddae95-workload-certs\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-kqr6c\" (UID: \"bc40c202-21a0-44ec-b556-9deab9ddae95\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-kqr6c"
Oct 30 04:17:54.109848 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:17:54.109844    4461 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bc40c202-21a0-44ec-b556-9deab9ddae95-istio-token\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-kqr6c\" (UID: \"bc40c202-21a0-44ec-b556-9deab9ddae95\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-kqr6c"
Oct 30 04:17:54.109965 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:17:54.109878    4461 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc40c202-21a0-44ec-b556-9deab9ddae95-credential-socket\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-kqr6c\" (UID: \"bc40c202-21a0-44ec-b556-9deab9ddae95\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-kqr6c"
Oct 30 04:17:54.109965 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:17:54.109895    4461 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-btmj5\" (UniqueName: \"kubernetes.io/projected/bc40c202-21a0-44ec-b556-9deab9ddae95-kube-api-access-btmj5\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-kqr6c\" (UID: \"bc40c202-21a0-44ec-b556-9deab9ddae95\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-kqr6c"
Oct 30 04:17:54.109965 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:17:54.109937    4461 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bc40c202-21a0-44ec-b556-9deab9ddae95-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-kqr6c\" (UID: \"bc40c202-21a0-44ec-b556-9deab9ddae95\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-kqr6c"
Oct 30 04:17:54.110067 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:17:54.109966    4461 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bc40c202-21a0-44ec-b556-9deab9ddae95-istio-data\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-kqr6c\" (UID: \"bc40c202-21a0-44ec-b556-9deab9ddae95\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-kqr6c"
Oct 30 04:17:54.124953 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:17:54.124917    4461 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Oct 30 04:17:54.125091 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:17:54.124966    4461 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Oct 30 04:17:54.125091 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:17:54.124989    4461 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Oct 30 04:17:54.125091 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:17:54.125081    4461 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"openshift-ai-inference-openshift-default-dockercfg-pzw79\""
Oct 30 04:17:54.211184 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:17:54.211150    4461 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bc40c202-21a0-44ec-b556-9deab9ddae95-istio-token\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-kqr6c\" (UID: \"bc40c202-21a0-44ec-b556-9deab9ddae95\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-kqr6c"
Oct 30 04:17:54.211329 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:17:54.211194    4461 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc40c202-21a0-44ec-b556-9deab9ddae95-credential-socket\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-kqr6c\" (UID: \"bc40c202-21a0-44ec-b556-9deab9ddae95\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-kqr6c"
Oct 30 04:17:54.211329 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:17:54.211211    4461 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-btmj5\" (UniqueName: \"kubernetes.io/projected/bc40c202-21a0-44ec-b556-9deab9ddae95-kube-api-access-btmj5\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-kqr6c\" (UID: \"bc40c202-21a0-44ec-b556-9deab9ddae95\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-kqr6c"
Oct 30 04:17:54.211329 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:17:54.211286    4461 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bc40c202-21a0-44ec-b556-9deab9ddae95-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-kqr6c\" (UID: \"bc40c202-21a0-44ec-b556-9deab9ddae95\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-kqr6c"
Oct 30 04:17:54.211329 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:17:54.211319    4461 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bc40c202-21a0-44ec-b556-9deab9ddae95-istio-data\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-kqr6c\" (UID: \"bc40c202-21a0-44ec-b556-9deab9ddae95\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-kqr6c"
Oct 30 04:17:54.211539 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:17:54.211360    4461 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc40c202-21a0-44ec-b556-9deab9ddae95-workload-socket\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-kqr6c\" (UID: \"bc40c202-21a0-44ec-b556-9deab9ddae95\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-kqr6c"
Oct 30 04:17:54.211539 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:17:54.211382    4461 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bc40c202-21a0-44ec-b556-9deab9ddae95-istio-podinfo\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-kqr6c\" (UID: \"bc40c202-21a0-44ec-b556-9deab9ddae95\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-kqr6c"
Oct 30 04:17:54.211539 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:17:54.211410    4461 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bc40c202-21a0-44ec-b556-9deab9ddae95-istio-envoy\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-kqr6c\" (UID: \"bc40c202-21a0-44ec-b556-9deab9ddae95\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-kqr6c"
Oct 30 04:17:54.211539 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:17:54.211490    4461 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bc40c202-21a0-44ec-b556-9deab9ddae95-workload-certs\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-kqr6c\" (UID: \"bc40c202-21a0-44ec-b556-9deab9ddae95\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-kqr6c"
Oct 30 04:17:54.211762 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:17:54.211628    4461 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc40c202-21a0-44ec-b556-9deab9ddae95-credential-socket\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-kqr6c\" (UID: \"bc40c202-21a0-44ec-b556-9deab9ddae95\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-kqr6c"
Oct 30 04:17:54.211762 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:17:54.211730    4461 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bc40c202-21a0-44ec-b556-9deab9ddae95-istio-data\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-kqr6c\" (UID: \"bc40c202-21a0-44ec-b556-9deab9ddae95\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-kqr6c"
Oct 30 04:17:54.211841 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:17:54.211797    4461 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc40c202-21a0-44ec-b556-9deab9ddae95-workload-socket\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-kqr6c\" (UID: \"bc40c202-21a0-44ec-b556-9deab9ddae95\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-kqr6c"
Oct 30 04:17:54.211916 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:17:54.211890    4461 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bc40c202-21a0-44ec-b556-9deab9ddae95-workload-certs\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-kqr6c\" (UID: \"bc40c202-21a0-44ec-b556-9deab9ddae95\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-kqr6c"
Oct 30 04:17:54.212139 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:17:54.212115    4461 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bc40c202-21a0-44ec-b556-9deab9ddae95-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-kqr6c\" (UID: \"bc40c202-21a0-44ec-b556-9deab9ddae95\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-kqr6c"
Oct 30 04:17:54.213662 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:17:54.213644    4461 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bc40c202-21a0-44ec-b556-9deab9ddae95-istio-envoy\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-kqr6c\" (UID: \"bc40c202-21a0-44ec-b556-9deab9ddae95\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-kqr6c"
Oct 30 04:17:54.213968 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:17:54.213952    4461 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bc40c202-21a0-44ec-b556-9deab9ddae95-istio-podinfo\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-kqr6c\" (UID: \"bc40c202-21a0-44ec-b556-9deab9ddae95\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-kqr6c"
Oct 30 04:17:54.234958 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:17:54.234928    4461 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bc40c202-21a0-44ec-b556-9deab9ddae95-istio-token\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-kqr6c\" (UID: \"bc40c202-21a0-44ec-b556-9deab9ddae95\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-kqr6c"
Oct 30 04:17:54.235173 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:17:54.235154    4461 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-btmj5\" (UniqueName: \"kubernetes.io/projected/bc40c202-21a0-44ec-b556-9deab9ddae95-kube-api-access-btmj5\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-kqr6c\" (UID: \"bc40c202-21a0-44ec-b556-9deab9ddae95\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-kqr6c"
Oct 30 04:17:54.337009 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:17:54.336922    4461 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-kqr6c"
Oct 30 04:17:54.504060 ip-10-0-143-156 kubenswrapper[4461]: E1030 04:17:54.504023    4461 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2k85b" containerName="extractor"
Oct 30 04:17:54.649372 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:17:54.649329    4461 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-kqr6c"]
Oct 30 04:17:54.652030 ip-10-0-143-156 kubenswrapper[4461]: W1030 04:17:54.652002    4461 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbc40c202_21a0_44ec_b556_9deab9ddae95.slice/crio-578e39beba463be8b7f935908de7e979dd6fcb0084ce2c352cf0d0e4603b3692 WatchSource:0}: Error finding container 578e39beba463be8b7f935908de7e979dd6fcb0084ce2c352cf0d0e4603b3692: Status 404 returned error can't find the container with id 578e39beba463be8b7f935908de7e979dd6fcb0084ce2c352cf0d0e4603b3692
Oct 30 04:17:54.654270 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:17:54.654237    4461 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 04:17:54.654372 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:17:54.654298    4461 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 04:17:54.654372 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:17:54.654326    4461 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 04:17:54.683558 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:17:54.683532    4461 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-kqr6c" event={"ID":"bc40c202-21a0-44ec-b556-9deab9ddae95","Type":"ContainerStarted","Data":"578e39beba463be8b7f935908de7e979dd6fcb0084ce2c352cf0d0e4603b3692"}
Oct 30 04:17:55.691638 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:17:55.691595    4461 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-kqr6c" event={"ID":"bc40c202-21a0-44ec-b556-9deab9ddae95","Type":"ContainerStarted","Data":"c4b00b0773172e16b73df4627ada87c834ab890800c82f573ab79e6652d577b0"}
Oct 30 04:17:55.750778 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:17:55.750730    4461 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-kqr6c" podStartSLOduration=2.750718921 podStartE2EDuration="2.750718921s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:17:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:17:55.749672823 +0000 UTC m=+4943.799478028" watchObservedRunningTime="2026-10-30 04:17:55.750718921 +0000 UTC m=+4943.800524091"
Oct 30 04:17:56.337150 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:17:56.337111    4461 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-kqr6c"
Oct 30 04:17:56.338560 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:17:56.338535    4461 patch_prober.go:29] interesting pod/openshift-ai-inference-openshift-default-5b4f8cc59b-kqr6c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body=
Oct 30 04:17:56.338679 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:17:56.338584    4461 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-kqr6c" podUID="bc40c202-21a0-44ec-b556-9deab9ddae95" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused"
Oct 30 04:17:57.337756 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:17:57.337720    4461 patch_prober.go:29] interesting pod/openshift-ai-inference-openshift-default-5b4f8cc59b-kqr6c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body=
Oct 30 04:17:57.338252 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:17:57.337778    4461 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-kqr6c" podUID="bc40c202-21a0-44ec-b556-9deab9ddae95" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused"
Oct 30 04:17:58.338278 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:17:58.338244    4461 patch_prober.go:29] interesting pod/openshift-ai-inference-openshift-default-5b4f8cc59b-kqr6c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body=
Oct 30 04:17:58.338661 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:17:58.338295    4461 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-kqr6c" podUID="bc40c202-21a0-44ec-b556-9deab9ddae95" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused"
Oct 30 04:17:59.337518 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:17:59.337474    4461 patch_prober.go:29] interesting pod/openshift-ai-inference-openshift-default-5b4f8cc59b-kqr6c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body=
Oct 30 04:17:59.337836 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:17:59.337536    4461 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-kqr6c" podUID="bc40c202-21a0-44ec-b556-9deab9ddae95" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused"
Oct 30 04:18:00.337429 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:18:00.337393    4461 patch_prober.go:29] interesting pod/openshift-ai-inference-openshift-default-5b4f8cc59b-kqr6c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body=
Oct 30 04:18:00.337826 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:18:00.337443    4461 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-kqr6c" podUID="bc40c202-21a0-44ec-b556-9deab9ddae95" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused"
Oct 30 04:18:01.341848 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:18:01.341817    4461 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-kqr6c"
Oct 30 04:18:01.342287 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:18:01.342122    4461 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-kqr6c"
Oct 30 04:18:01.342787 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:18:01.342770    4461 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-kqr6c"
Oct 30 04:18:09.068900 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:18:09.068863    4461 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-ljhv4"]
Oct 30 04:18:09.072587 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:18:09.072566    4461 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-ljhv4"
Oct 30 04:18:09.084944 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:18:09.084916    4461 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-99z99"]
Oct 30 04:18:09.087895 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:18:09.087876    4461 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-99z99"
Oct 30 04:18:09.097755 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:18:09.097731    4461 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-ljhv4"]
Oct 30 04:18:09.124287 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:18:09.124239    4461 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-99z99"]
Oct 30 04:18:09.155579 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:18:09.155541    4461 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a4af4d2f-3158-4eda-933d-faa3882b72d6-workload-certs\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-99z99\" (UID: \"a4af4d2f-3158-4eda-933d-faa3882b72d6\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-99z99"
Oct 30 04:18:09.155759 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:18:09.155597    4461 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/75acdd43-c913-440c-adeb-473c8d51a0db-istio-envoy\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-ljhv4\" (UID: \"75acdd43-c913-440c-adeb-473c8d51a0db\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-ljhv4"
Oct 30 04:18:09.155759 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:18:09.155677    4461 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/75acdd43-c913-440c-adeb-473c8d51a0db-istio-podinfo\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-ljhv4\" (UID: \"75acdd43-c913-440c-adeb-473c8d51a0db\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-ljhv4"
Oct 30 04:18:09.155832 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:18:09.155768    4461 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4af4d2f-3158-4eda-933d-faa3882b72d6-credential-socket\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-99z99\" (UID: \"a4af4d2f-3158-4eda-933d-faa3882b72d6\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-99z99"
Oct 30 04:18:09.155832 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:18:09.155795    4461 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a4af4d2f-3158-4eda-933d-faa3882b72d6-istio-podinfo\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-99z99\" (UID: \"a4af4d2f-3158-4eda-933d-faa3882b72d6\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-99z99"
Oct 30 04:18:09.155832 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:18:09.155814    4461 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tqw8q\" (UniqueName: \"kubernetes.io/projected/75acdd43-c913-440c-adeb-473c8d51a0db-kube-api-access-tqw8q\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-ljhv4\" (UID: \"75acdd43-c913-440c-adeb-473c8d51a0db\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-ljhv4"
Oct 30 04:18:09.156019 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:18:09.155830    4461 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/75acdd43-c913-440c-adeb-473c8d51a0db-istio-token\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-ljhv4\" (UID: \"75acdd43-c913-440c-adeb-473c8d51a0db\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-ljhv4"
Oct 30 04:18:09.156019 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:18:09.155882    4461 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a4af4d2f-3158-4eda-933d-faa3882b72d6-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-99z99\" (UID: \"a4af4d2f-3158-4eda-933d-faa3882b72d6\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-99z99"
Oct 30 04:18:09.156019 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:18:09.155908    4461 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a4af4d2f-3158-4eda-933d-faa3882b72d6-istio-data\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-99z99\" (UID: \"a4af4d2f-3158-4eda-933d-faa3882b72d6\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-99z99"
Oct 30 04:18:09.156019 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:18:09.155927    4461 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/75acdd43-c913-440c-adeb-473c8d51a0db-workload-socket\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-ljhv4\" (UID: \"75acdd43-c913-440c-adeb-473c8d51a0db\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-ljhv4"
Oct 30 04:18:09.156019 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:18:09.156009    4461 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4af4d2f-3158-4eda-933d-faa3882b72d6-workload-socket\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-99z99\" (UID: \"a4af4d2f-3158-4eda-933d-faa3882b72d6\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-99z99"
Oct 30 04:18:09.156313 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:18:09.156030    4461 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/75acdd43-c913-440c-adeb-473c8d51a0db-istio-data\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-ljhv4\" (UID: \"75acdd43-c913-440c-adeb-473c8d51a0db\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-ljhv4"
Oct 30 04:18:09.156313 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:18:09.156048    4461 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a4af4d2f-3158-4eda-933d-faa3882b72d6-istio-envoy\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-99z99\" (UID: \"a4af4d2f-3158-4eda-933d-faa3882b72d6\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-99z99"
Oct 30 04:18:09.156313 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:18:09.156077    4461 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/75acdd43-c913-440c-adeb-473c8d51a0db-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-ljhv4\" (UID: \"75acdd43-c913-440c-adeb-473c8d51a0db\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-ljhv4"
Oct 30 04:18:09.156313 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:18:09.156257    4461 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a4af4d2f-3158-4eda-933d-faa3882b72d6-istio-token\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-99z99\" (UID: \"a4af4d2f-3158-4eda-933d-faa3882b72d6\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-99z99"
Oct 30 04:18:09.156313 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:18:09.156291    4461 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/75acdd43-c913-440c-adeb-473c8d51a0db-credential-socket\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-ljhv4\" (UID: \"75acdd43-c913-440c-adeb-473c8d51a0db\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-ljhv4"
Oct 30 04:18:09.156501 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:18:09.156344    4461 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v8r45\" (UniqueName: \"kubernetes.io/projected/a4af4d2f-3158-4eda-933d-faa3882b72d6-kube-api-access-v8r45\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-99z99\" (UID: \"a4af4d2f-3158-4eda-933d-faa3882b72d6\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-99z99"
Oct 30 04:18:09.156501 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:18:09.156374    4461 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/75acdd43-c913-440c-adeb-473c8d51a0db-workload-certs\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-ljhv4\" (UID: \"75acdd43-c913-440c-adeb-473c8d51a0db\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-ljhv4"
Oct 30 04:18:09.257322 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:18:09.257285    4461 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/75acdd43-c913-440c-adeb-473c8d51a0db-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-ljhv4\" (UID: \"75acdd43-c913-440c-adeb-473c8d51a0db\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-ljhv4"
Oct 30 04:18:09.257516 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:18:09.257331    4461 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a4af4d2f-3158-4eda-933d-faa3882b72d6-istio-token\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-99z99\" (UID: \"a4af4d2f-3158-4eda-933d-faa3882b72d6\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-99z99"
Oct 30 04:18:09.257516 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:18:09.257451    4461 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/75acdd43-c913-440c-adeb-473c8d51a0db-credential-socket\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-ljhv4\" (UID: \"75acdd43-c913-440c-adeb-473c8d51a0db\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-ljhv4"
Oct 30 04:18:09.257516 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:18:09.257491    4461 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v8r45\" (UniqueName: \"kubernetes.io/projected/a4af4d2f-3158-4eda-933d-faa3882b72d6-kube-api-access-v8r45\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-99z99\" (UID: \"a4af4d2f-3158-4eda-933d-faa3882b72d6\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-99z99"
Oct 30 04:18:09.257690 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:18:09.257523    4461 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/75acdd43-c913-440c-adeb-473c8d51a0db-workload-certs\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-ljhv4\" (UID: \"75acdd43-c913-440c-adeb-473c8d51a0db\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-ljhv4"
Oct 30 04:18:09.257690 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:18:09.257555    4461 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a4af4d2f-3158-4eda-933d-faa3882b72d6-workload-certs\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-99z99\" (UID: \"a4af4d2f-3158-4eda-933d-faa3882b72d6\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-99z99"
Oct 30 04:18:09.257690 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:18:09.257596    4461 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/75acdd43-c913-440c-adeb-473c8d51a0db-istio-envoy\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-ljhv4\" (UID: \"75acdd43-c913-440c-adeb-473c8d51a0db\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-ljhv4"
Oct 30 04:18:09.257690 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:18:09.257624    4461 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/75acdd43-c913-440c-adeb-473c8d51a0db-istio-podinfo\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-ljhv4\" (UID: \"75acdd43-c913-440c-adeb-473c8d51a0db\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-ljhv4"
Oct 30 04:18:09.257690 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:18:09.257654    4461 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4af4d2f-3158-4eda-933d-faa3882b72d6-credential-socket\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-99z99\" (UID: \"a4af4d2f-3158-4eda-933d-faa3882b72d6\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-99z99"
Oct 30 04:18:09.258004 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:18:09.257692    4461 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a4af4d2f-3158-4eda-933d-faa3882b72d6-istio-podinfo\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-99z99\" (UID: \"a4af4d2f-3158-4eda-933d-faa3882b72d6\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-99z99"
Oct 30 04:18:09.258004 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:18:09.257723    4461 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tqw8q\" (UniqueName: \"kubernetes.io/projected/75acdd43-c913-440c-adeb-473c8d51a0db-kube-api-access-tqw8q\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-ljhv4\" (UID: \"75acdd43-c913-440c-adeb-473c8d51a0db\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-ljhv4"
Oct 30 04:18:09.258004 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:18:09.257754    4461 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/75acdd43-c913-440c-adeb-473c8d51a0db-istio-token\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-ljhv4\" (UID: \"75acdd43-c913-440c-adeb-473c8d51a0db\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-ljhv4"
Oct 30 04:18:09.258004 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:18:09.257777    4461 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a4af4d2f-3158-4eda-933d-faa3882b72d6-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-99z99\" (UID: \"a4af4d2f-3158-4eda-933d-faa3882b72d6\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-99z99"
Oct 30 04:18:09.258004 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:18:09.257821    4461 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a4af4d2f-3158-4eda-933d-faa3882b72d6-istio-data\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-99z99\" (UID: \"a4af4d2f-3158-4eda-933d-faa3882b72d6\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-99z99"
Oct 30 04:18:09.258293 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:18:09.258278    4461 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/75acdd43-c913-440c-adeb-473c8d51a0db-workload-socket\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-ljhv4\" (UID: \"75acdd43-c913-440c-adeb-473c8d51a0db\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-ljhv4"
Oct 30 04:18:09.258353 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:18:09.258309    4461 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a4af4d2f-3158-4eda-933d-faa3882b72d6-workload-certs\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-99z99\" (UID: \"a4af4d2f-3158-4eda-933d-faa3882b72d6\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-99z99"
Oct 30 04:18:09.258353 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:18:09.258337    4461 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4af4d2f-3158-4eda-933d-faa3882b72d6-workload-socket\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-99z99\" (UID: \"a4af4d2f-3158-4eda-933d-faa3882b72d6\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-99z99"
Oct 30 04:18:09.258487 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:18:09.258409    4461 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/75acdd43-c913-440c-adeb-473c8d51a0db-istio-data\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-ljhv4\" (UID: \"75acdd43-c913-440c-adeb-473c8d51a0db\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-ljhv4"
Oct 30 04:18:09.258487 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:18:09.258439    4461 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4af4d2f-3158-4eda-933d-faa3882b72d6-credential-socket\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-99z99\" (UID: \"a4af4d2f-3158-4eda-933d-faa3882b72d6\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-99z99"
Oct 30 04:18:09.258487 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:18:09.258454    4461 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a4af4d2f-3158-4eda-933d-faa3882b72d6-istio-envoy\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-99z99\" (UID: \"a4af4d2f-3158-4eda-933d-faa3882b72d6\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-99z99"
Oct 30 04:18:09.258487 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:18:09.258453    4461 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/75acdd43-c913-440c-adeb-473c8d51a0db-workload-certs\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-ljhv4\" (UID: \"75acdd43-c913-440c-adeb-473c8d51a0db\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-ljhv4"
Oct 30 04:18:09.258689 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:18:09.258663    4461 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4af4d2f-3158-4eda-933d-faa3882b72d6-workload-socket\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-99z99\" (UID: \"a4af4d2f-3158-4eda-933d-faa3882b72d6\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-99z99"
Oct 30 04:18:09.258826 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:18:09.258733    4461 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/75acdd43-c913-440c-adeb-473c8d51a0db-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-ljhv4\" (UID: \"75acdd43-c913-440c-adeb-473c8d51a0db\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-ljhv4"
Oct 30 04:18:09.259037 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:18:09.259009    4461 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/75acdd43-c913-440c-adeb-473c8d51a0db-workload-socket\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-ljhv4\" (UID: \"75acdd43-c913-440c-adeb-473c8d51a0db\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-ljhv4"
Oct 30 04:18:09.260044 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:18:09.259190    4461 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a4af4d2f-3158-4eda-933d-faa3882b72d6-istio-data\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-99z99\" (UID: \"a4af4d2f-3158-4eda-933d-faa3882b72d6\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-99z99"
Oct 30 04:18:09.260044 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:18:09.259290    4461 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/75acdd43-c913-440c-adeb-473c8d51a0db-credential-socket\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-ljhv4\" (UID: \"75acdd43-c913-440c-adeb-473c8d51a0db\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-ljhv4"
Oct 30 04:18:09.260044 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:18:09.259706    4461 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/75acdd43-c913-440c-adeb-473c8d51a0db-istio-data\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-ljhv4\" (UID: \"75acdd43-c913-440c-adeb-473c8d51a0db\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-ljhv4"
Oct 30 04:18:09.260044 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:18:09.259990    4461 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a4af4d2f-3158-4eda-933d-faa3882b72d6-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-99z99\" (UID: \"a4af4d2f-3158-4eda-933d-faa3882b72d6\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-99z99"
Oct 30 04:18:09.261304 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:18:09.261276    4461 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/75acdd43-c913-440c-adeb-473c8d51a0db-istio-envoy\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-ljhv4\" (UID: \"75acdd43-c913-440c-adeb-473c8d51a0db\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-ljhv4"
Oct 30 04:18:09.262121 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:18:09.261904    4461 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/75acdd43-c913-440c-adeb-473c8d51a0db-istio-podinfo\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-ljhv4\" (UID: \"75acdd43-c913-440c-adeb-473c8d51a0db\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-ljhv4"
Oct 30 04:18:09.265444 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:18:09.262849    4461 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a4af4d2f-3158-4eda-933d-faa3882b72d6-istio-podinfo\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-99z99\" (UID: \"a4af4d2f-3158-4eda-933d-faa3882b72d6\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-99z99"
Oct 30 04:18:09.265960 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:18:09.265937    4461 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a4af4d2f-3158-4eda-933d-faa3882b72d6-istio-envoy\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-99z99\" (UID: \"a4af4d2f-3158-4eda-933d-faa3882b72d6\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-99z99"
Oct 30 04:18:09.281162 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:18:09.281128    4461 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a4af4d2f-3158-4eda-933d-faa3882b72d6-istio-token\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-99z99\" (UID: \"a4af4d2f-3158-4eda-933d-faa3882b72d6\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-99z99"
Oct 30 04:18:09.281270 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:18:09.281199    4461 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tqw8q\" (UniqueName: \"kubernetes.io/projected/75acdd43-c913-440c-adeb-473c8d51a0db-kube-api-access-tqw8q\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-ljhv4\" (UID: \"75acdd43-c913-440c-adeb-473c8d51a0db\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-ljhv4"
Oct 30 04:18:09.281362 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:18:09.281346    4461 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v8r45\" (UniqueName: \"kubernetes.io/projected/a4af4d2f-3158-4eda-933d-faa3882b72d6-kube-api-access-v8r45\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-99z99\" (UID: \"a4af4d2f-3158-4eda-933d-faa3882b72d6\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-99z99"
Oct 30 04:18:09.282000 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:18:09.281978    4461 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/75acdd43-c913-440c-adeb-473c8d51a0db-istio-token\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-ljhv4\" (UID: \"75acdd43-c913-440c-adeb-473c8d51a0db\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-ljhv4"
Oct 30 04:18:09.386491 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:18:09.386395    4461 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-ljhv4"
Oct 30 04:18:09.400953 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:18:09.400908    4461 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-99z99"
Oct 30 04:18:09.713746 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:18:09.713662    4461 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-ljhv4"]
Oct 30 04:18:09.716359 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:18:09.716328    4461 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-99z99"]
Oct 30 04:18:09.717057 ip-10-0-143-156 kubenswrapper[4461]: W1030 04:18:09.717032    4461 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod75acdd43_c913_440c_adeb_473c8d51a0db.slice/crio-8b73df52cfddbc2e553588a1789e97861fb75f2f5a08af22cf4395118ecefa0c WatchSource:0}: Error finding container 8b73df52cfddbc2e553588a1789e97861fb75f2f5a08af22cf4395118ecefa0c: Status 404 returned error can't find the container with id 8b73df52cfddbc2e553588a1789e97861fb75f2f5a08af22cf4395118ecefa0c
Oct 30 04:18:09.719128 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:18:09.719083    4461 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 04:18:09.719198 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:18:09.719162    4461 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 04:18:09.719198 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:18:09.719193    4461 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 04:18:09.729409 ip-10-0-143-156 kubenswrapper[4461]: W1030 04:18:09.729383    4461 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda4af4d2f_3158_4eda_933d_faa3882b72d6.slice/crio-b0a49b3ac71c9ae5150eae202e4165c801ee67c9c7d70b9419055136c4b3a802 WatchSource:0}: Error finding container b0a49b3ac71c9ae5150eae202e4165c801ee67c9c7d70b9419055136c4b3a802: Status 404 returned error can't find the container with id b0a49b3ac71c9ae5150eae202e4165c801ee67c9c7d70b9419055136c4b3a802
Oct 30 04:18:09.731343 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:18:09.731315    4461 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 04:18:09.731428 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:18:09.731388    4461 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 04:18:09.731489 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:18:09.731432    4461 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 04:18:09.752148 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:18:09.752124    4461 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-99z99" event={"ID":"a4af4d2f-3158-4eda-933d-faa3882b72d6","Type":"ContainerStarted","Data":"b0a49b3ac71c9ae5150eae202e4165c801ee67c9c7d70b9419055136c4b3a802"}
Oct 30 04:18:09.753178 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:18:09.753156    4461 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-ljhv4" event={"ID":"75acdd43-c913-440c-adeb-473c8d51a0db","Type":"ContainerStarted","Data":"8b73df52cfddbc2e553588a1789e97861fb75f2f5a08af22cf4395118ecefa0c"}
Oct 30 04:18:10.759615 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:18:10.759567    4461 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-ljhv4" event={"ID":"75acdd43-c913-440c-adeb-473c8d51a0db","Type":"ContainerStarted","Data":"51c887c526b619228cab026d0a1a8fd1e6332d9bdd81024a2c62ed7baa5cf7d6"}
Oct 30 04:18:10.764119 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:18:10.761267    4461 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-99z99" event={"ID":"a4af4d2f-3158-4eda-933d-faa3882b72d6","Type":"ContainerStarted","Data":"c1996807db794194249a080083bc7ef5aa75423b7a67b952451b98625e49774f"}
Oct 30 04:18:10.854987 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:18:10.854932    4461 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-ljhv4" podStartSLOduration=1.85491966 podStartE2EDuration="1.85491966s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:18:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:18:10.841965882 +0000 UTC m=+4958.891771088" watchObservedRunningTime="2026-10-30 04:18:10.85491966 +0000 UTC m=+4958.904724836"
Oct 30 04:18:11.388121 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:18:11.387472    4461 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-ljhv4"
Oct 30 04:18:11.396120 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:18:11.394157    4461 patch_prober.go:29] interesting pod/openshift-ai-inference-openshift-default-5b4f8cc59b-ljhv4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body=
Oct 30 04:18:11.396120 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:18:11.394221    4461 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-ljhv4" podUID="75acdd43-c913-440c-adeb-473c8d51a0db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused"
Oct 30 04:18:11.414119 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:18:11.404510    4461 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-99z99"
Oct 30 04:18:11.414119 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:18:11.404627    4461 patch_prober.go:29] interesting pod/openshift-ai-inference-openshift-default-5b4f8cc59b-99z99 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body=
Oct 30 04:18:11.414119 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:18:11.404669    4461 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-99z99" podUID="a4af4d2f-3158-4eda-933d-faa3882b72d6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused"
Oct 30 04:18:12.391019 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:18:12.390562    4461 patch_prober.go:29] interesting pod/openshift-ai-inference-openshift-default-5b4f8cc59b-ljhv4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body=
Oct 30 04:18:12.391019 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:18:12.390629    4461 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-ljhv4" podUID="75acdd43-c913-440c-adeb-473c8d51a0db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused"
Oct 30 04:18:12.405650 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:18:12.405507    4461 patch_prober.go:29] interesting pod/openshift-ai-inference-openshift-default-5b4f8cc59b-99z99 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body=
Oct 30 04:18:12.405650 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:18:12.405590    4461 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-99z99" podUID="a4af4d2f-3158-4eda-933d-faa3882b72d6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused"
Oct 30 04:18:13.390122 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:18:13.389362    4461 patch_prober.go:29] interesting pod/openshift-ai-inference-openshift-default-5b4f8cc59b-ljhv4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body=
Oct 30 04:18:13.390122 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:18:13.389423    4461 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-ljhv4" podUID="75acdd43-c913-440c-adeb-473c8d51a0db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused"
Oct 30 04:18:13.410692 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:18:13.405003    4461 patch_prober.go:29] interesting pod/openshift-ai-inference-openshift-default-5b4f8cc59b-99z99 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body=
Oct 30 04:18:13.410692 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:18:13.405062    4461 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-99z99" podUID="a4af4d2f-3158-4eda-933d-faa3882b72d6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused"
Oct 30 04:18:14.390553 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:18:14.390509    4461 patch_prober.go:29] interesting pod/openshift-ai-inference-openshift-default-5b4f8cc59b-ljhv4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body=
Oct 30 04:18:14.390867 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:18:14.390836    4461 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-ljhv4" podUID="75acdd43-c913-440c-adeb-473c8d51a0db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused"
Oct 30 04:18:14.403479 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:18:14.403449    4461 patch_prober.go:29] interesting pod/openshift-ai-inference-openshift-default-5b4f8cc59b-99z99 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body=
Oct 30 04:18:14.403689 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:18:14.403662    4461 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-99z99" podUID="a4af4d2f-3158-4eda-933d-faa3882b72d6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused"
Oct 30 04:18:15.388751 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:18:15.388699    4461 patch_prober.go:29] interesting pod/openshift-ai-inference-openshift-default-5b4f8cc59b-ljhv4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body=
Oct 30 04:18:15.389420 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:18:15.389364    4461 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-ljhv4" podUID="75acdd43-c913-440c-adeb-473c8d51a0db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused"
Oct 30 04:18:15.402771 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:18:15.402731    4461 patch_prober.go:29] interesting pod/openshift-ai-inference-openshift-default-5b4f8cc59b-99z99 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body=
Oct 30 04:18:15.403114 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:18:15.403054    4461 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-99z99" podUID="a4af4d2f-3158-4eda-933d-faa3882b72d6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused"
Oct 30 04:18:16.334665 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:18:16.334597    4461 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-99z99" podStartSLOduration=7.334578869 podStartE2EDuration="7.334578869s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:18:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:18:10.889911808 +0000 UTC m=+4958.939716986" watchObservedRunningTime="2026-10-30 04:18:16.334578869 +0000 UTC m=+4964.384384046"
Oct 30 04:18:16.334861 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:18:16.334764    4461 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-5d77dc844-lw2zh"]
Oct 30 04:18:16.343976 ip-10-0-143-156 kubenswrapper[4461]: E1030 04:18:16.343929    4461 log.go:32] "ExecSync cmd from runtime service failed" err="rpc error: code = Unknown desc = command error: , stdout: , stderr: , exit code -1" containerID="fe0da9000fb4eea8b6db60888f83b6670b7198c384c96de418edee1ebcaa9466" cmd=["/bin/sleep","15"]
Oct 30 04:18:16.344152 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:18:16.343998    4461 handlers.go:85] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-5d77dc844-lw2zh" message="" err="rpc error: code = Unknown desc = command error: , stdout: , stderr: , exit code -1"
Oct 30 04:18:16.344152 ip-10-0-143-156 kubenswrapper[4461]: E1030 04:18:16.344013    4461 kuberuntime_container.go:780] "PreStop hook failed" err="rpc error: code = Unknown desc = command error: , stdout: , stderr: , exit code -1" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-5d77dc844-lw2zh" podUID="87a3b7a4-a34a-4e46-b39f-b57c67e0e8fc" containerName="main" containerID="cri-o://fe0da9000fb4eea8b6db60888f83b6670b7198c384c96de418edee1ebcaa9466"
Oct 30 04:18:16.344152 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:18:16.344054    4461 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-5d77dc844-lw2zh" podUID="87a3b7a4-a34a-4e46-b39f-b57c67e0e8fc" containerName="main" containerID="cri-o://fe0da9000fb4eea8b6db60888f83b6670b7198c384c96de418edee1ebcaa9466" gracePeriod=60
Oct 30 04:18:16.383277 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:18:16.383241    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 04:18:16.391049 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:18:16.391021    4461 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-ljhv4"
Oct 30 04:18:16.391404 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:18:16.391299    4461 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-ljhv4"
Oct 30 04:18:16.392017 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:18:16.392000    4461 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-ljhv4"
Oct 30 04:18:16.404773 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:18:16.404748    4461 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-99z99"
Oct 30 04:18:16.405035 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:18:16.405016    4461 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-99z99"
Oct 30 04:18:16.405601 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:18:16.405582    4461 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-99z99"
Oct 30 04:18:16.655780 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:18:16.655741    4461 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-5d77dc844-lw2zh" podUID="87a3b7a4-a34a-4e46-b39f-b57c67e0e8fc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/health\": dial tcp 10.133.0.58:8000: connect: connection refused"
Oct 30 04:18:16.762922 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:18:16.762888    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 04:18:16.834426 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:18:16.834395    4461 generic.go:356] "Generic (PLEG): container finished" podID="87a3b7a4-a34a-4e46-b39f-b57c67e0e8fc" containerID="fe0da9000fb4eea8b6db60888f83b6670b7198c384c96de418edee1ebcaa9466" exitCode=0
Oct 30 04:18:16.834614 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:18:16.834467    4461 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-5d77dc844-lw2zh" event={"ID":"87a3b7a4-a34a-4e46-b39f-b57c67e0e8fc","Type":"ContainerDied","Data":"fe0da9000fb4eea8b6db60888f83b6670b7198c384c96de418edee1ebcaa9466"}
Oct 30 04:18:17.061545 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:18:17.061507    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 04:18:17.090993 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:18:17.090965    4461 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-5d77dc844-lw2zh"
Oct 30 04:18:17.171449 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:18:17.171416    4461 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/87a3b7a4-a34a-4e46-b39f-b57c67e0e8fc-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/87a3b7a4-a34a-4e46-b39f-b57c67e0e8fc-tmp-dir\") pod \"87a3b7a4-a34a-4e46-b39f-b57c67e0e8fc\" (UID: \"87a3b7a4-a34a-4e46-b39f-b57c67e0e8fc\") "
Oct 30 04:18:17.171647 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:18:17.171456    4461 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/87a3b7a4-a34a-4e46-b39f-b57c67e0e8fc-dshm\" (UniqueName: \"kubernetes.io/empty-dir/87a3b7a4-a34a-4e46-b39f-b57c67e0e8fc-dshm\") pod \"87a3b7a4-a34a-4e46-b39f-b57c67e0e8fc\" (UID: \"87a3b7a4-a34a-4e46-b39f-b57c67e0e8fc\") "
Oct 30 04:18:17.171647 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:18:17.171510    4461 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/87a3b7a4-a34a-4e46-b39f-b57c67e0e8fc-kube-api-access-n5xjv\" (UniqueName: \"kubernetes.io/projected/87a3b7a4-a34a-4e46-b39f-b57c67e0e8fc-kube-api-access-n5xjv\") pod \"87a3b7a4-a34a-4e46-b39f-b57c67e0e8fc\" (UID: \"87a3b7a4-a34a-4e46-b39f-b57c67e0e8fc\") "
Oct 30 04:18:17.171647 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:18:17.171540    4461 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/87a3b7a4-a34a-4e46-b39f-b57c67e0e8fc-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/87a3b7a4-a34a-4e46-b39f-b57c67e0e8fc-model-cache\") pod \"87a3b7a4-a34a-4e46-b39f-b57c67e0e8fc\" (UID: \"87a3b7a4-a34a-4e46-b39f-b57c67e0e8fc\") "
Oct 30 04:18:17.171647 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:18:17.171618    4461 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/87a3b7a4-a34a-4e46-b39f-b57c67e0e8fc-home\" (UniqueName: \"kubernetes.io/empty-dir/87a3b7a4-a34a-4e46-b39f-b57c67e0e8fc-home\") pod \"87a3b7a4-a34a-4e46-b39f-b57c67e0e8fc\" (UID: \"87a3b7a4-a34a-4e46-b39f-b57c67e0e8fc\") "
Oct 30 04:18:17.171814 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:18:17.171675    4461 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/87a3b7a4-a34a-4e46-b39f-b57c67e0e8fc-tls-certs\" (UniqueName: \"kubernetes.io/secret/87a3b7a4-a34a-4e46-b39f-b57c67e0e8fc-tls-certs\") pod \"87a3b7a4-a34a-4e46-b39f-b57c67e0e8fc\" (UID: \"87a3b7a4-a34a-4e46-b39f-b57c67e0e8fc\") "
Oct 30 04:18:17.171814 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:18:17.171736    4461 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/87a3b7a4-a34a-4e46-b39f-b57c67e0e8fc-tmp-dir" pod "87a3b7a4-a34a-4e46-b39f-b57c67e0e8fc" (UID: "87a3b7a4-a34a-4e46-b39f-b57c67e0e8fc"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:18:17.171897 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:18:17.171860    4461 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/87a3b7a4-a34a-4e46-b39f-b57c67e0e8fc-model-cache" pod "87a3b7a4-a34a-4e46-b39f-b57c67e0e8fc" (UID: "87a3b7a4-a34a-4e46-b39f-b57c67e0e8fc"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:18:17.171993 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:18:17.171971    4461 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/87a3b7a4-a34a-4e46-b39f-b57c67e0e8fc-tmp-dir\") on node \"ip-10-0-143-156.ec2.internal\" DevicePath \"\""
Oct 30 04:18:17.171993 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:18:17.171992    4461 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/87a3b7a4-a34a-4e46-b39f-b57c67e0e8fc-model-cache\") on node \"ip-10-0-143-156.ec2.internal\" DevicePath \"\""
Oct 30 04:18:17.172190 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:18:17.172165    4461 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/87a3b7a4-a34a-4e46-b39f-b57c67e0e8fc-home" pod "87a3b7a4-a34a-4e46-b39f-b57c67e0e8fc" (UID: "87a3b7a4-a34a-4e46-b39f-b57c67e0e8fc"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:18:17.173871 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:18:17.173849    4461 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/87a3b7a4-a34a-4e46-b39f-b57c67e0e8fc-dshm" pod "87a3b7a4-a34a-4e46-b39f-b57c67e0e8fc" (UID: "87a3b7a4-a34a-4e46-b39f-b57c67e0e8fc"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:18:17.173956 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:18:17.173875    4461 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/87a3b7a4-a34a-4e46-b39f-b57c67e0e8fc-tls-certs" pod "87a3b7a4-a34a-4e46-b39f-b57c67e0e8fc" (UID: "87a3b7a4-a34a-4e46-b39f-b57c67e0e8fc"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:18:17.173997 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:18:17.173959    4461 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/87a3b7a4-a34a-4e46-b39f-b57c67e0e8fc-kube-api-access-n5xjv" pod "87a3b7a4-a34a-4e46-b39f-b57c67e0e8fc" (UID: "87a3b7a4-a34a-4e46-b39f-b57c67e0e8fc"). InnerVolumeSpecName "kube-api-access-n5xjv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:18:17.273413 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:18:17.273379    4461 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/87a3b7a4-a34a-4e46-b39f-b57c67e0e8fc-home\") on node \"ip-10-0-143-156.ec2.internal\" DevicePath \"\""
Oct 30 04:18:17.273413 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:18:17.273412    4461 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/87a3b7a4-a34a-4e46-b39f-b57c67e0e8fc-tls-certs\") on node \"ip-10-0-143-156.ec2.internal\" DevicePath \"\""
Oct 30 04:18:17.273632 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:18:17.273425    4461 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/87a3b7a4-a34a-4e46-b39f-b57c67e0e8fc-dshm\") on node \"ip-10-0-143-156.ec2.internal\" DevicePath \"\""
Oct 30 04:18:17.273632 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:18:17.273437    4461 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n5xjv\" (UniqueName: \"kubernetes.io/projected/87a3b7a4-a34a-4e46-b39f-b57c67e0e8fc-kube-api-access-n5xjv\") on node \"ip-10-0-143-156.ec2.internal\" DevicePath \"\""
Oct 30 04:18:17.387500 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:18:17.387439    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 04:18:17.677297 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:18:17.677198    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 04:18:17.839310 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:18:17.839283    4461 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-5d77dc844-lw2zh"
Oct 30 04:18:17.839509 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:18:17.839310    4461 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-5d77dc844-lw2zh" event={"ID":"87a3b7a4-a34a-4e46-b39f-b57c67e0e8fc","Type":"ContainerDied","Data":"311bfac4642f6a877eb28eb28fc99c148dcf8ad503a3c4a7ec4575b2f79d0530"}
Oct 30 04:18:17.839509 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:18:17.839369    4461 scope.go:122] "RemoveContainer" containerID="fe0da9000fb4eea8b6db60888f83b6670b7198c384c96de418edee1ebcaa9466"
Oct 30 04:18:17.891190 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:18:17.891158    4461 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-5d77dc844-lw2zh"]
Oct 30 04:18:17.903752 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:18:17.903720    4461 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-5d77dc844-lw2zh"]
Oct 30 04:18:17.966755 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:18:17.966724    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 04:18:18.115080 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:18:18.115046    4461 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="87a3b7a4-a34a-4e46-b39f-b57c67e0e8fc" path="/var/lib/kubelet/pods/87a3b7a4-a34a-4e46-b39f-b57c67e0e8fc/volumes"
Oct 30 04:18:18.260942 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:18:18.260850    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 04:18:18.560906 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:18:18.560818    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 04:18:18.884272 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:18:18.884183    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 04:18:20.450247 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:18:20.450208    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 04:18:20.733610 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:18:20.733574    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 04:18:23.294832 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:18:23.294780    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 04:18:24.504177 ip-10-0-143-156 kubenswrapper[4461]: E1030 04:18:24.504142    4461 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2k85b" containerName="extractor"
Oct 30 04:18:30.869780 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:18:30.869746    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 04:18:33.830034 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:18:33.829988    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 04:18:51.119922 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:18:51.119841    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 04:18:54.504682 ip-10-0-143-156 kubenswrapper[4461]: E1030 04:18:54.504640    4461 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2k85b" containerName="extractor"
Oct 30 04:19:04.485697 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:19:04.485655    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 04:19:08.270702 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:19:08.270667    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 04:19:08.305879 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:19:08.305849    4461 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-v2szj/must-gather-tzjl5"]
Oct 30 04:19:08.306233 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:19:08.306217    4461 cpu_manager.go:414] "containerMap: removing container" podUID="87a3b7a4-a34a-4e46-b39f-b57c67e0e8fc" containerName="main"
Oct 30 04:19:08.306288 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:19:08.306237    4461 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="87a3b7a4-a34a-4e46-b39f-b57c67e0e8fc" containerName="main"
Oct 30 04:19:08.306328 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:19:08.306302    4461 memory_manager.go:365] "RemoveStaleState removing state" podUID="87a3b7a4-a34a-4e46-b39f-b57c67e0e8fc" containerName="main"
Oct 30 04:19:08.309171 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:19:08.309156    4461 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-v2szj/must-gather-tzjl5"
Oct 30 04:19:08.324385 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:19:08.324363    4461 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-v2szj\"/\"openshift-service-ca.crt\""
Oct 30 04:19:08.331846 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:19:08.331823    4461 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/41c1eb75-8da1-4781-9941-9683f0321ad4-must-gather-output\") pod \"must-gather-tzjl5\" (UID: \"41c1eb75-8da1-4781-9941-9683f0321ad4\") " pod="openshift-must-gather-v2szj/must-gather-tzjl5"
Oct 30 04:19:08.331979 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:19:08.331962    4461 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wxfgt\" (UniqueName: \"kubernetes.io/projected/41c1eb75-8da1-4781-9941-9683f0321ad4-kube-api-access-wxfgt\") pod \"must-gather-tzjl5\" (UID: \"41c1eb75-8da1-4781-9941-9683f0321ad4\") " pod="openshift-must-gather-v2szj/must-gather-tzjl5"
Oct 30 04:19:08.337019 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:19:08.336999    4461 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-v2szj\"/\"default-dockercfg-lfr66\""
Oct 30 04:19:08.337152 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:19:08.337035    4461 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-v2szj\"/\"kube-root-ca.crt\""
Oct 30 04:19:08.338402 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:19:08.338377    4461 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-v2szj/must-gather-tzjl5"]
Oct 30 04:19:08.432898 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:19:08.432851    4461 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wxfgt\" (UniqueName: \"kubernetes.io/projected/41c1eb75-8da1-4781-9941-9683f0321ad4-kube-api-access-wxfgt\") pod \"must-gather-tzjl5\" (UID: \"41c1eb75-8da1-4781-9941-9683f0321ad4\") " pod="openshift-must-gather-v2szj/must-gather-tzjl5"
Oct 30 04:19:08.432898 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:19:08.432913    4461 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/41c1eb75-8da1-4781-9941-9683f0321ad4-must-gather-output\") pod \"must-gather-tzjl5\" (UID: \"41c1eb75-8da1-4781-9941-9683f0321ad4\") " pod="openshift-must-gather-v2szj/must-gather-tzjl5"
Oct 30 04:19:08.433235 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:19:08.433219    4461 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/41c1eb75-8da1-4781-9941-9683f0321ad4-must-gather-output\") pod \"must-gather-tzjl5\" (UID: \"41c1eb75-8da1-4781-9941-9683f0321ad4\") " pod="openshift-must-gather-v2szj/must-gather-tzjl5"
Oct 30 04:19:08.456458 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:19:08.456425    4461 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wxfgt\" (UniqueName: \"kubernetes.io/projected/41c1eb75-8da1-4781-9941-9683f0321ad4-kube-api-access-wxfgt\") pod \"must-gather-tzjl5\" (UID: \"41c1eb75-8da1-4781-9941-9683f0321ad4\") " pod="openshift-must-gather-v2szj/must-gather-tzjl5"
Oct 30 04:19:08.587124 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:19:08.587013    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 04:19:08.622667 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:19:08.622645    4461 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-v2szj/must-gather-tzjl5"
Oct 30 04:19:08.919582 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:19:08.919552    4461 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-v2szj/must-gather-tzjl5"]
Oct 30 04:19:08.922237 ip-10-0-143-156 kubenswrapper[4461]: W1030 04:19:08.922205    4461 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod41c1eb75_8da1_4781_9941_9683f0321ad4.slice/crio-e1e8494955a7d54b6ad9eddaa93345283ed5194b2b505c8f42193d2484d46ffc WatchSource:0}: Error finding container e1e8494955a7d54b6ad9eddaa93345283ed5194b2b505c8f42193d2484d46ffc: Status 404 returned error can't find the container with id e1e8494955a7d54b6ad9eddaa93345283ed5194b2b505c8f42193d2484d46ffc
Oct 30 04:19:08.924062 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:19:08.924044    4461 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 04:19:09.044245 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:19:09.044207    4461 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-v2szj/must-gather-tzjl5" event={"ID":"41c1eb75-8da1-4781-9941-9683f0321ad4","Type":"ContainerStarted","Data":"e1e8494955a7d54b6ad9eddaa93345283ed5194b2b505c8f42193d2484d46ffc"}
Oct 30 04:19:11.183125 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:19:11.183075    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 04:19:15.092425 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:19:15.092338    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 04:19:16.078063 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:19:16.078008    4461 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-v2szj/must-gather-tzjl5" event={"ID":"41c1eb75-8da1-4781-9941-9683f0321ad4","Type":"ContainerStarted","Data":"534fd247c3d5043bee6742aa20b9a19a1b28f15292808ed3e46f667f5ed3b3f7"}
Oct 30 04:19:16.078063 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:19:16.078064    4461 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-v2szj/must-gather-tzjl5" event={"ID":"41c1eb75-8da1-4781-9941-9683f0321ad4","Type":"ContainerStarted","Data":"d00d3b6f8237128101dda693e6c8a9f19ba6215c991c1c762b59bacc3107d165"}
Oct 30 04:19:16.119826 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:19:16.119774    4461 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-v2szj/must-gather-tzjl5" podStartSLOduration=1.9836999789999998 podStartE2EDuration="8.119759672s" totalImagesPullingTime="6.136059693s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:19:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:19:16.118220551 +0000 UTC m=+5024.168025731" watchObservedRunningTime="2026-10-30 04:19:16.119759672 +0000 UTC m=+5024.169564874"
Oct 30 04:19:17.093878 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:19:17.093833    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 04:19:17.389540 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:19:17.389457    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 04:19:19.571544 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:19:19.571507    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 04:19:20.723520 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:19:20.723489    4461 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log" limit=1024
Oct 30 04:19:24.504664 ip-10-0-143-156 kubenswrapper[4461]: E1030 04:19:24.504616    4461 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2k85b" containerName="extractor"
Oct 30 04:19:44.408217 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:19:44.408183    4461 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-5b4f8cc59b-99z99_a4af4d2f-3158-4eda-933d-faa3882b72d6/istio-proxy/0.log"
Oct 30 04:19:44.832642 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:19:44.832617    4461 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-5b4f8cc59b-kqr6c_bc40c202-21a0-44ec-b556-9deab9ddae95/istio-proxy/0.log"
Oct 30 04:19:44.897547 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:19:44.897518    4461 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-5b4f8cc59b-ljhv4_75acdd43-c913-440c-adeb-473c8d51a0db/istio-proxy/0.log"
Oct 30 04:19:46.955983 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:19:46.955945    4461 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-5b4f8cc59b-99z99_a4af4d2f-3158-4eda-933d-faa3882b72d6/istio-proxy/0.log"
Oct 30 04:19:47.214625 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:19:47.214603    4461 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-5b4f8cc59b-kqr6c_bc40c202-21a0-44ec-b556-9deab9ddae95/istio-proxy/0.log"
Oct 30 04:19:47.279042 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:19:47.279008    4461 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-5b4f8cc59b-ljhv4_75acdd43-c913-440c-adeb-473c8d51a0db/istio-proxy/0.log"
Oct 30 04:19:49.045524 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:19:49.045490    4461 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-66498fdf6c-gvm55_94fdc4cf-9363-4925-b3df-01c4487615e6/manager/0.log"
Oct 30 04:19:49.171291 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:19:49.171263    4461 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-44jp4_49dc33fe-1441-4d7c-822e-4472497f6685/kuadrant-console-plugin/0.log"
Oct 30 04:19:49.328550 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:19:49.328471    4461 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-gkz9m_2f5b6e59-9823-4958-846b-7b9b481114ef/limitador/0.log"
Oct 30 04:19:51.220363 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:19:51.220331    4461 generic.go:356] "Generic (PLEG): container finished" podID="41c1eb75-8da1-4781-9941-9683f0321ad4" containerID="d00d3b6f8237128101dda693e6c8a9f19ba6215c991c1c762b59bacc3107d165" exitCode=0
Oct 30 04:19:51.220786 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:19:51.220386    4461 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-v2szj/must-gather-tzjl5" event={"ID":"41c1eb75-8da1-4781-9941-9683f0321ad4","Type":"ContainerDied","Data":"d00d3b6f8237128101dda693e6c8a9f19ba6215c991c1c762b59bacc3107d165"}
Oct 30 04:19:51.220786 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:19:51.220730    4461 scope.go:122] "RemoveContainer" containerID="d00d3b6f8237128101dda693e6c8a9f19ba6215c991c1c762b59bacc3107d165"
Oct 30 04:19:51.426020 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:19:51.425991    4461 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-v2szj_must-gather-tzjl5_41c1eb75-8da1-4781-9941-9683f0321ad4/gather/0.log"
Oct 30 04:19:54.503088 ip-10-0-143-156 kubenswrapper[4461]: E1030 04:19:54.503045    4461 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2k85b" containerName="extractor"
Oct 30 04:19:57.338577 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:19:57.338540    4461 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-must-gather-v2szj/must-gather-tzjl5"]
Oct 30 04:19:57.339002 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:19:57.338728    4461 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-must-gather-v2szj/must-gather-tzjl5" podUID="41c1eb75-8da1-4781-9941-9683f0321ad4" containerName="copy" containerID="cri-o://534fd247c3d5043bee6742aa20b9a19a1b28f15292808ed3e46f667f5ed3b3f7" gracePeriod=2
Oct 30 04:19:57.351631 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:19:57.351600    4461 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-must-gather-v2szj/must-gather-tzjl5"]
Oct 30 04:19:57.352641 ip-10-0-143-156 kubenswrapper[4461]: E1030 04:19:57.352604    4461 status_manager.go:1045] "Failed to get status for pod" err="pods \"must-gather-tzjl5\" is forbidden: User \"system:node:ip-10-0-143-156.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-v2szj\": no relationship found between node 'ip-10-0-143-156.ec2.internal' and this object" podUID="41c1eb75-8da1-4781-9941-9683f0321ad4" pod="openshift-must-gather-v2szj/must-gather-tzjl5"
Oct 30 04:19:57.521037 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:19:57.521003    4461 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-vbnwt_98f9efe3-a732-4ba4-8543-941633d22170/global-pull-secret-syncer/0.log"
Oct 30 04:19:57.581958 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:19:57.581935    4461 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-v2szj_must-gather-tzjl5_41c1eb75-8da1-4781-9941-9683f0321ad4/copy/0.log"
Oct 30 04:19:57.583725 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:19:57.583705    4461 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-v2szj/must-gather-tzjl5"
Oct 30 04:19:57.597541 ip-10-0-143-156 kubenswrapper[4461]: E1030 04:19:57.597485    4461 status_manager.go:1045] "Failed to get status for pod" err="pods \"must-gather-tzjl5\" is forbidden: User \"system:node:ip-10-0-143-156.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-v2szj\": no relationship found between node 'ip-10-0-143-156.ec2.internal' and this object" podUID="41c1eb75-8da1-4781-9941-9683f0321ad4" pod="openshift-must-gather-v2szj/must-gather-tzjl5"
Oct 30 04:19:57.689309 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:19:57.689280    4461 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/41c1eb75-8da1-4781-9941-9683f0321ad4-kube-api-access-wxfgt\" (UniqueName: \"kubernetes.io/projected/41c1eb75-8da1-4781-9941-9683f0321ad4-kube-api-access-wxfgt\") pod \"41c1eb75-8da1-4781-9941-9683f0321ad4\" (UID: \"41c1eb75-8da1-4781-9941-9683f0321ad4\") "
Oct 30 04:19:57.689469 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:19:57.689377    4461 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/41c1eb75-8da1-4781-9941-9683f0321ad4-must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/41c1eb75-8da1-4781-9941-9683f0321ad4-must-gather-output\") pod \"41c1eb75-8da1-4781-9941-9683f0321ad4\" (UID: \"41c1eb75-8da1-4781-9941-9683f0321ad4\") "
Oct 30 04:19:57.691479 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:19:57.691457    4461 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/41c1eb75-8da1-4781-9941-9683f0321ad4-kube-api-access-wxfgt" pod "41c1eb75-8da1-4781-9941-9683f0321ad4" (UID: "41c1eb75-8da1-4781-9941-9683f0321ad4"). InnerVolumeSpecName "kube-api-access-wxfgt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:19:57.696771 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:19:57.696743    4461 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/41c1eb75-8da1-4781-9941-9683f0321ad4-must-gather-output" pod "41c1eb75-8da1-4781-9941-9683f0321ad4" (UID: "41c1eb75-8da1-4781-9941-9683f0321ad4"). InnerVolumeSpecName "must-gather-output". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:19:57.790504 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:19:57.790464    4461 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wxfgt\" (UniqueName: \"kubernetes.io/projected/41c1eb75-8da1-4781-9941-9683f0321ad4-kube-api-access-wxfgt\") on node \"ip-10-0-143-156.ec2.internal\" DevicePath \"\""
Oct 30 04:19:57.790504 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:19:57.790494    4461 reconciler_common.go:299] "Volume detached for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/41c1eb75-8da1-4781-9941-9683f0321ad4-must-gather-output\") on node \"ip-10-0-143-156.ec2.internal\" DevicePath \"\""
Oct 30 04:19:58.049894 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:19:58.049863    4461 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-djpqs_f593f4c2-de6c-42db-b451-d0ade01317f7/konnectivity-agent/0.log"
Oct 30 04:19:58.114651 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:19:58.114619    4461 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="41c1eb75-8da1-4781-9941-9683f0321ad4" path="/var/lib/kubelet/pods/41c1eb75-8da1-4781-9941-9683f0321ad4/volumes"
Oct 30 04:19:58.250714 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:19:58.250670    4461 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-v2szj_must-gather-tzjl5_41c1eb75-8da1-4781-9941-9683f0321ad4/copy/0.log"
Oct 30 04:19:58.251305 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:19:58.251264    4461 generic.go:356] "Generic (PLEG): container finished" podID="41c1eb75-8da1-4781-9941-9683f0321ad4" containerID="534fd247c3d5043bee6742aa20b9a19a1b28f15292808ed3e46f667f5ed3b3f7" exitCode=143
Oct 30 04:19:58.251418 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:19:58.251348    4461 scope.go:122] "RemoveContainer" containerID="534fd247c3d5043bee6742aa20b9a19a1b28f15292808ed3e46f667f5ed3b3f7"
Oct 30 04:19:58.251688 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:19:58.251534    4461 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-v2szj/must-gather-tzjl5"
Oct 30 04:19:58.259463 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:19:58.259445    4461 scope.go:122] "RemoveContainer" containerID="d00d3b6f8237128101dda693e6c8a9f19ba6215c991c1c762b59bacc3107d165"
Oct 30 04:19:58.272730 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:19:58.272709    4461 scope.go:122] "RemoveContainer" containerID="534fd247c3d5043bee6742aa20b9a19a1b28f15292808ed3e46f667f5ed3b3f7"
Oct 30 04:19:58.272995 ip-10-0-143-156 kubenswrapper[4461]: E1030 04:19:58.272973    4461 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"534fd247c3d5043bee6742aa20b9a19a1b28f15292808ed3e46f667f5ed3b3f7\": container with ID starting with 534fd247c3d5043bee6742aa20b9a19a1b28f15292808ed3e46f667f5ed3b3f7 not found: ID does not exist" containerID="534fd247c3d5043bee6742aa20b9a19a1b28f15292808ed3e46f667f5ed3b3f7"
Oct 30 04:19:58.273068 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:19:58.273005    4461 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"534fd247c3d5043bee6742aa20b9a19a1b28f15292808ed3e46f667f5ed3b3f7"} err="failed to get container status \"534fd247c3d5043bee6742aa20b9a19a1b28f15292808ed3e46f667f5ed3b3f7\": rpc error: code = NotFound desc = could not find container \"534fd247c3d5043bee6742aa20b9a19a1b28f15292808ed3e46f667f5ed3b3f7\": container with ID starting with 534fd247c3d5043bee6742aa20b9a19a1b28f15292808ed3e46f667f5ed3b3f7 not found: ID does not exist"
Oct 30 04:19:58.273068 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:19:58.273031    4461 scope.go:122] "RemoveContainer" containerID="d00d3b6f8237128101dda693e6c8a9f19ba6215c991c1c762b59bacc3107d165"
Oct 30 04:19:58.273318 ip-10-0-143-156 kubenswrapper[4461]: E1030 04:19:58.273295    4461 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d00d3b6f8237128101dda693e6c8a9f19ba6215c991c1c762b59bacc3107d165\": container with ID starting with d00d3b6f8237128101dda693e6c8a9f19ba6215c991c1c762b59bacc3107d165 not found: ID does not exist" containerID="d00d3b6f8237128101dda693e6c8a9f19ba6215c991c1c762b59bacc3107d165"
Oct 30 04:19:58.273358 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:19:58.273326    4461 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d00d3b6f8237128101dda693e6c8a9f19ba6215c991c1c762b59bacc3107d165"} err="failed to get container status \"d00d3b6f8237128101dda693e6c8a9f19ba6215c991c1c762b59bacc3107d165\": rpc error: code = NotFound desc = could not find container \"d00d3b6f8237128101dda693e6c8a9f19ba6215c991c1c762b59bacc3107d165\": container with ID starting with d00d3b6f8237128101dda693e6c8a9f19ba6215c991c1c762b59bacc3107d165 not found: ID does not exist"
Oct 30 04:19:58.654502 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:19:58.654468    4461 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-143-156.ec2.internal_fee27b4a19dca64d5627dff27844f71f/haproxy/0.log"
Oct 30 04:20:05.096949 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:20:05.096919    4461 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-66498fdf6c-gvm55_94fdc4cf-9363-4925-b3df-01c4487615e6/manager/0.log"
Oct 30 04:20:05.341900 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:20:05.341870    4461 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-44jp4_49dc33fe-1441-4d7c-822e-4472497f6685/kuadrant-console-plugin/0.log"
Oct 30 04:20:05.635528 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:20:05.635503    4461 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-gkz9m_2f5b6e59-9823-4958-846b-7b9b481114ef/limitador/0.log"
Oct 30 04:20:07.809534 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:20:07.809502    4461 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-lgh6m/perf-node-gather-daemonset-59482"]
Oct 30 04:20:07.809937 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:20:07.809804    4461 cpu_manager.go:414] "containerMap: removing container" podUID="41c1eb75-8da1-4781-9941-9683f0321ad4" containerName="gather"
Oct 30 04:20:07.809937 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:20:07.809815    4461 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="41c1eb75-8da1-4781-9941-9683f0321ad4" containerName="gather"
Oct 30 04:20:07.809937 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:20:07.809830    4461 cpu_manager.go:414] "containerMap: removing container" podUID="41c1eb75-8da1-4781-9941-9683f0321ad4" containerName="copy"
Oct 30 04:20:07.809937 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:20:07.809835    4461 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="41c1eb75-8da1-4781-9941-9683f0321ad4" containerName="copy"
Oct 30 04:20:07.809937 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:20:07.809896    4461 memory_manager.go:365] "RemoveStaleState removing state" podUID="41c1eb75-8da1-4781-9941-9683f0321ad4" containerName="gather"
Oct 30 04:20:07.809937 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:20:07.809904    4461 memory_manager.go:365] "RemoveStaleState removing state" podUID="41c1eb75-8da1-4781-9941-9683f0321ad4" containerName="copy"
Oct 30 04:20:07.814506 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:20:07.814485    4461 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-lgh6m/perf-node-gather-daemonset-59482"
Oct 30 04:20:07.832321 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:20:07.832289    4461 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-lgh6m/perf-node-gather-daemonset-59482"]
Oct 30 04:20:07.833668 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:20:07.833641    4461 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-lgh6m\"/\"openshift-service-ca.crt\""
Oct 30 04:20:07.843526 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:20:07.843501    4461 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-lgh6m\"/\"kube-root-ca.crt\""
Oct 30 04:20:07.843671 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:20:07.843610    4461 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-lgh6m\"/\"default-dockercfg-bgns5\""
Oct 30 04:20:07.878299 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:20:07.878258    4461 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/e2e4e09f-5da6-4149-ad98-94aaddc3e15b-proc\") pod \"perf-node-gather-daemonset-59482\" (UID: \"e2e4e09f-5da6-4149-ad98-94aaddc3e15b\") " pod="openshift-must-gather-lgh6m/perf-node-gather-daemonset-59482"
Oct 30 04:20:07.878299 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:20:07.878301    4461 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/e2e4e09f-5da6-4149-ad98-94aaddc3e15b-podres\") pod \"perf-node-gather-daemonset-59482\" (UID: \"e2e4e09f-5da6-4149-ad98-94aaddc3e15b\") " pod="openshift-must-gather-lgh6m/perf-node-gather-daemonset-59482"
Oct 30 04:20:07.878505 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:20:07.878317    4461 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e2e4e09f-5da6-4149-ad98-94aaddc3e15b-sys\") pod \"perf-node-gather-daemonset-59482\" (UID: \"e2e4e09f-5da6-4149-ad98-94aaddc3e15b\") " pod="openshift-must-gather-lgh6m/perf-node-gather-daemonset-59482"
Oct 30 04:20:07.878505 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:20:07.878454    4461 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e2e4e09f-5da6-4149-ad98-94aaddc3e15b-lib-modules\") pod \"perf-node-gather-daemonset-59482\" (UID: \"e2e4e09f-5da6-4149-ad98-94aaddc3e15b\") " pod="openshift-must-gather-lgh6m/perf-node-gather-daemonset-59482"
Oct 30 04:20:07.878576 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:20:07.878508    4461 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g6qnb\" (UniqueName: \"kubernetes.io/projected/e2e4e09f-5da6-4149-ad98-94aaddc3e15b-kube-api-access-g6qnb\") pod \"perf-node-gather-daemonset-59482\" (UID: \"e2e4e09f-5da6-4149-ad98-94aaddc3e15b\") " pod="openshift-must-gather-lgh6m/perf-node-gather-daemonset-59482"
Oct 30 04:20:07.979593 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:20:07.979551    4461 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/e2e4e09f-5da6-4149-ad98-94aaddc3e15b-proc\") pod \"perf-node-gather-daemonset-59482\" (UID: \"e2e4e09f-5da6-4149-ad98-94aaddc3e15b\") " pod="openshift-must-gather-lgh6m/perf-node-gather-daemonset-59482"
Oct 30 04:20:07.979593 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:20:07.979593    4461 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/e2e4e09f-5da6-4149-ad98-94aaddc3e15b-podres\") pod \"perf-node-gather-daemonset-59482\" (UID: \"e2e4e09f-5da6-4149-ad98-94aaddc3e15b\") " pod="openshift-must-gather-lgh6m/perf-node-gather-daemonset-59482"
Oct 30 04:20:07.979825 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:20:07.979612    4461 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e2e4e09f-5da6-4149-ad98-94aaddc3e15b-sys\") pod \"perf-node-gather-daemonset-59482\" (UID: \"e2e4e09f-5da6-4149-ad98-94aaddc3e15b\") " pod="openshift-must-gather-lgh6m/perf-node-gather-daemonset-59482"
Oct 30 04:20:07.979825 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:20:07.979649    4461 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e2e4e09f-5da6-4149-ad98-94aaddc3e15b-lib-modules\") pod \"perf-node-gather-daemonset-59482\" (UID: \"e2e4e09f-5da6-4149-ad98-94aaddc3e15b\") " pod="openshift-must-gather-lgh6m/perf-node-gather-daemonset-59482"
Oct 30 04:20:07.979825 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:20:07.979688    4461 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g6qnb\" (UniqueName: \"kubernetes.io/projected/e2e4e09f-5da6-4149-ad98-94aaddc3e15b-kube-api-access-g6qnb\") pod \"perf-node-gather-daemonset-59482\" (UID: \"e2e4e09f-5da6-4149-ad98-94aaddc3e15b\") " pod="openshift-must-gather-lgh6m/perf-node-gather-daemonset-59482"
Oct 30 04:20:07.979825 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:20:07.979703    4461 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/e2e4e09f-5da6-4149-ad98-94aaddc3e15b-podres\") pod \"perf-node-gather-daemonset-59482\" (UID: \"e2e4e09f-5da6-4149-ad98-94aaddc3e15b\") " pod="openshift-must-gather-lgh6m/perf-node-gather-daemonset-59482"
Oct 30 04:20:07.979825 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:20:07.979703    4461 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/e2e4e09f-5da6-4149-ad98-94aaddc3e15b-proc\") pod \"perf-node-gather-daemonset-59482\" (UID: \"e2e4e09f-5da6-4149-ad98-94aaddc3e15b\") " pod="openshift-must-gather-lgh6m/perf-node-gather-daemonset-59482"
Oct 30 04:20:07.979825 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:20:07.979766    4461 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e2e4e09f-5da6-4149-ad98-94aaddc3e15b-lib-modules\") pod \"perf-node-gather-daemonset-59482\" (UID: \"e2e4e09f-5da6-4149-ad98-94aaddc3e15b\") " pod="openshift-must-gather-lgh6m/perf-node-gather-daemonset-59482"
Oct 30 04:20:07.979825 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:20:07.979769    4461 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e2e4e09f-5da6-4149-ad98-94aaddc3e15b-sys\") pod \"perf-node-gather-daemonset-59482\" (UID: \"e2e4e09f-5da6-4149-ad98-94aaddc3e15b\") " pod="openshift-must-gather-lgh6m/perf-node-gather-daemonset-59482"
Oct 30 04:20:08.003946 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:20:08.003911    4461 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g6qnb\" (UniqueName: \"kubernetes.io/projected/e2e4e09f-5da6-4149-ad98-94aaddc3e15b-kube-api-access-g6qnb\") pod \"perf-node-gather-daemonset-59482\" (UID: \"e2e4e09f-5da6-4149-ad98-94aaddc3e15b\") " pod="openshift-must-gather-lgh6m/perf-node-gather-daemonset-59482"
Oct 30 04:20:08.125978 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:20:08.125889    4461 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-lgh6m/perf-node-gather-daemonset-59482"
Oct 30 04:20:08.315776 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:20:08.315739    4461 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-7d8666798c-cnjhs_9e87e1b3-bbad-44a8-9684-83aed2d040d5/metrics-server/0.log"
Oct 30 04:20:08.449216 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:20:08.449189    4461 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-lgh6m/perf-node-gather-daemonset-59482"]
Oct 30 04:20:08.451649 ip-10-0-143-156 kubenswrapper[4461]: W1030 04:20:08.451617    4461 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pode2e4e09f_5da6_4149_ad98_94aaddc3e15b.slice/crio-b6ff618b7fba7e64c6ee4f3b0e8b11f0b0878bde46992ed04f7709585fc05ed3 WatchSource:0}: Error finding container b6ff618b7fba7e64c6ee4f3b0e8b11f0b0878bde46992ed04f7709585fc05ed3: Status 404 returned error can't find the container with id b6ff618b7fba7e64c6ee4f3b0e8b11f0b0878bde46992ed04f7709585fc05ed3
Oct 30 04:20:08.466541 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:20:08.466514    4461 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-77f4f76594-st7wb_924e9f2a-f91e-43b5-ad1c-ea75d8aaa765/monitoring-plugin/0.log"
Oct 30 04:20:08.609134 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:20:08.609091    4461 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-n769x_61b66fcf-4ed1-40d0-acb5-d0490b21fd6b/node-exporter/0.log"
Oct 30 04:20:08.728092 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:20:08.728060    4461 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-n769x_61b66fcf-4ed1-40d0-acb5-d0490b21fd6b/kube-rbac-proxy/0.log"
Oct 30 04:20:08.845826 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:20:08.845798    4461 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-n769x_61b66fcf-4ed1-40d0-acb5-d0490b21fd6b/init-textfile/0.log"
Oct 30 04:20:09.302937 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:20:09.302902    4461 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-lgh6m/perf-node-gather-daemonset-59482" event={"ID":"e2e4e09f-5da6-4149-ad98-94aaddc3e15b","Type":"ContainerStarted","Data":"fc9045cd0db40ebc5603bc34977199d082982a091e825239d4815b756cfc1397"}
Oct 30 04:20:09.303142 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:20:09.302953    4461 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-lgh6m/perf-node-gather-daemonset-59482" event={"ID":"e2e4e09f-5da6-4149-ad98-94aaddc3e15b","Type":"ContainerStarted","Data":"b6ff618b7fba7e64c6ee4f3b0e8b11f0b0878bde46992ed04f7709585fc05ed3"}
Oct 30 04:20:09.303142 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:20:09.302998    4461 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-lgh6m/perf-node-gather-daemonset-59482"
Oct 30 04:20:09.345023 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:20:09.344972    4461 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-lgh6m/perf-node-gather-daemonset-59482" podStartSLOduration=2.344957675 podStartE2EDuration="2.344957675s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 04:20:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:20:09.343825057 +0000 UTC m=+5077.393630215" watchObservedRunningTime="2026-10-30 04:20:09.344957675 +0000 UTC m=+5077.394762852"
Oct 30 04:20:10.083369 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:20:10.083335    4461 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_4c1f3a6e-55b5-49a6-a08b-e0a6ef968f92/prometheus/0.log"
Oct 30 04:20:10.181249 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:20:10.181220    4461 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_4c1f3a6e-55b5-49a6-a08b-e0a6ef968f92/config-reloader/0.log"
Oct 30 04:20:10.299337 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:20:10.299310    4461 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_4c1f3a6e-55b5-49a6-a08b-e0a6ef968f92/thanos-sidecar/0.log"
Oct 30 04:20:10.416985 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:20:10.416890    4461 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_4c1f3a6e-55b5-49a6-a08b-e0a6ef968f92/kube-rbac-proxy-web/0.log"
Oct 30 04:20:10.534467 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:20:10.534441    4461 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_4c1f3a6e-55b5-49a6-a08b-e0a6ef968f92/kube-rbac-proxy/0.log"
Oct 30 04:20:10.656399 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:20:10.656372    4461 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_4c1f3a6e-55b5-49a6-a08b-e0a6ef968f92/kube-rbac-proxy-thanos/0.log"
Oct 30 04:20:10.776134 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:20:10.776091    4461 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_4c1f3a6e-55b5-49a6-a08b-e0a6ef968f92/init-config-reloader/0.log"
Oct 30 04:20:10.900707 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:20:10.900678    4461 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-9cd89b74c-jrs99_759f6f70-c39d-4607-92df-1ef50553b00f/prometheus-operator/0.log"
Oct 30 04:20:11.014475 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:20:11.014443    4461 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-9cd89b74c-jrs99_759f6f70-c39d-4607-92df-1ef50553b00f/kube-rbac-proxy/0.log"
Oct 30 04:20:15.315924 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:20:15.315847    4461 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-lgh6m/perf-node-gather-daemonset-59482"
Oct 30 04:20:19.528859 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:20:19.528829    4461 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-wjll7_5f9fb000-9531-41f1-b3cf-7e79ca1a24d7/dns/0.log"
Oct 30 04:20:19.645530 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:20:19.645498    4461 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-wjll7_5f9fb000-9531-41f1-b3cf-7e79ca1a24d7/kube-rbac-proxy/0.log"
Oct 30 04:20:20.248922 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:20:20.248892    4461 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-zhb7b_0e6f015d-0a2d-4579-ad4d-a88b8c11c3b9/dns-node-resolver/0.log"
Oct 30 04:20:21.483183 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:20:21.483154    4461 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-7fbbd597bb-twtnh_21d9fcc4-833e-4196-a646-c1f0d0c38313/registry/0.log"
Oct 30 04:20:21.629067 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:20:21.629030    4461 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-9mddj_487caf0f-5f2a-4e93-9c06-de9af02c470b/node-ca/0.log"
Oct 30 04:20:22.652178 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:20:22.652143    4461 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-5b4f8cc59b-99z99_a4af4d2f-3158-4eda-933d-faa3882b72d6/istio-proxy/0.log"
Oct 30 04:20:22.907773 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:20:22.907703    4461 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-5b4f8cc59b-kqr6c_bc40c202-21a0-44ec-b556-9deab9ddae95/istio-proxy/0.log"
Oct 30 04:20:22.971856 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:20:22.971830    4461 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-5b4f8cc59b-ljhv4_75acdd43-c913-440c-adeb-473c8d51a0db/istio-proxy/0.log"
Oct 30 04:20:24.135403 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:20:24.135371    4461 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-5b4f8cc59b-99z99_a4af4d2f-3158-4eda-933d-faa3882b72d6/istio-proxy/0.log"
Oct 30 04:20:24.503667 ip-10-0-143-156 kubenswrapper[4461]: E1030 04:20:24.503626    4461 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2k85b" containerName="extractor"
Oct 30 04:20:24.626191 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:20:24.626163    4461 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-5b4f8cc59b-kqr6c_bc40c202-21a0-44ec-b556-9deab9ddae95/istio-proxy/0.log"
Oct 30 04:20:24.749439 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:20:24.749411    4461 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-5b4f8cc59b-ljhv4_75acdd43-c913-440c-adeb-473c8d51a0db/istio-proxy/0.log"
Oct 30 04:20:25.322997 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:20:25.322971    4461 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-5b4f8cc59b-99z99_a4af4d2f-3158-4eda-933d-faa3882b72d6/istio-proxy/0.log"
Oct 30 04:20:25.578276 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:20:25.578201    4461 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-5b4f8cc59b-kqr6c_bc40c202-21a0-44ec-b556-9deab9ddae95/istio-proxy/0.log"
Oct 30 04:20:25.643908 ip-10-0-143-156 kubenswrapper[4461]: I1030 04:20:25.643880    4461 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-5b4f8cc59b-ljhv4_75acdd43-c913-440c-adeb-473c8d51a0db/istio-proxy/0.log"