Oct 31 07:26:26.172903 ip-10-0-139-83 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Oct 31 07:26:26.173053 ip-10-0-139-83 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Oct 31 07:26:26.173143 ip-10-0-139-83 systemd[1]: kubelet.service: Failed with result 'resources'.
Oct 31 07:26:26.173728 ip-10-0-139-83 systemd[1]: Failed to start Kubernetes Kubelet.
Oct 31 07:26:36.352807 ip-10-0-139-83 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found.
Oct 31 07:26:36.352822 ip-10-0-139-83 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot faed6606cdad45f4a07ab407dea0d5e9 --
Oct 31 07:28:23.186586 ip-10-0-139-83 systemd[1]: Starting Kubernetes Kubelet...
Oct 31 07:28:23.606166 ip-10-0-139-83 kubenswrapper[2573]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 31 07:28:23.606166 ip-10-0-139-83 kubenswrapper[2573]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Oct 31 07:28:23.606166 ip-10-0-139-83 kubenswrapper[2573]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 31 07:28:23.606166 ip-10-0-139-83 kubenswrapper[2573]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 31 07:28:23.611216 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.611101    2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 31 07:28:23.611216 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.611216    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 31 07:28:23.611289 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.611222    2573 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 31 07:28:23.611289 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.611227    2573 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 31 07:28:23.611289 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.611231    2573 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 31 07:28:23.611289 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.611234    2573 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 31 07:28:23.611289 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.611237    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 31 07:28:23.611289 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.611240    2573 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 31 07:28:23.611289 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.611242    2573 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 31 07:28:23.611289 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.611245    2573 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 31 07:28:23.611289 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.611247    2573 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 31 07:28:23.611289 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.611250    2573 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 31 07:28:23.611289 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.611253    2573 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 31 07:28:23.611289 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.611255    2573 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 31 07:28:23.611289 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.611258    2573 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 31 07:28:23.611289 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.611267    2573 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 31 07:28:23.611289 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.611270    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 31 07:28:23.611289 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.611272    2573 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 31 07:28:23.611289 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.611275    2573 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 31 07:28:23.611289 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.611277    2573 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 31 07:28:23.611289 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.611280    2573 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 31 07:28:23.611732 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.611282    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 31 07:28:23.611732 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.611285    2573 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 31 07:28:23.611732 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.611288    2573 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 31 07:28:23.611732 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.611290    2573 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 31 07:28:23.611732 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.611293    2573 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 31 07:28:23.611732 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.611296    2573 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 31 07:28:23.611732 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.611299    2573 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 31 07:28:23.611732 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.611301    2573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 31 07:28:23.611732 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.611304    2573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 31 07:28:23.611732 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.611307    2573 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 31 07:28:23.611732 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.611309    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 31 07:28:23.611732 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.611313    2573 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 31 07:28:23.611732 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.611315    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 31 07:28:23.611732 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.611318    2573 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 31 07:28:23.611732 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.611320    2573 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 31 07:28:23.611732 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.611337    2573 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 31 07:28:23.611732 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.611341    2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 31 07:28:23.611732 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.611345    2573 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 31 07:28:23.611732 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.611348    2573 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 31 07:28:23.611732 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.611352    2573 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 31 07:28:23.612227 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.611358    2573 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 31 07:28:23.612227 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.611361    2573 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 31 07:28:23.612227 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.611364    2573 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 31 07:28:23.612227 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.611367    2573 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 31 07:28:23.612227 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.611369    2573 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 31 07:28:23.612227 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.611371    2573 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 31 07:28:23.612227 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.611374    2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 31 07:28:23.612227 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.611377    2573 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 31 07:28:23.612227 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.611380    2573 feature_gate.go:386] unrecognized feature gate: Example2
Oct 31 07:28:23.612227 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.611382    2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 31 07:28:23.612227 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.611385    2573 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 31 07:28:23.612227 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.611388    2573 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 31 07:28:23.612227 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.611391    2573 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 31 07:28:23.612227 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.611394    2573 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 31 07:28:23.612227 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.611396    2573 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 31 07:28:23.612227 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.611399    2573 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 31 07:28:23.612227 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.611402    2573 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 31 07:28:23.612227 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.611405    2573 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 31 07:28:23.612227 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.611407    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 31 07:28:23.612227 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.611410    2573 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 31 07:28:23.612706 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.611413    2573 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 31 07:28:23.612706 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.611415    2573 feature_gate.go:386] unrecognized feature gate: Example
Oct 31 07:28:23.612706 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.611418    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 31 07:28:23.612706 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.611420    2573 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 31 07:28:23.612706 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.611423    2573 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 31 07:28:23.612706 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.611426    2573 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 31 07:28:23.612706 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.611428    2573 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 31 07:28:23.612706 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.611431    2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 31 07:28:23.612706 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.611434    2573 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 31 07:28:23.612706 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.611436    2573 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 31 07:28:23.612706 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.611439    2573 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 31 07:28:23.612706 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.611441    2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 31 07:28:23.612706 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.611443    2573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 31 07:28:23.612706 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.611446    2573 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 31 07:28:23.612706 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.611449    2573 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 31 07:28:23.612706 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.611451    2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 31 07:28:23.612706 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.611454    2573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 31 07:28:23.612706 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.611456    2573 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 31 07:28:23.612706 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.611458    2573 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 31 07:28:23.612706 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.611461    2573 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 31 07:28:23.613161 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.611464    2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 31 07:28:23.613161 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.611466    2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 31 07:28:23.613161 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.611469    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 31 07:28:23.613161 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.611472    2573 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 31 07:28:23.613161 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.611475    2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 31 07:28:23.613161 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.611477    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 31 07:28:23.613161 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.611480    2573 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 31 07:28:23.613161 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.611482    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 31 07:28:23.613161 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.611485    2573 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 31 07:28:23.613161 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.611487    2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 31 07:28:23.613161 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.611490    2573 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 31 07:28:23.613161 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.611492    2573 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 31 07:28:23.613161 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.611495    2573 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 31 07:28:23.613161 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.611497    2573 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 31 07:28:23.613161 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.611500    2573 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 31 07:28:23.613161 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.611502    2573 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 31 07:28:23.613161 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.611505    2573 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 31 07:28:23.613161 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.611509    2573 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 31 07:28:23.613161 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.611511    2573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 31 07:28:23.613602 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.611514    2573 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 31 07:28:23.613602 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.611517    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 31 07:28:23.613602 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.611519    2573 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 31 07:28:23.613602 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.611521    2573 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 31 07:28:23.613602 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.611524    2573 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 31 07:28:23.613602 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.611526    2573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 31 07:28:23.615512 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.615485    2573 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 31 07:28:23.615512 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.615508    2573 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 31 07:28:23.615512 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.615513    2573 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 31 07:28:23.615512 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.615517    2573 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 31 07:28:23.615512 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.615521    2573 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 31 07:28:23.615805 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.615525    2573 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 31 07:28:23.615805 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.615530    2573 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 31 07:28:23.615805 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.615534    2573 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 31 07:28:23.615805 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.615538    2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 31 07:28:23.615805 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.615542    2573 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 31 07:28:23.615805 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.615546    2573 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 31 07:28:23.615805 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.615549    2573 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 31 07:28:23.615805 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.615553    2573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 31 07:28:23.615805 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.615557    2573 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 31 07:28:23.615805 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.615561    2573 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 31 07:28:23.615805 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.615565    2573 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 31 07:28:23.615805 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.615568    2573 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 31 07:28:23.615805 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.615572    2573 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 31 07:28:23.615805 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.615576    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 31 07:28:23.615805 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.615580    2573 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 31 07:28:23.615805 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.615584    2573 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 31 07:28:23.615805 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.615588    2573 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 31 07:28:23.615805 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.615592    2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 31 07:28:23.615805 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.615596    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 31 07:28:23.615805 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.615599    2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 31 07:28:23.616611 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.615603    2573 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 31 07:28:23.616611 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.615607    2573 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 31 07:28:23.616611 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.615610    2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 31 07:28:23.616611 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.615614    2573 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 31 07:28:23.616611 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.615618    2573 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 31 07:28:23.616611 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.615622    2573 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 31 07:28:23.616611 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.615627    2573 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 31 07:28:23.616611 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.615633    2573 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 31 07:28:23.616611 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.615637    2573 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 31 07:28:23.616611 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.615641    2573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 31 07:28:23.616611 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.615645    2573 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 31 07:28:23.616611 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.615649    2573 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 31 07:28:23.616611 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.615654    2573 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 31 07:28:23.616611 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.615662    2573 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 31 07:28:23.616611 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.615667    2573 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 31 07:28:23.616611 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.615671    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 31 07:28:23.616611 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.615675    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 31 07:28:23.616611 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.615681    2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 31 07:28:23.617515 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.615685    2573 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 31 07:28:23.617515 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.615689    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 31 07:28:23.617515 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.615693    2573 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 31 07:28:23.617515 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.615697    2573 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 31 07:28:23.617515 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.615702    2573 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 31 07:28:23.617515 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.615706    2573 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 31 07:28:23.617515 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.615710    2573 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 31 07:28:23.617515 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.615714    2573 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 31 07:28:23.617515 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.615718    2573 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 31 07:28:23.617515 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.615722    2573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 31 07:28:23.617515 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.615726    2573 feature_gate.go:386] unrecognized feature gate: Example2
Oct 31 07:28:23.617515 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.615730    2573 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 31 07:28:23.617515 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.615734    2573 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 31 07:28:23.617515 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.615738    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 31 07:28:23.617515 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.615743    2573 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 31 07:28:23.617515 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.615747    2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 31 07:28:23.617515 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.615751    2573 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 31 07:28:23.617515 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.615755    2573 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 31 07:28:23.617515 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.615759    2573 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 31 07:28:23.617515 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.615763    2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 31 07:28:23.618016 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.615767    2573 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 31 07:28:23.618016 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.615771    2573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 31 07:28:23.618016 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.615775    2573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 31 07:28:23.618016 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.615779    2573 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 31 07:28:23.618016 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.615783    2573 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 31 07:28:23.618016 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.615787    2573 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 31 07:28:23.618016 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.615794    2573 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 31 07:28:23.618016 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.615800    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 31 07:28:23.618016 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.615807    2573 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 31 07:28:23.618016 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.615813    2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 31 07:28:23.618016 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.615818    2573 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 31 07:28:23.618016 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.615822    2573 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 31 07:28:23.618016 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.615826    2573 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 31 07:28:23.618016 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.615830    2573 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 31 07:28:23.618016 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.615834    2573 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 31 07:28:23.618016 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.615839    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 31 07:28:23.618016 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.615843    2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 31 07:28:23.618016 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.615847    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 31 07:28:23.618016 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.615851    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 31 07:28:23.618517 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.615856    2573 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 31 07:28:23.618517 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.615862    2573 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 31 07:28:23.618517 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.615866    2573 feature_gate.go:386] unrecognized feature gate: Example
Oct 31 07:28:23.618517 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.615870    2573 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 31 07:28:23.618517 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.615874    2573 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 31 07:28:23.618517 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.615878    2573 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 31 07:28:23.618517 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.615883    2573 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 31 07:28:23.618517 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.615887    2573 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 31 07:28:23.618517 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.615891    2573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 31 07:28:23.618517 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.615895    2573 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 31 07:28:23.618517 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.615899    2573 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 31 07:28:23.618517 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.615903    2573 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 31 07:28:23.618517 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.615907    2573 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 31 07:28:23.618517 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.615911    2573 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 31 07:28:23.618517 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.615916    2573 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 31 07:28:23.618517 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.615921    2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 31 07:28:23.618517 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.615928    2573 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 31 07:28:23.618517 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.615932    2573 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 31 07:28:23.618517 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.615936    2573 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 31 07:28:23.618517 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.615940    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 31 07:28:23.619065 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.615945    2573 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 31 07:28:23.619065 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.615949    2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 31 07:28:23.619065 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.615953    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 31 07:28:23.619065 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.615958    2573 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 31 07:28:23.619065 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.616862    2573 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 31 07:28:23.619065 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.616873    2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 31 07:28:23.619065 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.616879    2573 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 31 07:28:23.619065 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.616885    2573 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 31 07:28:23.619065 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.616890    2573 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 31 07:28:23.619065 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.616897    2573 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 31 07:28:23.619065 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.616903    2573 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 31 07:28:23.619065 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.616908    2573 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 31 07:28:23.619065 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.616913    2573 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 31 07:28:23.619065 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.616920    2573 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 31 07:28:23.619065 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.616925    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 31 07:28:23.619065 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.616930    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 31 07:28:23.619065 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.616934    2573 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 31 07:28:23.619065 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.616938    2573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 31 07:28:23.619065 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.616942    2573 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 31 07:28:23.619860 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.616946    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 31 07:28:23.619860 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.616950    2573 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 31 07:28:23.619860 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.616955    2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 31 07:28:23.619860 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.616959    2573 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 31 07:28:23.619860 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.616963    2573 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 31 07:28:23.619860 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.616968    2573 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 31 07:28:23.619860 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.616972    2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 31 07:28:23.619860 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.616976    2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 31 07:28:23.619860 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.616980    2573 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 31 07:28:23.619860 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.616984    2573 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 31 07:28:23.619860 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.616989    2573 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 31 07:28:23.619860 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.616993    2573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 31 07:28:23.619860 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.616997    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 31 07:28:23.619860 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.617002    2573 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 31 07:28:23.619860 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.617006    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 31 07:28:23.619860 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.617011    2573 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 31 07:28:23.619860 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.617015    2573 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 31 07:28:23.619860 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.617019    2573 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 31 07:28:23.619860 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.617023    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 31 07:28:23.620663 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.617027    2573 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 31 07:28:23.620663 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.617031    2573 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 31 07:28:23.620663 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.617035    2573 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 31 07:28:23.620663 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.617039    2573 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 31 07:28:23.620663 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.617043    2573 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 31 07:28:23.620663 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.617047    2573 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 31 07:28:23.620663 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.617051    2573 feature_gate.go:386] unrecognized feature gate: Example2
Oct 31 07:28:23.620663 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.617055    2573 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 31 07:28:23.620663 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.617060    2573 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 31 07:28:23.620663 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.617064    2573 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 31 07:28:23.620663 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.617068    2573 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 31 07:28:23.620663 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.617072    2573 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 31 07:28:23.620663 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.617076    2573 feature_gate.go:386] unrecognized feature gate: Example
Oct 31 07:28:23.620663 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.617080    2573 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 31 07:28:23.620663 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.617085    2573 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 31 07:28:23.620663 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.617089    2573 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 31 07:28:23.620663 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.617093    2573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 31 07:28:23.620663 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.617097    2573 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 31 07:28:23.620663 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.617101    2573 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 31 07:28:23.620663 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.617105    2573 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 31 07:28:23.621476 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.617109    2573 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 31 07:28:23.621476 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.617113    2573 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 31 07:28:23.621476 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.617117    2573 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 31 07:28:23.621476 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.617121    2573 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 31 07:28:23.621476 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.617127    2573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 31 07:28:23.621476 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.617131    2573 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 31 07:28:23.621476 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.617135    2573 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 31 07:28:23.621476 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.617139    2573 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 31 07:28:23.621476 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.617146    2573 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 31 07:28:23.621476 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.617152    2573 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 31 07:28:23.621476 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.617157    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 31 07:28:23.621476 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.617161    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 31 07:28:23.621476 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.617165    2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 31 07:28:23.621476 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.617169    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 31 07:28:23.621476 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.617173    2573 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 31 07:28:23.621476 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.617177    2573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 31 07:28:23.621476 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.617182    2573 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 31 07:28:23.621476 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.617185    2573 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 31 07:28:23.621476 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.617190    2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 31 07:28:23.621476 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.617193    2573 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 31 07:28:23.621959 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.617197    2573 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 31 07:28:23.621959 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.617202    2573 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 31 07:28:23.621959 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.617207    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 31 07:28:23.621959 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.617211    2573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 31 07:28:23.621959 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.617215    2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 31 07:28:23.621959 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.617219    2573 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 31 07:28:23.621959 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.617223    2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 31 07:28:23.621959 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.617227    2573 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 31 07:28:23.621959 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.617232    2573 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 31 07:28:23.621959 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.617235    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 31 07:28:23.621959 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.617239    2573 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 31 07:28:23.621959 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.617243    2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 31 07:28:23.621959 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.617247    2573 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 31 07:28:23.621959 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.617251    2573 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 31 07:28:23.621959 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.617255    2573 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 31 07:28:23.621959 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.617259    2573 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 31 07:28:23.621959 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.617263    2573 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 31 07:28:23.621959 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.617267    2573 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 31 07:28:23.621959 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.617271    2573 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 31 07:28:23.621959 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.617275    2573 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 31 07:28:23.622641 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.617281    2573 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 31 07:28:23.622641 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.617285    2573 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 31 07:28:23.622641 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.617289    2573 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 31 07:28:23.622641 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.617293    2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 31 07:28:23.622641 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.617297    2573 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 31 07:28:23.622641 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.617301    2573 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 31 07:28:23.622641 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.617305    2573 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 31 07:28:23.622641 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.617309    2573 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 31 07:28:23.622641 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.617313    2573 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 31 07:28:23.622641 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.617317    2573 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 31 07:28:23.622641 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.617321    2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 31 07:28:23.622641 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.617344    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 31 07:28:23.622641 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:23.618968    2573 flags.go:64] FLAG: --address="0.0.0.0"
Oct 31 07:28:23.622641 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:23.618986    2573 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Oct 31 07:28:23.622641 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:23.618999    2573 flags.go:64] FLAG: --anonymous-auth="true"
Oct 31 07:28:23.622641 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:23.619006    2573 flags.go:64] FLAG: --application-metrics-count-limit="100"
Oct 31 07:28:23.622641 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:23.619013    2573 flags.go:64] FLAG: --authentication-token-webhook="false"
Oct 31 07:28:23.622641 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:23.619018    2573 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Oct 31 07:28:23.622641 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:23.619024    2573 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Oct 31 07:28:23.622641 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:23.619031    2573 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Oct 31 07:28:23.622641 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:23.619037    2573 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Oct 31 07:28:23.623259 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:23.619042    2573 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Oct 31 07:28:23.623259 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:23.619047    2573 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Oct 31 07:28:23.623259 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:23.619052    2573 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Oct 31 07:28:23.623259 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:23.619057    2573 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Oct 31 07:28:23.623259 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:23.619062    2573 flags.go:64] FLAG: --cgroup-root=""
Oct 31 07:28:23.623259 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:23.619066    2573 flags.go:64] FLAG: --cgroups-per-qos="true"
Oct 31 07:28:23.623259 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:23.619071    2573 flags.go:64] FLAG: --client-ca-file=""
Oct 31 07:28:23.623259 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:23.619075    2573 flags.go:64] FLAG: --cloud-provider="external"
Oct 31 07:28:23.623259 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:23.619080    2573 flags.go:64] FLAG: --cluster-dns="[]"
Oct 31 07:28:23.623259 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:23.619088    2573 flags.go:64] FLAG: --cluster-domain=""
Oct 31 07:28:23.623259 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:23.619092    2573 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Oct 31 07:28:23.623259 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:23.619097    2573 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Oct 31 07:28:23.623259 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:23.619102    2573 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Oct 31 07:28:23.623259 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:23.619107    2573 flags.go:64] FLAG: --container-log-max-files="5"
Oct 31 07:28:23.623259 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:23.619120    2573 flags.go:64] FLAG: --container-log-max-size="10Mi"
Oct 31 07:28:23.623259 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:23.619126    2573 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Oct 31 07:28:23.623259 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:23.619131    2573 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Oct 31 07:28:23.623259 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:23.619136    2573 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Oct 31 07:28:23.623259 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:23.619140    2573 flags.go:64] FLAG: --contention-profiling="false"
Oct 31 07:28:23.623259 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:23.619146    2573 flags.go:64] FLAG: --cpu-cfs-quota="true"
Oct 31 07:28:23.623259 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:23.619150    2573 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Oct 31 07:28:23.623259 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:23.619155    2573 flags.go:64] FLAG: --cpu-manager-policy="none"
Oct 31 07:28:23.623259 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:23.619160    2573 flags.go:64] FLAG: --cpu-manager-policy-options=""
Oct 31 07:28:23.623259 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:23.619166    2573 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Oct 31 07:28:23.623865 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:23.619172    2573 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Oct 31 07:28:23.623865 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:23.619176    2573 flags.go:64] FLAG: --enable-debugging-handlers="true"
Oct 31 07:28:23.623865 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:23.619181    2573 flags.go:64] FLAG: --enable-load-reader="false"
Oct 31 07:28:23.623865 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:23.619186    2573 flags.go:64] FLAG: --enable-server="true"
Oct 31 07:28:23.623865 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:23.619191    2573 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Oct 31 07:28:23.623865 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:23.619197    2573 flags.go:64] FLAG: --event-burst="100"
Oct 31 07:28:23.623865 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:23.619202    2573 flags.go:64] FLAG: --event-qps="50"
Oct 31 07:28:23.623865 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:23.619206    2573 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Oct 31 07:28:23.623865 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:23.619211    2573 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Oct 31 07:28:23.623865 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:23.619216    2573 flags.go:64] FLAG: --eviction-hard=""
Oct 31 07:28:23.623865 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:23.619223    2573 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Oct 31 07:28:23.623865 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:23.619228    2573 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Oct 31 07:28:23.623865 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:23.619233    2573 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Oct 31 07:28:23.623865 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:23.619238    2573 flags.go:64] FLAG: --eviction-soft=""
Oct 31 07:28:23.623865 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:23.619243    2573 flags.go:64] FLAG: --eviction-soft-grace-period=""
Oct 31 07:28:23.623865 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:23.619247    2573 flags.go:64] FLAG: --exit-on-lock-contention="false"
Oct 31 07:28:23.623865 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:23.619252    2573 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Oct 31 07:28:23.623865 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:23.619257    2573 flags.go:64] FLAG: --experimental-mounter-path=""
Oct 31 07:28:23.623865 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:23.619261    2573 flags.go:64] FLAG: --fail-cgroupv1="true"
Oct 31 07:28:23.623865 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:23.619266    2573 flags.go:64] FLAG: --fail-swap-on="true"
Oct 31 07:28:23.623865 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:23.619270    2573 flags.go:64] FLAG: --feature-gates=""
Oct 31 07:28:23.623865 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:23.619282    2573 flags.go:64] FLAG: --file-check-frequency="20s"
Oct 31 07:28:23.623865 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:23.619287    2573 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Oct 31 07:28:23.623865 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:23.619292    2573 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Oct 31 07:28:23.623865 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:23.619297    2573 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Oct 31 07:28:23.624545 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:23.619301    2573 flags.go:64] FLAG: --healthz-port="10248"
Oct 31 07:28:23.624545 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:23.619306    2573 flags.go:64] FLAG: --help="false"
Oct 31 07:28:23.624545 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:23.619311    2573 flags.go:64] FLAG: --hostname-override="ip-10-0-139-83.us-west-1.compute.internal"
Oct 31 07:28:23.624545 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:23.619316    2573 flags.go:64] FLAG: --housekeeping-interval="10s"
Oct 31 07:28:23.624545 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:23.619320    2573 flags.go:64] FLAG: --http-check-frequency="20s"
Oct 31 07:28:23.624545 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:23.619343    2573 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Oct 31 07:28:23.624545 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:23.619349    2573 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Oct 31 07:28:23.624545 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:23.619355    2573 flags.go:64] FLAG: --image-gc-high-threshold="85"
Oct 31 07:28:23.624545 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:23.619359    2573 flags.go:64] FLAG: --image-gc-low-threshold="80"
Oct 31 07:28:23.624545 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:23.619364    2573 flags.go:64] FLAG: --image-service-endpoint=""
Oct 31 07:28:23.624545 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:23.619369    2573 flags.go:64] FLAG: --kernel-memcg-notification="false"
Oct 31 07:28:23.624545 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:23.619374    2573 flags.go:64] FLAG: --kube-api-burst="100"
Oct 31 07:28:23.624545 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:23.619379    2573 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Oct 31 07:28:23.624545 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:23.619383    2573 flags.go:64] FLAG: --kube-api-qps="50"
Oct 31 07:28:23.624545 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:23.619387    2573 flags.go:64] FLAG: --kube-reserved=""
Oct 31 07:28:23.624545 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:23.619392    2573 flags.go:64] FLAG: --kube-reserved-cgroup=""
Oct 31 07:28:23.624545 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:23.619397    2573 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Oct 31 07:28:23.624545 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:23.619401    2573 flags.go:64] FLAG: --kubelet-cgroups=""
Oct 31 07:28:23.624545 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:23.619406    2573 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Oct 31 07:28:23.624545 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:23.619410    2573 flags.go:64] FLAG: --lock-file=""
Oct 31 07:28:23.624545 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:23.619414    2573 flags.go:64] FLAG: --log-cadvisor-usage="false"
Oct 31 07:28:23.624545 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:23.619420    2573 flags.go:64] FLAG: --log-flush-frequency="5s"
Oct 31 07:28:23.624545 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:23.619424    2573 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Oct 31 07:28:23.624545 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:23.619435    2573 flags.go:64] FLAG: --log-json-split-stream="false"
Oct 31 07:28:23.625105 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:23.619439    2573 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Oct 31 07:28:23.625105 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:23.619444    2573 flags.go:64] FLAG: --log-text-split-stream="false"
Oct 31 07:28:23.625105 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:23.619449    2573 flags.go:64] FLAG: --logging-format="text"
Oct 31 07:28:23.625105 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:23.619453    2573 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Oct 31 07:28:23.625105 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:23.619458    2573 flags.go:64] FLAG: --make-iptables-util-chains="true"
Oct 31 07:28:23.625105 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:23.619463    2573 flags.go:64] FLAG: --manifest-url=""
Oct 31 07:28:23.625105 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:23.619467    2573 flags.go:64] FLAG: --manifest-url-header=""
Oct 31 07:28:23.625105 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:23.619473    2573 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Oct 31 07:28:23.625105 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:23.619478    2573 flags.go:64] FLAG: --max-open-files="1000000"
Oct 31 07:28:23.625105 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:23.619484    2573 flags.go:64] FLAG: --max-pods="110"
Oct 31 07:28:23.625105 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:23.619489    2573 flags.go:64] FLAG: --maximum-dead-containers="-1"
Oct 31 07:28:23.625105 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:23.619494    2573 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Oct 31 07:28:23.625105 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:23.619499    2573 flags.go:64] FLAG: --memory-manager-policy="None"
Oct 31 07:28:23.625105 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:23.619504    2573 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Oct 31 07:28:23.625105 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:23.619508    2573 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Oct 31 07:28:23.625105 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:23.619513    2573 flags.go:64] FLAG: --node-ip="0.0.0.0"
Oct 31 07:28:23.625105 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:23.619518    2573 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Oct 31 07:28:23.625105 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:23.619531    2573 flags.go:64] FLAG: --node-status-max-images="50"
Oct 31 07:28:23.625105 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:23.619536    2573 flags.go:64] FLAG: --node-status-update-frequency="10s"
Oct 31 07:28:23.625105 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:23.619541    2573 flags.go:64] FLAG: --oom-score-adj="-999"
Oct 31 07:28:23.625105 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:23.619546    2573 flags.go:64] FLAG: --pod-cidr=""
Oct 31 07:28:23.625105 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:23.619551    2573 flags.go:64] FLAG: --pod-manifest-path=""
Oct 31 07:28:23.625105 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:23.619555    2573 flags.go:64] FLAG: --pod-max-pids="-1"
Oct 31 07:28:23.625105 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:23.619560    2573 flags.go:64] FLAG: --pods-per-core="0"
Oct 31 07:28:23.625105 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:23.619564    2573 flags.go:64] FLAG: --port="10250"
Oct 31 07:28:23.625704 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:23.619569    2573 flags.go:64] FLAG: --protect-kernel-defaults="false"
Oct 31 07:28:23.625704 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:23.619574    2573 flags.go:64] FLAG: --provider-id="aws:///us-west-1a/i-0c45e2b77a29ed30d"
Oct 31 07:28:23.625704 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:23.619579    2573 flags.go:64] FLAG: --qos-reserved=""
Oct 31 07:28:23.625704 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:23.619584    2573 flags.go:64] FLAG: --read-only-port="10255"
Oct 31 07:28:23.625704 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:23.619588    2573 flags.go:64] FLAG: --register-node="true"
Oct 31 07:28:23.625704 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:23.619593    2573 flags.go:64] FLAG: --register-with-taints=""
Oct 31 07:28:23.625704 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:23.619599    2573 flags.go:64] FLAG: --registry-burst="10"
Oct 31 07:28:23.625704 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:23.619604    2573 flags.go:64] FLAG: --registry-qps="5"
Oct 31 07:28:23.625704 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:23.619608    2573 flags.go:64] FLAG: --reserved-cpus=""
Oct 31 07:28:23.625704 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:23.619612    2573 flags.go:64] FLAG: --reserved-memory=""
Oct 31 07:28:23.625704 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:23.619618    2573 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Oct 31 07:28:23.625704 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:23.619623    2573 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Oct 31 07:28:23.625704 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:23.619628    2573 flags.go:64] FLAG: --rotate-certificates="false"
Oct 31 07:28:23.625704 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:23.619632    2573 flags.go:64] FLAG: --rotate-server-certificates="false"
Oct 31 07:28:23.625704 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:23.619637    2573 flags.go:64] FLAG: --runonce="false"
Oct 31 07:28:23.625704 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:23.619641    2573 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Oct 31 07:28:23.625704 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:23.619646    2573 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Oct 31 07:28:23.625704 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:23.619651    2573 flags.go:64] FLAG: --seccomp-default="false"
Oct 31 07:28:23.625704 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:23.619655    2573 flags.go:64] FLAG: --serialize-image-pulls="true"
Oct 31 07:28:23.625704 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:23.619660    2573 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Oct 31 07:28:23.625704 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:23.619664    2573 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Oct 31 07:28:23.625704 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:23.619669    2573 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Oct 31 07:28:23.625704 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:23.619674    2573 flags.go:64] FLAG: --storage-driver-password="root"
Oct 31 07:28:23.625704 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:23.619679    2573 flags.go:64] FLAG: --storage-driver-secure="false"
Oct 31 07:28:23.625704 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:23.619684    2573 flags.go:64] FLAG: --storage-driver-table="stats"
Oct 31 07:28:23.625704 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:23.619688    2573 flags.go:64] FLAG: --storage-driver-user="root"
Oct 31 07:28:23.626348 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:23.619693    2573 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Oct 31 07:28:23.626348 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:23.619700    2573 flags.go:64] FLAG: --sync-frequency="1m0s"
Oct 31 07:28:23.626348 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:23.619704    2573 flags.go:64] FLAG: --system-cgroups=""
Oct 31 07:28:23.626348 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:23.619708    2573 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Oct 31 07:28:23.626348 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:23.619717    2573 flags.go:64] FLAG: --system-reserved-cgroup=""
Oct 31 07:28:23.626348 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:23.619721    2573 flags.go:64] FLAG: --tls-cert-file=""
Oct 31 07:28:23.626348 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:23.619726    2573 flags.go:64] FLAG: --tls-cipher-suites="[]"
Oct 31 07:28:23.626348 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:23.619733    2573 flags.go:64] FLAG: --tls-min-version=""
Oct 31 07:28:23.626348 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:23.619738    2573 flags.go:64] FLAG: --tls-private-key-file=""
Oct 31 07:28:23.626348 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:23.619744    2573 flags.go:64] FLAG: --topology-manager-policy="none"
Oct 31 07:28:23.626348 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:23.619749    2573 flags.go:64] FLAG: --topology-manager-policy-options=""
Oct 31 07:28:23.626348 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:23.619754    2573 flags.go:64] FLAG: --topology-manager-scope="container"
Oct 31 07:28:23.626348 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:23.619758    2573 flags.go:64] FLAG: --v="2"
Oct 31 07:28:23.626348 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:23.619765    2573 flags.go:64] FLAG: --version="false"
Oct 31 07:28:23.626348 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:23.619772    2573 flags.go:64] FLAG: --vmodule=""
Oct 31 07:28:23.626348 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:23.619778    2573 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Oct 31 07:28:23.626348 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:23.619783    2573 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Oct 31 07:28:23.626348 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.620004    2573 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 31 07:28:23.626348 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.620011    2573 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 31 07:28:23.626348 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.620015    2573 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 31 07:28:23.626348 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.620020    2573 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 31 07:28:23.626348 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.620024    2573 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 31 07:28:23.626348 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.620029    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 31 07:28:23.626348 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.620033    2573 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 31 07:28:23.626986 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.620037    2573 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 31 07:28:23.626986 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.620043    2573 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 31 07:28:23.626986 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.620048    2573 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 31 07:28:23.626986 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.620052    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 31 07:28:23.626986 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.620056    2573 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 31 07:28:23.626986 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.620061    2573 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 31 07:28:23.626986 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.620065    2573 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 31 07:28:23.626986 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.620069    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 31 07:28:23.626986 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.620073    2573 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 31 07:28:23.626986 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.620077    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 31 07:28:23.626986 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.620082    2573 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 31 07:28:23.626986 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.620086    2573 feature_gate.go:386] unrecognized feature gate: Example2
Oct 31 07:28:23.626986 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.620090    2573 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 31 07:28:23.626986 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.620094    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 31 07:28:23.626986 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.620098    2573 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 31 07:28:23.626986 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.620102    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 31 07:28:23.626986 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.620106    2573 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 31 07:28:23.626986 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.620112    2573 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 31 07:28:23.626986 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.620116    2573 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 31 07:28:23.627540 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.620120    2573 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 31 07:28:23.627540 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.620124    2573 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 31 07:28:23.627540 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.620128    2573 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 31 07:28:23.627540 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.620132    2573 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 31 07:28:23.627540 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.620137    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 31 07:28:23.627540 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.620141    2573 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 31 07:28:23.627540 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.620145    2573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 31 07:28:23.627540 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.620149    2573 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 31 07:28:23.627540 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.620153    2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 31 07:28:23.627540 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.620157    2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 31 07:28:23.627540 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.620161    2573 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 31 07:28:23.627540 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.620165    2573 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 31 07:28:23.627540 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.620169    2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 31 07:28:23.627540 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.620173    2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 31 07:28:23.627540 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.620179    2573 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 31 07:28:23.627540 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.620183    2573 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 31 07:28:23.627540 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.620187    2573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 31 07:28:23.627540 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.620191    2573 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 31 07:28:23.627540 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.620195    2573 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 31 07:28:23.627540 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.620200    2573 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 31 07:28:23.627999 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.620204    2573 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 31 07:28:23.627999 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.620208    2573 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 31 07:28:23.627999 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.620213    2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 31 07:28:23.627999 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.620217    2573 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 31 07:28:23.627999 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.620225    2573 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 31 07:28:23.627999 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.620231    2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 31 07:28:23.627999 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.620236    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 31 07:28:23.627999 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.620240    2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 31 07:28:23.627999 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.620244    2573 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 31 07:28:23.627999 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.620249    2573 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 31 07:28:23.627999 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.620256    2573 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 31 07:28:23.627999 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.620260    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 31 07:28:23.627999 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.620264    2573 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 31 07:28:23.627999 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.620268    2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 31 07:28:23.627999 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.620272    2573 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 31 07:28:23.627999 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.620276    2573 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 31 07:28:23.627999 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.620282    2573 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 31 07:28:23.627999 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.620288    2573 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 31 07:28:23.628434 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.620293    2573 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 31 07:28:23.628434 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.620298    2573 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 31 07:28:23.628434 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.620302    2573 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 31 07:28:23.628434 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.620306    2573 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 31 07:28:23.628434 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.620310    2573 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 31 07:28:23.628434 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.620315    2573 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 31 07:28:23.628434 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.620319    2573 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 31 07:28:23.628434 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.620339    2573 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 31 07:28:23.628434 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.620346    2573 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 31 07:28:23.628434 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.620350    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 31 07:28:23.628434 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.620354    2573 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 31 07:28:23.628434 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.620358    2573 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 31 07:28:23.628434 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.620362    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 31 07:28:23.628434 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.620366    2573 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 31 07:28:23.628434 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.620370    2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 31 07:28:23.628434 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.620374    2573 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 31 07:28:23.628434 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.620378    2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 31 07:28:23.628434 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.620382    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 31 07:28:23.628434 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.620386    2573 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 31 07:28:23.628434 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.620392    2573 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 31 07:28:23.628887 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.620395    2573 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 31 07:28:23.628887 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.620400    2573 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 31 07:28:23.628887 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.620404    2573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 31 07:28:23.628887 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.620407    2573 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 31 07:28:23.628887 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.620413    2573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 31 07:28:23.628887 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.620417    2573 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 31 07:28:23.628887 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.620421    2573 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 31 07:28:23.628887 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.620425    2573 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 31 07:28:23.628887 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.620429    2573 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 31 07:28:23.628887 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.620433    2573 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 31 07:28:23.628887 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.620438    2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 31 07:28:23.628887 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.620442    2573 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 31 07:28:23.628887 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.620446    2573 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 31 07:28:23.628887 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.620450    2573 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 31 07:28:23.628887 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.620455    2573 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 31 07:28:23.628887 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.620459    2573 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 31 07:28:23.628887 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.620463    2573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 31 07:28:23.628887 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.620467    2573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 31 07:28:23.628887 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.620471    2573 feature_gate.go:386] unrecognized feature gate: Example
Oct 31 07:28:23.629319 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.620476    2573 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 31 07:28:23.629319 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.620482    2573 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 31 07:28:23.629319 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.620486    2573 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 31 07:28:23.629319 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:23.620574    2573 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 31 07:28:23.629319 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.620645    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 31 07:28:23.629319 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.620653    2573 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 31 07:28:23.629319 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.620658    2573 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 31 07:28:23.629319 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.620662    2573 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 31 07:28:23.629319 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.620670    2573 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 31 07:28:23.629319 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.620674    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 31 07:28:23.629319 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.620678    2573 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 31 07:28:23.629319 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.620682    2573 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 31 07:28:23.629319 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.620686    2573 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 31 07:28:23.629319 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.620691    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 31 07:28:23.629319 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.620696    2573 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 31 07:28:23.629319 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.620700    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 31 07:28:23.629702 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.620704    2573 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 31 07:28:23.629702 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.620710    2573 feature_gate.go:386] unrecognized feature gate: Example2
Oct 31 07:28:23.629702 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.620714    2573 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 31 07:28:23.629702 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.620718    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 31 07:28:23.629702 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.620722    2573 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 31 07:28:23.629702 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.620727    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 31 07:28:23.629702 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.620730    2573 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 31 07:28:23.629702 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.620734    2573 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 31 07:28:23.629702 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.620738    2573 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 31 07:28:23.629702 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.620743    2573 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 31 07:28:23.629702 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.620747    2573 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 31 07:28:23.629702 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.620751    2573 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 31 07:28:23.629702 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.620755    2573 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 31 07:28:23.629702 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.620759    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 31 07:28:23.629702 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.620763    2573 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 31 07:28:23.629702 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.620767    2573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 31 07:28:23.629702 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.620771    2573 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 31 07:28:23.629702 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.620776    2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 31 07:28:23.629702 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.620781    2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 31 07:28:23.630161 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.620785    2573 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 31 07:28:23.630161 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.620790    2573 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 31 07:28:23.630161 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.620794    2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 31 07:28:23.630161 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.620798    2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 31 07:28:23.630161 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.620802    2573 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 31 07:28:23.630161 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.620806    2573 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 31 07:28:23.630161 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.620810    2573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 31 07:28:23.630161 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.620815    2573 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 31 07:28:23.630161 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.620819    2573 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 31 07:28:23.630161 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.620823    2573 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 31 07:28:23.630161 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.620828    2573 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 31 07:28:23.630161 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.620833    2573 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 31 07:28:23.630161 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.620837    2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 31 07:28:23.630161 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.620841    2573 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 31 07:28:23.630161 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.620849    2573 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 31 07:28:23.630161 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.620856    2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 31 07:28:23.630161 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.620863    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 31 07:28:23.630161 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.620867    2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 31 07:28:23.630161 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.620872    2573 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 31 07:28:23.630618 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.620876    2573 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 31 07:28:23.630618 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.620880    2573 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 31 07:28:23.630618 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.620884    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 31 07:28:23.630618 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.620889    2573 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 31 07:28:23.630618 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.620893    2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 31 07:28:23.630618 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.620897    2573 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 31 07:28:23.630618 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.620901    2573 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 31 07:28:23.630618 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.620907    2573 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 31 07:28:23.630618 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.620914    2573 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 31 07:28:23.630618 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.620919    2573 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 31 07:28:23.630618 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.620923    2573 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 31 07:28:23.630618 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.620928    2573 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 31 07:28:23.630618 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.620934    2573 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 31 07:28:23.630618 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.620938    2573 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 31 07:28:23.630618 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.620942    2573 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 31 07:28:23.630618 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.620946    2573 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 31 07:28:23.630618 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.620951    2573 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 31 07:28:23.630618 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.620955    2573 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 31 07:28:23.630618 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.620958    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 31 07:28:23.631068 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.620962    2573 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 31 07:28:23.631068 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.620967    2573 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 31 07:28:23.631068 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.620971    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 31 07:28:23.631068 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.620975    2573 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 31 07:28:23.631068 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.620979    2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 31 07:28:23.631068 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.620983    2573 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 31 07:28:23.631068 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.620989    2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 31 07:28:23.631068 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.620993    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 31 07:28:23.631068 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.620998    2573 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 31 07:28:23.631068 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.621002    2573 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 31 07:28:23.631068 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.621006    2573 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 31 07:28:23.631068 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.621010    2573 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 31 07:28:23.631068 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.621014    2573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 31 07:28:23.631068 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.621018    2573 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 31 07:28:23.631068 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.621022    2573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 31 07:28:23.631068 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.621026    2573 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 31 07:28:23.631068 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.621031    2573 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 31 07:28:23.631068 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.621035    2573 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 31 07:28:23.631068 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.621039    2573 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 31 07:28:23.631068 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.621042    2573 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 31 07:28:23.631535 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.621046    2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 31 07:28:23.631535 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.621051    2573 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 31 07:28:23.631535 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.621055    2573 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 31 07:28:23.631535 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.621059    2573 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 31 07:28:23.631535 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.621064    2573 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 31 07:28:23.631535 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.621069    2573 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 31 07:28:23.631535 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.621074    2573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 31 07:28:23.631535 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.621078    2573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 31 07:28:23.631535 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.621082    2573 feature_gate.go:386] unrecognized feature gate: Example
Oct 31 07:28:23.631535 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.621087    2573 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 31 07:28:23.631535 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.621091    2573 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 31 07:28:23.631535 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.621095    2573 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 31 07:28:23.631535 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.621099    2573 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 31 07:28:23.631535 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.621104    2573 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 31 07:28:23.631535 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.621108    2573 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 31 07:28:23.631535 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.621112    2573 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 31 07:28:23.631535 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.621116    2573 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 31 07:28:23.631535 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:23.628940    2573 server.go:525] "Kubelet version" kubeletVersion="v1.35.9"
Oct 31 07:28:23.631535 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:23.628956    2573 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Oct 31 07:28:23.631535 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:23.628974    2573 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Oct 31 07:28:23.632008 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:23.628978    2573 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Oct 31 07:28:23.632008 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.629036    2573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 31 07:28:23.632008 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.629042    2573 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 31 07:28:23.632008 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.629045    2573 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 31 07:28:23.632008 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.629048    2573 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 31 07:28:23.632008 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.629050    2573 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 31 07:28:23.632008 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.629054    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 31 07:28:23.632008 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.629057    2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 31 07:28:23.632008 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.629059    2573 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 31 07:28:23.632008 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.629062    2573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 31 07:28:23.632008 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.629064    2573 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 31 07:28:23.632008 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.629067    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 31 07:28:23.632008 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.629069    2573 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 31 07:28:23.632008 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.629072    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 31 07:28:23.632008 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.629075    2573 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 31 07:28:23.632008 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.629077    2573 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 31 07:28:23.632008 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.629080    2573 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 31 07:28:23.632008 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.629082    2573 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 31 07:28:23.632008 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.629085    2573 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 31 07:28:23.632450 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.629087    2573 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 31 07:28:23.632450 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.629090    2573 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 31 07:28:23.632450 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.629093    2573 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 31 07:28:23.632450 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.629096    2573 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 31 07:28:23.632450 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.629098    2573 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 31 07:28:23.632450 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.629101    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 31 07:28:23.632450 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.629104    2573 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 31 07:28:23.632450 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.629107    2573 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 31 07:28:23.632450 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.629109    2573 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 31 07:28:23.632450 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.629112    2573 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 31 07:28:23.632450 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.629114    2573 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 31 07:28:23.632450 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.629118    2573 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 31 07:28:23.632450 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.629120    2573 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 31 07:28:23.632450 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.629123    2573 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 31 07:28:23.632450 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.629126    2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 31 07:28:23.632450 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.629128    2573 feature_gate.go:386] unrecognized feature gate: Example2
Oct 31 07:28:23.632450 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.629131    2573 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 31 07:28:23.632450 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.629134    2573 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 31 07:28:23.632450 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.629136    2573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 31 07:28:23.632450 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.629139    2573 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 31 07:28:23.632895 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.629143    2573 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 31 07:28:23.632895 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.629146    2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 31 07:28:23.632895 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.629149    2573 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 31 07:28:23.632895 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.629152    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 31 07:28:23.632895 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.629154    2573 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 31 07:28:23.632895 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.629157    2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 31 07:28:23.632895 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.629160    2573 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 31 07:28:23.632895 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.629163    2573 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 31 07:28:23.632895 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.629165    2573 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 31 07:28:23.632895 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.629168    2573 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 31 07:28:23.632895 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.629171    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 31 07:28:23.632895 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.629173    2573 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 31 07:28:23.632895 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.629176    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 31 07:28:23.632895 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.629178    2573 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 31 07:28:23.632895 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.629181    2573 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 31 07:28:23.632895 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.629184    2573 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 31 07:28:23.632895 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.629186    2573 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 31 07:28:23.632895 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.629189    2573 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 31 07:28:23.632895 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.629192    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 31 07:28:23.633322 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.629194    2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 31 07:28:23.633322 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.629197    2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 31 07:28:23.633322 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.629199    2573 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 31 07:28:23.633322 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.629202    2573 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 31 07:28:23.633322 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.629204    2573 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 31 07:28:23.633322 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.629208    2573 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 31 07:28:23.633322 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.629210    2573 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 31 07:28:23.633322 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.629213    2573 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 31 07:28:23.633322 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.629217    2573 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 31 07:28:23.633322 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.629221    2573 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 31 07:28:23.633322 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.629225    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 31 07:28:23.633322 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.629228    2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 31 07:28:23.633322 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.629231    2573 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 31 07:28:23.633322 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.629234    2573 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 31 07:28:23.633322 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.629237    2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 31 07:28:23.633322 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.629239    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 31 07:28:23.633322 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.629241    2573 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 31 07:28:23.633322 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.629244    2573 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 31 07:28:23.633322 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.629246    2573 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 31 07:28:23.633802 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.629249    2573 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 31 07:28:23.633802 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.629251    2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 31 07:28:23.633802 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.629254    2573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 31 07:28:23.633802 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.629256    2573 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 31 07:28:23.633802 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.629259    2573 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 31 07:28:23.633802 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.629262    2573 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 31 07:28:23.633802 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.629264    2573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 31 07:28:23.633802 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.629266    2573 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 31 07:28:23.633802 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.629269    2573 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 31 07:28:23.633802 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.629271    2573 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 31 07:28:23.633802 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.629273    2573 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 31 07:28:23.633802 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.629276    2573 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 31 07:28:23.633802 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.629278    2573 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 31 07:28:23.633802 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.629281    2573 feature_gate.go:386] unrecognized feature gate: Example
Oct 31 07:28:23.633802 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.629284    2573 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 31 07:28:23.633802 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.629286    2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 31 07:28:23.633802 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.629289    2573 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 31 07:28:23.633802 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.629291    2573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 31 07:28:23.633802 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.629294    2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 31 07:28:23.634241 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.629297    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 31 07:28:23.634241 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.629299    2573 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 31 07:28:23.634241 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.629302    2573 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 31 07:28:23.634241 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.629304    2573 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 31 07:28:23.634241 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.629307    2573 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 31 07:28:23.634241 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.629309    2573 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 31 07:28:23.634241 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.629312    2573 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 31 07:28:23.634241 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.629315    2573 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 31 07:28:23.634241 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.629317    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 31 07:28:23.634241 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.629319    2573 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 31 07:28:23.634241 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.629322    2573 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 31 07:28:23.634241 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:23.629365    2573 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 31 07:28:23.634241 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.629503    2573 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 31 07:28:23.634241 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.629508    2573 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 31 07:28:23.634241 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.629511    2573 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 31 07:28:23.634241 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.629513    2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 31 07:28:23.634644 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.629516    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 31 07:28:23.634644 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.629519    2573 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 31 07:28:23.634644 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.629521    2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 31 07:28:23.634644 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.629524    2573 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 31 07:28:23.634644 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.629526    2573 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 31 07:28:23.634644 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.629529    2573 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 31 07:28:23.634644 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.629531    2573 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 31 07:28:23.634644 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.629534    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 31 07:28:23.634644 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.629536    2573 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 31 07:28:23.634644 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.629539    2573 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 31 07:28:23.634644 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.629542    2573 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 31 07:28:23.634644 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.629544    2573 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 31 07:28:23.634644 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.629547    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 31 07:28:23.634644 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.629549    2573 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 31 07:28:23.634644 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.629552    2573 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 31 07:28:23.634644 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.629554    2573 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 31 07:28:23.634644 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.629556    2573 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 31 07:28:23.634644 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.629559    2573 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 31 07:28:23.634644 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.629562    2573 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 31 07:28:23.634644 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.629564    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 31 07:28:23.635141 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.629567    2573 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 31 07:28:23.635141 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.629570    2573 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 31 07:28:23.635141 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.629572    2573 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 31 07:28:23.635141 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.629575    2573 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 31 07:28:23.635141 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.629577    2573 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 31 07:28:23.635141 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.629579    2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 31 07:28:23.635141 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.629582    2573 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 31 07:28:23.635141 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.629584    2573 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 31 07:28:23.635141 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.629587    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 31 07:28:23.635141 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.629589    2573 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 31 07:28:23.635141 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.629592    2573 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 31 07:28:23.635141 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.629594    2573 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 31 07:28:23.635141 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.629597    2573 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 31 07:28:23.635141 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.629600    2573 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 31 07:28:23.635141 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.629602    2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 31 07:28:23.635141 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.629605    2573 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 31 07:28:23.635141 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.629607    2573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 31 07:28:23.635141 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.629610    2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 31 07:28:23.635141 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.629612    2573 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 31 07:28:23.635604 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.629614    2573 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 31 07:28:23.635604 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.629617    2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 31 07:28:23.635604 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.629619    2573 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 31 07:28:23.635604 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.629622    2573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 31 07:28:23.635604 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.629624    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 31 07:28:23.635604 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.629626    2573 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 31 07:28:23.635604 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.629629    2573 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 31 07:28:23.635604 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.629631    2573 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 31 07:28:23.635604 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.629634    2573 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 31 07:28:23.635604 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.629637    2573 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 31 07:28:23.635604 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.629641    2573 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 31 07:28:23.635604 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.629644    2573 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 31 07:28:23.635604 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.629647    2573 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 31 07:28:23.635604 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.629649    2573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 31 07:28:23.635604 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.629652    2573 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 31 07:28:23.635604 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.629654    2573 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 31 07:28:23.635604 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.629657    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 31 07:28:23.635604 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.629659    2573 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 31 07:28:23.635604 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.629662    2573 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 31 07:28:23.636027 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.629664    2573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 31 07:28:23.636027 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.629667    2573 feature_gate.go:386] unrecognized feature gate: Example
Oct 31 07:28:23.636027 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.629669    2573 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 31 07:28:23.636027 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.629672    2573 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 31 07:28:23.636027 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.629674    2573 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 31 07:28:23.636027 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.629677    2573 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 31 07:28:23.636027 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.629679    2573 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 31 07:28:23.636027 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.629682    2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 31 07:28:23.636027 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.629684    2573 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 31 07:28:23.636027 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.629687    2573 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 31 07:28:23.636027 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.629689    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 31 07:28:23.636027 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.629692    2573 feature_gate.go:386] unrecognized feature gate: Example2
Oct 31 07:28:23.636027 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.629694    2573 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 31 07:28:23.636027 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.629697    2573 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 31 07:28:23.636027 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.629699    2573 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 31 07:28:23.636027 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.629703    2573 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 31 07:28:23.636027 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.629707    2573 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 31 07:28:23.636027 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.629710    2573 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 31 07:28:23.636027 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.629713    2573 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 31 07:28:23.636489 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.629716    2573 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 31 07:28:23.636489 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.629726    2573 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 31 07:28:23.636489 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.629730    2573 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 31 07:28:23.636489 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.629732    2573 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 31 07:28:23.636489 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.629735    2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 31 07:28:23.636489 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.629737    2573 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 31 07:28:23.636489 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.629741    2573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 31 07:28:23.636489 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.629743    2573 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 31 07:28:23.636489 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.629748    2573 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 31 07:28:23.636489 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.629751    2573 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 31 07:28:23.636489 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.629754    2573 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 31 07:28:23.636489 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.629775    2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 31 07:28:23.636489 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.629778    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 31 07:28:23.636489 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.629781    2573 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 31 07:28:23.636489 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.629783    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 31 07:28:23.636489 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.629786    2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 31 07:28:23.636489 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.629788    2573 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 31 07:28:23.636489 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.629791    2573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 31 07:28:23.636489 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.629793    2573 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 31 07:28:23.636913 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.629796    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 31 07:28:23.636913 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.629799    2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 31 07:28:23.636913 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.629802    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 31 07:28:23.636913 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.629804    2573 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 31 07:28:23.636913 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.629807    2573 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 31 07:28:23.636913 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.629810    2573 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 31 07:28:23.636913 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:23.629833    2573 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 31 07:28:23.636913 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.629872    2573 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 31 07:28:23.636913 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.629876    2573 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 31 07:28:23.636913 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.629879    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 31 07:28:23.636913 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.629882    2573 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 31 07:28:23.636913 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.629885    2573 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 31 07:28:23.636913 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.629888    2573 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 31 07:28:23.636913 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.629890    2573 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 31 07:28:23.636913 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.629893    2573 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 31 07:28:23.636913 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.629895    2573 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 31 07:28:23.637272 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.629898    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 31 07:28:23.637272 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.629900    2573 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 31 07:28:23.637272 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.629903    2573 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 31 07:28:23.637272 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.629906    2573 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 31 07:28:23.637272 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.629908    2573 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 31 07:28:23.637272 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.629911    2573 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 31 07:28:23.637272 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.629915    2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 31 07:28:23.637272 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.629918    2573 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 31 07:28:23.637272 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.629920    2573 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 31 07:28:23.637272 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.629923    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 31 07:28:23.637272 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.629925    2573 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 31 07:28:23.637272 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.629928    2573 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 31 07:28:23.637272 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.629931    2573 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 31 07:28:23.637272 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.629933    2573 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 31 07:28:23.637272 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.629936    2573 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 31 07:28:23.637272 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.629938    2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 31 07:28:23.637272 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.629941    2573 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 31 07:28:23.637272 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.629943    2573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 31 07:28:23.637272 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.629946    2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 31 07:28:23.637779 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.629948    2573 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 31 07:28:23.637779 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.629951    2573 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 31 07:28:23.637779 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.629953    2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 31 07:28:23.637779 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.629956    2573 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 31 07:28:23.637779 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.629959    2573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 31 07:28:23.637779 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.629962    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 31 07:28:23.637779 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.629964    2573 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 31 07:28:23.637779 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.629967    2573 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 31 07:28:23.637779 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.629970    2573 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 31 07:28:23.637779 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.629972    2573 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 31 07:28:23.637779 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.629976    2573 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 31 07:28:23.637779 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.629980    2573 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 31 07:28:23.637779 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.629983    2573 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 31 07:28:23.637779 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.629986    2573 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 31 07:28:23.637779 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.629990    2573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 31 07:28:23.637779 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.629992    2573 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 31 07:28:23.637779 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.629995    2573 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 31 07:28:23.637779 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.629997    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 31 07:28:23.637779 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.630001    2573 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 31 07:28:23.638243 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.630006    2573 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 31 07:28:23.638243 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.630008    2573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 31 07:28:23.638243 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.630011    2573 feature_gate.go:386] unrecognized feature gate: Example
Oct 31 07:28:23.638243 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.630014    2573 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 31 07:28:23.638243 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.630016    2573 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 31 07:28:23.638243 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.630019    2573 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 31 07:28:23.638243 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.630021    2573 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 31 07:28:23.638243 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.630024    2573 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 31 07:28:23.638243 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.630026    2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 31 07:28:23.638243 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.630029    2573 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 31 07:28:23.638243 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.630031    2573 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 31 07:28:23.638243 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.630035    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 31 07:28:23.638243 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.630037    2573 feature_gate.go:386] unrecognized feature gate: Example2
Oct 31 07:28:23.638243 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.630040    2573 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 31 07:28:23.638243 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.630042    2573 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 31 07:28:23.638243 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.630045    2573 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 31 07:28:23.638243 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.630049    2573 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 31 07:28:23.638243 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.630052    2573 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 31 07:28:23.638243 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.630056    2573 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 31 07:28:23.638862 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.630059    2573 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 31 07:28:23.638862 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.630061    2573 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 31 07:28:23.638862 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.630064    2573 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 31 07:28:23.638862 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.630067    2573 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 31 07:28:23.638862 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.630069    2573 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 31 07:28:23.638862 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.630071    2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 31 07:28:23.638862 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.630074    2573 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 31 07:28:23.638862 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.630076    2573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 31 07:28:23.638862 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.630080    2573 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 31 07:28:23.638862 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.630083    2573 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 31 07:28:23.638862 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.630085    2573 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 31 07:28:23.638862 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.630088    2573 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 31 07:28:23.638862 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.630091    2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 31 07:28:23.638862 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.630095    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 31 07:28:23.638862 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.630098    2573 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 31 07:28:23.638862 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.630100    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 31 07:28:23.638862 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.630103    2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 31 07:28:23.638862 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.630106    2573 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 31 07:28:23.638862 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.630108    2573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 31 07:28:23.639508 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.630111    2573 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 31 07:28:23.639508 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.630113    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 31 07:28:23.639508 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.630116    2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 31 07:28:23.639508 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.630119    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 31 07:28:23.639508 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.630122    2573 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 31 07:28:23.639508 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.630124    2573 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 31 07:28:23.639508 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.630127    2573 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 31 07:28:23.639508 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.630129    2573 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 31 07:28:23.639508 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.630132    2573 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 31 07:28:23.639508 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.630135    2573 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 31 07:28:23.639508 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.630137    2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 31 07:28:23.639508 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.630140    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 31 07:28:23.639508 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.630142    2573 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 31 07:28:23.639508 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.630145    2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 31 07:28:23.639508 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.630147    2573 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 31 07:28:23.639508 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.630150    2573 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 31 07:28:23.639508 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.630152    2573 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 31 07:28:23.639508 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.630155    2573 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 31 07:28:23.639508 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.630157    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 31 07:28:23.639508 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.630160    2573 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 31 07:28:23.640047 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:23.630162    2573 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 31 07:28:23.640047 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:23.630220    2573 server.go:957] "Client rotation is on, will bootstrap in background"
Oct 31 07:28:23.640047 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:23.632538    2573 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Oct 31 07:28:23.640047 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:23.633540    2573 server.go:1014] "Starting client certificate rotation"
Oct 31 07:28:23.640047 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:23.633634    2573 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Oct 31 07:28:23.640047 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:23.633668    2573 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Oct 31 07:28:23.640047 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:23.635666    2573 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 31 07:28:23.640047 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:23.636801    2573 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 31 07:28:23.649938 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:23.649920    2573 log.go:25] "Validated CRI v1 runtime API"
Oct 31 07:28:23.653751 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:23.653735    2573 log.go:25] "Validated CRI v1 image API"
Oct 31 07:28:23.654611 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:23.654590    2573 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Oct 31 07:28:23.658559 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:23.658536    2573 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 bc678eae-8b6f-494f-90bb-737d90244670:/dev/nvme0n1p3 eec75f1d-0616-4685-bf4f-0a76eaa0b8d4:/dev/nvme0n1p4]
Oct 31 07:28:23.658604 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:23.658560    2573 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}]
Oct 31 07:28:23.664089 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:23.663959    2573 manager.go:217] Machine: {Timestamp:2026-10-31 07:28:23.663507694 +0000 UTC m=+0.354278510 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3123539 MemoryCapacity:32812007424 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2e19ef47dea3198edfb2a67ef8bd76 SystemUUID:ec2e19ef-47de-a319-8edf-b2a67ef8bd76 BootID:faed6606-cdad-45f4-a07a-b407dea0d5e9 Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16406003712 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6562402304 Type:vfs Inodes:819200 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16406003712 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:28:50:23:a4:5b Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:28:50:23:a4:5b Speed:0 Mtu:9001} {Name:ovs-system MacAddress:8e:74:45:93:74:17 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812007424 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Oct 31 07:28:23.664089 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:23.664085    2573 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Oct 31 07:28:23.664200 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:23.664189    2573 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Oct 31 07:28:23.667765 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:23.667739    2573 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Oct 31 07:28:23.667885 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:23.667766    2573 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-139-83.us-west-1.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Oct 31 07:28:23.668573 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:23.668564    2573 topology_manager.go:143] "Creating topology manager with none policy"
Oct 31 07:28:23.668612 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:23.668575    2573 container_manager_linux.go:311] "Creating device plugin manager"
Oct 31 07:28:23.668612 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:23.668588    2573 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 31 07:28:23.668612 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:23.668601    2573 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 31 07:28:23.669817 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:23.669808    2573 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager"
Oct 31 07:28:23.674160 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:23.674148    2573 state_mem.go:41] "Initialized" logger="CPUManager state memory"
Oct 31 07:28:23.674282 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:23.674274    2573 server.go:1235] "Using root directory" path="/var/lib/kubelet"
Oct 31 07:28:23.675089 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:23.675080    2573 kubelet.go:493] "Attempting to sync node with API server"
Oct 31 07:28:23.675120 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:23.675093    2573 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests"
Oct 31 07:28:23.675120 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:23.675103    2573 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Oct 31 07:28:23.675120 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:23.675111    2573 kubelet.go:405] "Adding apiserver pod source"
Oct 31 07:28:23.675120 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:23.675120    2573 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Oct 31 07:28:23.675430 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:23.675418    2573 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 31 07:28:23.677692 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:23.677671    2573 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1"
Oct 31 07:28:23.684787 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:23.684772    2573 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 31 07:28:23.685678 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:23.685667    2573 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Oct 31 07:28:23.685710 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:23.685696    2573 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Oct 31 07:28:23.685879 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:23.685869    2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Oct 31 07:28:23.685907 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:23.685884    2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Oct 31 07:28:23.685907 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:23.685891    2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Oct 31 07:28:23.685907 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:23.685897    2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Oct 31 07:28:23.685907 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:23.685902    2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Oct 31 07:28:23.685907 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:23.685907    2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Oct 31 07:28:23.686611 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:23.686602    2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Oct 31 07:28:23.686639 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:23.686611    2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Oct 31 07:28:23.686639 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:23.686619    2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Oct 31 07:28:23.686639 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:23.686625    2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Oct 31 07:28:23.686639 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:23.686634    2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Oct 31 07:28:23.686786 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:23.686642    2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Oct 31 07:28:23.688227 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:23.688218    2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Oct 31 07:28:23.688265 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:23.688229    2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Oct 31 07:28:23.691891 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:23.691866    2573 server.go:1263] "Started kubelet"
Oct 31 07:28:23.692004 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:23.691952    2573 server.go:182] "Starting to listen" address="0.0.0.0" port=10250
Oct 31 07:28:23.692232 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:23.691958    2573 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Oct 31 07:28:23.692304 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:23.692252    2573 server_v1.go:49] "podresources" method="list" useActivePods=true
Oct 31 07:28:23.692591 ip-10-0-139-83 systemd[1]: Started Kubernetes Kubelet.
Oct 31 07:28:23.693742 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:23.693726    2573 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Oct 31 07:28:23.696178 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:23.696159    2573 server.go:317] "Adding debug handlers to kubelet server"
Oct 31 07:28:23.698621 ip-10-0-139-83 kubenswrapper[2573]: E1031 07:28:23.698594    2573 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Oct 31 07:28:23.701837 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:23.701821    2573 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Oct 31 07:28:23.702440 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:23.702422    2573 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer"
Oct 31 07:28:23.703463 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:23.703284    2573 volume_manager.go:309] "The desired_state_of_world populator starts"
Oct 31 07:28:23.703463 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:23.703317    2573 volume_manager.go:311] "Starting Kubelet Volume Manager"
Oct 31 07:28:23.703463 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:23.703405    2573 factory.go:153] Registering CRI-O factory
Oct 31 07:28:23.703463 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:23.703414    2573 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Oct 31 07:28:23.703463 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:23.703418    2573 factory.go:223] Registration of the crio container factory successfully
Oct 31 07:28:23.703463 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:23.703461    2573 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Oct 31 07:28:23.703463 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:23.703470    2573 factory.go:55] Registering systemd factory
Oct 31 07:28:23.703463 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:23.703473    2573 reconstruct.go:97] "Volume reconstruction finished"
Oct 31 07:28:23.703463 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:23.703478    2573 factory.go:223] Registration of the systemd container factory successfully
Oct 31 07:28:23.703463 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:23.703481    2573 reconciler.go:29] "Reconciler: start to sync state"
Oct 31 07:28:23.703463 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:23.703495    2573 factory.go:103] Registering Raw factory
Oct 31 07:28:23.703463 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:23.703509    2573 manager.go:1199] Started watching for new ooms in manager
Oct 31 07:28:23.704080 ip-10-0-139-83 kubenswrapper[2573]: E1031 07:28:23.703466    2573 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-139-83.us-west-1.compute.internal\" not found"
Oct 31 07:28:23.704499 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:23.704482    2573 manager.go:319] Starting recovery of all containers
Oct 31 07:28:23.714077 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:23.714061    2573 manager.go:324] Recovery completed
Oct 31 07:28:23.718116 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:23.718101    2573 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 31 07:28:23.719813 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:23.719795    2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-83.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Oct 31 07:28:23.719861 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:23.719833    2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-83.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Oct 31 07:28:23.719861 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:23.719847    2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-83.us-west-1.compute.internal" event="NodeHasSufficientPID"
Oct 31 07:28:23.720352 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:23.720340    2573 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 31 07:28:23.720415 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:23.720338    2573 cpu_manager.go:226] "Starting" policy="none"
Oct 31 07:28:23.720449 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:23.720419    2573 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s"
Oct 31 07:28:23.720449 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:23.720438    2573 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory"
Oct 31 07:28:23.720971 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:23.720956    2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-83.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Oct 31 07:28:23.721032 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:23.720983    2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-83.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Oct 31 07:28:23.721032 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:23.720996    2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-83.us-west-1.compute.internal" event="NodeHasSufficientPID"
Oct 31 07:28:23.722579 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:23.722563    2573 policy_none.go:50] "Start"
Oct 31 07:28:23.722579 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:23.722581    2573 memory_manager.go:187] "Starting memorymanager" policy="None"
Oct 31 07:28:23.722664 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:23.722591    2573 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Oct 31 07:28:23.724471 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:23.724459    2573 policy_none.go:44] "Start"
Oct 31 07:28:23.763998 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:23.763836    2573 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"}
Oct 31 07:28:23.768218 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:23.768205    2573 manager.go:341] "Starting Device Plugin manager"
Oct 31 07:28:23.768884 ip-10-0-139-83 kubenswrapper[2573]: E1031 07:28:23.768866    2573 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Oct 31 07:28:23.768968 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:23.768881    2573 server.go:87] "Starting device plugin registration server"
Oct 31 07:28:23.769154 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:23.769141    2573 eviction_manager.go:194] "Eviction manager: starting control loop"
Oct 31 07:28:23.769203 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:23.769156    2573 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Oct 31 07:28:23.769421 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:23.769250    2573 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Oct 31 07:28:23.769421 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:23.769351    2573 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts"
Oct 31 07:28:23.769421 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:23.769359    2573 plugin_manager.go:121] "Starting Kubelet Plugin Manager"
Oct 31 07:28:23.770212 ip-10-0-139-83 kubenswrapper[2573]: E1031 07:28:23.770191    2573 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Oct 31 07:28:23.770309 ip-10-0-139-83 kubenswrapper[2573]: E1031 07:28:23.770256    2573 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-139-83.us-west-1.compute.internal\" not found"
Oct 31 07:28:23.832459 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:23.832438    2573 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-139-83.us-west-1.compute.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Oct 31 07:28:23.832459 ip-10-0-139-83 kubenswrapper[2573]: E1031 07:28:23.832444    2573 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-139-83.us-west-1.compute.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Oct 31 07:28:23.834384 ip-10-0-139-83 kubenswrapper[2573]: E1031 07:28:23.832434    2573 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-139-83.us-west-1.compute.internal.18e38ca1eb79c6d5  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-139-83.us-west-1.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-139-83.us-west-1.compute.internal,},FirstTimestamp:2026-10-31 07:28:23.691839189 +0000 UTC m=+0.382610004,LastTimestamp:2026-10-31 07:28:23.691839189 +0000 UTC m=+0.382610004,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-139-83.us-west-1.compute.internal,}"
Oct 31 07:28:23.848912 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:23.848880    2573 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Oct 31 07:28:23.850226 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:23.850211    2573 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Oct 31 07:28:23.850226 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:23.850232    2573 status_manager.go:249] "Starting to sync pod status with apiserver"
Oct 31 07:28:23.850387 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:23.850253    2573 kubelet.go:2521] "Starting kubelet main sync loop"
Oct 31 07:28:23.850387 ip-10-0-139-83 kubenswrapper[2573]: E1031 07:28:23.850288    2573 kubelet.go:2545] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Oct 31 07:28:23.869979 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:23.869937    2573 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 31 07:28:23.871264 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:23.871247    2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-83.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Oct 31 07:28:23.871344 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:23.871277    2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-83.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Oct 31 07:28:23.871344 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:23.871287    2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-83.us-west-1.compute.internal" event="NodeHasSufficientPID"
Oct 31 07:28:23.871344 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:23.871307    2573 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-139-83.us-west-1.compute.internal"
Oct 31 07:28:23.906253 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:23.906237    2573 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Oct 31 07:28:23.906342 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:23.906297    2573 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-wpkxb"
Oct 31 07:28:23.950793 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:23.950768    2573 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-83.us-west-1.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-139-83.us-west-1.compute.internal"]
Oct 31 07:28:23.950878 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:23.950823    2573 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 31 07:28:23.951802 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:23.951783    2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-83.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Oct 31 07:28:23.951885 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:23.951812    2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-83.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Oct 31 07:28:23.951885 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:23.951825    2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-83.us-west-1.compute.internal" event="NodeHasSufficientPID"
Oct 31 07:28:23.953945 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:23.953933    2573 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 31 07:28:23.954101 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:23.954087    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-83.us-west-1.compute.internal"
Oct 31 07:28:23.954144 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:23.954118    2573 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 31 07:28:23.954670 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:23.954656    2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-83.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Oct 31 07:28:23.954716 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:23.954665    2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-83.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Oct 31 07:28:23.954716 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:23.954679    2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-83.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Oct 31 07:28:23.954716 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:23.954688    2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-83.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Oct 31 07:28:23.954716 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:23.954699    2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-83.us-west-1.compute.internal" event="NodeHasSufficientPID"
Oct 31 07:28:23.954716 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:23.954713    2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-83.us-west-1.compute.internal" event="NodeHasSufficientPID"
Oct 31 07:28:23.956758 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:23.956743    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-83.us-west-1.compute.internal"
Oct 31 07:28:23.956832 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:23.956766    2573 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 31 07:28:23.957953 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:23.957937    2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-83.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Oct 31 07:28:23.958022 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:23.957972    2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-83.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Oct 31 07:28:23.958022 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:23.957988    2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-83.us-west-1.compute.internal" event="NodeHasSufficientPID"
Oct 31 07:28:23.984670 ip-10-0-139-83 kubenswrapper[2573]: E1031 07:28:23.984650    2573 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-139-83.us-west-1.compute.internal\" not found" node="ip-10-0-139-83.us-west-1.compute.internal"
Oct 31 07:28:23.987045 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:23.987029    2573 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 31 07:28:23.987120 ip-10-0-139-83 kubenswrapper[2573]: E1031 07:28:23.987058    2573 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-139-83.us-west-1.compute.internal\" not found" node="ip-10-0-139-83.us-west-1.compute.internal"
Oct 31 07:28:23.988218 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:23.988205    2573 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 31 07:28:24.004511 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:24.004489    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/8dfeb47fa079b05125409320442d08bf-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-83.us-west-1.compute.internal\" (UID: \"8dfeb47fa079b05125409320442d08bf\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-83.us-west-1.compute.internal"
Oct 31 07:28:24.004579 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:24.004520    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8dfeb47fa079b05125409320442d08bf-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-83.us-west-1.compute.internal\" (UID: \"8dfeb47fa079b05125409320442d08bf\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-83.us-west-1.compute.internal"
Oct 31 07:28:24.085111 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:24.085084    2573 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 31 07:28:24.086452 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:24.086434    2573 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-139-83.us-west-1.compute.internal"
Oct 31 07:28:24.086668 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:24.086652    2573 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 31 07:28:24.104447 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:24.104431    2573 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-83.us-west-1.compute.internal"
Oct 31 07:28:24.105543 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:24.105526    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/8dfeb47fa079b05125409320442d08bf-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-83.us-west-1.compute.internal\" (UID: \"8dfeb47fa079b05125409320442d08bf\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-83.us-west-1.compute.internal"
Oct 31 07:28:24.105618 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:24.105554    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8dfeb47fa079b05125409320442d08bf-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-83.us-west-1.compute.internal\" (UID: \"8dfeb47fa079b05125409320442d08bf\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-83.us-west-1.compute.internal"
Oct 31 07:28:24.105618 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:24.105611    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8dfeb47fa079b05125409320442d08bf-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-83.us-west-1.compute.internal\" (UID: \"8dfeb47fa079b05125409320442d08bf\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-83.us-west-1.compute.internal"
Oct 31 07:28:24.105705 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:24.105633    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/8dfeb47fa079b05125409320442d08bf-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-83.us-west-1.compute.internal\" (UID: \"8dfeb47fa079b05125409320442d08bf\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-83.us-west-1.compute.internal"
Oct 31 07:28:24.105705 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:24.105650    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/2183ffc19200ec816330ede9883f3bf6-config\") pod \"kube-apiserver-proxy-ip-10-0-139-83.us-west-1.compute.internal\" (UID: \"2183ffc19200ec816330ede9883f3bf6\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-83.us-west-1.compute.internal"
Oct 31 07:28:24.206764 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:24.206700    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/2183ffc19200ec816330ede9883f3bf6-config\") pod \"kube-apiserver-proxy-ip-10-0-139-83.us-west-1.compute.internal\" (UID: \"2183ffc19200ec816330ede9883f3bf6\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-83.us-west-1.compute.internal"
Oct 31 07:28:24.206764 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:24.206757    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/2183ffc19200ec816330ede9883f3bf6-config\") pod \"kube-apiserver-proxy-ip-10-0-139-83.us-west-1.compute.internal\" (UID: \"2183ffc19200ec816330ede9883f3bf6\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-83.us-west-1.compute.internal"
Oct 31 07:28:24.208803 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:24.208789    2573 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 31 07:28:24.209344 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:24.209320    2573 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-83.us-west-1.compute.internal"
Oct 31 07:28:24.265774 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:24.265739    2573 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 31 07:28:24.286926 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:24.286910    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-83.us-west-1.compute.internal"
Oct 31 07:28:24.289738 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:24.289713    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-83.us-west-1.compute.internal"
Oct 31 07:28:24.633400 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:24.633317    2573 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Oct 31 07:28:24.633882 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:24.633486    2573 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received"
Oct 31 07:28:24.633882 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:24.633495    2573 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received"
Oct 31 07:28:24.633882 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:24.633528    2573 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received"
Oct 31 07:28:24.633882 ip-10-0-139-83 kubenswrapper[2573]: E1031 07:28:24.633506    2573 event.go:368] "Unable to write event (may retry after sleeping)" err="Patch \"https://adf9bd3603f5d4feb94c1d7340fbfba2-7a76273f21d70fcd.elb.us-east-2.amazonaws.com:6443/api/v1/namespaces/default/events/ip-10-0-139-83.us-west-1.compute.internal.18e38ca1ed24a388\": read tcp 10.0.139.83:60198->16.58.112.7:6443: use of closed network connection" event="&Event{ObjectMeta:{ip-10-0-139-83.us-west-1.compute.internal.18e38ca1ed24a388  default   4817 0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-139-83.us-west-1.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-139-83.us-west-1.compute.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-139-83.us-west-1.compute.internal,},FirstTimestamp:2026-10-31 07:28:23 +0000 UTC,LastTimestamp:2026-10-31 07:28:23.951796945 +0000 UTC m=+0.642567760,Count:4,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-139-83.us-west-1.compute.internal,}"
Oct 31 07:28:24.675901 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:24.675882    2573 apiserver.go:52] "Watching apiserver"
Oct 31 07:28:24.702393 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:24.702373    2573 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Oct 31 07:28:24.910294 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:24.910220    2573 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-30 07:23:24 +0000 UTC" deadline="2028-05-28 06:26:19.239870485 +0000 UTC"
Oct 31 07:28:24.910294 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:24.910250    2573 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13798h57m54.329623473s"
Oct 31 07:28:24.938989 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:24.938961    2573 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Oct 31 07:28:24.939286 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:24.939268    2573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-multus/multus-additional-cni-plugins-dh6w8","openshift-multus/multus-pmnqx","openshift-multus/network-metrics-daemon-r2zbv","kube-system/global-pull-secret-syncer-kht4l","openshift-cluster-node-tuning-operator/tuned-cpn4m","kube-system/kube-apiserver-proxy-ip-10-0-139-83.us-west-1.compute.internal","openshift-dns/node-resolver-tnw4j","openshift-network-diagnostics/network-check-target-9zp4l","openshift-network-operator/iptables-alerter-65hzp","openshift-ovn-kubernetes/ovnkube-node-5p7xw","kube-system/konnectivity-agent-h4k2m","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-797gj","openshift-image-registry/node-ca-th7x7","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-83.us-west-1.compute.internal"]
Oct 31 07:28:24.941712 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:24.941689    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-cpn4m"
Oct 31 07:28:24.943775 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:24.943755    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-797gj"
Oct 31 07:28:24.946007 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:24.945990    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-5p7xw"
Oct 31 07:28:24.948140 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:24.948124    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-65hzp"
Oct 31 07:28:24.950154 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:24.950141    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kht4l"
Oct 31 07:28:24.950224 ip-10-0-139-83 kubenswrapper[2573]: E1031 07:28:24.950204    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kht4l" podUID="77256436-39ff-4c2a-8390-4e9fa7d68486"
Oct 31 07:28:24.952265 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:24.952249    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-tnw4j"
Oct 31 07:28:24.954604 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:24.954573    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9zp4l"
Oct 31 07:28:24.954660 ip-10-0-139-83 kubenswrapper[2573]: E1031 07:28:24.954634    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9zp4l" podUID="f1ba68f6-d87c-4492-a5eb-5f2d3a60eb1c"
Oct 31 07:28:24.956932 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:24.956911    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-dh6w8"
Oct 31 07:28:24.959177 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:24.959160    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r2zbv"
Oct 31 07:28:24.959267 ip-10-0-139-83 kubenswrapper[2573]: E1031 07:28:24.959219    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-r2zbv" podUID="c3ad95fd-e997-459c-93fd-27c4c030ea0d"
Oct 31 07:28:24.961312 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:24.961296    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-h4k2m"
Oct 31 07:28:24.963477 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:24.963458    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-pmnqx"
Oct 31 07:28:24.965967 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:24.965951    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-th7x7"
Oct 31 07:28:24.994422 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:24.994387    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Oct 31 07:28:24.994422 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:24.994404    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Oct 31 07:28:24.994422 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:24.994388    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-w8pvl\""
Oct 31 07:28:24.995541 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:24.995522    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Oct 31 07:28:24.995671 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:24.995566    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Oct 31 07:28:24.995756 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:24.995713    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Oct 31 07:28:24.996096 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:24.996082    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-kwsqb\""
Oct 31 07:28:24.998036 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:24.998016    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Oct 31 07:28:24.998141 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:24.998088    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Oct 31 07:28:24.998141 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:24.998128    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Oct 31 07:28:24.998244 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:24.998134    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Oct 31 07:28:24.998244 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:24.998187    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Oct 31 07:28:24.998244 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:24.998185    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Oct 31 07:28:24.998244 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:24.998205    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-gttml\""
Oct 31 07:28:24.999892 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:24.999873    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-46dh7\""
Oct 31 07:28:24.999892 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:24.999889    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Oct 31 07:28:25.000018 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:24.999896    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Oct 31 07:28:25.000018 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:24.999890    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Oct 31 07:28:25.004673 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:25.004660    2573 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Oct 31 07:28:25.009812 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:25.009698    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/3fa078c7-4bf2-4e76-9692-ae460e8fadfa-run\") pod \"tuned-cpn4m\" (UID: \"3fa078c7-4bf2-4e76-9692-ae460e8fadfa\") " pod="openshift-cluster-node-tuning-operator/tuned-cpn4m"
Oct 31 07:28:25.009812 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:25.009779    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/3fa078c7-4bf2-4e76-9692-ae460e8fadfa-lib-modules\") pod \"tuned-cpn4m\" (UID: \"3fa078c7-4bf2-4e76-9692-ae460e8fadfa\") " pod="openshift-cluster-node-tuning-operator/tuned-cpn4m"
Oct 31 07:28:25.009812 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:25.009810    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/3fa078c7-4bf2-4e76-9692-ae460e8fadfa-etc-tuned\") pod \"tuned-cpn4m\" (UID: \"3fa078c7-4bf2-4e76-9692-ae460e8fadfa\") " pod="openshift-cluster-node-tuning-operator/tuned-cpn4m"
Oct 31 07:28:25.010067 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:25.009835    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/9dad8dd1-cfc5-412d-bf9a-bac3c9162af1-device-dir\") pod \"aws-ebs-csi-driver-node-797gj\" (UID: \"9dad8dd1-cfc5-412d-bf9a-bac3c9162af1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-797gj"
Oct 31 07:28:25.010067 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:25.009868    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/26659955-3909-4171-90b8-21b0ad6eef64-host-slash\") pod \"iptables-alerter-65hzp\" (UID: \"26659955-3909-4171-90b8-21b0ad6eef64\") " pod="openshift-network-operator/iptables-alerter-65hzp"
Oct 31 07:28:25.010067 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:25.009896    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/4bfbebfd-2be7-438f-b88d-273f24751f8c-env-overrides\") pod \"ovnkube-node-5p7xw\" (UID: \"4bfbebfd-2be7-438f-b88d-273f24751f8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-5p7xw"
Oct 31 07:28:25.010067 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:25.010037    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/08198bc1-314f-4881-833c-e633bac15e4a-cni-binary-copy\") pod \"multus-pmnqx\" (UID: \"08198bc1-314f-4881-833c-e633bac15e4a\") " pod="openshift-multus/multus-pmnqx"
Oct 31 07:28:25.010351 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:25.010103    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/08198bc1-314f-4881-833c-e633bac15e4a-multus-daemon-config\") pod \"multus-pmnqx\" (UID: \"08198bc1-314f-4881-833c-e633bac15e4a\") " pod="openshift-multus/multus-pmnqx"
Oct 31 07:28:25.010351 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:25.010143    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3fa078c7-4bf2-4e76-9692-ae460e8fadfa-sys\") pod \"tuned-cpn4m\" (UID: \"3fa078c7-4bf2-4e76-9692-ae460e8fadfa\") " pod="openshift-cluster-node-tuning-operator/tuned-cpn4m"
Oct 31 07:28:25.010449 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:25.010402    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cq7rv\" (UniqueName: \"kubernetes.io/projected/3fa078c7-4bf2-4e76-9692-ae460e8fadfa-kube-api-access-cq7rv\") pod \"tuned-cpn4m\" (UID: \"3fa078c7-4bf2-4e76-9692-ae460e8fadfa\") " pod="openshift-cluster-node-tuning-operator/tuned-cpn4m"
Oct 31 07:28:25.010449 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:25.010439    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/9dad8dd1-cfc5-412d-bf9a-bac3c9162af1-registration-dir\") pod \"aws-ebs-csi-driver-node-797gj\" (UID: \"9dad8dd1-cfc5-412d-bf9a-bac3c9162af1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-797gj"
Oct 31 07:28:25.010542 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:25.010473    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/26659955-3909-4171-90b8-21b0ad6eef64-iptables-alerter-script\") pod \"iptables-alerter-65hzp\" (UID: \"26659955-3909-4171-90b8-21b0ad6eef64\") " pod="openshift-network-operator/iptables-alerter-65hzp"
Oct 31 07:28:25.010588 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:25.010541    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4bfbebfd-2be7-438f-b88d-273f24751f8c-etc-openvswitch\") pod \"ovnkube-node-5p7xw\" (UID: \"4bfbebfd-2be7-438f-b88d-273f24751f8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-5p7xw"
Oct 31 07:28:25.010588 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:25.010568    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/08198bc1-314f-4881-833c-e633bac15e4a-etc-kubernetes\") pod \"multus-pmnqx\" (UID: \"08198bc1-314f-4881-833c-e633bac15e4a\") " pod="openshift-multus/multus-pmnqx"
Oct 31 07:28:25.010714 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:25.010601    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5n5lv\" (UniqueName: \"kubernetes.io/projected/9dad8dd1-cfc5-412d-bf9a-bac3c9162af1-kube-api-access-5n5lv\") pod \"aws-ebs-csi-driver-node-797gj\" (UID: \"9dad8dd1-cfc5-412d-bf9a-bac3c9162af1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-797gj"
Oct 31 07:28:25.010714 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:25.010631    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/77256436-39ff-4c2a-8390-4e9fa7d68486-dbus\") pod \"global-pull-secret-syncer-kht4l\" (UID: \"77256436-39ff-4c2a-8390-4e9fa7d68486\") " pod="kube-system/global-pull-secret-syncer-kht4l"
Oct 31 07:28:25.010714 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:25.010694    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/c4233c69-4213-43be-8809-76612b6b6f3e-hosts-file\") pod \"node-resolver-tnw4j\" (UID: \"c4233c69-4213-43be-8809-76612b6b6f3e\") " pod="openshift-dns/node-resolver-tnw4j"
Oct 31 07:28:25.010841 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:25.010770    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c3ad95fd-e997-459c-93fd-27c4c030ea0d-metrics-certs\") pod \"network-metrics-daemon-r2zbv\" (UID: \"c3ad95fd-e997-459c-93fd-27c4c030ea0d\") " pod="openshift-multus/network-metrics-daemon-r2zbv"
Oct 31 07:28:25.010841 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:25.010810    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4bfbebfd-2be7-438f-b88d-273f24751f8c-run-openvswitch\") pod \"ovnkube-node-5p7xw\" (UID: \"4bfbebfd-2be7-438f-b88d-273f24751f8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-5p7xw"
Oct 31 07:28:25.010931 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:25.010838    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/4bfbebfd-2be7-438f-b88d-273f24751f8c-log-socket\") pod \"ovnkube-node-5p7xw\" (UID: \"4bfbebfd-2be7-438f-b88d-273f24751f8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-5p7xw"
Oct 31 07:28:25.010931 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:25.010878    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/4bfbebfd-2be7-438f-b88d-273f24751f8c-ovn-node-metrics-cert\") pod \"ovnkube-node-5p7xw\" (UID: \"4bfbebfd-2be7-438f-b88d-273f24751f8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-5p7xw"
Oct 31 07:28:25.010931 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:25.010907    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5hqbq\" (UniqueName: \"kubernetes.io/projected/c4233c69-4213-43be-8809-76612b6b6f3e-kube-api-access-5hqbq\") pod \"node-resolver-tnw4j\" (UID: \"c4233c69-4213-43be-8809-76612b6b6f3e\") " pod="openshift-dns/node-resolver-tnw4j"
Oct 31 07:28:25.011051 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:25.010937    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/4bfbebfd-2be7-438f-b88d-273f24751f8c-run-ovn\") pod \"ovnkube-node-5p7xw\" (UID: \"4bfbebfd-2be7-438f-b88d-273f24751f8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-5p7xw"
Oct 31 07:28:25.011051 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:25.010963    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/4bfbebfd-2be7-438f-b88d-273f24751f8c-ovnkube-script-lib\") pod \"ovnkube-node-5p7xw\" (UID: \"4bfbebfd-2be7-438f-b88d-273f24751f8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-5p7xw"
Oct 31 07:28:25.011051 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:25.010991    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cz8tm\" (UniqueName: \"kubernetes.io/projected/4bfbebfd-2be7-438f-b88d-273f24751f8c-kube-api-access-cz8tm\") pod \"ovnkube-node-5p7xw\" (UID: \"4bfbebfd-2be7-438f-b88d-273f24751f8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-5p7xw"
Oct 31 07:28:25.011051 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:25.011019    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5-agent-certs\") pod \"konnectivity-agent-h4k2m\" (UID: \"2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5\") " pod="kube-system/konnectivity-agent-h4k2m"
Oct 31 07:28:25.011214 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:25.011048    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xs9lt\" (UniqueName: \"kubernetes.io/projected/08198bc1-314f-4881-833c-e633bac15e4a-kube-api-access-xs9lt\") pod \"multus-pmnqx\" (UID: \"08198bc1-314f-4881-833c-e633bac15e4a\") " pod="openshift-multus/multus-pmnqx"
Oct 31 07:28:25.011214 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:25.011142    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/fc64d31a-f48a-4293-a4de-262045d0b7e8-serviceca\") pod \"node-ca-th7x7\" (UID: \"fc64d31a-f48a-4293-a4de-262045d0b7e8\") " pod="openshift-image-registry/node-ca-th7x7"
Oct 31 07:28:25.011214 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:25.011170    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/3fa078c7-4bf2-4e76-9692-ae460e8fadfa-etc-sysctl-d\") pod \"tuned-cpn4m\" (UID: \"3fa078c7-4bf2-4e76-9692-ae460e8fadfa\") " pod="openshift-cluster-node-tuning-operator/tuned-cpn4m"
Oct 31 07:28:25.011214 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:25.011197    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3fa078c7-4bf2-4e76-9692-ae460e8fadfa-tmp\") pod \"tuned-cpn4m\" (UID: \"3fa078c7-4bf2-4e76-9692-ae460e8fadfa\") " pod="openshift-cluster-node-tuning-operator/tuned-cpn4m"
Oct 31 07:28:25.011401 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:25.011219    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/4bfbebfd-2be7-438f-b88d-273f24751f8c-host-run-netns\") pod \"ovnkube-node-5p7xw\" (UID: \"4bfbebfd-2be7-438f-b88d-273f24751f8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-5p7xw"
Oct 31 07:28:25.011401 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:25.011248    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/4bfbebfd-2be7-438f-b88d-273f24751f8c-host-cni-netd\") pod \"ovnkube-node-5p7xw\" (UID: \"4bfbebfd-2be7-438f-b88d-273f24751f8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-5p7xw"
Oct 31 07:28:25.011401 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:25.011278    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/9c72279a-708f-46df-8915-9995c5aea14d-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-dh6w8\" (UID: \"9c72279a-708f-46df-8915-9995c5aea14d\") " pod="openshift-multus/multus-additional-cni-plugins-dh6w8"
Oct 31 07:28:25.011401 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:25.011312    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/08198bc1-314f-4881-833c-e633bac15e4a-host-var-lib-cni-bin\") pod \"multus-pmnqx\" (UID: \"08198bc1-314f-4881-833c-e633bac15e4a\") " pod="openshift-multus/multus-pmnqx"
Oct 31 07:28:25.011401 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:25.011394    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kk4nh\" (UniqueName: \"kubernetes.io/projected/fc64d31a-f48a-4293-a4de-262045d0b7e8-kube-api-access-kk4nh\") pod \"node-ca-th7x7\" (UID: \"fc64d31a-f48a-4293-a4de-262045d0b7e8\") " pod="openshift-image-registry/node-ca-th7x7"
Oct 31 07:28:25.011613 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:25.011423    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3fa078c7-4bf2-4e76-9692-ae460e8fadfa-etc-kubernetes\") pod \"tuned-cpn4m\" (UID: \"3fa078c7-4bf2-4e76-9692-ae460e8fadfa\") " pod="openshift-cluster-node-tuning-operator/tuned-cpn4m"
Oct 31 07:28:25.011613 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:25.011461    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/9dad8dd1-cfc5-412d-bf9a-bac3c9162af1-sys-fs\") pod \"aws-ebs-csi-driver-node-797gj\" (UID: \"9dad8dd1-cfc5-412d-bf9a-bac3c9162af1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-797gj"
Oct 31 07:28:25.011613 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:25.011491    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/9c72279a-708f-46df-8915-9995c5aea14d-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-dh6w8\" (UID: \"9c72279a-708f-46df-8915-9995c5aea14d\") " pod="openshift-multus/multus-additional-cni-plugins-dh6w8"
Oct 31 07:28:25.011613 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:25.011525    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/08198bc1-314f-4881-833c-e633bac15e4a-host-var-lib-kubelet\") pod \"multus-pmnqx\" (UID: \"08198bc1-314f-4881-833c-e633bac15e4a\") " pod="openshift-multus/multus-pmnqx"
Oct 31 07:28:25.011613 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:25.011555    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/4bfbebfd-2be7-438f-b88d-273f24751f8c-host-kubelet\") pod \"ovnkube-node-5p7xw\" (UID: \"4bfbebfd-2be7-438f-b88d-273f24751f8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-5p7xw"
Oct 31 07:28:25.011613 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:25.011583    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4bfbebfd-2be7-438f-b88d-273f24751f8c-host-slash\") pod \"ovnkube-node-5p7xw\" (UID: \"4bfbebfd-2be7-438f-b88d-273f24751f8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-5p7xw"
Oct 31 07:28:25.011857 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:25.011615    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/9c72279a-708f-46df-8915-9995c5aea14d-cni-binary-copy\") pod \"multus-additional-cni-plugins-dh6w8\" (UID: \"9c72279a-708f-46df-8915-9995c5aea14d\") " pod="openshift-multus/multus-additional-cni-plugins-dh6w8"
Oct 31 07:28:25.011857 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:25.011646    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/08198bc1-314f-4881-833c-e633bac15e4a-system-cni-dir\") pod \"multus-pmnqx\" (UID: \"08198bc1-314f-4881-833c-e633bac15e4a\") " pod="openshift-multus/multus-pmnqx"
Oct 31 07:28:25.011857 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:25.011671    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/08198bc1-314f-4881-833c-e633bac15e4a-hostroot\") pod \"multus-pmnqx\" (UID: \"08198bc1-314f-4881-833c-e633bac15e4a\") " pod="openshift-multus/multus-pmnqx"
Oct 31 07:28:25.011857 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:25.011703    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/08198bc1-314f-4881-833c-e633bac15e4a-host-run-multus-certs\") pod \"multus-pmnqx\" (UID: \"08198bc1-314f-4881-833c-e633bac15e4a\") " pod="openshift-multus/multus-pmnqx"
Oct 31 07:28:25.011857 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:25.011731    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/3fa078c7-4bf2-4e76-9692-ae460e8fadfa-etc-sysctl-conf\") pod \"tuned-cpn4m\" (UID: \"3fa078c7-4bf2-4e76-9692-ae460e8fadfa\") " pod="openshift-cluster-node-tuning-operator/tuned-cpn4m"
Oct 31 07:28:25.011857 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:25.011755    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/9dad8dd1-cfc5-412d-bf9a-bac3c9162af1-kubelet-dir\") pod \"aws-ebs-csi-driver-node-797gj\" (UID: \"9dad8dd1-cfc5-412d-bf9a-bac3c9162af1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-797gj"
Oct 31 07:28:25.012117 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:25.011906    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/4bfbebfd-2be7-438f-b88d-273f24751f8c-systemd-units\") pod \"ovnkube-node-5p7xw\" (UID: \"4bfbebfd-2be7-438f-b88d-273f24751f8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-5p7xw"
Oct 31 07:28:25.012504 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:25.012485    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/4bfbebfd-2be7-438f-b88d-273f24751f8c-host-cni-bin\") pod \"ovnkube-node-5p7xw\" (UID: \"4bfbebfd-2be7-438f-b88d-273f24751f8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-5p7xw"
Oct 31 07:28:25.012878 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:25.012845    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/08198bc1-314f-4881-833c-e633bac15e4a-host-run-netns\") pod \"multus-pmnqx\" (UID: \"08198bc1-314f-4881-833c-e633bac15e4a\") " pod="openshift-multus/multus-pmnqx"
Oct 31 07:28:25.012934 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:25.012883    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3fa078c7-4bf2-4e76-9692-ae460e8fadfa-var-lib-kubelet\") pod \"tuned-cpn4m\" (UID: \"3fa078c7-4bf2-4e76-9692-ae460e8fadfa\") " pod="openshift-cluster-node-tuning-operator/tuned-cpn4m"
Oct 31 07:28:25.012968 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:25.012951    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/4bfbebfd-2be7-438f-b88d-273f24751f8c-ovnkube-config\") pod \"ovnkube-node-5p7xw\" (UID: \"4bfbebfd-2be7-438f-b88d-273f24751f8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-5p7xw"
Oct 31 07:28:25.012999 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:25.012989    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5-konnectivity-ca\") pod \"konnectivity-agent-h4k2m\" (UID: \"2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5\") " pod="kube-system/konnectivity-agent-h4k2m"
Oct 31 07:28:25.013031 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:25.013012    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/08198bc1-314f-4881-833c-e633bac15e4a-cnibin\") pod \"multus-pmnqx\" (UID: \"08198bc1-314f-4881-833c-e633bac15e4a\") " pod="openshift-multus/multus-pmnqx"
Oct 31 07:28:25.013063 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:25.013052    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/08198bc1-314f-4881-833c-e633bac15e4a-host-var-lib-cni-multus\") pod \"multus-pmnqx\" (UID: \"08198bc1-314f-4881-833c-e633bac15e4a\") " pod="openshift-multus/multus-pmnqx"
Oct 31 07:28:25.013153 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:25.013084    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/9dad8dd1-cfc5-412d-bf9a-bac3c9162af1-socket-dir\") pod \"aws-ebs-csi-driver-node-797gj\" (UID: \"9dad8dd1-cfc5-412d-bf9a-bac3c9162af1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-797gj"
Oct 31 07:28:25.013153 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:25.013110    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ghkb9\" (UniqueName: \"kubernetes.io/projected/f1ba68f6-d87c-4492-a5eb-5f2d3a60eb1c-kube-api-access-ghkb9\") pod \"network-check-target-9zp4l\" (UID: \"f1ba68f6-d87c-4492-a5eb-5f2d3a60eb1c\") " pod="openshift-network-diagnostics/network-check-target-9zp4l"
Oct 31 07:28:25.013153 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:25.013142    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9c72279a-708f-46df-8915-9995c5aea14d-system-cni-dir\") pod \"multus-additional-cni-plugins-dh6w8\" (UID: \"9c72279a-708f-46df-8915-9995c5aea14d\") " pod="openshift-multus/multus-additional-cni-plugins-dh6w8"
Oct 31 07:28:25.013247 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:25.013163    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/3fa078c7-4bf2-4e76-9692-ae460e8fadfa-etc-sysconfig\") pod \"tuned-cpn4m\" (UID: \"3fa078c7-4bf2-4e76-9692-ae460e8fadfa\") " pod="openshift-cluster-node-tuning-operator/tuned-cpn4m"
Oct 31 07:28:25.013247 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:25.013185    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/3fa078c7-4bf2-4e76-9692-ae460e8fadfa-etc-systemd\") pod \"tuned-cpn4m\" (UID: \"3fa078c7-4bf2-4e76-9692-ae460e8fadfa\") " pod="openshift-cluster-node-tuning-operator/tuned-cpn4m"
Oct 31 07:28:25.013247 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:25.013212    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4bfbebfd-2be7-438f-b88d-273f24751f8c-var-lib-openvswitch\") pod \"ovnkube-node-5p7xw\" (UID: \"4bfbebfd-2be7-438f-b88d-273f24751f8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-5p7xw"
Oct 31 07:28:25.013247 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:25.013239    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4bfbebfd-2be7-438f-b88d-273f24751f8c-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-5p7xw\" (UID: \"4bfbebfd-2be7-438f-b88d-273f24751f8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-5p7xw"
Oct 31 07:28:25.013391 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:25.013265    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/fc64d31a-f48a-4293-a4de-262045d0b7e8-host\") pod \"node-ca-th7x7\" (UID: \"fc64d31a-f48a-4293-a4de-262045d0b7e8\") " pod="openshift-image-registry/node-ca-th7x7"
Oct 31 07:28:25.013391 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:25.013279    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jbdqx\" (UniqueName: \"kubernetes.io/projected/26659955-3909-4171-90b8-21b0ad6eef64-kube-api-access-jbdqx\") pod \"iptables-alerter-65hzp\" (UID: \"26659955-3909-4171-90b8-21b0ad6eef64\") " pod="openshift-network-operator/iptables-alerter-65hzp"
Oct 31 07:28:25.013391 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:25.013370    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/4bfbebfd-2be7-438f-b88d-273f24751f8c-node-log\") pod \"ovnkube-node-5p7xw\" (UID: \"4bfbebfd-2be7-438f-b88d-273f24751f8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-5p7xw"
Oct 31 07:28:25.013491 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:25.013413    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4bfbebfd-2be7-438f-b88d-273f24751f8c-host-run-ovn-kubernetes\") pod \"ovnkube-node-5p7xw\" (UID: \"4bfbebfd-2be7-438f-b88d-273f24751f8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-5p7xw"
Oct 31 07:28:25.013491 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:25.013439    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/08198bc1-314f-4881-833c-e633bac15e4a-multus-conf-dir\") pod \"multus-pmnqx\" (UID: \"08198bc1-314f-4881-833c-e633bac15e4a\") " pod="openshift-multus/multus-pmnqx"
Oct 31 07:28:25.013491 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:25.013462    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/77256436-39ff-4c2a-8390-4e9fa7d68486-kubelet-config\") pod \"global-pull-secret-syncer-kht4l\" (UID: \"77256436-39ff-4c2a-8390-4e9fa7d68486\") " pod="kube-system/global-pull-secret-syncer-kht4l"
Oct 31 07:28:25.013491 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:25.013479    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/3fa078c7-4bf2-4e76-9692-ae460e8fadfa-host\") pod \"tuned-cpn4m\" (UID: \"3fa078c7-4bf2-4e76-9692-ae460e8fadfa\") " pod="openshift-cluster-node-tuning-operator/tuned-cpn4m"
Oct 31 07:28:25.013624 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:25.013504    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/77256436-39ff-4c2a-8390-4e9fa7d68486-original-pull-secret\") pod \"global-pull-secret-syncer-kht4l\" (UID: \"77256436-39ff-4c2a-8390-4e9fa7d68486\") " pod="kube-system/global-pull-secret-syncer-kht4l"
Oct 31 07:28:25.013624 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:25.013559    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kxwb9\" (UniqueName: \"kubernetes.io/projected/c3ad95fd-e997-459c-93fd-27c4c030ea0d-kube-api-access-kxwb9\") pod \"network-metrics-daemon-r2zbv\" (UID: \"c3ad95fd-e997-459c-93fd-27c4c030ea0d\") " pod="openshift-multus/network-metrics-daemon-r2zbv"
Oct 31 07:28:25.013624 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:25.013582    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9c72279a-708f-46df-8915-9995c5aea14d-tuning-conf-dir\") pod \"multus-additional-cni-plugins-dh6w8\" (UID: \"9c72279a-708f-46df-8915-9995c5aea14d\") " pod="openshift-multus/multus-additional-cni-plugins-dh6w8"
Oct 31 07:28:25.013624 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:25.013600    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h4gzh\" (UniqueName: \"kubernetes.io/projected/9c72279a-708f-46df-8915-9995c5aea14d-kube-api-access-h4gzh\") pod \"multus-additional-cni-plugins-dh6w8\" (UID: \"9c72279a-708f-46df-8915-9995c5aea14d\") " pod="openshift-multus/multus-additional-cni-plugins-dh6w8"
Oct 31 07:28:25.013830 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:25.013627    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c4233c69-4213-43be-8809-76612b6b6f3e-tmp-dir\") pod \"node-resolver-tnw4j\" (UID: \"c4233c69-4213-43be-8809-76612b6b6f3e\") " pod="openshift-dns/node-resolver-tnw4j"
Oct 31 07:28:25.013830 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:25.013663    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/3fa078c7-4bf2-4e76-9692-ae460e8fadfa-etc-modprobe-d\") pod \"tuned-cpn4m\" (UID: \"3fa078c7-4bf2-4e76-9692-ae460e8fadfa\") " pod="openshift-cluster-node-tuning-operator/tuned-cpn4m"
Oct 31 07:28:25.013830 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:25.013689    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/4bfbebfd-2be7-438f-b88d-273f24751f8c-run-systemd\") pod \"ovnkube-node-5p7xw\" (UID: \"4bfbebfd-2be7-438f-b88d-273f24751f8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-5p7xw"
Oct 31 07:28:25.013830 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:25.013705    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/08198bc1-314f-4881-833c-e633bac15e4a-multus-cni-dir\") pod \"multus-pmnqx\" (UID: \"08198bc1-314f-4881-833c-e633bac15e4a\") " pod="openshift-multus/multus-pmnqx"
Oct 31 07:28:25.013830 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:25.013747    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/9dad8dd1-cfc5-412d-bf9a-bac3c9162af1-etc-selinux\") pod \"aws-ebs-csi-driver-node-797gj\" (UID: \"9dad8dd1-cfc5-412d-bf9a-bac3c9162af1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-797gj"
Oct 31 07:28:25.013830 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:25.013797    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9c72279a-708f-46df-8915-9995c5aea14d-cnibin\") pod \"multus-additional-cni-plugins-dh6w8\" (UID: \"9c72279a-708f-46df-8915-9995c5aea14d\") " pod="openshift-multus/multus-additional-cni-plugins-dh6w8"
Oct 31 07:28:25.013830 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:25.013818    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/08198bc1-314f-4881-833c-e633bac15e4a-multus-socket-dir-parent\") pod \"multus-pmnqx\" (UID: \"08198bc1-314f-4881-833c-e633bac15e4a\") " pod="openshift-multus/multus-pmnqx"
Oct 31 07:28:25.014032 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:25.013846    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/08198bc1-314f-4881-833c-e633bac15e4a-host-run-k8s-cni-cncf-io\") pod \"multus-pmnqx\" (UID: \"08198bc1-314f-4881-833c-e633bac15e4a\") " pod="openshift-multus/multus-pmnqx"
Oct 31 07:28:25.041407 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:25.041388    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Oct 31 07:28:25.041625 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:25.041611    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Oct 31 07:28:25.041663 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:25.041652    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-7fk4f\""
Oct 31 07:28:25.044468 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:25.044454    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-pvmcp\""
Oct 31 07:28:25.044521 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:25.044484    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Oct 31 07:28:25.044521 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:25.044496    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Oct 31 07:28:25.044521 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:25.044512    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Oct 31 07:28:25.044647 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:25.044535    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Oct 31 07:28:25.044647 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:25.044547    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Oct 31 07:28:25.044647 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:25.044556    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Oct 31 07:28:25.045943 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:25.045927    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-ldrdc\""
Oct 31 07:28:25.046009 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:25.045998    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Oct 31 07:28:25.046009 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:25.046006    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Oct 31 07:28:25.046101 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:25.046043    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Oct 31 07:28:25.048261 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:25.048249    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Oct 31 07:28:25.048319 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:25.048298    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Oct 31 07:28:25.048319 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:25.048311    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-jtbjn\""
Oct 31 07:28:25.048426 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:25.048358    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-798mv\""
Oct 31 07:28:25.048426 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:25.048374    2573 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Oct 31 07:28:25.048426 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:25.048420    2573 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-m59vf"
Oct 31 07:28:25.114583 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:25.114560    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/9dad8dd1-cfc5-412d-bf9a-bac3c9162af1-device-dir\") pod \"aws-ebs-csi-driver-node-797gj\" (UID: \"9dad8dd1-cfc5-412d-bf9a-bac3c9162af1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-797gj"
Oct 31 07:28:25.114673 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:25.114584    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/26659955-3909-4171-90b8-21b0ad6eef64-host-slash\") pod \"iptables-alerter-65hzp\" (UID: \"26659955-3909-4171-90b8-21b0ad6eef64\") " pod="openshift-network-operator/iptables-alerter-65hzp"
Oct 31 07:28:25.114673 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:25.114599    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/4bfbebfd-2be7-438f-b88d-273f24751f8c-env-overrides\") pod \"ovnkube-node-5p7xw\" (UID: \"4bfbebfd-2be7-438f-b88d-273f24751f8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-5p7xw"
Oct 31 07:28:25.114673 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:25.114614    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/08198bc1-314f-4881-833c-e633bac15e4a-cni-binary-copy\") pod \"multus-pmnqx\" (UID: \"08198bc1-314f-4881-833c-e633bac15e4a\") " pod="openshift-multus/multus-pmnqx"
Oct 31 07:28:25.114673 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:25.114628    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/08198bc1-314f-4881-833c-e633bac15e4a-multus-daemon-config\") pod \"multus-pmnqx\" (UID: \"08198bc1-314f-4881-833c-e633bac15e4a\") " pod="openshift-multus/multus-pmnqx"
Oct 31 07:28:25.114824 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:25.114673    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/26659955-3909-4171-90b8-21b0ad6eef64-host-slash\") pod \"iptables-alerter-65hzp\" (UID: \"26659955-3909-4171-90b8-21b0ad6eef64\") " pod="openshift-network-operator/iptables-alerter-65hzp"
Oct 31 07:28:25.114824 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:25.114693    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/9dad8dd1-cfc5-412d-bf9a-bac3c9162af1-device-dir\") pod \"aws-ebs-csi-driver-node-797gj\" (UID: \"9dad8dd1-cfc5-412d-bf9a-bac3c9162af1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-797gj"
Oct 31 07:28:25.114824 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:25.114758    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3fa078c7-4bf2-4e76-9692-ae460e8fadfa-sys\") pod \"tuned-cpn4m\" (UID: \"3fa078c7-4bf2-4e76-9692-ae460e8fadfa\") " pod="openshift-cluster-node-tuning-operator/tuned-cpn4m"
Oct 31 07:28:25.114824 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:25.114786    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cq7rv\" (UniqueName: \"kubernetes.io/projected/3fa078c7-4bf2-4e76-9692-ae460e8fadfa-kube-api-access-cq7rv\") pod \"tuned-cpn4m\" (UID: \"3fa078c7-4bf2-4e76-9692-ae460e8fadfa\") " pod="openshift-cluster-node-tuning-operator/tuned-cpn4m"
Oct 31 07:28:25.114824 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:25.114812    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/9dad8dd1-cfc5-412d-bf9a-bac3c9162af1-registration-dir\") pod \"aws-ebs-csi-driver-node-797gj\" (UID: \"9dad8dd1-cfc5-412d-bf9a-bac3c9162af1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-797gj"
Oct 31 07:28:25.115031 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:25.114839    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/26659955-3909-4171-90b8-21b0ad6eef64-iptables-alerter-script\") pod \"iptables-alerter-65hzp\" (UID: \"26659955-3909-4171-90b8-21b0ad6eef64\") " pod="openshift-network-operator/iptables-alerter-65hzp"
Oct 31 07:28:25.115031 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:25.114888    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3fa078c7-4bf2-4e76-9692-ae460e8fadfa-sys\") pod \"tuned-cpn4m\" (UID: \"3fa078c7-4bf2-4e76-9692-ae460e8fadfa\") " pod="openshift-cluster-node-tuning-operator/tuned-cpn4m"
Oct 31 07:28:25.115031 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:25.114922    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/9dad8dd1-cfc5-412d-bf9a-bac3c9162af1-registration-dir\") pod \"aws-ebs-csi-driver-node-797gj\" (UID: \"9dad8dd1-cfc5-412d-bf9a-bac3c9162af1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-797gj"
Oct 31 07:28:25.115031 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:25.115000    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4bfbebfd-2be7-438f-b88d-273f24751f8c-etc-openvswitch\") pod \"ovnkube-node-5p7xw\" (UID: \"4bfbebfd-2be7-438f-b88d-273f24751f8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-5p7xw"
Oct 31 07:28:25.115235 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:25.115037    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/08198bc1-314f-4881-833c-e633bac15e4a-etc-kubernetes\") pod \"multus-pmnqx\" (UID: \"08198bc1-314f-4881-833c-e633bac15e4a\") " pod="openshift-multus/multus-pmnqx"
Oct 31 07:28:25.115235 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:25.115064    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5n5lv\" (UniqueName: \"kubernetes.io/projected/9dad8dd1-cfc5-412d-bf9a-bac3c9162af1-kube-api-access-5n5lv\") pod \"aws-ebs-csi-driver-node-797gj\" (UID: \"9dad8dd1-cfc5-412d-bf9a-bac3c9162af1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-797gj"
Oct 31 07:28:25.115235 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:25.115088    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/77256436-39ff-4c2a-8390-4e9fa7d68486-dbus\") pod \"global-pull-secret-syncer-kht4l\" (UID: \"77256436-39ff-4c2a-8390-4e9fa7d68486\") " pod="kube-system/global-pull-secret-syncer-kht4l"
Oct 31 07:28:25.115235 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:25.115113    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/c4233c69-4213-43be-8809-76612b6b6f3e-hosts-file\") pod \"node-resolver-tnw4j\" (UID: \"c4233c69-4213-43be-8809-76612b6b6f3e\") " pod="openshift-dns/node-resolver-tnw4j"
Oct 31 07:28:25.115235 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:25.115133    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/08198bc1-314f-4881-833c-e633bac15e4a-etc-kubernetes\") pod \"multus-pmnqx\" (UID: \"08198bc1-314f-4881-833c-e633bac15e4a\") " pod="openshift-multus/multus-pmnqx"
Oct 31 07:28:25.115235 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:25.115138    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c3ad95fd-e997-459c-93fd-27c4c030ea0d-metrics-certs\") pod \"network-metrics-daemon-r2zbv\" (UID: \"c3ad95fd-e997-459c-93fd-27c4c030ea0d\") " pod="openshift-multus/network-metrics-daemon-r2zbv"
Oct 31 07:28:25.115235 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:25.115192    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4bfbebfd-2be7-438f-b88d-273f24751f8c-run-openvswitch\") pod \"ovnkube-node-5p7xw\" (UID: \"4bfbebfd-2be7-438f-b88d-273f24751f8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-5p7xw"
Oct 31 07:28:25.115235 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:25.115197    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4bfbebfd-2be7-438f-b88d-273f24751f8c-etc-openvswitch\") pod \"ovnkube-node-5p7xw\" (UID: \"4bfbebfd-2be7-438f-b88d-273f24751f8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-5p7xw"
Oct 31 07:28:25.115235 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:25.115216    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/4bfbebfd-2be7-438f-b88d-273f24751f8c-log-socket\") pod \"ovnkube-node-5p7xw\" (UID: \"4bfbebfd-2be7-438f-b88d-273f24751f8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-5p7xw"
Oct 31 07:28:25.115235 ip-10-0-139-83 kubenswrapper[2573]: E1031 07:28:25.115221    2573 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 31 07:28:25.115235 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:25.115226    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/08198bc1-314f-4881-833c-e633bac15e4a-cni-binary-copy\") pod \"multus-pmnqx\" (UID: \"08198bc1-314f-4881-833c-e633bac15e4a\") " pod="openshift-multus/multus-pmnqx"
Oct 31 07:28:25.115775 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:25.115239    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/4bfbebfd-2be7-438f-b88d-273f24751f8c-ovn-node-metrics-cert\") pod \"ovnkube-node-5p7xw\" (UID: \"4bfbebfd-2be7-438f-b88d-273f24751f8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-5p7xw"
Oct 31 07:28:25.115775 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:25.115267    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5hqbq\" (UniqueName: \"kubernetes.io/projected/c4233c69-4213-43be-8809-76612b6b6f3e-kube-api-access-5hqbq\") pod \"node-resolver-tnw4j\" (UID: \"c4233c69-4213-43be-8809-76612b6b6f3e\") " pod="openshift-dns/node-resolver-tnw4j"
Oct 31 07:28:25.115775 ip-10-0-139-83 kubenswrapper[2573]: E1031 07:28:25.115288    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c3ad95fd-e997-459c-93fd-27c4c030ea0d-metrics-certs podName:c3ad95fd-e997-459c-93fd-27c4c030ea0d nodeName:}" failed. No retries permitted until 2026-10-31 07:28:25.615264121 +0000 UTC m=+2.306034939 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c3ad95fd-e997-459c-93fd-27c4c030ea0d-metrics-certs") pod "network-metrics-daemon-r2zbv" (UID: "c3ad95fd-e997-459c-93fd-27c4c030ea0d") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 31 07:28:25.115775 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:25.115304    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/08198bc1-314f-4881-833c-e633bac15e4a-multus-daemon-config\") pod \"multus-pmnqx\" (UID: \"08198bc1-314f-4881-833c-e633bac15e4a\") " pod="openshift-multus/multus-pmnqx"
Oct 31 07:28:25.115775 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:25.115160    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/4bfbebfd-2be7-438f-b88d-273f24751f8c-env-overrides\") pod \"ovnkube-node-5p7xw\" (UID: \"4bfbebfd-2be7-438f-b88d-273f24751f8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-5p7xw"
Oct 31 07:28:25.115775 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:25.115345    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/4bfbebfd-2be7-438f-b88d-273f24751f8c-run-ovn\") pod \"ovnkube-node-5p7xw\" (UID: \"4bfbebfd-2be7-438f-b88d-273f24751f8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-5p7xw"
Oct 31 07:28:25.115775 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:25.115352    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/c4233c69-4213-43be-8809-76612b6b6f3e-hosts-file\") pod \"node-resolver-tnw4j\" (UID: \"c4233c69-4213-43be-8809-76612b6b6f3e\") " pod="openshift-dns/node-resolver-tnw4j"
Oct 31 07:28:25.115775 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:25.115320    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4bfbebfd-2be7-438f-b88d-273f24751f8c-run-openvswitch\") pod \"ovnkube-node-5p7xw\" (UID: \"4bfbebfd-2be7-438f-b88d-273f24751f8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-5p7xw"
Oct 31 07:28:25.115775 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:25.115364    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/77256436-39ff-4c2a-8390-4e9fa7d68486-dbus\") pod \"global-pull-secret-syncer-kht4l\" (UID: \"77256436-39ff-4c2a-8390-4e9fa7d68486\") " pod="kube-system/global-pull-secret-syncer-kht4l"
Oct 31 07:28:25.115775 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:25.115377    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/4bfbebfd-2be7-438f-b88d-273f24751f8c-log-socket\") pod \"ovnkube-node-5p7xw\" (UID: \"4bfbebfd-2be7-438f-b88d-273f24751f8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-5p7xw"
Oct 31 07:28:25.115775 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:25.115386    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/4bfbebfd-2be7-438f-b88d-273f24751f8c-ovnkube-script-lib\") pod \"ovnkube-node-5p7xw\" (UID: \"4bfbebfd-2be7-438f-b88d-273f24751f8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-5p7xw"
Oct 31 07:28:25.115775 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:25.115411    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cz8tm\" (UniqueName: \"kubernetes.io/projected/4bfbebfd-2be7-438f-b88d-273f24751f8c-kube-api-access-cz8tm\") pod \"ovnkube-node-5p7xw\" (UID: \"4bfbebfd-2be7-438f-b88d-273f24751f8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-5p7xw"
Oct 31 07:28:25.115775 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:25.115426    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5-agent-certs\") pod \"konnectivity-agent-h4k2m\" (UID: \"2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5\") " pod="kube-system/konnectivity-agent-h4k2m"
Oct 31 07:28:25.115775 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:25.115442    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xs9lt\" (UniqueName: \"kubernetes.io/projected/08198bc1-314f-4881-833c-e633bac15e4a-kube-api-access-xs9lt\") pod \"multus-pmnqx\" (UID: \"08198bc1-314f-4881-833c-e633bac15e4a\") " pod="openshift-multus/multus-pmnqx"
Oct 31 07:28:25.115775 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:25.115372    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/26659955-3909-4171-90b8-21b0ad6eef64-iptables-alerter-script\") pod \"iptables-alerter-65hzp\" (UID: \"26659955-3909-4171-90b8-21b0ad6eef64\") " pod="openshift-network-operator/iptables-alerter-65hzp"
Oct 31 07:28:25.115775 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:25.115385    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/4bfbebfd-2be7-438f-b88d-273f24751f8c-run-ovn\") pod \"ovnkube-node-5p7xw\" (UID: \"4bfbebfd-2be7-438f-b88d-273f24751f8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-5p7xw"
Oct 31 07:28:25.115775 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:25.115480    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/fc64d31a-f48a-4293-a4de-262045d0b7e8-serviceca\") pod \"node-ca-th7x7\" (UID: \"fc64d31a-f48a-4293-a4de-262045d0b7e8\") " pod="openshift-image-registry/node-ca-th7x7"
Oct 31 07:28:25.116464 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:25.115507    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/3fa078c7-4bf2-4e76-9692-ae460e8fadfa-etc-sysctl-d\") pod \"tuned-cpn4m\" (UID: \"3fa078c7-4bf2-4e76-9692-ae460e8fadfa\") " pod="openshift-cluster-node-tuning-operator/tuned-cpn4m"
Oct 31 07:28:25.116464 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:25.115530    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3fa078c7-4bf2-4e76-9692-ae460e8fadfa-tmp\") pod \"tuned-cpn4m\" (UID: \"3fa078c7-4bf2-4e76-9692-ae460e8fadfa\") " pod="openshift-cluster-node-tuning-operator/tuned-cpn4m"
Oct 31 07:28:25.116464 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:25.115552    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/4bfbebfd-2be7-438f-b88d-273f24751f8c-host-run-netns\") pod \"ovnkube-node-5p7xw\" (UID: \"4bfbebfd-2be7-438f-b88d-273f24751f8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-5p7xw"
Oct 31 07:28:25.116464 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:25.115574    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/4bfbebfd-2be7-438f-b88d-273f24751f8c-host-cni-netd\") pod \"ovnkube-node-5p7xw\" (UID: \"4bfbebfd-2be7-438f-b88d-273f24751f8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-5p7xw"
Oct 31 07:28:25.116464 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:25.115598    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/9c72279a-708f-46df-8915-9995c5aea14d-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-dh6w8\" (UID: \"9c72279a-708f-46df-8915-9995c5aea14d\") " pod="openshift-multus/multus-additional-cni-plugins-dh6w8"
Oct 31 07:28:25.116464 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:25.115622    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/08198bc1-314f-4881-833c-e633bac15e4a-host-var-lib-cni-bin\") pod \"multus-pmnqx\" (UID: \"08198bc1-314f-4881-833c-e633bac15e4a\") " pod="openshift-multus/multus-pmnqx"
Oct 31 07:28:25.116464 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:25.115645    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kk4nh\" (UniqueName: \"kubernetes.io/projected/fc64d31a-f48a-4293-a4de-262045d0b7e8-kube-api-access-kk4nh\") pod \"node-ca-th7x7\" (UID: \"fc64d31a-f48a-4293-a4de-262045d0b7e8\") " pod="openshift-image-registry/node-ca-th7x7"
Oct 31 07:28:25.116464 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:25.115675    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/4bfbebfd-2be7-438f-b88d-273f24751f8c-host-cni-netd\") pod \"ovnkube-node-5p7xw\" (UID: \"4bfbebfd-2be7-438f-b88d-273f24751f8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-5p7xw"
Oct 31 07:28:25.116464 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:25.115687    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3fa078c7-4bf2-4e76-9692-ae460e8fadfa-etc-kubernetes\") pod \"tuned-cpn4m\" (UID: \"3fa078c7-4bf2-4e76-9692-ae460e8fadfa\") " pod="openshift-cluster-node-tuning-operator/tuned-cpn4m"
Oct 31 07:28:25.116464 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:25.115695    2573 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Oct 31 07:28:25.116464 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:25.115719    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/9dad8dd1-cfc5-412d-bf9a-bac3c9162af1-sys-fs\") pod \"aws-ebs-csi-driver-node-797gj\" (UID: \"9dad8dd1-cfc5-412d-bf9a-bac3c9162af1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-797gj"
Oct 31 07:28:25.116464 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:25.115739    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3fa078c7-4bf2-4e76-9692-ae460e8fadfa-etc-kubernetes\") pod \"tuned-cpn4m\" (UID: \"3fa078c7-4bf2-4e76-9692-ae460e8fadfa\") " pod="openshift-cluster-node-tuning-operator/tuned-cpn4m"
Oct 31 07:28:25.116464 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:25.115746    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/3fa078c7-4bf2-4e76-9692-ae460e8fadfa-etc-sysctl-d\") pod \"tuned-cpn4m\" (UID: \"3fa078c7-4bf2-4e76-9692-ae460e8fadfa\") " pod="openshift-cluster-node-tuning-operator/tuned-cpn4m"
Oct 31 07:28:25.116464 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:25.115812    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/9dad8dd1-cfc5-412d-bf9a-bac3c9162af1-sys-fs\") pod \"aws-ebs-csi-driver-node-797gj\" (UID: \"9dad8dd1-cfc5-412d-bf9a-bac3c9162af1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-797gj"
Oct 31 07:28:25.116464 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:25.115814    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/08198bc1-314f-4881-833c-e633bac15e4a-host-var-lib-cni-bin\") pod \"multus-pmnqx\" (UID: \"08198bc1-314f-4881-833c-e633bac15e4a\") " pod="openshift-multus/multus-pmnqx"
Oct 31 07:28:25.116464 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:25.115745    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/9c72279a-708f-46df-8915-9995c5aea14d-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-dh6w8\" (UID: \"9c72279a-708f-46df-8915-9995c5aea14d\") " pod="openshift-multus/multus-additional-cni-plugins-dh6w8"
Oct 31 07:28:25.116464 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:25.115850    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/08198bc1-314f-4881-833c-e633bac15e4a-host-var-lib-kubelet\") pod \"multus-pmnqx\" (UID: \"08198bc1-314f-4881-833c-e633bac15e4a\") " pod="openshift-multus/multus-pmnqx"
Oct 31 07:28:25.116464 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:25.115856    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/4bfbebfd-2be7-438f-b88d-273f24751f8c-host-run-netns\") pod \"ovnkube-node-5p7xw\" (UID: \"4bfbebfd-2be7-438f-b88d-273f24751f8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-5p7xw"
Oct 31 07:28:25.117416 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:25.115876    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/4bfbebfd-2be7-438f-b88d-273f24751f8c-host-kubelet\") pod \"ovnkube-node-5p7xw\" (UID: \"4bfbebfd-2be7-438f-b88d-273f24751f8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-5p7xw"
Oct 31 07:28:25.117416 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:25.115898    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4bfbebfd-2be7-438f-b88d-273f24751f8c-host-slash\") pod \"ovnkube-node-5p7xw\" (UID: \"4bfbebfd-2be7-438f-b88d-273f24751f8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-5p7xw"
Oct 31 07:28:25.117416 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:25.115922    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/9c72279a-708f-46df-8915-9995c5aea14d-cni-binary-copy\") pod \"multus-additional-cni-plugins-dh6w8\" (UID: \"9c72279a-708f-46df-8915-9995c5aea14d\") " pod="openshift-multus/multus-additional-cni-plugins-dh6w8"
Oct 31 07:28:25.117416 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:25.115932    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/4bfbebfd-2be7-438f-b88d-273f24751f8c-ovnkube-script-lib\") pod \"ovnkube-node-5p7xw\" (UID: \"4bfbebfd-2be7-438f-b88d-273f24751f8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-5p7xw"
Oct 31 07:28:25.117416 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:25.115947    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/fc64d31a-f48a-4293-a4de-262045d0b7e8-serviceca\") pod \"node-ca-th7x7\" (UID: \"fc64d31a-f48a-4293-a4de-262045d0b7e8\") " pod="openshift-image-registry/node-ca-th7x7"
Oct 31 07:28:25.117416 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:25.115945    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/08198bc1-314f-4881-833c-e633bac15e4a-system-cni-dir\") pod \"multus-pmnqx\" (UID: \"08198bc1-314f-4881-833c-e633bac15e4a\") " pod="openshift-multus/multus-pmnqx"
Oct 31 07:28:25.117416 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:25.115981    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/08198bc1-314f-4881-833c-e633bac15e4a-host-var-lib-kubelet\") pod \"multus-pmnqx\" (UID: \"08198bc1-314f-4881-833c-e633bac15e4a\") " pod="openshift-multus/multus-pmnqx"
Oct 31 07:28:25.117416 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:25.115990    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/08198bc1-314f-4881-833c-e633bac15e4a-hostroot\") pod \"multus-pmnqx\" (UID: \"08198bc1-314f-4881-833c-e633bac15e4a\") " pod="openshift-multus/multus-pmnqx"
Oct 31 07:28:25.117416 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:25.115997    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/4bfbebfd-2be7-438f-b88d-273f24751f8c-host-kubelet\") pod \"ovnkube-node-5p7xw\" (UID: \"4bfbebfd-2be7-438f-b88d-273f24751f8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-5p7xw"
Oct 31 07:28:25.117416 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:25.116030    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/08198bc1-314f-4881-833c-e633bac15e4a-host-run-multus-certs\") pod \"multus-pmnqx\" (UID: \"08198bc1-314f-4881-833c-e633bac15e4a\") " pod="openshift-multus/multus-pmnqx"
Oct 31 07:28:25.117416 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:25.116036    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/08198bc1-314f-4881-833c-e633bac15e4a-hostroot\") pod \"multus-pmnqx\" (UID: \"08198bc1-314f-4881-833c-e633bac15e4a\") " pod="openshift-multus/multus-pmnqx"
Oct 31 07:28:25.117416 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:25.116041    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4bfbebfd-2be7-438f-b88d-273f24751f8c-host-slash\") pod \"ovnkube-node-5p7xw\" (UID: \"4bfbebfd-2be7-438f-b88d-273f24751f8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-5p7xw"
Oct 31 07:28:25.117416 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:25.116054    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/3fa078c7-4bf2-4e76-9692-ae460e8fadfa-etc-sysctl-conf\") pod \"tuned-cpn4m\" (UID: \"3fa078c7-4bf2-4e76-9692-ae460e8fadfa\") " pod="openshift-cluster-node-tuning-operator/tuned-cpn4m"
Oct 31 07:28:25.117416 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:25.116076    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/08198bc1-314f-4881-833c-e633bac15e4a-system-cni-dir\") pod \"multus-pmnqx\" (UID: \"08198bc1-314f-4881-833c-e633bac15e4a\") " pod="openshift-multus/multus-pmnqx"
Oct 31 07:28:25.117416 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:25.116079    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/9dad8dd1-cfc5-412d-bf9a-bac3c9162af1-kubelet-dir\") pod \"aws-ebs-csi-driver-node-797gj\" (UID: \"9dad8dd1-cfc5-412d-bf9a-bac3c9162af1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-797gj"
Oct 31 07:28:25.117416 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:25.116077    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/08198bc1-314f-4881-833c-e633bac15e4a-host-run-multus-certs\") pod \"multus-pmnqx\" (UID: \"08198bc1-314f-4881-833c-e633bac15e4a\") " pod="openshift-multus/multus-pmnqx"
Oct 31 07:28:25.117416 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:25.116102    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/4bfbebfd-2be7-438f-b88d-273f24751f8c-systemd-units\") pod \"ovnkube-node-5p7xw\" (UID: \"4bfbebfd-2be7-438f-b88d-273f24751f8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-5p7xw"
Oct 31 07:28:25.117416 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:25.116122    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/4bfbebfd-2be7-438f-b88d-273f24751f8c-host-cni-bin\") pod \"ovnkube-node-5p7xw\" (UID: \"4bfbebfd-2be7-438f-b88d-273f24751f8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-5p7xw"
Oct 31 07:28:25.118285 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:25.116143    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/08198bc1-314f-4881-833c-e633bac15e4a-host-run-netns\") pod \"multus-pmnqx\" (UID: \"08198bc1-314f-4881-833c-e633bac15e4a\") " pod="openshift-multus/multus-pmnqx"
Oct 31 07:28:25.118285 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:25.116163    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3fa078c7-4bf2-4e76-9692-ae460e8fadfa-var-lib-kubelet\") pod \"tuned-cpn4m\" (UID: \"3fa078c7-4bf2-4e76-9692-ae460e8fadfa\") " pod="openshift-cluster-node-tuning-operator/tuned-cpn4m"
Oct 31 07:28:25.118285 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:25.116186    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/4bfbebfd-2be7-438f-b88d-273f24751f8c-ovnkube-config\") pod \"ovnkube-node-5p7xw\" (UID: \"4bfbebfd-2be7-438f-b88d-273f24751f8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-5p7xw"
Oct 31 07:28:25.118285 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:25.116200    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/3fa078c7-4bf2-4e76-9692-ae460e8fadfa-etc-sysctl-conf\") pod \"tuned-cpn4m\" (UID: \"3fa078c7-4bf2-4e76-9692-ae460e8fadfa\") " pod="openshift-cluster-node-tuning-operator/tuned-cpn4m"
Oct 31 07:28:25.118285 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:25.116210    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5-konnectivity-ca\") pod \"konnectivity-agent-h4k2m\" (UID: \"2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5\") " pod="kube-system/konnectivity-agent-h4k2m"
Oct 31 07:28:25.118285 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:25.116239    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/4bfbebfd-2be7-438f-b88d-273f24751f8c-host-cni-bin\") pod \"ovnkube-node-5p7xw\" (UID: \"4bfbebfd-2be7-438f-b88d-273f24751f8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-5p7xw"
Oct 31 07:28:25.118285 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:25.116233    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/08198bc1-314f-4881-833c-e633bac15e4a-cnibin\") pod \"multus-pmnqx\" (UID: \"08198bc1-314f-4881-833c-e633bac15e4a\") " pod="openshift-multus/multus-pmnqx"
Oct 31 07:28:25.118285 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:25.116283    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/9dad8dd1-cfc5-412d-bf9a-bac3c9162af1-kubelet-dir\") pod \"aws-ebs-csi-driver-node-797gj\" (UID: \"9dad8dd1-cfc5-412d-bf9a-bac3c9162af1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-797gj"
Oct 31 07:28:25.118285 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:25.116295    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/08198bc1-314f-4881-833c-e633bac15e4a-host-var-lib-cni-multus\") pod \"multus-pmnqx\" (UID: \"08198bc1-314f-4881-833c-e633bac15e4a\") " pod="openshift-multus/multus-pmnqx"
Oct 31 07:28:25.118285 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:25.116317    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/4bfbebfd-2be7-438f-b88d-273f24751f8c-systemd-units\") pod \"ovnkube-node-5p7xw\" (UID: \"4bfbebfd-2be7-438f-b88d-273f24751f8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-5p7xw"
Oct 31 07:28:25.118285 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:25.116320    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/9dad8dd1-cfc5-412d-bf9a-bac3c9162af1-socket-dir\") pod \"aws-ebs-csi-driver-node-797gj\" (UID: \"9dad8dd1-cfc5-412d-bf9a-bac3c9162af1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-797gj"
Oct 31 07:28:25.118285 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:25.116367    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ghkb9\" (UniqueName: \"kubernetes.io/projected/f1ba68f6-d87c-4492-a5eb-5f2d3a60eb1c-kube-api-access-ghkb9\") pod \"network-check-target-9zp4l\" (UID: \"f1ba68f6-d87c-4492-a5eb-5f2d3a60eb1c\") " pod="openshift-network-diagnostics/network-check-target-9zp4l"
Oct 31 07:28:25.118285 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:25.116382    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/9c72279a-708f-46df-8915-9995c5aea14d-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-dh6w8\" (UID: \"9c72279a-708f-46df-8915-9995c5aea14d\") " pod="openshift-multus/multus-additional-cni-plugins-dh6w8"
Oct 31 07:28:25.118285 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:25.116394    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9c72279a-708f-46df-8915-9995c5aea14d-system-cni-dir\") pod \"multus-additional-cni-plugins-dh6w8\" (UID: \"9c72279a-708f-46df-8915-9995c5aea14d\") " pod="openshift-multus/multus-additional-cni-plugins-dh6w8"
Oct 31 07:28:25.118285 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:25.116413    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/08198bc1-314f-4881-833c-e633bac15e4a-host-run-netns\") pod \"multus-pmnqx\" (UID: \"08198bc1-314f-4881-833c-e633bac15e4a\") " pod="openshift-multus/multus-pmnqx"
Oct 31 07:28:25.118285 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:25.116417    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/3fa078c7-4bf2-4e76-9692-ae460e8fadfa-etc-sysconfig\") pod \"tuned-cpn4m\" (UID: \"3fa078c7-4bf2-4e76-9692-ae460e8fadfa\") " pod="openshift-cluster-node-tuning-operator/tuned-cpn4m"
Oct 31 07:28:25.118285 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:25.116454    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3fa078c7-4bf2-4e76-9692-ae460e8fadfa-var-lib-kubelet\") pod \"tuned-cpn4m\" (UID: \"3fa078c7-4bf2-4e76-9692-ae460e8fadfa\") " pod="openshift-cluster-node-tuning-operator/tuned-cpn4m"
Oct 31 07:28:25.119066 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:25.116463    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/3fa078c7-4bf2-4e76-9692-ae460e8fadfa-etc-systemd\") pod \"tuned-cpn4m\" (UID: \"3fa078c7-4bf2-4e76-9692-ae460e8fadfa\") " pod="openshift-cluster-node-tuning-operator/tuned-cpn4m"
Oct 31 07:28:25.119066 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:25.116485    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4bfbebfd-2be7-438f-b88d-273f24751f8c-var-lib-openvswitch\") pod \"ovnkube-node-5p7xw\" (UID: \"4bfbebfd-2be7-438f-b88d-273f24751f8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-5p7xw"
Oct 31 07:28:25.119066 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:25.116509    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4bfbebfd-2be7-438f-b88d-273f24751f8c-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-5p7xw\" (UID: \"4bfbebfd-2be7-438f-b88d-273f24751f8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-5p7xw"
Oct 31 07:28:25.119066 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:25.116515    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/9c72279a-708f-46df-8915-9995c5aea14d-cni-binary-copy\") pod \"multus-additional-cni-plugins-dh6w8\" (UID: \"9c72279a-708f-46df-8915-9995c5aea14d\") " pod="openshift-multus/multus-additional-cni-plugins-dh6w8"
Oct 31 07:28:25.119066 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:25.116366    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/9c72279a-708f-46df-8915-9995c5aea14d-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-dh6w8\" (UID: \"9c72279a-708f-46df-8915-9995c5aea14d\") " pod="openshift-multus/multus-additional-cni-plugins-dh6w8"
Oct 31 07:28:25.119066 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:25.116542    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/9dad8dd1-cfc5-412d-bf9a-bac3c9162af1-socket-dir\") pod \"aws-ebs-csi-driver-node-797gj\" (UID: \"9dad8dd1-cfc5-412d-bf9a-bac3c9162af1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-797gj"
Oct 31 07:28:25.119066 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:25.116535    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/fc64d31a-f48a-4293-a4de-262045d0b7e8-host\") pod \"node-ca-th7x7\" (UID: \"fc64d31a-f48a-4293-a4de-262045d0b7e8\") " pod="openshift-image-registry/node-ca-th7x7"
Oct 31 07:28:25.119066 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:25.116570    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/fc64d31a-f48a-4293-a4de-262045d0b7e8-host\") pod \"node-ca-th7x7\" (UID: \"fc64d31a-f48a-4293-a4de-262045d0b7e8\") " pod="openshift-image-registry/node-ca-th7x7"
Oct 31 07:28:25.119066 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:25.116585    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jbdqx\" (UniqueName: \"kubernetes.io/projected/26659955-3909-4171-90b8-21b0ad6eef64-kube-api-access-jbdqx\") pod \"iptables-alerter-65hzp\" (UID: \"26659955-3909-4171-90b8-21b0ad6eef64\") " pod="openshift-network-operator/iptables-alerter-65hzp"
Oct 31 07:28:25.119066 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:25.116610    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/4bfbebfd-2be7-438f-b88d-273f24751f8c-node-log\") pod \"ovnkube-node-5p7xw\" (UID: \"4bfbebfd-2be7-438f-b88d-273f24751f8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-5p7xw"
Oct 31 07:28:25.119066 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:25.116613    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/3fa078c7-4bf2-4e76-9692-ae460e8fadfa-etc-systemd\") pod \"tuned-cpn4m\" (UID: \"3fa078c7-4bf2-4e76-9692-ae460e8fadfa\") " pod="openshift-cluster-node-tuning-operator/tuned-cpn4m"
Oct 31 07:28:25.119066 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:25.116634    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4bfbebfd-2be7-438f-b88d-273f24751f8c-host-run-ovn-kubernetes\") pod \"ovnkube-node-5p7xw\" (UID: \"4bfbebfd-2be7-438f-b88d-273f24751f8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-5p7xw"
Oct 31 07:28:25.119066 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:25.116649    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4bfbebfd-2be7-438f-b88d-273f24751f8c-var-lib-openvswitch\") pod \"ovnkube-node-5p7xw\" (UID: \"4bfbebfd-2be7-438f-b88d-273f24751f8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-5p7xw"
Oct 31 07:28:25.119066 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:25.116661    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/08198bc1-314f-4881-833c-e633bac15e4a-multus-conf-dir\") pod \"multus-pmnqx\" (UID: \"08198bc1-314f-4881-833c-e633bac15e4a\") " pod="openshift-multus/multus-pmnqx"
Oct 31 07:28:25.119066 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:25.116684    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4bfbebfd-2be7-438f-b88d-273f24751f8c-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-5p7xw\" (UID: \"4bfbebfd-2be7-438f-b88d-273f24751f8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-5p7xw"
Oct 31 07:28:25.119066 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:25.116765    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/08198bc1-314f-4881-833c-e633bac15e4a-cnibin\") pod \"multus-pmnqx\" (UID: \"08198bc1-314f-4881-833c-e633bac15e4a\") " pod="openshift-multus/multus-pmnqx"
Oct 31 07:28:25.119066 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:25.116771    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/08198bc1-314f-4881-833c-e633bac15e4a-host-var-lib-cni-multus\") pod \"multus-pmnqx\" (UID: \"08198bc1-314f-4881-833c-e633bac15e4a\") " pod="openshift-multus/multus-pmnqx"
Oct 31 07:28:25.119541 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:25.116808    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9c72279a-708f-46df-8915-9995c5aea14d-system-cni-dir\") pod \"multus-additional-cni-plugins-dh6w8\" (UID: \"9c72279a-708f-46df-8915-9995c5aea14d\") " pod="openshift-multus/multus-additional-cni-plugins-dh6w8"
Oct 31 07:28:25.119541 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:25.116850    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/4bfbebfd-2be7-438f-b88d-273f24751f8c-node-log\") pod \"ovnkube-node-5p7xw\" (UID: \"4bfbebfd-2be7-438f-b88d-273f24751f8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-5p7xw"
Oct 31 07:28:25.119541 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:25.116851    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/77256436-39ff-4c2a-8390-4e9fa7d68486-kubelet-config\") pod \"global-pull-secret-syncer-kht4l\" (UID: \"77256436-39ff-4c2a-8390-4e9fa7d68486\") " pod="kube-system/global-pull-secret-syncer-kht4l"
Oct 31 07:28:25.119541 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:25.116873    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/08198bc1-314f-4881-833c-e633bac15e4a-multus-conf-dir\") pod \"multus-pmnqx\" (UID: \"08198bc1-314f-4881-833c-e633bac15e4a\") " pod="openshift-multus/multus-pmnqx"
Oct 31 07:28:25.119541 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:25.116888    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4bfbebfd-2be7-438f-b88d-273f24751f8c-host-run-ovn-kubernetes\") pod \"ovnkube-node-5p7xw\" (UID: \"4bfbebfd-2be7-438f-b88d-273f24751f8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-5p7xw"
Oct 31 07:28:25.119541 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:25.116910    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/77256436-39ff-4c2a-8390-4e9fa7d68486-kubelet-config\") pod \"global-pull-secret-syncer-kht4l\" (UID: \"77256436-39ff-4c2a-8390-4e9fa7d68486\") " pod="kube-system/global-pull-secret-syncer-kht4l"
Oct 31 07:28:25.119541 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:25.116915    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/3fa078c7-4bf2-4e76-9692-ae460e8fadfa-host\") pod \"tuned-cpn4m\" (UID: \"3fa078c7-4bf2-4e76-9692-ae460e8fadfa\") " pod="openshift-cluster-node-tuning-operator/tuned-cpn4m"
Oct 31 07:28:25.119541 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:25.116937    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/3fa078c7-4bf2-4e76-9692-ae460e8fadfa-host\") pod \"tuned-cpn4m\" (UID: \"3fa078c7-4bf2-4e76-9692-ae460e8fadfa\") " pod="openshift-cluster-node-tuning-operator/tuned-cpn4m"
Oct 31 07:28:25.119541 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:25.116947    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/77256436-39ff-4c2a-8390-4e9fa7d68486-original-pull-secret\") pod \"global-pull-secret-syncer-kht4l\" (UID: \"77256436-39ff-4c2a-8390-4e9fa7d68486\") " pod="kube-system/global-pull-secret-syncer-kht4l"
Oct 31 07:28:25.119541 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:25.116964    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/3fa078c7-4bf2-4e76-9692-ae460e8fadfa-etc-sysconfig\") pod \"tuned-cpn4m\" (UID: \"3fa078c7-4bf2-4e76-9692-ae460e8fadfa\") " pod="openshift-cluster-node-tuning-operator/tuned-cpn4m"
Oct 31 07:28:25.119541 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:25.116972    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kxwb9\" (UniqueName: \"kubernetes.io/projected/c3ad95fd-e997-459c-93fd-27c4c030ea0d-kube-api-access-kxwb9\") pod \"network-metrics-daemon-r2zbv\" (UID: \"c3ad95fd-e997-459c-93fd-27c4c030ea0d\") " pod="openshift-multus/network-metrics-daemon-r2zbv"
Oct 31 07:28:25.119541 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:25.116996    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9c72279a-708f-46df-8915-9995c5aea14d-tuning-conf-dir\") pod \"multus-additional-cni-plugins-dh6w8\" (UID: \"9c72279a-708f-46df-8915-9995c5aea14d\") " pod="openshift-multus/multus-additional-cni-plugins-dh6w8"
Oct 31 07:28:25.119541 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:25.117017    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h4gzh\" (UniqueName: \"kubernetes.io/projected/9c72279a-708f-46df-8915-9995c5aea14d-kube-api-access-h4gzh\") pod \"multus-additional-cni-plugins-dh6w8\" (UID: \"9c72279a-708f-46df-8915-9995c5aea14d\") " pod="openshift-multus/multus-additional-cni-plugins-dh6w8"
Oct 31 07:28:25.119541 ip-10-0-139-83 kubenswrapper[2573]: E1031 07:28:25.117024    2573 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 31 07:28:25.119541 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:25.117039    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c4233c69-4213-43be-8809-76612b6b6f3e-tmp-dir\") pod \"node-resolver-tnw4j\" (UID: \"c4233c69-4213-43be-8809-76612b6b6f3e\") " pod="openshift-dns/node-resolver-tnw4j"
Oct 31 07:28:25.119541 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:25.117061    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/3fa078c7-4bf2-4e76-9692-ae460e8fadfa-etc-modprobe-d\") pod \"tuned-cpn4m\" (UID: \"3fa078c7-4bf2-4e76-9692-ae460e8fadfa\") " pod="openshift-cluster-node-tuning-operator/tuned-cpn4m"
Oct 31 07:28:25.119541 ip-10-0-139-83 kubenswrapper[2573]: E1031 07:28:25.117109    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/77256436-39ff-4c2a-8390-4e9fa7d68486-original-pull-secret podName:77256436-39ff-4c2a-8390-4e9fa7d68486 nodeName:}" failed. No retries permitted until 2026-10-31 07:28:25.61706634 +0000 UTC m=+2.307837156 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/77256436-39ff-4c2a-8390-4e9fa7d68486-original-pull-secret") pod "global-pull-secret-syncer-kht4l" (UID: "77256436-39ff-4c2a-8390-4e9fa7d68486") : object "kube-system"/"original-pull-secret" not registered
Oct 31 07:28:25.119970 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:25.117131    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/3fa078c7-4bf2-4e76-9692-ae460e8fadfa-etc-modprobe-d\") pod \"tuned-cpn4m\" (UID: \"3fa078c7-4bf2-4e76-9692-ae460e8fadfa\") " pod="openshift-cluster-node-tuning-operator/tuned-cpn4m"
Oct 31 07:28:25.119970 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:25.117284    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/4bfbebfd-2be7-438f-b88d-273f24751f8c-ovnkube-config\") pod \"ovnkube-node-5p7xw\" (UID: \"4bfbebfd-2be7-438f-b88d-273f24751f8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-5p7xw"
Oct 31 07:28:25.119970 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:25.117293    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/4bfbebfd-2be7-438f-b88d-273f24751f8c-run-systemd\") pod \"ovnkube-node-5p7xw\" (UID: \"4bfbebfd-2be7-438f-b88d-273f24751f8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-5p7xw"
Oct 31 07:28:25.119970 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:25.117322    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/08198bc1-314f-4881-833c-e633bac15e4a-multus-cni-dir\") pod \"multus-pmnqx\" (UID: \"08198bc1-314f-4881-833c-e633bac15e4a\") " pod="openshift-multus/multus-pmnqx"
Oct 31 07:28:25.119970 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:25.117343    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5-konnectivity-ca\") pod \"konnectivity-agent-h4k2m\" (UID: \"2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5\") " pod="kube-system/konnectivity-agent-h4k2m"
Oct 31 07:28:25.119970 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:25.117363    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/4bfbebfd-2be7-438f-b88d-273f24751f8c-run-systemd\") pod \"ovnkube-node-5p7xw\" (UID: \"4bfbebfd-2be7-438f-b88d-273f24751f8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-5p7xw"
Oct 31 07:28:25.119970 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:25.117365    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/9dad8dd1-cfc5-412d-bf9a-bac3c9162af1-etc-selinux\") pod \"aws-ebs-csi-driver-node-797gj\" (UID: \"9dad8dd1-cfc5-412d-bf9a-bac3c9162af1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-797gj"
Oct 31 07:28:25.119970 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:25.117402    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/08198bc1-314f-4881-833c-e633bac15e4a-multus-cni-dir\") pod \"multus-pmnqx\" (UID: \"08198bc1-314f-4881-833c-e633bac15e4a\") " pod="openshift-multus/multus-pmnqx"
Oct 31 07:28:25.119970 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:25.117401    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9c72279a-708f-46df-8915-9995c5aea14d-cnibin\") pod \"multus-additional-cni-plugins-dh6w8\" (UID: \"9c72279a-708f-46df-8915-9995c5aea14d\") " pod="openshift-multus/multus-additional-cni-plugins-dh6w8"
Oct 31 07:28:25.119970 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:25.117432    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9c72279a-708f-46df-8915-9995c5aea14d-cnibin\") pod \"multus-additional-cni-plugins-dh6w8\" (UID: \"9c72279a-708f-46df-8915-9995c5aea14d\") " pod="openshift-multus/multus-additional-cni-plugins-dh6w8"
Oct 31 07:28:25.119970 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:25.117433    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/08198bc1-314f-4881-833c-e633bac15e4a-multus-socket-dir-parent\") pod \"multus-pmnqx\" (UID: \"08198bc1-314f-4881-833c-e633bac15e4a\") " pod="openshift-multus/multus-pmnqx"
Oct 31 07:28:25.119970 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:25.117443    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/9dad8dd1-cfc5-412d-bf9a-bac3c9162af1-etc-selinux\") pod \"aws-ebs-csi-driver-node-797gj\" (UID: \"9dad8dd1-cfc5-412d-bf9a-bac3c9162af1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-797gj"
Oct 31 07:28:25.119970 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:25.117442    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c4233c69-4213-43be-8809-76612b6b6f3e-tmp-dir\") pod \"node-resolver-tnw4j\" (UID: \"c4233c69-4213-43be-8809-76612b6b6f3e\") " pod="openshift-dns/node-resolver-tnw4j"
Oct 31 07:28:25.119970 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:25.117468    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/08198bc1-314f-4881-833c-e633bac15e4a-host-run-k8s-cni-cncf-io\") pod \"multus-pmnqx\" (UID: \"08198bc1-314f-4881-833c-e633bac15e4a\") " pod="openshift-multus/multus-pmnqx"
Oct 31 07:28:25.119970 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:25.117495    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/3fa078c7-4bf2-4e76-9692-ae460e8fadfa-run\") pod \"tuned-cpn4m\" (UID: \"3fa078c7-4bf2-4e76-9692-ae460e8fadfa\") " pod="openshift-cluster-node-tuning-operator/tuned-cpn4m"
Oct 31 07:28:25.119970 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:25.117503    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/08198bc1-314f-4881-833c-e633bac15e4a-host-run-k8s-cni-cncf-io\") pod \"multus-pmnqx\" (UID: \"08198bc1-314f-4881-833c-e633bac15e4a\") " pod="openshift-multus/multus-pmnqx"
Oct 31 07:28:25.119970 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:25.117470    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/08198bc1-314f-4881-833c-e633bac15e4a-multus-socket-dir-parent\") pod \"multus-pmnqx\" (UID: \"08198bc1-314f-4881-833c-e633bac15e4a\") " pod="openshift-multus/multus-pmnqx"
Oct 31 07:28:25.119970 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:25.117520    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/3fa078c7-4bf2-4e76-9692-ae460e8fadfa-lib-modules\") pod \"tuned-cpn4m\" (UID: \"3fa078c7-4bf2-4e76-9692-ae460e8fadfa\") " pod="openshift-cluster-node-tuning-operator/tuned-cpn4m"
Oct 31 07:28:25.120442 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:25.117545    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/3fa078c7-4bf2-4e76-9692-ae460e8fadfa-etc-tuned\") pod \"tuned-cpn4m\" (UID: \"3fa078c7-4bf2-4e76-9692-ae460e8fadfa\") " pod="openshift-cluster-node-tuning-operator/tuned-cpn4m"
Oct 31 07:28:25.120442 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:25.117571    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/3fa078c7-4bf2-4e76-9692-ae460e8fadfa-run\") pod \"tuned-cpn4m\" (UID: \"3fa078c7-4bf2-4e76-9692-ae460e8fadfa\") " pod="openshift-cluster-node-tuning-operator/tuned-cpn4m"
Oct 31 07:28:25.120442 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:25.117647    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/3fa078c7-4bf2-4e76-9692-ae460e8fadfa-lib-modules\") pod \"tuned-cpn4m\" (UID: \"3fa078c7-4bf2-4e76-9692-ae460e8fadfa\") " pod="openshift-cluster-node-tuning-operator/tuned-cpn4m"
Oct 31 07:28:25.120442 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:25.118096    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9c72279a-708f-46df-8915-9995c5aea14d-tuning-conf-dir\") pod \"multus-additional-cni-plugins-dh6w8\" (UID: \"9c72279a-708f-46df-8915-9995c5aea14d\") " pod="openshift-multus/multus-additional-cni-plugins-dh6w8"
Oct 31 07:28:25.120442 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:25.119049    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3fa078c7-4bf2-4e76-9692-ae460e8fadfa-tmp\") pod \"tuned-cpn4m\" (UID: \"3fa078c7-4bf2-4e76-9692-ae460e8fadfa\") " pod="openshift-cluster-node-tuning-operator/tuned-cpn4m"
Oct 31 07:28:25.120442 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:25.119186    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5-agent-certs\") pod \"konnectivity-agent-h4k2m\" (UID: \"2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5\") " pod="kube-system/konnectivity-agent-h4k2m"
Oct 31 07:28:25.120442 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:25.119243    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/4bfbebfd-2be7-438f-b88d-273f24751f8c-ovn-node-metrics-cert\") pod \"ovnkube-node-5p7xw\" (UID: \"4bfbebfd-2be7-438f-b88d-273f24751f8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-5p7xw"
Oct 31 07:28:25.120442 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:25.119779    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/3fa078c7-4bf2-4e76-9692-ae460e8fadfa-etc-tuned\") pod \"tuned-cpn4m\" (UID: \"3fa078c7-4bf2-4e76-9692-ae460e8fadfa\") " pod="openshift-cluster-node-tuning-operator/tuned-cpn4m"
Oct 31 07:28:25.170677 ip-10-0-139-83 kubenswrapper[2573]: E1031 07:28:25.170624    2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 31 07:28:25.170677 ip-10-0-139-83 kubenswrapper[2573]: E1031 07:28:25.170647    2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 31 07:28:25.170677 ip-10-0-139-83 kubenswrapper[2573]: E1031 07:28:25.170659    2573 projected.go:196] Error preparing data for projected volume kube-api-access-ghkb9 for pod openshift-network-diagnostics/network-check-target-9zp4l: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 31 07:28:25.170893 ip-10-0-139-83 kubenswrapper[2573]: E1031 07:28:25.170708    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f1ba68f6-d87c-4492-a5eb-5f2d3a60eb1c-kube-api-access-ghkb9 podName:f1ba68f6-d87c-4492-a5eb-5f2d3a60eb1c nodeName:}" failed. No retries permitted until 2026-10-31 07:28:25.670691862 +0000 UTC m=+2.361462682 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-ghkb9" (UniqueName: "kubernetes.io/projected/f1ba68f6-d87c-4492-a5eb-5f2d3a60eb1c-kube-api-access-ghkb9") pod "network-check-target-9zp4l" (UID: "f1ba68f6-d87c-4492-a5eb-5f2d3a60eb1c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 31 07:28:25.173079 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:25.172885    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cq7rv\" (UniqueName: \"kubernetes.io/projected/3fa078c7-4bf2-4e76-9692-ae460e8fadfa-kube-api-access-cq7rv\") pod \"tuned-cpn4m\" (UID: \"3fa078c7-4bf2-4e76-9692-ae460e8fadfa\") " pod="openshift-cluster-node-tuning-operator/tuned-cpn4m"
Oct 31 07:28:25.173414 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:25.173391    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5n5lv\" (UniqueName: \"kubernetes.io/projected/9dad8dd1-cfc5-412d-bf9a-bac3c9162af1-kube-api-access-5n5lv\") pod \"aws-ebs-csi-driver-node-797gj\" (UID: \"9dad8dd1-cfc5-412d-bf9a-bac3c9162af1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-797gj"
Oct 31 07:28:25.174151 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:25.173386    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kk4nh\" (UniqueName: \"kubernetes.io/projected/fc64d31a-f48a-4293-a4de-262045d0b7e8-kube-api-access-kk4nh\") pod \"node-ca-th7x7\" (UID: \"fc64d31a-f48a-4293-a4de-262045d0b7e8\") " pod="openshift-image-registry/node-ca-th7x7"
Oct 31 07:28:25.174151 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:25.173953    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cz8tm\" (UniqueName: \"kubernetes.io/projected/4bfbebfd-2be7-438f-b88d-273f24751f8c-kube-api-access-cz8tm\") pod \"ovnkube-node-5p7xw\" (UID: \"4bfbebfd-2be7-438f-b88d-273f24751f8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-5p7xw"
Oct 31 07:28:25.174151 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:25.173991    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jbdqx\" (UniqueName: \"kubernetes.io/projected/26659955-3909-4171-90b8-21b0ad6eef64-kube-api-access-jbdqx\") pod \"iptables-alerter-65hzp\" (UID: \"26659955-3909-4171-90b8-21b0ad6eef64\") " pod="openshift-network-operator/iptables-alerter-65hzp"
Oct 31 07:28:25.174151 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:25.174060    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xs9lt\" (UniqueName: \"kubernetes.io/projected/08198bc1-314f-4881-833c-e633bac15e4a-kube-api-access-xs9lt\") pod \"multus-pmnqx\" (UID: \"08198bc1-314f-4881-833c-e633bac15e4a\") " pod="openshift-multus/multus-pmnqx"
Oct 31 07:28:25.174151 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:25.174113    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5hqbq\" (UniqueName: \"kubernetes.io/projected/c4233c69-4213-43be-8809-76612b6b6f3e-kube-api-access-5hqbq\") pod \"node-resolver-tnw4j\" (UID: \"c4233c69-4213-43be-8809-76612b6b6f3e\") " pod="openshift-dns/node-resolver-tnw4j"
Oct 31 07:28:25.174469 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:25.174455    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h4gzh\" (UniqueName: \"kubernetes.io/projected/9c72279a-708f-46df-8915-9995c5aea14d-kube-api-access-h4gzh\") pod \"multus-additional-cni-plugins-dh6w8\" (UID: \"9c72279a-708f-46df-8915-9995c5aea14d\") " pod="openshift-multus/multus-additional-cni-plugins-dh6w8"
Oct 31 07:28:25.175957 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:25.175938    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kxwb9\" (UniqueName: \"kubernetes.io/projected/c3ad95fd-e997-459c-93fd-27c4c030ea0d-kube-api-access-kxwb9\") pod \"network-metrics-daemon-r2zbv\" (UID: \"c3ad95fd-e997-459c-93fd-27c4c030ea0d\") " pod="openshift-multus/network-metrics-daemon-r2zbv"
Oct 31 07:28:25.266510 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:25.266479    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-cpn4m"
Oct 31 07:28:25.288135 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:25.288111    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-797gj"
Oct 31 07:28:25.315140 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:25.315114    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-5p7xw"
Oct 31 07:28:25.320186 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:25.320171    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-65hzp"
Oct 31 07:28:25.334793 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:25.334777    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-tnw4j"
Oct 31 07:28:25.363064 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:25.363039    2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8dfeb47fa079b05125409320442d08bf.slice/crio-9f2589536197b37032285af9622b146bd14cd852452e464e8eb73c0a610e5078 WatchSource:0}: Error finding container 9f2589536197b37032285af9622b146bd14cd852452e464e8eb73c0a610e5078: Status 404 returned error can't find the container with id 9f2589536197b37032285af9622b146bd14cd852452e464e8eb73c0a610e5078
Oct 31 07:28:25.363668 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:25.363651    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-dh6w8"
Oct 31 07:28:25.363759 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:25.363650    2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc4233c69_4213_43be_8809_76612b6b6f3e.slice/crio-37019c645339bd463a7dee60255920849f38a55d3c31eda8755469f8ada74ba0 WatchSource:0}: Error finding container 37019c645339bd463a7dee60255920849f38a55d3c31eda8755469f8ada74ba0: Status 404 returned error can't find the container with id 37019c645339bd463a7dee60255920849f38a55d3c31eda8755469f8ada74ba0
Oct 31 07:28:25.364670 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:25.364647    2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2183ffc19200ec816330ede9883f3bf6.slice/crio-e5d2dc100f3cacdb69fc74b690c1bbe741b4298bd41697263a403392aa95b8b8 WatchSource:0}: Error finding container e5d2dc100f3cacdb69fc74b690c1bbe741b4298bd41697263a403392aa95b8b8: Status 404 returned error can't find the container with id e5d2dc100f3cacdb69fc74b690c1bbe741b4298bd41697263a403392aa95b8b8
Oct 31 07:28:25.366738 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:25.366716    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-h4k2m"
Oct 31 07:28:25.368580 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:25.368549    2573 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 31 07:28:25.374824 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:25.374803    2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c72279a_708f_46df_8915_9995c5aea14d.slice/crio-82a11c7ec21dcf611cf9180470eb718c94c4332da4e8f2b47dcf174499bada7a WatchSource:0}: Error finding container 82a11c7ec21dcf611cf9180470eb718c94c4332da4e8f2b47dcf174499bada7a: Status 404 returned error can't find the container with id 82a11c7ec21dcf611cf9180470eb718c94c4332da4e8f2b47dcf174499bada7a
Oct 31 07:28:25.375834 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:25.375816    2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e6fa788_ffb3_4a3b_8f6c_e52c2f9655a5.slice/crio-5292ef84738cab81213f85cb2c9188469f27c00aa7758fb95524fa5c53ddac15 WatchSource:0}: Error finding container 5292ef84738cab81213f85cb2c9188469f27c00aa7758fb95524fa5c53ddac15: Status 404 returned error can't find the container with id 5292ef84738cab81213f85cb2c9188469f27c00aa7758fb95524fa5c53ddac15
Oct 31 07:28:25.381874 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:25.381857    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-pmnqx"
Oct 31 07:28:25.386401 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:25.386381    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-th7x7"
Oct 31 07:28:25.390104 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:25.390078    2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod08198bc1_314f_4881_833c_e633bac15e4a.slice/crio-69600ee490f34bf474850cf25f80679e6519843dc40baf37c5d57342b0cec097 WatchSource:0}: Error finding container 69600ee490f34bf474850cf25f80679e6519843dc40baf37c5d57342b0cec097: Status 404 returned error can't find the container with id 69600ee490f34bf474850cf25f80679e6519843dc40baf37c5d57342b0cec097
Oct 31 07:28:25.394315 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:25.394295    2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc64d31a_f48a_4293_a4de_262045d0b7e8.slice/crio-ecb08d77ee39577a396f19d406280850735fe9412808b0951b5d432f6275f230 WatchSource:0}: Error finding container ecb08d77ee39577a396f19d406280850735fe9412808b0951b5d432f6275f230: Status 404 returned error can't find the container with id ecb08d77ee39577a396f19d406280850735fe9412808b0951b5d432f6275f230
Oct 31 07:28:25.621390 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:25.621359    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/77256436-39ff-4c2a-8390-4e9fa7d68486-original-pull-secret\") pod \"global-pull-secret-syncer-kht4l\" (UID: \"77256436-39ff-4c2a-8390-4e9fa7d68486\") " pod="kube-system/global-pull-secret-syncer-kht4l"
Oct 31 07:28:25.621540 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:25.621409    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c3ad95fd-e997-459c-93fd-27c4c030ea0d-metrics-certs\") pod \"network-metrics-daemon-r2zbv\" (UID: \"c3ad95fd-e997-459c-93fd-27c4c030ea0d\") " pod="openshift-multus/network-metrics-daemon-r2zbv"
Oct 31 07:28:25.621540 ip-10-0-139-83 kubenswrapper[2573]: E1031 07:28:25.621475    2573 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 31 07:28:25.621615 ip-10-0-139-83 kubenswrapper[2573]: E1031 07:28:25.621540    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/77256436-39ff-4c2a-8390-4e9fa7d68486-original-pull-secret podName:77256436-39ff-4c2a-8390-4e9fa7d68486 nodeName:}" failed. No retries permitted until 2026-10-31 07:28:26.621525528 +0000 UTC m=+3.312296353 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/77256436-39ff-4c2a-8390-4e9fa7d68486-original-pull-secret") pod "global-pull-secret-syncer-kht4l" (UID: "77256436-39ff-4c2a-8390-4e9fa7d68486") : object "kube-system"/"original-pull-secret" not registered
Oct 31 07:28:25.621615 ip-10-0-139-83 kubenswrapper[2573]: E1031 07:28:25.621486    2573 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 31 07:28:25.621615 ip-10-0-139-83 kubenswrapper[2573]: E1031 07:28:25.621574    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c3ad95fd-e997-459c-93fd-27c4c030ea0d-metrics-certs podName:c3ad95fd-e997-459c-93fd-27c4c030ea0d nodeName:}" failed. No retries permitted until 2026-10-31 07:28:26.62156627 +0000 UTC m=+3.312337079 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c3ad95fd-e997-459c-93fd-27c4c030ea0d-metrics-certs") pod "network-metrics-daemon-r2zbv" (UID: "c3ad95fd-e997-459c-93fd-27c4c030ea0d") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 31 07:28:25.722341 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:25.722307    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ghkb9\" (UniqueName: \"kubernetes.io/projected/f1ba68f6-d87c-4492-a5eb-5f2d3a60eb1c-kube-api-access-ghkb9\") pod \"network-check-target-9zp4l\" (UID: \"f1ba68f6-d87c-4492-a5eb-5f2d3a60eb1c\") " pod="openshift-network-diagnostics/network-check-target-9zp4l"
Oct 31 07:28:25.722925 ip-10-0-139-83 kubenswrapper[2573]: E1031 07:28:25.722444    2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 31 07:28:25.722925 ip-10-0-139-83 kubenswrapper[2573]: E1031 07:28:25.722459    2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 31 07:28:25.722925 ip-10-0-139-83 kubenswrapper[2573]: E1031 07:28:25.722468    2573 projected.go:196] Error preparing data for projected volume kube-api-access-ghkb9 for pod openshift-network-diagnostics/network-check-target-9zp4l: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 31 07:28:25.722925 ip-10-0-139-83 kubenswrapper[2573]: E1031 07:28:25.722513    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f1ba68f6-d87c-4492-a5eb-5f2d3a60eb1c-kube-api-access-ghkb9 podName:f1ba68f6-d87c-4492-a5eb-5f2d3a60eb1c nodeName:}" failed. No retries permitted until 2026-10-31 07:28:26.722500525 +0000 UTC m=+3.413271330 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-ghkb9" (UniqueName: "kubernetes.io/projected/f1ba68f6-d87c-4492-a5eb-5f2d3a60eb1c-kube-api-access-ghkb9") pod "network-check-target-9zp4l" (UID: "f1ba68f6-d87c-4492-a5eb-5f2d3a60eb1c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 31 07:28:25.758890 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:25.758822    2573 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 31 07:28:25.856996 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:25.856951    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-th7x7" event={"ID":"fc64d31a-f48a-4293-a4de-262045d0b7e8","Type":"ContainerStarted","Data":"ecb08d77ee39577a396f19d406280850735fe9412808b0951b5d432f6275f230"}
Oct 31 07:28:25.858081 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:25.858054    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-pmnqx" event={"ID":"08198bc1-314f-4881-833c-e633bac15e4a","Type":"ContainerStarted","Data":"69600ee490f34bf474850cf25f80679e6519843dc40baf37c5d57342b0cec097"}
Oct 31 07:28:25.859144 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:25.859116    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-h4k2m" event={"ID":"2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5","Type":"ContainerStarted","Data":"5292ef84738cab81213f85cb2c9188469f27c00aa7758fb95524fa5c53ddac15"}
Oct 31 07:28:25.860223 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:25.860203    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-dh6w8" event={"ID":"9c72279a-708f-46df-8915-9995c5aea14d","Type":"ContainerStarted","Data":"82a11c7ec21dcf611cf9180470eb718c94c4332da4e8f2b47dcf174499bada7a"}
Oct 31 07:28:25.861336 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:25.861301    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-tnw4j" event={"ID":"c4233c69-4213-43be-8809-76612b6b6f3e","Type":"ContainerStarted","Data":"37019c645339bd463a7dee60255920849f38a55d3c31eda8755469f8ada74ba0"}
Oct 31 07:28:25.862341 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:25.862309    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-83.us-west-1.compute.internal" event={"ID":"2183ffc19200ec816330ede9883f3bf6","Type":"ContainerStarted","Data":"e5d2dc100f3cacdb69fc74b690c1bbe741b4298bd41697263a403392aa95b8b8"}
Oct 31 07:28:25.863265 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:25.863243    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-83.us-west-1.compute.internal" event={"ID":"8dfeb47fa079b05125409320442d08bf","Type":"ContainerStarted","Data":"9f2589536197b37032285af9622b146bd14cd852452e464e8eb73c0a610e5078"}
Oct 31 07:28:25.936299 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:25.936276    2573 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 31 07:28:26.052969 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:26.052888    2573 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-30 07:23:25 +0000 UTC" deadline="2028-07-01 21:02:45.721210184 +0000 UTC"
Oct 31 07:28:26.052969 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:26.052926    2573 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14629h34m19.668287634s"
Oct 31 07:28:26.129894 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:26.129867    2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4bfbebfd_2be7_438f_b88d_273f24751f8c.slice/crio-7df3a2737e0ecbaf7eb754ef7539c98a3a1aac973f357f73d3c8d852b7d2312f WatchSource:0}: Error finding container 7df3a2737e0ecbaf7eb754ef7539c98a3a1aac973f357f73d3c8d852b7d2312f: Status 404 returned error can't find the container with id 7df3a2737e0ecbaf7eb754ef7539c98a3a1aac973f357f73d3c8d852b7d2312f
Oct 31 07:28:26.130676 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:26.130657    2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3fa078c7_4bf2_4e76_9692_ae460e8fadfa.slice/crio-f5d9afbc1fc89adec0532733b9e5573011daea02a1f894ee021f5f878ff04ec0 WatchSource:0}: Error finding container f5d9afbc1fc89adec0532733b9e5573011daea02a1f894ee021f5f878ff04ec0: Status 404 returned error can't find the container with id f5d9afbc1fc89adec0532733b9e5573011daea02a1f894ee021f5f878ff04ec0
Oct 31 07:28:26.156626 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:26.156604    2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9dad8dd1_cfc5_412d_bf9a_bac3c9162af1.slice/crio-c8d95e23ad7960db9871e1b75389bf92563cfef5ab50bfece8472084ee3133ba WatchSource:0}: Error finding container c8d95e23ad7960db9871e1b75389bf92563cfef5ab50bfece8472084ee3133ba: Status 404 returned error can't find the container with id c8d95e23ad7960db9871e1b75389bf92563cfef5ab50bfece8472084ee3133ba
Oct 31 07:28:26.161626 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:26.161604    2573 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 31 07:28:26.182474 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:26.182455    2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod26659955_3909_4171_90b8_21b0ad6eef64.slice/crio-c1ffc96f411310ca3c9bbb972b3c1df1fe0467e4564c58e844cd9b804d40194c WatchSource:0}: Error finding container c1ffc96f411310ca3c9bbb972b3c1df1fe0467e4564c58e844cd9b804d40194c: Status 404 returned error can't find the container with id c1ffc96f411310ca3c9bbb972b3c1df1fe0467e4564c58e844cd9b804d40194c
Oct 31 07:28:26.630114 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:26.629892    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/77256436-39ff-4c2a-8390-4e9fa7d68486-original-pull-secret\") pod \"global-pull-secret-syncer-kht4l\" (UID: \"77256436-39ff-4c2a-8390-4e9fa7d68486\") " pod="kube-system/global-pull-secret-syncer-kht4l"
Oct 31 07:28:26.630389 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:26.630334    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c3ad95fd-e997-459c-93fd-27c4c030ea0d-metrics-certs\") pod \"network-metrics-daemon-r2zbv\" (UID: \"c3ad95fd-e997-459c-93fd-27c4c030ea0d\") " pod="openshift-multus/network-metrics-daemon-r2zbv"
Oct 31 07:28:26.634636 ip-10-0-139-83 kubenswrapper[2573]: E1031 07:28:26.634606    2573 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 31 07:28:26.634759 ip-10-0-139-83 kubenswrapper[2573]: E1031 07:28:26.634698    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/77256436-39ff-4c2a-8390-4e9fa7d68486-original-pull-secret podName:77256436-39ff-4c2a-8390-4e9fa7d68486 nodeName:}" failed. No retries permitted until 2026-10-31 07:28:28.634673865 +0000 UTC m=+5.325444670 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/77256436-39ff-4c2a-8390-4e9fa7d68486-original-pull-secret") pod "global-pull-secret-syncer-kht4l" (UID: "77256436-39ff-4c2a-8390-4e9fa7d68486") : object "kube-system"/"original-pull-secret" not registered
Oct 31 07:28:26.634905 ip-10-0-139-83 kubenswrapper[2573]: E1031 07:28:26.634888    2573 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 31 07:28:26.634963 ip-10-0-139-83 kubenswrapper[2573]: E1031 07:28:26.634951    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c3ad95fd-e997-459c-93fd-27c4c030ea0d-metrics-certs podName:c3ad95fd-e997-459c-93fd-27c4c030ea0d nodeName:}" failed. No retries permitted until 2026-10-31 07:28:28.634936557 +0000 UTC m=+5.325707366 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c3ad95fd-e997-459c-93fd-27c4c030ea0d-metrics-certs") pod "network-metrics-daemon-r2zbv" (UID: "c3ad95fd-e997-459c-93fd-27c4c030ea0d") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 31 07:28:26.731984 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:26.731944    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ghkb9\" (UniqueName: \"kubernetes.io/projected/f1ba68f6-d87c-4492-a5eb-5f2d3a60eb1c-kube-api-access-ghkb9\") pod \"network-check-target-9zp4l\" (UID: \"f1ba68f6-d87c-4492-a5eb-5f2d3a60eb1c\") " pod="openshift-network-diagnostics/network-check-target-9zp4l"
Oct 31 07:28:26.732773 ip-10-0-139-83 kubenswrapper[2573]: E1031 07:28:26.732126    2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 31 07:28:26.732773 ip-10-0-139-83 kubenswrapper[2573]: E1031 07:28:26.732145    2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 31 07:28:26.732773 ip-10-0-139-83 kubenswrapper[2573]: E1031 07:28:26.732158    2573 projected.go:196] Error preparing data for projected volume kube-api-access-ghkb9 for pod openshift-network-diagnostics/network-check-target-9zp4l: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 31 07:28:26.732773 ip-10-0-139-83 kubenswrapper[2573]: E1031 07:28:26.732215    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f1ba68f6-d87c-4492-a5eb-5f2d3a60eb1c-kube-api-access-ghkb9 podName:f1ba68f6-d87c-4492-a5eb-5f2d3a60eb1c nodeName:}" failed. No retries permitted until 2026-10-31 07:28:28.732196957 +0000 UTC m=+5.422967777 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-ghkb9" (UniqueName: "kubernetes.io/projected/f1ba68f6-d87c-4492-a5eb-5f2d3a60eb1c-kube-api-access-ghkb9") pod "network-check-target-9zp4l" (UID: "f1ba68f6-d87c-4492-a5eb-5f2d3a60eb1c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 31 07:28:26.851073 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:26.851036    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9zp4l"
Oct 31 07:28:26.851294 ip-10-0-139-83 kubenswrapper[2573]: E1031 07:28:26.851251    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9zp4l" podUID="f1ba68f6-d87c-4492-a5eb-5f2d3a60eb1c"
Oct 31 07:28:26.851374 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:26.851355    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kht4l"
Oct 31 07:28:26.851489 ip-10-0-139-83 kubenswrapper[2573]: E1031 07:28:26.851466    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kht4l" podUID="77256436-39ff-4c2a-8390-4e9fa7d68486"
Oct 31 07:28:26.851593 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:26.851574    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r2zbv"
Oct 31 07:28:26.851735 ip-10-0-139-83 kubenswrapper[2573]: E1031 07:28:26.851704    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-r2zbv" podUID="c3ad95fd-e997-459c-93fd-27c4c030ea0d"
Oct 31 07:28:26.869449 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:26.869312    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-65hzp" event={"ID":"26659955-3909-4171-90b8-21b0ad6eef64","Type":"ContainerStarted","Data":"c1ffc96f411310ca3c9bbb972b3c1df1fe0467e4564c58e844cd9b804d40194c"}
Oct 31 07:28:26.877142 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:26.877039    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-797gj" event={"ID":"9dad8dd1-cfc5-412d-bf9a-bac3c9162af1","Type":"ContainerStarted","Data":"c8d95e23ad7960db9871e1b75389bf92563cfef5ab50bfece8472084ee3133ba"}
Oct 31 07:28:26.887697 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:26.887586    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-cpn4m" event={"ID":"3fa078c7-4bf2-4e76-9692-ae460e8fadfa","Type":"ContainerStarted","Data":"f5d9afbc1fc89adec0532733b9e5573011daea02a1f894ee021f5f878ff04ec0"}
Oct 31 07:28:26.893309 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:26.893281    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-5p7xw" event={"ID":"4bfbebfd-2be7-438f-b88d-273f24751f8c","Type":"ContainerStarted","Data":"7df3a2737e0ecbaf7eb754ef7539c98a3a1aac973f357f73d3c8d852b7d2312f"}
Oct 31 07:28:27.053194 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:27.053123    2573 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-30 07:23:25 +0000 UTC" deadline="2028-05-02 03:17:23.138425327 +0000 UTC"
Oct 31 07:28:27.053194 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:27.053191    2573 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13171h48m56.085238746s"
Oct 31 07:28:28.647994 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:28.647497    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/77256436-39ff-4c2a-8390-4e9fa7d68486-original-pull-secret\") pod \"global-pull-secret-syncer-kht4l\" (UID: \"77256436-39ff-4c2a-8390-4e9fa7d68486\") " pod="kube-system/global-pull-secret-syncer-kht4l"
Oct 31 07:28:28.647994 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:28.647566    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c3ad95fd-e997-459c-93fd-27c4c030ea0d-metrics-certs\") pod \"network-metrics-daemon-r2zbv\" (UID: \"c3ad95fd-e997-459c-93fd-27c4c030ea0d\") " pod="openshift-multus/network-metrics-daemon-r2zbv"
Oct 31 07:28:28.647994 ip-10-0-139-83 kubenswrapper[2573]: E1031 07:28:28.647737    2573 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 31 07:28:28.647994 ip-10-0-139-83 kubenswrapper[2573]: E1031 07:28:28.647805    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c3ad95fd-e997-459c-93fd-27c4c030ea0d-metrics-certs podName:c3ad95fd-e997-459c-93fd-27c4c030ea0d nodeName:}" failed. No retries permitted until 2026-10-31 07:28:32.647783939 +0000 UTC m=+9.338554750 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c3ad95fd-e997-459c-93fd-27c4c030ea0d-metrics-certs") pod "network-metrics-daemon-r2zbv" (UID: "c3ad95fd-e997-459c-93fd-27c4c030ea0d") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 31 07:28:28.647994 ip-10-0-139-83 kubenswrapper[2573]: E1031 07:28:28.647887    2573 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 31 07:28:28.647994 ip-10-0-139-83 kubenswrapper[2573]: E1031 07:28:28.647923    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/77256436-39ff-4c2a-8390-4e9fa7d68486-original-pull-secret podName:77256436-39ff-4c2a-8390-4e9fa7d68486 nodeName:}" failed. No retries permitted until 2026-10-31 07:28:32.647910816 +0000 UTC m=+9.338681625 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/77256436-39ff-4c2a-8390-4e9fa7d68486-original-pull-secret") pod "global-pull-secret-syncer-kht4l" (UID: "77256436-39ff-4c2a-8390-4e9fa7d68486") : object "kube-system"/"original-pull-secret" not registered
Oct 31 07:28:28.748365 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:28.748312    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ghkb9\" (UniqueName: \"kubernetes.io/projected/f1ba68f6-d87c-4492-a5eb-5f2d3a60eb1c-kube-api-access-ghkb9\") pod \"network-check-target-9zp4l\" (UID: \"f1ba68f6-d87c-4492-a5eb-5f2d3a60eb1c\") " pod="openshift-network-diagnostics/network-check-target-9zp4l"
Oct 31 07:28:28.748525 ip-10-0-139-83 kubenswrapper[2573]: E1031 07:28:28.748505    2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 31 07:28:28.748608 ip-10-0-139-83 kubenswrapper[2573]: E1031 07:28:28.748534    2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 31 07:28:28.748608 ip-10-0-139-83 kubenswrapper[2573]: E1031 07:28:28.748548    2573 projected.go:196] Error preparing data for projected volume kube-api-access-ghkb9 for pod openshift-network-diagnostics/network-check-target-9zp4l: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 31 07:28:28.748608 ip-10-0-139-83 kubenswrapper[2573]: E1031 07:28:28.748605    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f1ba68f6-d87c-4492-a5eb-5f2d3a60eb1c-kube-api-access-ghkb9 podName:f1ba68f6-d87c-4492-a5eb-5f2d3a60eb1c nodeName:}" failed. No retries permitted until 2026-10-31 07:28:32.74858532 +0000 UTC m=+9.439356126 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-ghkb9" (UniqueName: "kubernetes.io/projected/f1ba68f6-d87c-4492-a5eb-5f2d3a60eb1c-kube-api-access-ghkb9") pod "network-check-target-9zp4l" (UID: "f1ba68f6-d87c-4492-a5eb-5f2d3a60eb1c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 31 07:28:28.850573 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:28.850526    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9zp4l"
Oct 31 07:28:28.850753 ip-10-0-139-83 kubenswrapper[2573]: E1031 07:28:28.850659    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9zp4l" podUID="f1ba68f6-d87c-4492-a5eb-5f2d3a60eb1c"
Oct 31 07:28:28.850753 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:28.850723    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kht4l"
Oct 31 07:28:28.850875 ip-10-0-139-83 kubenswrapper[2573]: E1031 07:28:28.850795    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kht4l" podUID="77256436-39ff-4c2a-8390-4e9fa7d68486"
Oct 31 07:28:28.850875 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:28.850846    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r2zbv"
Oct 31 07:28:28.850987 ip-10-0-139-83 kubenswrapper[2573]: E1031 07:28:28.850928    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-r2zbv" podUID="c3ad95fd-e997-459c-93fd-27c4c030ea0d"
Oct 31 07:28:29.909792 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:29.909452    2573 generic.go:356] "Generic (PLEG): container finished" podID="9c72279a-708f-46df-8915-9995c5aea14d" containerID="9969a8fe850f4420a949de006c685300fd4d2047c9eefef4e247bf296bc04038" exitCode=0
Oct 31 07:28:29.909792 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:29.909681    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-dh6w8" event={"ID":"9c72279a-708f-46df-8915-9995c5aea14d","Type":"ContainerDied","Data":"9969a8fe850f4420a949de006c685300fd4d2047c9eefef4e247bf296bc04038"}
Oct 31 07:28:30.850995 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:30.850951    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r2zbv"
Oct 31 07:28:30.851168 ip-10-0-139-83 kubenswrapper[2573]: E1031 07:28:30.851096    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-r2zbv" podUID="c3ad95fd-e997-459c-93fd-27c4c030ea0d"
Oct 31 07:28:30.851245 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:30.851171    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9zp4l"
Oct 31 07:28:30.851308 ip-10-0-139-83 kubenswrapper[2573]: E1031 07:28:30.851250    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9zp4l" podUID="f1ba68f6-d87c-4492-a5eb-5f2d3a60eb1c"
Oct 31 07:28:30.851308 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:30.851301    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kht4l"
Oct 31 07:28:30.851426 ip-10-0-139-83 kubenswrapper[2573]: E1031 07:28:30.851399    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kht4l" podUID="77256436-39ff-4c2a-8390-4e9fa7d68486"
Oct 31 07:28:30.929631 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:30.929534    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-83.us-west-1.compute.internal" event={"ID":"2183ffc19200ec816330ede9883f3bf6","Type":"ContainerStarted","Data":"cd2467be2b735e8292bf12bac2cb19105c276629b61450f504271e37fb9c7e7b"}
Oct 31 07:28:30.936637 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:30.936604    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-h4k2m" event={"ID":"2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5","Type":"ContainerStarted","Data":"f07b979e3df55a03ef3e9e4b905adad3538402005b82a787741df123e084fecd"}
Oct 31 07:28:30.961089 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:30.961050    2573 generic.go:356] "Generic (PLEG): container finished" podID="8dfeb47fa079b05125409320442d08bf" containerID="e3679e2424b009b37ae06bb097193ef296d3a36862993a093b27fbf5d970d554" exitCode=0
Oct 31 07:28:30.961239 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:30.961109    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-83.us-west-1.compute.internal" event={"ID":"8dfeb47fa079b05125409320442d08bf","Type":"ContainerDied","Data":"e3679e2424b009b37ae06bb097193ef296d3a36862993a093b27fbf5d970d554"}
Oct 31 07:28:31.052799 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:31.052391    2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-83.us-west-1.compute.internal" podStartSLOduration=6.052370985 podStartE2EDuration="6.052370985s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 07:28:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 07:28:31.050989276 +0000 UTC m=+7.741760106" watchObservedRunningTime="2026-10-31 07:28:31.052370985 +0000 UTC m=+7.743141814"
Oct 31 07:28:31.966703 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:31.966668    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-th7x7" event={"ID":"fc64d31a-f48a-4293-a4de-262045d0b7e8","Type":"ContainerStarted","Data":"6e92606ba2723ad7f80960f4d7c33928d05851f6783cd1447b1e2e12931b61dd"}
Oct 31 07:28:32.086640 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:32.086575    2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-h4k2m" podStartSLOduration=2.295009511 podStartE2EDuration="7.086554498s" totalImagesPullingTime="4.791544987s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 07:28:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 07:28:31.174005511 +0000 UTC m=+7.864776342" watchObservedRunningTime="2026-10-31 07:28:32.086554498 +0000 UTC m=+8.777325330"
Oct 31 07:28:32.210780 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:32.210134    2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-th7x7" podStartSLOduration=1.147924063 podStartE2EDuration="7.210110279s" totalImagesPullingTime="6.062186216s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 07:28:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 07:28:32.087878463 +0000 UTC m=+8.778649285" watchObservedRunningTime="2026-10-31 07:28:32.210110279 +0000 UTC m=+8.900881108"
Oct 31 07:28:32.693562 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:32.693477    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c3ad95fd-e997-459c-93fd-27c4c030ea0d-metrics-certs\") pod \"network-metrics-daemon-r2zbv\" (UID: \"c3ad95fd-e997-459c-93fd-27c4c030ea0d\") " pod="openshift-multus/network-metrics-daemon-r2zbv"
Oct 31 07:28:32.693754 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:32.693586    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/77256436-39ff-4c2a-8390-4e9fa7d68486-original-pull-secret\") pod \"global-pull-secret-syncer-kht4l\" (UID: \"77256436-39ff-4c2a-8390-4e9fa7d68486\") " pod="kube-system/global-pull-secret-syncer-kht4l"
Oct 31 07:28:32.693754 ip-10-0-139-83 kubenswrapper[2573]: E1031 07:28:32.693742    2573 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 31 07:28:32.693875 ip-10-0-139-83 kubenswrapper[2573]: E1031 07:28:32.693811    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/77256436-39ff-4c2a-8390-4e9fa7d68486-original-pull-secret podName:77256436-39ff-4c2a-8390-4e9fa7d68486 nodeName:}" failed. No retries permitted until 2026-10-31 07:28:40.693788962 +0000 UTC m=+17.384559770 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/77256436-39ff-4c2a-8390-4e9fa7d68486-original-pull-secret") pod "global-pull-secret-syncer-kht4l" (UID: "77256436-39ff-4c2a-8390-4e9fa7d68486") : object "kube-system"/"original-pull-secret" not registered
Oct 31 07:28:32.693940 ip-10-0-139-83 kubenswrapper[2573]: E1031 07:28:32.693887    2573 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 31 07:28:32.693940 ip-10-0-139-83 kubenswrapper[2573]: E1031 07:28:32.693925    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c3ad95fd-e997-459c-93fd-27c4c030ea0d-metrics-certs podName:c3ad95fd-e997-459c-93fd-27c4c030ea0d nodeName:}" failed. No retries permitted until 2026-10-31 07:28:40.693912803 +0000 UTC m=+17.384683611 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c3ad95fd-e997-459c-93fd-27c4c030ea0d-metrics-certs") pod "network-metrics-daemon-r2zbv" (UID: "c3ad95fd-e997-459c-93fd-27c4c030ea0d") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 31 07:28:32.794295 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:32.793974    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ghkb9\" (UniqueName: \"kubernetes.io/projected/f1ba68f6-d87c-4492-a5eb-5f2d3a60eb1c-kube-api-access-ghkb9\") pod \"network-check-target-9zp4l\" (UID: \"f1ba68f6-d87c-4492-a5eb-5f2d3a60eb1c\") " pod="openshift-network-diagnostics/network-check-target-9zp4l"
Oct 31 07:28:32.794295 ip-10-0-139-83 kubenswrapper[2573]: E1031 07:28:32.794164    2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 31 07:28:32.794295 ip-10-0-139-83 kubenswrapper[2573]: E1031 07:28:32.794185    2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 31 07:28:32.794295 ip-10-0-139-83 kubenswrapper[2573]: E1031 07:28:32.794198    2573 projected.go:196] Error preparing data for projected volume kube-api-access-ghkb9 for pod openshift-network-diagnostics/network-check-target-9zp4l: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 31 07:28:32.794295 ip-10-0-139-83 kubenswrapper[2573]: E1031 07:28:32.794256    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f1ba68f6-d87c-4492-a5eb-5f2d3a60eb1c-kube-api-access-ghkb9 podName:f1ba68f6-d87c-4492-a5eb-5f2d3a60eb1c nodeName:}" failed. No retries permitted until 2026-10-31 07:28:40.794236418 +0000 UTC m=+17.485007229 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-ghkb9" (UniqueName: "kubernetes.io/projected/f1ba68f6-d87c-4492-a5eb-5f2d3a60eb1c-kube-api-access-ghkb9") pod "network-check-target-9zp4l" (UID: "f1ba68f6-d87c-4492-a5eb-5f2d3a60eb1c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 31 07:28:32.851737 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:32.851314    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9zp4l"
Oct 31 07:28:32.851737 ip-10-0-139-83 kubenswrapper[2573]: E1031 07:28:32.851458    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9zp4l" podUID="f1ba68f6-d87c-4492-a5eb-5f2d3a60eb1c"
Oct 31 07:28:32.851737 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:32.851521    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kht4l"
Oct 31 07:28:32.851737 ip-10-0-139-83 kubenswrapper[2573]: E1031 07:28:32.851571    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kht4l" podUID="77256436-39ff-4c2a-8390-4e9fa7d68486"
Oct 31 07:28:32.851737 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:32.851612    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r2zbv"
Oct 31 07:28:32.851737 ip-10-0-139-83 kubenswrapper[2573]: E1031 07:28:32.851662    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-r2zbv" podUID="c3ad95fd-e997-459c-93fd-27c4c030ea0d"
Oct 31 07:28:32.973914 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:32.973877    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-83.us-west-1.compute.internal" event={"ID":"8dfeb47fa079b05125409320442d08bf","Type":"ContainerStarted","Data":"7147ad859942ca65bf1b1ede84537503a2044ebcd89d053724d075b9b70bda3f"}
Oct 31 07:28:33.098670 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:33.098593    2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-83.us-west-1.compute.internal" podStartSLOduration=8.098572883 podStartE2EDuration="8.098572883s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 07:28:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 07:28:33.09771011 +0000 UTC m=+9.788480941" watchObservedRunningTime="2026-10-31 07:28:33.098572883 +0000 UTC m=+9.789343711"
Oct 31 07:28:33.979145 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:33.979000    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-797gj" event={"ID":"9dad8dd1-cfc5-412d-bf9a-bac3c9162af1","Type":"ContainerStarted","Data":"ad840500072aacdfd8acdf98d5fbf503d33cbf220bc36d2ae8e1698c3db989c9"}
Oct 31 07:28:34.562570 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:34.562533    2573 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-h4k2m"
Oct 31 07:28:34.563712 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:34.563464    2573 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-h4k2m"
Oct 31 07:28:34.853640 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:34.850768    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kht4l"
Oct 31 07:28:34.853640 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:34.850772    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9zp4l"
Oct 31 07:28:34.853640 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:34.850918    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r2zbv"
Oct 31 07:28:34.853640 ip-10-0-139-83 kubenswrapper[2573]: E1031 07:28:34.850924    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kht4l" podUID="77256436-39ff-4c2a-8390-4e9fa7d68486"
Oct 31 07:28:34.853640 ip-10-0-139-83 kubenswrapper[2573]: E1031 07:28:34.851004    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9zp4l" podUID="f1ba68f6-d87c-4492-a5eb-5f2d3a60eb1c"
Oct 31 07:28:34.853640 ip-10-0-139-83 kubenswrapper[2573]: E1031 07:28:34.851100    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-r2zbv" podUID="c3ad95fd-e997-459c-93fd-27c4c030ea0d"
Oct 31 07:28:34.984390 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:34.984335    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-pmnqx" event={"ID":"08198bc1-314f-4881-833c-e633bac15e4a","Type":"ContainerStarted","Data":"3dbd7f63dafa4ef1edc1e4b30eaafe5df8fc51b157ecb9b32cb4ff0cb01f224d"}
Oct 31 07:28:34.989495 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:34.989467    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-tnw4j" event={"ID":"c4233c69-4213-43be-8809-76612b6b6f3e","Type":"ContainerStarted","Data":"76090f760dd0ff0c58a0a3e20d5e1fb5ed00ed586656cfaec71ec93685f08311"}
Oct 31 07:28:34.989821 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:34.989798    2573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-h4k2m"
Oct 31 07:28:34.989947 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:34.989874    2573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-h4k2m"
Oct 31 07:28:35.208431 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:35.206751    2573 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Oct 31 07:28:35.231241 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:35.231192    2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-pmnqx" podStartSLOduration=1.045641072 podStartE2EDuration="10.231176041s" totalImagesPullingTime="9.185534969s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 07:28:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 07:28:35.108297686 +0000 UTC m=+11.799068539" watchObservedRunningTime="2026-10-31 07:28:35.231176041 +0000 UTC m=+11.921946868"
Oct 31 07:28:35.353159 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:35.352923    2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-tnw4j" podStartSLOduration=1.129745456 podStartE2EDuration="10.352908175s" totalImagesPullingTime="9.223162719s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 07:28:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 07:28:35.231694674 +0000 UTC m=+11.922465501" watchObservedRunningTime="2026-10-31 07:28:35.352908175 +0000 UTC m=+12.043679002"
Oct 31 07:28:35.787556 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:35.787421    2573 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-31T07:28:35.206770503Z","UUID":"bfc7f0dd-68b0-437e-9152-f79e96d59d9b","Handler":null,"Name":"","Endpoint":""}
Oct 31 07:28:35.789283 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:35.789259    2573 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Oct 31 07:28:35.789448 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:35.789292    2573 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Oct 31 07:28:35.993778 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:35.993700    2573 generic.go:356] "Generic (PLEG): container finished" podID="9c72279a-708f-46df-8915-9995c5aea14d" containerID="1984fc27154b51abf200eca7b767ac5b05187e6919cc510f5d47ba5be76a1a54" exitCode=0
Oct 31 07:28:35.994214 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:35.993994    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-dh6w8" event={"ID":"9c72279a-708f-46df-8915-9995c5aea14d","Type":"ContainerDied","Data":"1984fc27154b51abf200eca7b767ac5b05187e6919cc510f5d47ba5be76a1a54"}
Oct 31 07:28:35.997992 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:35.997961    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-797gj" event={"ID":"9dad8dd1-cfc5-412d-bf9a-bac3c9162af1","Type":"ContainerStarted","Data":"d26ac441066bbad7aa5c70de997b35daff24bcda1a6aa8a2265c46581f1f36b0"}
Oct 31 07:28:36.850976 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:36.850861    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9zp4l"
Oct 31 07:28:36.850976 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:36.850932    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r2zbv"
Oct 31 07:28:36.851200 ip-10-0-139-83 kubenswrapper[2573]: E1031 07:28:36.851059    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9zp4l" podUID="f1ba68f6-d87c-4492-a5eb-5f2d3a60eb1c"
Oct 31 07:28:36.851200 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:36.851096    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kht4l"
Oct 31 07:28:36.851280 ip-10-0-139-83 kubenswrapper[2573]: E1031 07:28:36.851219    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-r2zbv" podUID="c3ad95fd-e997-459c-93fd-27c4c030ea0d"
Oct 31 07:28:36.851321 ip-10-0-139-83 kubenswrapper[2573]: E1031 07:28:36.851300    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kht4l" podUID="77256436-39ff-4c2a-8390-4e9fa7d68486"
Oct 31 07:28:37.001646 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:37.001616    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-65hzp" event={"ID":"26659955-3909-4171-90b8-21b0ad6eef64","Type":"ContainerStarted","Data":"ec508f6ad689aa7e9bf64c45daf85c90c3d7ed3abd56fd13205ed84d92e9d4f7"}
Oct 31 07:28:37.114110 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:37.114008    2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-65hzp" podStartSLOduration=4.320343075 podStartE2EDuration="13.113981162s" totalImagesPullingTime="8.793638087s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 07:28:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 07:28:37.113183725 +0000 UTC m=+13.803954549" watchObservedRunningTime="2026-10-31 07:28:37.113981162 +0000 UTC m=+13.804751988"
Oct 31 07:28:38.006740 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:38.006663    2573 generic.go:356] "Generic (PLEG): container finished" podID="9c72279a-708f-46df-8915-9995c5aea14d" containerID="acf8ef1de7748f37a3862bc7e78950c0776180ce342a1c0eb6c64b0fdb516e73" exitCode=0
Oct 31 07:28:38.007364 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:38.006748    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-dh6w8" event={"ID":"9c72279a-708f-46df-8915-9995c5aea14d","Type":"ContainerDied","Data":"acf8ef1de7748f37a3862bc7e78950c0776180ce342a1c0eb6c64b0fdb516e73"}
Oct 31 07:28:38.009748 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:38.009714    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-797gj" event={"ID":"9dad8dd1-cfc5-412d-bf9a-bac3c9162af1","Type":"ContainerStarted","Data":"4c3dc042ecaf5b48ea4fd2b760b7c1c395048be6ac5f255ad1ccac5f8fc850f2"}
Oct 31 07:28:38.011375 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:38.011315    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-cpn4m" event={"ID":"3fa078c7-4bf2-4e76-9692-ae460e8fadfa","Type":"ContainerStarted","Data":"a7f70009f2cac4bf4e329ab8bf1a55c6f6ed79dd9b9cd376946cb05b1f4e442a"}
Oct 31 07:28:38.237815 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:38.237752    2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-cpn4m" podStartSLOduration=2.96722111 podStartE2EDuration="14.23773884s" totalImagesPullingTime="11.27051773s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 07:28:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 07:28:38.237086693 +0000 UTC m=+14.927857517" watchObservedRunningTime="2026-10-31 07:28:38.23773884 +0000 UTC m=+14.928509665"
Oct 31 07:28:38.350996 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:38.350905    2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-797gj" podStartSLOduration=3.973535686 podStartE2EDuration="14.350886616s" totalImagesPullingTime="10.37735093s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 07:28:24 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 07:28:38.350576602 +0000 UTC m=+15.041347427" watchObservedRunningTime="2026-10-31 07:28:38.350886616 +0000 UTC m=+15.041657443"
Oct 31 07:28:38.851287 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:38.851259    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9zp4l"
Oct 31 07:28:38.851486 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:38.851300    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kht4l"
Oct 31 07:28:38.851486 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:38.851378    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r2zbv"
Oct 31 07:28:38.851486 ip-10-0-139-83 kubenswrapper[2573]: E1031 07:28:38.851394    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9zp4l" podUID="f1ba68f6-d87c-4492-a5eb-5f2d3a60eb1c"
Oct 31 07:28:38.851486 ip-10-0-139-83 kubenswrapper[2573]: E1031 07:28:38.851470    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kht4l" podUID="77256436-39ff-4c2a-8390-4e9fa7d68486"
Oct 31 07:28:38.851656 ip-10-0-139-83 kubenswrapper[2573]: E1031 07:28:38.851538    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-r2zbv" podUID="c3ad95fd-e997-459c-93fd-27c4c030ea0d"
Oct 31 07:28:40.018574 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:40.018289    2573 generic.go:356] "Generic (PLEG): container finished" podID="9c72279a-708f-46df-8915-9995c5aea14d" containerID="bfe4d1e06bbdacaeac725652d553ddb60e20fa727a3734cdbfe038de3598eb2f" exitCode=0
Oct 31 07:28:40.018574 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:40.018370    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-dh6w8" event={"ID":"9c72279a-708f-46df-8915-9995c5aea14d","Type":"ContainerDied","Data":"bfe4d1e06bbdacaeac725652d553ddb60e20fa727a3734cdbfe038de3598eb2f"}
Oct 31 07:28:40.763948 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:40.763914    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/77256436-39ff-4c2a-8390-4e9fa7d68486-original-pull-secret\") pod \"global-pull-secret-syncer-kht4l\" (UID: \"77256436-39ff-4c2a-8390-4e9fa7d68486\") " pod="kube-system/global-pull-secret-syncer-kht4l"
Oct 31 07:28:40.764126 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:40.763963    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c3ad95fd-e997-459c-93fd-27c4c030ea0d-metrics-certs\") pod \"network-metrics-daemon-r2zbv\" (UID: \"c3ad95fd-e997-459c-93fd-27c4c030ea0d\") " pod="openshift-multus/network-metrics-daemon-r2zbv"
Oct 31 07:28:40.764126 ip-10-0-139-83 kubenswrapper[2573]: E1031 07:28:40.764092    2573 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 31 07:28:40.764126 ip-10-0-139-83 kubenswrapper[2573]: E1031 07:28:40.764095    2573 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 31 07:28:40.764254 ip-10-0-139-83 kubenswrapper[2573]: E1031 07:28:40.764160    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c3ad95fd-e997-459c-93fd-27c4c030ea0d-metrics-certs podName:c3ad95fd-e997-459c-93fd-27c4c030ea0d nodeName:}" failed. No retries permitted until 2026-10-31 07:28:56.764143687 +0000 UTC m=+33.454914504 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c3ad95fd-e997-459c-93fd-27c4c030ea0d-metrics-certs") pod "network-metrics-daemon-r2zbv" (UID: "c3ad95fd-e997-459c-93fd-27c4c030ea0d") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 31 07:28:40.764254 ip-10-0-139-83 kubenswrapper[2573]: E1031 07:28:40.764174    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/77256436-39ff-4c2a-8390-4e9fa7d68486-original-pull-secret podName:77256436-39ff-4c2a-8390-4e9fa7d68486 nodeName:}" failed. No retries permitted until 2026-10-31 07:28:56.764168271 +0000 UTC m=+33.454939073 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/77256436-39ff-4c2a-8390-4e9fa7d68486-original-pull-secret") pod "global-pull-secret-syncer-kht4l" (UID: "77256436-39ff-4c2a-8390-4e9fa7d68486") : object "kube-system"/"original-pull-secret" not registered
Oct 31 07:28:40.850700 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:40.850667    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9zp4l"
Oct 31 07:28:40.850700 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:40.850690    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kht4l"
Oct 31 07:28:40.850917 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:40.850765    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r2zbv"
Oct 31 07:28:40.850917 ip-10-0-139-83 kubenswrapper[2573]: E1031 07:28:40.850790    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9zp4l" podUID="f1ba68f6-d87c-4492-a5eb-5f2d3a60eb1c"
Oct 31 07:28:40.850917 ip-10-0-139-83 kubenswrapper[2573]: E1031 07:28:40.850898    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-r2zbv" podUID="c3ad95fd-e997-459c-93fd-27c4c030ea0d"
Oct 31 07:28:40.851080 ip-10-0-139-83 kubenswrapper[2573]: E1031 07:28:40.850945    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kht4l" podUID="77256436-39ff-4c2a-8390-4e9fa7d68486"
Oct 31 07:28:40.864834 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:40.864806    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ghkb9\" (UniqueName: \"kubernetes.io/projected/f1ba68f6-d87c-4492-a5eb-5f2d3a60eb1c-kube-api-access-ghkb9\") pod \"network-check-target-9zp4l\" (UID: \"f1ba68f6-d87c-4492-a5eb-5f2d3a60eb1c\") " pod="openshift-network-diagnostics/network-check-target-9zp4l"
Oct 31 07:28:40.864997 ip-10-0-139-83 kubenswrapper[2573]: E1031 07:28:40.864979    2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 31 07:28:40.865057 ip-10-0-139-83 kubenswrapper[2573]: E1031 07:28:40.865005    2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 31 07:28:40.865057 ip-10-0-139-83 kubenswrapper[2573]: E1031 07:28:40.865017    2573 projected.go:196] Error preparing data for projected volume kube-api-access-ghkb9 for pod openshift-network-diagnostics/network-check-target-9zp4l: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 31 07:28:40.865139 ip-10-0-139-83 kubenswrapper[2573]: E1031 07:28:40.865074    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f1ba68f6-d87c-4492-a5eb-5f2d3a60eb1c-kube-api-access-ghkb9 podName:f1ba68f6-d87c-4492-a5eb-5f2d3a60eb1c nodeName:}" failed. No retries permitted until 2026-10-31 07:28:56.865055996 +0000 UTC m=+33.555826812 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-ghkb9" (UniqueName: "kubernetes.io/projected/f1ba68f6-d87c-4492-a5eb-5f2d3a60eb1c-kube-api-access-ghkb9") pod "network-check-target-9zp4l" (UID: "f1ba68f6-d87c-4492-a5eb-5f2d3a60eb1c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 31 07:28:42.026155 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:42.026125    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-5p7xw" event={"ID":"4bfbebfd-2be7-438f-b88d-273f24751f8c","Type":"ContainerStarted","Data":"4021d9aaeb4d41a6d4aacbec62476aff5df79b6c40550269b8f8b9a36375b073"}
Oct 31 07:28:42.026613 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:42.026198    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-5p7xw" event={"ID":"4bfbebfd-2be7-438f-b88d-273f24751f8c","Type":"ContainerStarted","Data":"1a8399fbdd54a7ef11a5364e302cbaa6cb1a5f88d809979caa8fd36717a1a050"}
Oct 31 07:28:42.026613 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:42.026231    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-5p7xw" event={"ID":"4bfbebfd-2be7-438f-b88d-273f24751f8c","Type":"ContainerStarted","Data":"f8b7db30c5a55a8481f6deb372912644f56a2f1b927ee410e5d44f30d53abc71"}
Oct 31 07:28:42.026613 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:42.026258    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-5p7xw" event={"ID":"4bfbebfd-2be7-438f-b88d-273f24751f8c","Type":"ContainerStarted","Data":"5747189ec15344c867c2276333d2726a31925a40a5175997bf25f9e2124a9b3d"}
Oct 31 07:28:42.026613 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:42.026287    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-5p7xw" event={"ID":"4bfbebfd-2be7-438f-b88d-273f24751f8c","Type":"ContainerStarted","Data":"08343bb014d311d318bdea25c735846e44d994833f141c55d861b4d29a166763"}
Oct 31 07:28:42.026613 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:42.026314    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-5p7xw" event={"ID":"4bfbebfd-2be7-438f-b88d-273f24751f8c","Type":"ContainerStarted","Data":"5a20851551391694d3acac96ae31ec373a3af87c6a1abf26e1b52997951d7cca"}
Oct 31 07:28:42.851149 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:42.850961    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9zp4l"
Oct 31 07:28:42.851321 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:42.850999    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kht4l"
Oct 31 07:28:42.851321 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:42.851032    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r2zbv"
Oct 31 07:28:42.851321 ip-10-0-139-83 kubenswrapper[2573]: E1031 07:28:42.851210    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9zp4l" podUID="f1ba68f6-d87c-4492-a5eb-5f2d3a60eb1c"
Oct 31 07:28:42.851321 ip-10-0-139-83 kubenswrapper[2573]: E1031 07:28:42.851309    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kht4l" podUID="77256436-39ff-4c2a-8390-4e9fa7d68486"
Oct 31 07:28:42.851479 ip-10-0-139-83 kubenswrapper[2573]: E1031 07:28:42.851386    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-r2zbv" podUID="c3ad95fd-e997-459c-93fd-27c4c030ea0d"
Oct 31 07:28:43.030787 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:43.030759    2573 generic.go:356] "Generic (PLEG): container finished" podID="9c72279a-708f-46df-8915-9995c5aea14d" containerID="3f7eeca7807c5d5ea10ecc01516192cae358f4c8a3ce534d2ec240a880a25abc" exitCode=0
Oct 31 07:28:43.031128 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:43.030812    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-dh6w8" event={"ID":"9c72279a-708f-46df-8915-9995c5aea14d","Type":"ContainerDied","Data":"3f7eeca7807c5d5ea10ecc01516192cae358f4c8a3ce534d2ec240a880a25abc"}
Oct 31 07:28:44.036892 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:44.036863    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-5p7xw" event={"ID":"4bfbebfd-2be7-438f-b88d-273f24751f8c","Type":"ContainerStarted","Data":"86abf9933cb77b2aa692df3aa815ba2901831c9f738e0d514afb31ae42bf4f0d"}
Oct 31 07:28:44.040150 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:44.040117    2573 generic.go:356] "Generic (PLEG): container finished" podID="9c72279a-708f-46df-8915-9995c5aea14d" containerID="5983783fae630464092023d00a7fd5341a04199f277281a2a7f800c2c9041fe5" exitCode=0
Oct 31 07:28:44.040229 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:44.040167    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-dh6w8" event={"ID":"9c72279a-708f-46df-8915-9995c5aea14d","Type":"ContainerDied","Data":"5983783fae630464092023d00a7fd5341a04199f277281a2a7f800c2c9041fe5"}
Oct 31 07:28:44.850959 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:44.850930    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9zp4l"
Oct 31 07:28:44.851114 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:44.850971    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kht4l"
Oct 31 07:28:44.851114 ip-10-0-139-83 kubenswrapper[2573]: E1031 07:28:44.851045    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9zp4l" podUID="f1ba68f6-d87c-4492-a5eb-5f2d3a60eb1c"
Oct 31 07:28:44.851114 ip-10-0-139-83 kubenswrapper[2573]: E1031 07:28:44.851077    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kht4l" podUID="77256436-39ff-4c2a-8390-4e9fa7d68486"
Oct 31 07:28:44.851208 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:44.851119    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r2zbv"
Oct 31 07:28:44.851208 ip-10-0-139-83 kubenswrapper[2573]: E1031 07:28:44.851171    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-r2zbv" podUID="c3ad95fd-e997-459c-93fd-27c4c030ea0d"
Oct 31 07:28:45.047172 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:45.047138    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-dh6w8" event={"ID":"9c72279a-708f-46df-8915-9995c5aea14d","Type":"ContainerStarted","Data":"5ab192d471d96c352d370b941685c914564da9566e0474e9413cc908fca2f114"}
Oct 31 07:28:45.163923 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:45.163830    2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-dh6w8" podStartSLOduration=6.949667756 podStartE2EDuration="20.163817528s" totalImagesPullingTime="13.214149772s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 07:28:25 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 07:28:45.163267481 +0000 UTC m=+21.854038304" watchObservedRunningTime="2026-10-31 07:28:45.163817528 +0000 UTC m=+21.854588351"
Oct 31 07:28:46.850626 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:46.850588    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9zp4l"
Oct 31 07:28:46.851149 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:46.850608    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kht4l"
Oct 31 07:28:46.851149 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:46.850700    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r2zbv"
Oct 31 07:28:46.851149 ip-10-0-139-83 kubenswrapper[2573]: E1031 07:28:46.850701    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9zp4l" podUID="f1ba68f6-d87c-4492-a5eb-5f2d3a60eb1c"
Oct 31 07:28:46.851149 ip-10-0-139-83 kubenswrapper[2573]: E1031 07:28:46.850767    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kht4l" podUID="77256436-39ff-4c2a-8390-4e9fa7d68486"
Oct 31 07:28:46.851149 ip-10-0-139-83 kubenswrapper[2573]: E1031 07:28:46.850850    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-r2zbv" podUID="c3ad95fd-e997-459c-93fd-27c4c030ea0d"
Oct 31 07:28:47.055313 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:47.055286    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-5p7xw" event={"ID":"4bfbebfd-2be7-438f-b88d-273f24751f8c","Type":"ContainerStarted","Data":"37aadcfa9b099a338550a2ca8e718fab357bf797f8b98ea0ba4a4415172ca015"}
Oct 31 07:28:47.055575 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:47.055560    2573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-5p7xw"
Oct 31 07:28:47.055650 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:47.055628    2573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-5p7xw"
Oct 31 07:28:47.071146 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:47.071123    2573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-5p7xw"
Oct 31 07:28:47.329754 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:47.329305    2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-5p7xw" podStartSLOduration=7.838919808 podStartE2EDuration="23.329277182s" totalImagesPullingTime="15.490357374s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 07:28:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 07:28:47.273817159 +0000 UTC m=+23.964587985" watchObservedRunningTime="2026-10-31 07:28:47.329277182 +0000 UTC m=+24.020048007"
Oct 31 07:28:48.058006 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:48.057981    2573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-5p7xw"
Oct 31 07:28:48.073608 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:48.073588    2573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-5p7xw"
Oct 31 07:28:48.842426 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:48.842392    2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-kht4l"]
Oct 31 07:28:48.842610 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:48.842565    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kht4l"
Oct 31 07:28:48.842693 ip-10-0-139-83 kubenswrapper[2573]: E1031 07:28:48.842672    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kht4l" podUID="77256436-39ff-4c2a-8390-4e9fa7d68486"
Oct 31 07:28:48.849741 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:48.849518    2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-9zp4l"]
Oct 31 07:28:48.849741 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:48.849626    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9zp4l"
Oct 31 07:28:48.849741 ip-10-0-139-83 kubenswrapper[2573]: E1031 07:28:48.849704    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9zp4l" podUID="f1ba68f6-d87c-4492-a5eb-5f2d3a60eb1c"
Oct 31 07:28:48.850121 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:48.850101    2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-r2zbv"]
Oct 31 07:28:48.850205 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:48.850194    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r2zbv"
Oct 31 07:28:48.850301 ip-10-0-139-83 kubenswrapper[2573]: E1031 07:28:48.850281    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-r2zbv" podUID="c3ad95fd-e997-459c-93fd-27c4c030ea0d"
Oct 31 07:28:49.059148 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:49.059116    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r2zbv"
Oct 31 07:28:49.059611 ip-10-0-139-83 kubenswrapper[2573]: E1031 07:28:49.059243    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-r2zbv" podUID="c3ad95fd-e997-459c-93fd-27c4c030ea0d"
Oct 31 07:28:50.850575 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:50.850549    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9zp4l"
Oct 31 07:28:50.850962 ip-10-0-139-83 kubenswrapper[2573]: E1031 07:28:50.850715    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9zp4l" podUID="f1ba68f6-d87c-4492-a5eb-5f2d3a60eb1c"
Oct 31 07:28:50.850962 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:50.850595    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r2zbv"
Oct 31 07:28:50.850962 ip-10-0-139-83 kubenswrapper[2573]: E1031 07:28:50.850830    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-r2zbv" podUID="c3ad95fd-e997-459c-93fd-27c4c030ea0d"
Oct 31 07:28:50.850962 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:50.850575    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kht4l"
Oct 31 07:28:50.850962 ip-10-0-139-83 kubenswrapper[2573]: E1031 07:28:50.850905    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kht4l" podUID="77256436-39ff-4c2a-8390-4e9fa7d68486"
Oct 31 07:28:52.851243 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:52.851205    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9zp4l"
Oct 31 07:28:52.851243 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:52.851246    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r2zbv"
Oct 31 07:28:52.851610 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:52.851215    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kht4l"
Oct 31 07:28:52.851610 ip-10-0-139-83 kubenswrapper[2573]: E1031 07:28:52.851315    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9zp4l" podUID="f1ba68f6-d87c-4492-a5eb-5f2d3a60eb1c"
Oct 31 07:28:52.851610 ip-10-0-139-83 kubenswrapper[2573]: E1031 07:28:52.851386    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kht4l" podUID="77256436-39ff-4c2a-8390-4e9fa7d68486"
Oct 31 07:28:52.851610 ip-10-0-139-83 kubenswrapper[2573]: E1031 07:28:52.851448    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-r2zbv" podUID="c3ad95fd-e997-459c-93fd-27c4c030ea0d"
Oct 31 07:28:54.581989 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:54.581968    2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-83.us-west-1.compute.internal" event="NodeReady"
Oct 31 07:28:54.582342 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:54.582065    2573 kubelet_node_status.go:477] "Fast updating node status as it just became ready"
Oct 31 07:28:54.734789 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:54.734759    2573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-xcrsn"]
Oct 31 07:28:54.809057 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:54.809017    2573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-hxmpf"]
Oct 31 07:28:54.809909 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:54.809891    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-xcrsn"
Oct 31 07:28:54.821655 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:54.821636    2573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-f9gtg"]
Oct 31 07:28:54.821763 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:54.821749    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-hxmpf"
Oct 31 07:28:54.833699 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:54.833649    2573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-7c4986d449-dwc4p"]
Oct 31 07:28:54.833785 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:54.833740    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-f9gtg"
Oct 31 07:28:54.857007 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:54.856986    2573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-6vkbv"]
Oct 31 07:28:54.857120 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:54.857107    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-7c4986d449-dwc4p"
Oct 31 07:28:54.862231 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:54.862213    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"kube-root-ca.crt\""
Oct 31 07:28:54.862395 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:54.862277    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"openshift-service-ca.crt\""
Oct 31 07:28:54.877905 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:54.877888    2573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-65chl"]
Oct 31 07:28:54.877999 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:54.877961    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-6vkbv"
Oct 31 07:28:54.890950 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:54.890934    2573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-6c88c45d57-c2bgn"]
Oct 31 07:28:54.891040 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:54.891017    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r2zbv"
Oct 31 07:28:54.891088 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:54.891034    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kht4l"
Oct 31 07:28:54.891137 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:54.891119    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-65chl"
Oct 31 07:28:54.891284 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:54.891266    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9zp4l"
Oct 31 07:28:54.905263 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:54.905248    2573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-7886c7d6c6-qdc9f"]
Oct 31 07:28:54.905418 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:54.905400    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-6c88c45d57-c2bgn"
Oct 31 07:28:54.910362 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:54.909523    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\""
Oct 31 07:28:54.919353 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:54.919311    2573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-6k8mn"]
Oct 31 07:28:54.919434 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:54.919391    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7886c7d6c6-qdc9f"
Oct 31 07:28:54.931577 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:54.931561    2573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-hb49h"]
Oct 31 07:28:54.931674 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:54.931660    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-6k8mn"
Oct 31 07:28:54.949223 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:54.949201    2573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-7955c67bcc-pkntv"]
Oct 31 07:28:54.949398 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:54.949384    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-hb49h"
Oct 31 07:28:54.963107 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:54.963089    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\""
Oct 31 07:28:54.963390 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:54.963373    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Oct 31 07:28:54.963442 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:54.963386    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Oct 31 07:28:54.963442 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:54.963421    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\""
Oct 31 07:28:54.963442 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:54.963435    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\""
Oct 31 07:28:54.963521 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:54.963435    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Oct 31 07:28:54.963678 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:54.963654    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\""
Oct 31 07:28:54.963678 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:54.963660    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\""
Oct 31 07:28:54.963678 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:54.963672    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Oct 31 07:28:54.963895 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:54.963674    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-config\""
Oct 31 07:28:54.963895 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:54.963696    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Oct 31 07:28:54.963895 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:54.963764    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Oct 31 07:28:54.963895 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:54.963891    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\""
Oct 31 07:28:54.966679 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:54.966662    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k6576\" (UniqueName: \"kubernetes.io/projected/81829763-27b1-4a48-a568-cd5f6a39a953-kube-api-access-k6576\") pod \"insights-operator-7c4986d449-dwc4p\" (UID: \"81829763-27b1-4a48-a568-cd5f6a39a953\") " pod="openshift-insights/insights-operator-7c4986d449-dwc4p"
Oct 31 07:28:54.966744 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:54.966688    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/ca31519e-aa90-4104-a175-f9af2b3bbeb4-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-6vkbv\" (UID: \"ca31519e-aa90-4104-a175-f9af2b3bbeb4\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-6vkbv"
Oct 31 07:28:54.966744 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:54.966705    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/81829763-27b1-4a48-a568-cd5f6a39a953-tmp\") pod \"insights-operator-7c4986d449-dwc4p\" (UID: \"81829763-27b1-4a48-a568-cd5f6a39a953\") " pod="openshift-insights/insights-operator-7c4986d449-dwc4p"
Oct 31 07:28:54.966744 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:54.966721    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/81829763-27b1-4a48-a568-cd5f6a39a953-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-dwc4p\" (UID: \"81829763-27b1-4a48-a568-cd5f6a39a953\") " pod="openshift-insights/insights-operator-7c4986d449-dwc4p"
Oct 31 07:28:54.966824 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:54.966785    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/81829763-27b1-4a48-a568-cd5f6a39a953-service-ca-bundle\") pod \"insights-operator-7c4986d449-dwc4p\" (UID: \"81829763-27b1-4a48-a568-cd5f6a39a953\") " pod="openshift-insights/insights-operator-7c4986d449-dwc4p"
Oct 31 07:28:54.966872 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:54.966856    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sn2lx\" (UniqueName: \"kubernetes.io/projected/91363236-08db-4f54-ae0a-e68604fcecc6-kube-api-access-sn2lx\") pod \"kas-connection-checker-685895c7c9-xcrsn\" (UID: \"91363236-08db-4f54-ae0a-e68604fcecc6\") " pod="kube-system/kas-connection-checker-685895c7c9-xcrsn"
Oct 31 07:28:54.966906 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:54.966886    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n6p25\" (UniqueName: \"kubernetes.io/projected/ca31519e-aa90-4104-a175-f9af2b3bbeb4-kube-api-access-n6p25\") pod \"cluster-samples-operator-6ddb6c8b94-6vkbv\" (UID: \"ca31519e-aa90-4104-a175-f9af2b3bbeb4\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-6vkbv"
Oct 31 07:28:54.966906 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:54.966902    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l4nfd\" (UniqueName: \"kubernetes.io/projected/323c8334-9a56-4c4e-8a6e-ae80995810d0-kube-api-access-l4nfd\") pod \"kas-connection-checker-685895c7c9-hxmpf\" (UID: \"323c8334-9a56-4c4e-8a6e-ae80995810d0\") " pod="kube-system/kas-connection-checker-685895c7c9-hxmpf"
Oct 31 07:28:54.966956 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:54.966916    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q7vwq\" (UniqueName: \"kubernetes.io/projected/ee4a4a2a-7439-4c78-8858-d6ba0085fdde-kube-api-access-q7vwq\") pod \"volume-data-source-validator-58c9f5f46d-65chl\" (UID: \"ee4a4a2a-7439-4c78-8858-d6ba0085fdde\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-65chl"
Oct 31 07:28:54.966956 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:54.966938    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/81829763-27b1-4a48-a568-cd5f6a39a953-snapshots\") pod \"insights-operator-7c4986d449-dwc4p\" (UID: \"81829763-27b1-4a48-a568-cd5f6a39a953\") " pod="openshift-insights/insights-operator-7c4986d449-dwc4p"
Oct 31 07:28:54.967010 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:54.966987    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/81829763-27b1-4a48-a568-cd5f6a39a953-serving-cert\") pod \"insights-operator-7c4986d449-dwc4p\" (UID: \"81829763-27b1-4a48-a568-cd5f6a39a953\") " pod="openshift-insights/insights-operator-7c4986d449-dwc4p"
Oct 31 07:28:54.967095 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:54.967083    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dx598\" (UniqueName: \"kubernetes.io/projected/631cff9e-a1f1-4dd2-bdc8-ef36519c7c1b-kube-api-access-dx598\") pod \"kas-connection-checker-685895c7c9-f9gtg\" (UID: \"631cff9e-a1f1-4dd2-bdc8-ef36519c7c1b\") " pod="kube-system/kas-connection-checker-685895c7c9-f9gtg"
Oct 31 07:28:54.967132 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:54.967121    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/ca31519e-aa90-4104-a175-f9af2b3bbeb4-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-6vkbv\" (UID: \"ca31519e-aa90-4104-a175-f9af2b3bbeb4\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-6vkbv"
Oct 31 07:28:54.980376 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:54.980357    2573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-p9z4r"]
Oct 31 07:28:54.980460 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:54.980447    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7955c67bcc-pkntv"
Oct 31 07:28:55.002939 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:55.002923    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-p9z4r"
Oct 31 07:28:55.003020 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:55.002922    2573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-4jszp"]
Oct 31 07:28:55.013808 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:55.013787    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\""
Oct 31 07:28:55.014682 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:55.014663    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\""
Oct 31 07:28:55.014797 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:55.014781    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Oct 31 07:28:55.014881 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:55.014795    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\""
Oct 31 07:28:55.014881 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:55.014821    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-4d4mq\""
Oct 31 07:28:55.014881 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:55.014839    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\""
Oct 31 07:28:55.014881 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:55.014878    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Oct 31 07:28:55.015053 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:55.014903    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\""
Oct 31 07:28:55.015053 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:55.014942    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Oct 31 07:28:55.015053 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:55.014954    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\""
Oct 31 07:28:55.015053 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:55.014941    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\""
Oct 31 07:28:55.015053 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:55.014975    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Oct 31 07:28:55.015053 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:55.014993    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-bpwql\""
Oct 31 07:28:55.015053 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:55.015010    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Oct 31 07:28:55.015230 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:55.015158    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Oct 31 07:28:55.015230 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:55.015160    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-crxcw\""
Oct 31 07:28:55.015230 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:55.015204    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-5qqnh\""
Oct 31 07:28:55.017174 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:55.017159    2573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-j94dz"]
Oct 31 07:28:55.017263 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:55.017253    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-7cd56b8dbc-4jszp"
Oct 31 07:28:55.030798 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:55.030781    2573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-fqhzf"]
Oct 31 07:28:55.030888 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:55.030876    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-86c478bb88-j94dz"
Oct 31 07:28:55.032976 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:55.032940    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-w5nxl\""
Oct 31 07:28:55.051266 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:55.051248    2573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-kmcg5"]
Oct 31 07:28:55.051406 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:55.051392    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-fqhzf"
Oct 31 07:28:55.064680 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:55.064660    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Oct 31 07:28:55.064680 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:55.064675    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\""
Oct 31 07:28:55.064847 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:55.064666    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\""
Oct 31 07:28:55.064847 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:55.064685    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\""
Oct 31 07:28:55.064847 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:55.064731    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\""
Oct 31 07:28:55.064847 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:55.064787    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\""
Oct 31 07:28:55.065013 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:55.064932    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Oct 31 07:28:55.065013 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:55.064937    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\""
Oct 31 07:28:55.065082 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:55.065070    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\""
Oct 31 07:28:55.068138 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:55.068120    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/4db4c33f-0c58-40f8-bf74-51b3eae3c9db-config\") pod \"service-ca-operator-544d749c8c-hb49h\" (UID: \"4db4c33f-0c58-40f8-bf74-51b3eae3c9db\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-hb49h"
Oct 31 07:28:55.068208 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:55.068149    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/ca31519e-aa90-4104-a175-f9af2b3bbeb4-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-6vkbv\" (UID: \"ca31519e-aa90-4104-a175-f9af2b3bbeb4\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-6vkbv"
Oct 31 07:28:55.068208 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:55.068165    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/789f11d8-d951-4144-a589-a2bdbf6844e3-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-6k8mn\" (UID: \"789f11d8-d951-4144-a589-a2bdbf6844e3\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-6k8mn"
Oct 31 07:28:55.068208 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:55.068189    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v2zhk\" (UniqueName: \"kubernetes.io/projected/993cfed6-806d-44c5-bba8-c02203638fe7-kube-api-access-v2zhk\") pod \"image-registry-7886c7d6c6-qdc9f\" (UID: \"993cfed6-806d-44c5-bba8-c02203638fe7\") " pod="openshift-image-registry/image-registry-7886c7d6c6-qdc9f"
Oct 31 07:28:55.068361 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:55.068246    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/81829763-27b1-4a48-a568-cd5f6a39a953-tmp\") pod \"insights-operator-7c4986d449-dwc4p\" (UID: \"81829763-27b1-4a48-a568-cd5f6a39a953\") " pod="openshift-insights/insights-operator-7c4986d449-dwc4p"
Oct 31 07:28:55.068361 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:55.068264    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/81829763-27b1-4a48-a568-cd5f6a39a953-service-ca-bundle\") pod \"insights-operator-7c4986d449-dwc4p\" (UID: \"81829763-27b1-4a48-a568-cd5f6a39a953\") " pod="openshift-insights/insights-operator-7c4986d449-dwc4p"
Oct 31 07:28:55.068361 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:55.068285    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/28b59d93-b9ce-4ceb-9f3a-742525eb94c4-ca-trust-extracted\") pod \"image-registry-7955c67bcc-pkntv\" (UID: \"28b59d93-b9ce-4ceb-9f3a-742525eb94c4\") " pod="openshift-image-registry/image-registry-7955c67bcc-pkntv"
Oct 31 07:28:55.068551 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:55.068358    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/993cfed6-806d-44c5-bba8-c02203638fe7-bound-sa-token\") pod \"image-registry-7886c7d6c6-qdc9f\" (UID: \"993cfed6-806d-44c5-bba8-c02203638fe7\") " pod="openshift-image-registry/image-registry-7886c7d6c6-qdc9f"
Oct 31 07:28:55.068551 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:55.068420    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/993cfed6-806d-44c5-bba8-c02203638fe7-ca-trust-extracted\") pod \"image-registry-7886c7d6c6-qdc9f\" (UID: \"993cfed6-806d-44c5-bba8-c02203638fe7\") " pod="openshift-image-registry/image-registry-7886c7d6c6-qdc9f"
Oct 31 07:28:55.068551 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:55.068478    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/993cfed6-806d-44c5-bba8-c02203638fe7-registry-tls\") pod \"image-registry-7886c7d6c6-qdc9f\" (UID: \"993cfed6-806d-44c5-bba8-c02203638fe7\") " pod="openshift-image-registry/image-registry-7886c7d6c6-qdc9f"
Oct 31 07:28:55.068551 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:55.068504    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/993cfed6-806d-44c5-bba8-c02203638fe7-trusted-ca\") pod \"image-registry-7886c7d6c6-qdc9f\" (UID: \"993cfed6-806d-44c5-bba8-c02203638fe7\") " pod="openshift-image-registry/image-registry-7886c7d6c6-qdc9f"
Oct 31 07:28:55.068745 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:55.068569    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sn2lx\" (UniqueName: \"kubernetes.io/projected/91363236-08db-4f54-ae0a-e68604fcecc6-kube-api-access-sn2lx\") pod \"kas-connection-checker-685895c7c9-xcrsn\" (UID: \"91363236-08db-4f54-ae0a-e68604fcecc6\") " pod="kube-system/kas-connection-checker-685895c7c9-xcrsn"
Oct 31 07:28:55.068745 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:55.068610    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n6p25\" (UniqueName: \"kubernetes.io/projected/ca31519e-aa90-4104-a175-f9af2b3bbeb4-kube-api-access-n6p25\") pod \"cluster-samples-operator-6ddb6c8b94-6vkbv\" (UID: \"ca31519e-aa90-4104-a175-f9af2b3bbeb4\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-6vkbv"
Oct 31 07:28:55.068745 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:55.068650    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l4nfd\" (UniqueName: \"kubernetes.io/projected/323c8334-9a56-4c4e-8a6e-ae80995810d0-kube-api-access-l4nfd\") pod \"kas-connection-checker-685895c7c9-hxmpf\" (UID: \"323c8334-9a56-4c4e-8a6e-ae80995810d0\") " pod="kube-system/kas-connection-checker-685895c7c9-hxmpf"
Oct 31 07:28:55.068745 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:55.068687    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q7vwq\" (UniqueName: \"kubernetes.io/projected/ee4a4a2a-7439-4c78-8858-d6ba0085fdde-kube-api-access-q7vwq\") pod \"volume-data-source-validator-58c9f5f46d-65chl\" (UID: \"ee4a4a2a-7439-4c78-8858-d6ba0085fdde\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-65chl"
Oct 31 07:28:55.068745 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:55.068712    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/28b59d93-b9ce-4ceb-9f3a-742525eb94c4-installation-pull-secrets\") pod \"image-registry-7955c67bcc-pkntv\" (UID: \"28b59d93-b9ce-4ceb-9f3a-742525eb94c4\") " pod="openshift-image-registry/image-registry-7955c67bcc-pkntv"
Oct 31 07:28:55.068745 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:55.068739    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b44ed0e0-a72e-4746-a7de-ed8610da845d-service-ca-bundle\") pod \"router-default-6c88c45d57-c2bgn\" (UID: \"b44ed0e0-a72e-4746-a7de-ed8610da845d\") " pod="openshift-ingress/router-default-6c88c45d57-c2bgn"
Oct 31 07:28:55.068994 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:55.068766    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/28b59d93-b9ce-4ceb-9f3a-742525eb94c4-registry-certificates\") pod \"image-registry-7955c67bcc-pkntv\" (UID: \"28b59d93-b9ce-4ceb-9f3a-742525eb94c4\") " pod="openshift-image-registry/image-registry-7955c67bcc-pkntv"
Oct 31 07:28:55.068994 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:55.068814    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dx598\" (UniqueName: \"kubernetes.io/projected/631cff9e-a1f1-4dd2-bdc8-ef36519c7c1b-kube-api-access-dx598\") pod \"kas-connection-checker-685895c7c9-f9gtg\" (UID: \"631cff9e-a1f1-4dd2-bdc8-ef36519c7c1b\") " pod="kube-system/kas-connection-checker-685895c7c9-f9gtg"
Oct 31 07:28:55.068994 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:55.068903    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/81829763-27b1-4a48-a568-cd5f6a39a953-service-ca-bundle\") pod \"insights-operator-7c4986d449-dwc4p\" (UID: \"81829763-27b1-4a48-a568-cd5f6a39a953\") " pod="openshift-insights/insights-operator-7c4986d449-dwc4p"
Oct 31 07:28:55.068994 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:55.068933    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/ca31519e-aa90-4104-a175-f9af2b3bbeb4-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-6vkbv\" (UID: \"ca31519e-aa90-4104-a175-f9af2b3bbeb4\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-6vkbv"
Oct 31 07:28:55.068994 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:55.068965    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/b44ed0e0-a72e-4746-a7de-ed8610da845d-stats-auth\") pod \"router-default-6c88c45d57-c2bgn\" (UID: \"b44ed0e0-a72e-4746-a7de-ed8610da845d\") " pod="openshift-ingress/router-default-6c88c45d57-c2bgn"
Oct 31 07:28:55.069210 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:55.069009    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q2kvz\" (UniqueName: \"kubernetes.io/projected/4db4c33f-0c58-40f8-bf74-51b3eae3c9db-kube-api-access-q2kvz\") pod \"service-ca-operator-544d749c8c-hb49h\" (UID: \"4db4c33f-0c58-40f8-bf74-51b3eae3c9db\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-hb49h"
Oct 31 07:28:55.069210 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:55.069038    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/ca31519e-aa90-4104-a175-f9af2b3bbeb4-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-6vkbv\" (UID: \"ca31519e-aa90-4104-a175-f9af2b3bbeb4\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-6vkbv"
Oct 31 07:28:55.069210 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:55.069061    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/28b59d93-b9ce-4ceb-9f3a-742525eb94c4-image-registry-private-configuration\") pod \"image-registry-7955c67bcc-pkntv\" (UID: \"28b59d93-b9ce-4ceb-9f3a-742525eb94c4\") " pod="openshift-image-registry/image-registry-7955c67bcc-pkntv"
Oct 31 07:28:55.069210 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:55.069102    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k6576\" (UniqueName: \"kubernetes.io/projected/81829763-27b1-4a48-a568-cd5f6a39a953-kube-api-access-k6576\") pod \"insights-operator-7c4986d449-dwc4p\" (UID: \"81829763-27b1-4a48-a568-cd5f6a39a953\") " pod="openshift-insights/insights-operator-7c4986d449-dwc4p"
Oct 31 07:28:55.069210 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:55.069148    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/81829763-27b1-4a48-a568-cd5f6a39a953-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-dwc4p\" (UID: \"81829763-27b1-4a48-a568-cd5f6a39a953\") " pod="openshift-insights/insights-operator-7c4986d449-dwc4p"
Oct 31 07:28:55.069210 ip-10-0-139-83 kubenswrapper[2573]: E1031 07:28:55.069174    2573 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Oct 31 07:28:55.069210 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:55.069178    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\""
Oct 31 07:28:55.069210 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:55.069199    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/b44ed0e0-a72e-4746-a7de-ed8610da845d-default-certificate\") pod \"router-default-6c88c45d57-c2bgn\" (UID: \"b44ed0e0-a72e-4746-a7de-ed8610da845d\") " pod="openshift-ingress/router-default-6c88c45d57-c2bgn"
Oct 31 07:28:55.069609 ip-10-0-139-83 kubenswrapper[2573]: E1031 07:28:55.069230    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ca31519e-aa90-4104-a175-f9af2b3bbeb4-samples-operator-tls podName:ca31519e-aa90-4104-a175-f9af2b3bbeb4 nodeName:}" failed. No retries permitted until 2026-10-31 07:28:55.569211854 +0000 UTC m=+32.259982676 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/ca31519e-aa90-4104-a175-f9af2b3bbeb4-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-6vkbv" (UID: "ca31519e-aa90-4104-a175-f9af2b3bbeb4") : secret "samples-operator-tls" not found
Oct 31 07:28:55.069609 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:55.069410    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/993cfed6-806d-44c5-bba8-c02203638fe7-registry-certificates\") pod \"image-registry-7886c7d6c6-qdc9f\" (UID: \"993cfed6-806d-44c5-bba8-c02203638fe7\") " pod="openshift-image-registry/image-registry-7886c7d6c6-qdc9f"
Oct 31 07:28:55.069609 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:55.069449    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xvv26\" (UniqueName: \"kubernetes.io/projected/28b59d93-b9ce-4ceb-9f3a-742525eb94c4-kube-api-access-xvv26\") pod \"image-registry-7955c67bcc-pkntv\" (UID: \"28b59d93-b9ce-4ceb-9f3a-742525eb94c4\") " pod="openshift-image-registry/image-registry-7955c67bcc-pkntv"
Oct 31 07:28:55.069609 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:55.069477    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/81829763-27b1-4a48-a568-cd5f6a39a953-snapshots\") pod \"insights-operator-7c4986d449-dwc4p\" (UID: \"81829763-27b1-4a48-a568-cd5f6a39a953\") " pod="openshift-insights/insights-operator-7c4986d449-dwc4p"
Oct 31 07:28:55.069609 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:55.069510    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wqmxh\" (UniqueName: \"kubernetes.io/projected/789f11d8-d951-4144-a589-a2bdbf6844e3-kube-api-access-wqmxh\") pod \"cluster-monitoring-operator-884576c57-6k8mn\" (UID: \"789f11d8-d951-4144-a589-a2bdbf6844e3\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-6k8mn"
Oct 31 07:28:55.069609 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:55.069540    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/81829763-27b1-4a48-a568-cd5f6a39a953-tmp\") pod \"insights-operator-7c4986d449-dwc4p\" (UID: \"81829763-27b1-4a48-a568-cd5f6a39a953\") " pod="openshift-insights/insights-operator-7c4986d449-dwc4p"
Oct 31 07:28:55.069609 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:55.069577    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/81829763-27b1-4a48-a568-cd5f6a39a953-serving-cert\") pod \"insights-operator-7c4986d449-dwc4p\" (UID: \"81829763-27b1-4a48-a568-cd5f6a39a953\") " pod="openshift-insights/insights-operator-7c4986d449-dwc4p"
Oct 31 07:28:55.069609 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:55.069587    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\""
Oct 31 07:28:55.069947 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:55.069613    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\""
Oct 31 07:28:55.069947 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:55.069634    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/28b59d93-b9ce-4ceb-9f3a-742525eb94c4-registry-tls\") pod \"image-registry-7955c67bcc-pkntv\" (UID: \"28b59d93-b9ce-4ceb-9f3a-742525eb94c4\") " pod="openshift-image-registry/image-registry-7955c67bcc-pkntv"
Oct 31 07:28:55.069947 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:55.069660    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/993cfed6-806d-44c5-bba8-c02203638fe7-installation-pull-secrets\") pod \"image-registry-7886c7d6c6-qdc9f\" (UID: \"993cfed6-806d-44c5-bba8-c02203638fe7\") " pod="openshift-image-registry/image-registry-7886c7d6c6-qdc9f"
Oct 31 07:28:55.069947 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:55.069763    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/4db4c33f-0c58-40f8-bf74-51b3eae3c9db-serving-cert\") pod \"service-ca-operator-544d749c8c-hb49h\" (UID: \"4db4c33f-0c58-40f8-bf74-51b3eae3c9db\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-hb49h"
Oct 31 07:28:55.069947 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:55.069791    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/789f11d8-d951-4144-a589-a2bdbf6844e3-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-6k8mn\" (UID: \"789f11d8-d951-4144-a589-a2bdbf6844e3\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-6k8mn"
Oct 31 07:28:55.069947 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:55.069846    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b44ed0e0-a72e-4746-a7de-ed8610da845d-metrics-certs\") pod \"router-default-6c88c45d57-c2bgn\" (UID: \"b44ed0e0-a72e-4746-a7de-ed8610da845d\") " pod="openshift-ingress/router-default-6c88c45d57-c2bgn"
Oct 31 07:28:55.069947 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:55.069876    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9pvp2\" (UniqueName: \"kubernetes.io/projected/b44ed0e0-a72e-4746-a7de-ed8610da845d-kube-api-access-9pvp2\") pod \"router-default-6c88c45d57-c2bgn\" (UID: \"b44ed0e0-a72e-4746-a7de-ed8610da845d\") " pod="openshift-ingress/router-default-6c88c45d57-c2bgn"
Oct 31 07:28:55.069947 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:55.069911    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/993cfed6-806d-44c5-bba8-c02203638fe7-image-registry-private-configuration\") pod \"image-registry-7886c7d6c6-qdc9f\" (UID: \"993cfed6-806d-44c5-bba8-c02203638fe7\") " pod="openshift-image-registry/image-registry-7886c7d6c6-qdc9f"
Oct 31 07:28:55.069947 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:55.069948    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/28b59d93-b9ce-4ceb-9f3a-742525eb94c4-bound-sa-token\") pod \"image-registry-7955c67bcc-pkntv\" (UID: \"28b59d93-b9ce-4ceb-9f3a-742525eb94c4\") " pod="openshift-image-registry/image-registry-7955c67bcc-pkntv"
Oct 31 07:28:55.070321 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:55.069955    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/81829763-27b1-4a48-a568-cd5f6a39a953-snapshots\") pod \"insights-operator-7c4986d449-dwc4p\" (UID: \"81829763-27b1-4a48-a568-cd5f6a39a953\") " pod="openshift-insights/insights-operator-7c4986d449-dwc4p"
Oct 31 07:28:55.070321 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:55.070008    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/28b59d93-b9ce-4ceb-9f3a-742525eb94c4-trusted-ca\") pod \"image-registry-7955c67bcc-pkntv\" (UID: \"28b59d93-b9ce-4ceb-9f3a-742525eb94c4\") " pod="openshift-image-registry/image-registry-7955c67bcc-pkntv"
Oct 31 07:28:55.070321 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:55.070008    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/81829763-27b1-4a48-a568-cd5f6a39a953-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-dwc4p\" (UID: \"81829763-27b1-4a48-a568-cd5f6a39a953\") " pod="openshift-insights/insights-operator-7c4986d449-dwc4p"
Oct 31 07:28:55.070463 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:55.070382    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\""
Oct 31 07:28:55.073027 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:55.073012    2573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-9cmfx"]
Oct 31 07:28:55.073132 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:55.073112    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-kmcg5"
Oct 31 07:28:55.075668 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:55.075632    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/81829763-27b1-4a48-a568-cd5f6a39a953-serving-cert\") pod \"insights-operator-7c4986d449-dwc4p\" (UID: \"81829763-27b1-4a48-a568-cd5f6a39a953\") " pod="openshift-insights/insights-operator-7c4986d449-dwc4p"
Oct 31 07:28:55.101144 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:55.101123    2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-f9gtg"]
Oct 31 07:28:55.101207 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:55.101151    2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-xcrsn"]
Oct 31 07:28:55.101207 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:55.101166    2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-hxmpf"]
Oct 31 07:28:55.101207 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:55.101180    2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-7c4986d449-dwc4p"]
Oct 31 07:28:55.101207 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:55.101191    2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-6vkbv"]
Oct 31 07:28:55.101207 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:55.101208    2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-6k8mn"]
Oct 31 07:28:55.101407 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:55.101216    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-9cmfx"
Oct 31 07:28:55.101407 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:55.101220    2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-6c88c45d57-c2bgn"]
Oct 31 07:28:55.101407 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:55.101388    2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-7886c7d6c6-qdc9f"]
Oct 31 07:28:55.101526 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:55.101417    2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-j94dz"]
Oct 31 07:28:55.101526 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:55.101432    2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-fqhzf"]
Oct 31 07:28:55.101526 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:55.101445    2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-hb49h"]
Oct 31 07:28:55.101526 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:55.101457    2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-9cmfx"]
Oct 31 07:28:55.101526 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:55.101475    2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-65chl"]
Oct 31 07:28:55.101526 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:55.101493    2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-p9z4r"]
Oct 31 07:28:55.101526 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:55.101505    2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-4jszp"]
Oct 31 07:28:55.101526 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:55.101518    2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-7955c67bcc-pkntv"]
Oct 31 07:28:55.101526 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:55.101529    2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-kmcg5"]
Oct 31 07:28:55.110642 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:55.110622    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\""
Oct 31 07:28:55.110707 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:55.110641    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\""
Oct 31 07:28:55.110707 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:55.110663    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\""
Oct 31 07:28:55.110775 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:55.110721    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-x96s4\""
Oct 31 07:28:55.110775 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:55.110771    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-dockercfg-d69tc\""
Oct 31 07:28:55.125905 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:55.125878    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sn2lx\" (UniqueName: \"kubernetes.io/projected/91363236-08db-4f54-ae0a-e68604fcecc6-kube-api-access-sn2lx\") pod \"kas-connection-checker-685895c7c9-xcrsn\" (UID: \"91363236-08db-4f54-ae0a-e68604fcecc6\") " pod="kube-system/kas-connection-checker-685895c7c9-xcrsn"
Oct 31 07:28:55.126224 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:55.126202    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n6p25\" (UniqueName: \"kubernetes.io/projected/ca31519e-aa90-4104-a175-f9af2b3bbeb4-kube-api-access-n6p25\") pod \"cluster-samples-operator-6ddb6c8b94-6vkbv\" (UID: \"ca31519e-aa90-4104-a175-f9af2b3bbeb4\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-6vkbv"
Oct 31 07:28:55.126398 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:55.126378    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l4nfd\" (UniqueName: \"kubernetes.io/projected/323c8334-9a56-4c4e-8a6e-ae80995810d0-kube-api-access-l4nfd\") pod \"kas-connection-checker-685895c7c9-hxmpf\" (UID: \"323c8334-9a56-4c4e-8a6e-ae80995810d0\") " pod="kube-system/kas-connection-checker-685895c7c9-hxmpf"
Oct 31 07:28:55.126829 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:55.126804    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q7vwq\" (UniqueName: \"kubernetes.io/projected/ee4a4a2a-7439-4c78-8858-d6ba0085fdde-kube-api-access-q7vwq\") pod \"volume-data-source-validator-58c9f5f46d-65chl\" (UID: \"ee4a4a2a-7439-4c78-8858-d6ba0085fdde\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-65chl"
Oct 31 07:28:55.126896 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:55.126870    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dx598\" (UniqueName: \"kubernetes.io/projected/631cff9e-a1f1-4dd2-bdc8-ef36519c7c1b-kube-api-access-dx598\") pod \"kas-connection-checker-685895c7c9-f9gtg\" (UID: \"631cff9e-a1f1-4dd2-bdc8-ef36519c7c1b\") " pod="kube-system/kas-connection-checker-685895c7c9-f9gtg"
Oct 31 07:28:55.127058 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:55.127041    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k6576\" (UniqueName: \"kubernetes.io/projected/81829763-27b1-4a48-a568-cd5f6a39a953-kube-api-access-k6576\") pod \"insights-operator-7c4986d449-dwc4p\" (UID: \"81829763-27b1-4a48-a568-cd5f6a39a953\") " pod="openshift-insights/insights-operator-7c4986d449-dwc4p"
Oct 31 07:28:55.134124 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:55.134107    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-xcrsn"
Oct 31 07:28:55.157392 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:55.157370    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-hxmpf"
Oct 31 07:28:55.164281 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:55.164264    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\""
Oct 31 07:28:55.164371 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:55.164292    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Oct 31 07:28:55.164371 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:55.164294    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Oct 31 07:28:55.164371 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:55.164296    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Oct 31 07:28:55.170659 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:55.170640    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g2r66\" (UniqueName: \"kubernetes.io/projected/2536c1c8-8a46-4af3-bc6d-81454a908d3f-kube-api-access-g2r66\") pod \"ingress-canary-kmcg5\" (UID: \"2536c1c8-8a46-4af3-bc6d-81454a908d3f\") " pod="openshift-ingress-canary/ingress-canary-kmcg5"
Oct 31 07:28:55.170759 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:55.170689    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/b44ed0e0-a72e-4746-a7de-ed8610da845d-default-certificate\") pod \"router-default-6c88c45d57-c2bgn\" (UID: \"b44ed0e0-a72e-4746-a7de-ed8610da845d\") " pod="openshift-ingress/router-default-6c88c45d57-c2bgn"
Oct 31 07:28:55.170823 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:55.170802    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/993cfed6-806d-44c5-bba8-c02203638fe7-registry-certificates\") pod \"image-registry-7886c7d6c6-qdc9f\" (UID: \"993cfed6-806d-44c5-bba8-c02203638fe7\") " pod="openshift-image-registry/image-registry-7886c7d6c6-qdc9f"
Oct 31 07:28:55.170872 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:55.170829    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/cb9e4575-9265-43b5-bf02-7843611d6563-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-p9z4r\" (UID: \"cb9e4575-9265-43b5-bf02-7843611d6563\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-p9z4r"
Oct 31 07:28:55.170872 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:55.170850    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kr9wm\" (UniqueName: \"kubernetes.io/projected/cb9e4575-9265-43b5-bf02-7843611d6563-kube-api-access-kr9wm\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-p9z4r\" (UID: \"cb9e4575-9265-43b5-bf02-7843611d6563\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-p9z4r"
Oct 31 07:28:55.170967 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:55.170868    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/b1db0e3b-e85d-4769-8012-cf4b6f740d9d-config\") pod \"console-operator-7cd56b8dbc-4jszp\" (UID: \"b1db0e3b-e85d-4769-8012-cf4b6f740d9d\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-4jszp"
Oct 31 07:28:55.170967 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:55.170914    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xvv26\" (UniqueName: \"kubernetes.io/projected/28b59d93-b9ce-4ceb-9f3a-742525eb94c4-kube-api-access-xvv26\") pod \"image-registry-7955c67bcc-pkntv\" (UID: \"28b59d93-b9ce-4ceb-9f3a-742525eb94c4\") " pod="openshift-image-registry/image-registry-7955c67bcc-pkntv"
Oct 31 07:28:55.170967 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:55.170941    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b1db0e3b-e85d-4769-8012-cf4b6f740d9d-trusted-ca\") pod \"console-operator-7cd56b8dbc-4jszp\" (UID: \"b1db0e3b-e85d-4769-8012-cf4b6f740d9d\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-4jszp"
Oct 31 07:28:55.170967 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:55.170961    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wqmxh\" (UniqueName: \"kubernetes.io/projected/789f11d8-d951-4144-a589-a2bdbf6844e3-kube-api-access-wqmxh\") pod \"cluster-monitoring-operator-884576c57-6k8mn\" (UID: \"789f11d8-d951-4144-a589-a2bdbf6844e3\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-6k8mn"
Oct 31 07:28:55.171133 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:55.171009    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/28b59d93-b9ce-4ceb-9f3a-742525eb94c4-registry-tls\") pod \"image-registry-7955c67bcc-pkntv\" (UID: \"28b59d93-b9ce-4ceb-9f3a-742525eb94c4\") " pod="openshift-image-registry/image-registry-7955c67bcc-pkntv"
Oct 31 07:28:55.171133 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:55.171042    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/993cfed6-806d-44c5-bba8-c02203638fe7-installation-pull-secrets\") pod \"image-registry-7886c7d6c6-qdc9f\" (UID: \"993cfed6-806d-44c5-bba8-c02203638fe7\") " pod="openshift-image-registry/image-registry-7886c7d6c6-qdc9f"
Oct 31 07:28:55.171133 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:55.171068    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/4db4c33f-0c58-40f8-bf74-51b3eae3c9db-serving-cert\") pod \"service-ca-operator-544d749c8c-hb49h\" (UID: \"4db4c33f-0c58-40f8-bf74-51b3eae3c9db\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-hb49h"
Oct 31 07:28:55.171133 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:55.171094    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/789f11d8-d951-4144-a589-a2bdbf6844e3-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-6k8mn\" (UID: \"789f11d8-d951-4144-a589-a2bdbf6844e3\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-6k8mn"
Oct 31 07:28:55.171133 ip-10-0-139-83 kubenswrapper[2573]: E1031 07:28:55.171103    2573 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 31 07:28:55.171133 ip-10-0-139-83 kubenswrapper[2573]: E1031 07:28:55.171116    2573 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7955c67bcc-pkntv: secret "image-registry-tls" not found
Oct 31 07:28:55.171133 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:55.171122    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b44ed0e0-a72e-4746-a7de-ed8610da845d-metrics-certs\") pod \"router-default-6c88c45d57-c2bgn\" (UID: \"b44ed0e0-a72e-4746-a7de-ed8610da845d\") " pod="openshift-ingress/router-default-6c88c45d57-c2bgn"
Oct 31 07:28:55.171474 ip-10-0-139-83 kubenswrapper[2573]: E1031 07:28:55.171175    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/28b59d93-b9ce-4ceb-9f3a-742525eb94c4-registry-tls podName:28b59d93-b9ce-4ceb-9f3a-742525eb94c4 nodeName:}" failed. No retries permitted until 2026-10-31 07:28:55.671156711 +0000 UTC m=+32.361927528 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/28b59d93-b9ce-4ceb-9f3a-742525eb94c4-registry-tls") pod "image-registry-7955c67bcc-pkntv" (UID: "28b59d93-b9ce-4ceb-9f3a-742525eb94c4") : secret "image-registry-tls" not found
Oct 31 07:28:55.171474 ip-10-0-139-83 kubenswrapper[2573]: E1031 07:28:55.171202    2573 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Oct 31 07:28:55.171474 ip-10-0-139-83 kubenswrapper[2573]: E1031 07:28:55.171223    2573 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Oct 31 07:28:55.171474 ip-10-0-139-83 kubenswrapper[2573]: E1031 07:28:55.171266    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/789f11d8-d951-4144-a589-a2bdbf6844e3-cluster-monitoring-operator-tls podName:789f11d8-d951-4144-a589-a2bdbf6844e3 nodeName:}" failed. No retries permitted until 2026-10-31 07:28:55.671249612 +0000 UTC m=+32.362020429 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/789f11d8-d951-4144-a589-a2bdbf6844e3-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-6k8mn" (UID: "789f11d8-d951-4144-a589-a2bdbf6844e3") : secret "cluster-monitoring-operator-tls" not found
Oct 31 07:28:55.171474 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:55.171284    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9pvp2\" (UniqueName: \"kubernetes.io/projected/b44ed0e0-a72e-4746-a7de-ed8610da845d-kube-api-access-9pvp2\") pod \"router-default-6c88c45d57-c2bgn\" (UID: \"b44ed0e0-a72e-4746-a7de-ed8610da845d\") " pod="openshift-ingress/router-default-6c88c45d57-c2bgn"
Oct 31 07:28:55.171474 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:55.171312    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/993cfed6-806d-44c5-bba8-c02203638fe7-image-registry-private-configuration\") pod \"image-registry-7886c7d6c6-qdc9f\" (UID: \"993cfed6-806d-44c5-bba8-c02203638fe7\") " pod="openshift-image-registry/image-registry-7886c7d6c6-qdc9f"
Oct 31 07:28:55.171474 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:55.171355    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/28b59d93-b9ce-4ceb-9f3a-742525eb94c4-bound-sa-token\") pod \"image-registry-7955c67bcc-pkntv\" (UID: \"28b59d93-b9ce-4ceb-9f3a-742525eb94c4\") " pod="openshift-image-registry/image-registry-7955c67bcc-pkntv"
Oct 31 07:28:55.171474 ip-10-0-139-83 kubenswrapper[2573]: E1031 07:28:55.171398    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b44ed0e0-a72e-4746-a7de-ed8610da845d-metrics-certs podName:b44ed0e0-a72e-4746-a7de-ed8610da845d nodeName:}" failed. No retries permitted until 2026-10-31 07:28:55.671385627 +0000 UTC m=+32.362156437 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b44ed0e0-a72e-4746-a7de-ed8610da845d-metrics-certs") pod "router-default-6c88c45d57-c2bgn" (UID: "b44ed0e0-a72e-4746-a7de-ed8610da845d") : secret "router-metrics-certs-default" not found
Oct 31 07:28:55.171474 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:55.171418    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/28b59d93-b9ce-4ceb-9f3a-742525eb94c4-trusted-ca\") pod \"image-registry-7955c67bcc-pkntv\" (UID: \"28b59d93-b9ce-4ceb-9f3a-742525eb94c4\") " pod="openshift-image-registry/image-registry-7955c67bcc-pkntv"
Oct 31 07:28:55.171474 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:55.171446    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/4db4c33f-0c58-40f8-bf74-51b3eae3c9db-config\") pod \"service-ca-operator-544d749c8c-hb49h\" (UID: \"4db4c33f-0c58-40f8-bf74-51b3eae3c9db\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-hb49h"
Oct 31 07:28:55.171474 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:55.171473    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/789f11d8-d951-4144-a589-a2bdbf6844e3-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-6k8mn\" (UID: \"789f11d8-d951-4144-a589-a2bdbf6844e3\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-6k8mn"
Oct 31 07:28:55.172081 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:55.171500    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v2zhk\" (UniqueName: \"kubernetes.io/projected/993cfed6-806d-44c5-bba8-c02203638fe7-kube-api-access-v2zhk\") pod \"image-registry-7886c7d6c6-qdc9f\" (UID: \"993cfed6-806d-44c5-bba8-c02203638fe7\") " pod="openshift-image-registry/image-registry-7886c7d6c6-qdc9f"
Oct 31 07:28:55.172081 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:55.171528    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/28b59d93-b9ce-4ceb-9f3a-742525eb94c4-ca-trust-extracted\") pod \"image-registry-7955c67bcc-pkntv\" (UID: \"28b59d93-b9ce-4ceb-9f3a-742525eb94c4\") " pod="openshift-image-registry/image-registry-7955c67bcc-pkntv"
Oct 31 07:28:55.172081 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:55.171554    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/993cfed6-806d-44c5-bba8-c02203638fe7-bound-sa-token\") pod \"image-registry-7886c7d6c6-qdc9f\" (UID: \"993cfed6-806d-44c5-bba8-c02203638fe7\") " pod="openshift-image-registry/image-registry-7886c7d6c6-qdc9f"
Oct 31 07:28:55.172081 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:55.171554    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/993cfed6-806d-44c5-bba8-c02203638fe7-registry-certificates\") pod \"image-registry-7886c7d6c6-qdc9f\" (UID: \"993cfed6-806d-44c5-bba8-c02203638fe7\") " pod="openshift-image-registry/image-registry-7886c7d6c6-qdc9f"
Oct 31 07:28:55.172081 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:55.171592    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/993cfed6-806d-44c5-bba8-c02203638fe7-ca-trust-extracted\") pod \"image-registry-7886c7d6c6-qdc9f\" (UID: \"993cfed6-806d-44c5-bba8-c02203638fe7\") " pod="openshift-image-registry/image-registry-7886c7d6c6-qdc9f"
Oct 31 07:28:55.172081 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:55.171623    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/993cfed6-806d-44c5-bba8-c02203638fe7-registry-tls\") pod \"image-registry-7886c7d6c6-qdc9f\" (UID: \"993cfed6-806d-44c5-bba8-c02203638fe7\") " pod="openshift-image-registry/image-registry-7886c7d6c6-qdc9f"
Oct 31 07:28:55.172081 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:55.171648    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/993cfed6-806d-44c5-bba8-c02203638fe7-trusted-ca\") pod \"image-registry-7886c7d6c6-qdc9f\" (UID: \"993cfed6-806d-44c5-bba8-c02203638fe7\") " pod="openshift-image-registry/image-registry-7886c7d6c6-qdc9f"
Oct 31 07:28:55.172081 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:55.171674    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bfzv8\" (UniqueName: \"kubernetes.io/projected/ce8cac52-291b-47fb-a7b6-4cf7562449b9-kube-api-access-bfzv8\") pod \"network-check-source-6f78d59f99-fqhzf\" (UID: \"ce8cac52-291b-47fb-a7b6-4cf7562449b9\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-fqhzf"
Oct 31 07:28:55.172081 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:55.171701    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2536c1c8-8a46-4af3-bc6d-81454a908d3f-cert\") pod \"ingress-canary-kmcg5\" (UID: \"2536c1c8-8a46-4af3-bc6d-81454a908d3f\") " pod="openshift-ingress-canary/ingress-canary-kmcg5"
Oct 31 07:28:55.172081 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:55.171747    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/28b59d93-b9ce-4ceb-9f3a-742525eb94c4-installation-pull-secrets\") pod \"image-registry-7955c67bcc-pkntv\" (UID: \"28b59d93-b9ce-4ceb-9f3a-742525eb94c4\") " pod="openshift-image-registry/image-registry-7955c67bcc-pkntv"
Oct 31 07:28:55.172081 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:55.171771    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b44ed0e0-a72e-4746-a7de-ed8610da845d-service-ca-bundle\") pod \"router-default-6c88c45d57-c2bgn\" (UID: \"b44ed0e0-a72e-4746-a7de-ed8610da845d\") " pod="openshift-ingress/router-default-6c88c45d57-c2bgn"
Oct 31 07:28:55.172081 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:55.171795    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/314c29aa-db6b-4f7f-86b4-c8952ce09c84-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-j94dz\" (UID: \"314c29aa-db6b-4f7f-86b4-c8952ce09c84\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-j94dz"
Oct 31 07:28:55.172081 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:55.171823    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/28b59d93-b9ce-4ceb-9f3a-742525eb94c4-registry-certificates\") pod \"image-registry-7955c67bcc-pkntv\" (UID: \"28b59d93-b9ce-4ceb-9f3a-742525eb94c4\") " pod="openshift-image-registry/image-registry-7955c67bcc-pkntv"
Oct 31 07:28:55.172081 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:55.171847    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-czqtq\" (UniqueName: \"kubernetes.io/projected/b1db0e3b-e85d-4769-8012-cf4b6f740d9d-kube-api-access-czqtq\") pod \"console-operator-7cd56b8dbc-4jszp\" (UID: \"b1db0e3b-e85d-4769-8012-cf4b6f740d9d\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-4jszp"
Oct 31 07:28:55.172081 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:55.171890    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/b44ed0e0-a72e-4746-a7de-ed8610da845d-stats-auth\") pod \"router-default-6c88c45d57-c2bgn\" (UID: \"b44ed0e0-a72e-4746-a7de-ed8610da845d\") " pod="openshift-ingress/router-default-6c88c45d57-c2bgn"
Oct 31 07:28:55.172081 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:55.171913    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q2kvz\" (UniqueName: \"kubernetes.io/projected/4db4c33f-0c58-40f8-bf74-51b3eae3c9db-kube-api-access-q2kvz\") pod \"service-ca-operator-544d749c8c-hb49h\" (UID: \"4db4c33f-0c58-40f8-bf74-51b3eae3c9db\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-hb49h"
Oct 31 07:28:55.172979 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:55.171940    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/cb9e4575-9265-43b5-bf02-7843611d6563-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-p9z4r\" (UID: \"cb9e4575-9265-43b5-bf02-7843611d6563\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-p9z4r"
Oct 31 07:28:55.172979 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:55.171980    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/28b59d93-b9ce-4ceb-9f3a-742525eb94c4-image-registry-private-configuration\") pod \"image-registry-7955c67bcc-pkntv\" (UID: \"28b59d93-b9ce-4ceb-9f3a-742525eb94c4\") " pod="openshift-image-registry/image-registry-7955c67bcc-pkntv"
Oct 31 07:28:55.172979 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:55.172009    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/314c29aa-db6b-4f7f-86b4-c8952ce09c84-nginx-conf\") pod \"networking-console-plugin-86c478bb88-j94dz\" (UID: \"314c29aa-db6b-4f7f-86b4-c8952ce09c84\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-j94dz"
Oct 31 07:28:55.172979 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:55.172045    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/b1db0e3b-e85d-4769-8012-cf4b6f740d9d-serving-cert\") pod \"console-operator-7cd56b8dbc-4jszp\" (UID: \"b1db0e3b-e85d-4769-8012-cf4b6f740d9d\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-4jszp"
Oct 31 07:28:55.172979 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:55.172201    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/789f11d8-d951-4144-a589-a2bdbf6844e3-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-6k8mn\" (UID: \"789f11d8-d951-4144-a589-a2bdbf6844e3\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-6k8mn"
Oct 31 07:28:55.172979 ip-10-0-139-83 kubenswrapper[2573]: E1031 07:28:55.172296    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b44ed0e0-a72e-4746-a7de-ed8610da845d-service-ca-bundle podName:b44ed0e0-a72e-4746-a7de-ed8610da845d nodeName:}" failed. No retries permitted until 2026-10-31 07:28:55.672283163 +0000 UTC m=+32.363053981 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/b44ed0e0-a72e-4746-a7de-ed8610da845d-service-ca-bundle") pod "router-default-6c88c45d57-c2bgn" (UID: "b44ed0e0-a72e-4746-a7de-ed8610da845d") : configmap references non-existent config key: service-ca.crt
Oct 31 07:28:55.172979 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:55.172620    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/28b59d93-b9ce-4ceb-9f3a-742525eb94c4-ca-trust-extracted\") pod \"image-registry-7955c67bcc-pkntv\" (UID: \"28b59d93-b9ce-4ceb-9f3a-742525eb94c4\") " pod="openshift-image-registry/image-registry-7955c67bcc-pkntv"
Oct 31 07:28:55.172979 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:55.172619    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/28b59d93-b9ce-4ceb-9f3a-742525eb94c4-trusted-ca\") pod \"image-registry-7955c67bcc-pkntv\" (UID: \"28b59d93-b9ce-4ceb-9f3a-742525eb94c4\") " pod="openshift-image-registry/image-registry-7955c67bcc-pkntv"
Oct 31 07:28:55.172979 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:55.172646    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/4db4c33f-0c58-40f8-bf74-51b3eae3c9db-config\") pod \"service-ca-operator-544d749c8c-hb49h\" (UID: \"4db4c33f-0c58-40f8-bf74-51b3eae3c9db\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-hb49h"
Oct 31 07:28:55.172979 ip-10-0-139-83 kubenswrapper[2573]: E1031 07:28:55.172730    2573 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 31 07:28:55.172979 ip-10-0-139-83 kubenswrapper[2573]: E1031 07:28:55.172743    2573 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7886c7d6c6-qdc9f: secret "image-registry-tls" not found
Oct 31 07:28:55.172979 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:55.172765    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/28b59d93-b9ce-4ceb-9f3a-742525eb94c4-registry-certificates\") pod \"image-registry-7955c67bcc-pkntv\" (UID: \"28b59d93-b9ce-4ceb-9f3a-742525eb94c4\") " pod="openshift-image-registry/image-registry-7955c67bcc-pkntv"
Oct 31 07:28:55.172979 ip-10-0-139-83 kubenswrapper[2573]: E1031 07:28:55.172777    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/993cfed6-806d-44c5-bba8-c02203638fe7-registry-tls podName:993cfed6-806d-44c5-bba8-c02203638fe7 nodeName:}" failed. No retries permitted until 2026-10-31 07:28:55.672764386 +0000 UTC m=+32.363535195 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/993cfed6-806d-44c5-bba8-c02203638fe7-registry-tls") pod "image-registry-7886c7d6c6-qdc9f" (UID: "993cfed6-806d-44c5-bba8-c02203638fe7") : secret "image-registry-tls" not found
Oct 31 07:28:55.172979 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:55.172918    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/993cfed6-806d-44c5-bba8-c02203638fe7-ca-trust-extracted\") pod \"image-registry-7886c7d6c6-qdc9f\" (UID: \"993cfed6-806d-44c5-bba8-c02203638fe7\") " pod="openshift-image-registry/image-registry-7886c7d6c6-qdc9f"
Oct 31 07:28:55.174188 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:55.174055    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-f9gtg"
Oct 31 07:28:55.174188 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:55.174087    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/993cfed6-806d-44c5-bba8-c02203638fe7-installation-pull-secrets\") pod \"image-registry-7886c7d6c6-qdc9f\" (UID: \"993cfed6-806d-44c5-bba8-c02203638fe7\") " pod="openshift-image-registry/image-registry-7886c7d6c6-qdc9f"
Oct 31 07:28:55.174321 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:55.174262    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/993cfed6-806d-44c5-bba8-c02203638fe7-image-registry-private-configuration\") pod \"image-registry-7886c7d6c6-qdc9f\" (UID: \"993cfed6-806d-44c5-bba8-c02203638fe7\") " pod="openshift-image-registry/image-registry-7886c7d6c6-qdc9f"
Oct 31 07:28:55.174536 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:55.174512    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/993cfed6-806d-44c5-bba8-c02203638fe7-trusted-ca\") pod \"image-registry-7886c7d6c6-qdc9f\" (UID: \"993cfed6-806d-44c5-bba8-c02203638fe7\") " pod="openshift-image-registry/image-registry-7886c7d6c6-qdc9f"
Oct 31 07:28:55.175549 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:55.175532    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/b44ed0e0-a72e-4746-a7de-ed8610da845d-default-certificate\") pod \"router-default-6c88c45d57-c2bgn\" (UID: \"b44ed0e0-a72e-4746-a7de-ed8610da845d\") " pod="openshift-ingress/router-default-6c88c45d57-c2bgn"
Oct 31 07:28:55.175758 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:55.175738    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/b44ed0e0-a72e-4746-a7de-ed8610da845d-stats-auth\") pod \"router-default-6c88c45d57-c2bgn\" (UID: \"b44ed0e0-a72e-4746-a7de-ed8610da845d\") " pod="openshift-ingress/router-default-6c88c45d57-c2bgn"
Oct 31 07:28:55.176202 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:55.176184    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/28b59d93-b9ce-4ceb-9f3a-742525eb94c4-installation-pull-secrets\") pod \"image-registry-7955c67bcc-pkntv\" (UID: \"28b59d93-b9ce-4ceb-9f3a-742525eb94c4\") " pod="openshift-image-registry/image-registry-7955c67bcc-pkntv"
Oct 31 07:28:55.176499 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:55.176480    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/4db4c33f-0c58-40f8-bf74-51b3eae3c9db-serving-cert\") pod \"service-ca-operator-544d749c8c-hb49h\" (UID: \"4db4c33f-0c58-40f8-bf74-51b3eae3c9db\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-hb49h"
Oct 31 07:28:55.177848 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:55.177821    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/28b59d93-b9ce-4ceb-9f3a-742525eb94c4-image-registry-private-configuration\") pod \"image-registry-7955c67bcc-pkntv\" (UID: \"28b59d93-b9ce-4ceb-9f3a-742525eb94c4\") " pod="openshift-image-registry/image-registry-7955c67bcc-pkntv"
Oct 31 07:28:55.186782 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:55.186758    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-7c4986d449-dwc4p"
Oct 31 07:28:55.211671 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:55.211646    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Oct 31 07:28:55.211801 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:55.211737    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-xb4h7\""
Oct 31 07:28:55.211801 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:55.211758    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Oct 31 07:28:55.211903 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:55.211817    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-m5kd5\""
Oct 31 07:28:55.230724 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:55.230655    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/28b59d93-b9ce-4ceb-9f3a-742525eb94c4-bound-sa-token\") pod \"image-registry-7955c67bcc-pkntv\" (UID: \"28b59d93-b9ce-4ceb-9f3a-742525eb94c4\") " pod="openshift-image-registry/image-registry-7955c67bcc-pkntv"
Oct 31 07:28:55.230880 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:55.230669    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9pvp2\" (UniqueName: \"kubernetes.io/projected/b44ed0e0-a72e-4746-a7de-ed8610da845d-kube-api-access-9pvp2\") pod \"router-default-6c88c45d57-c2bgn\" (UID: \"b44ed0e0-a72e-4746-a7de-ed8610da845d\") " pod="openshift-ingress/router-default-6c88c45d57-c2bgn"
Oct 31 07:28:55.231026 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:55.231008    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q2kvz\" (UniqueName: \"kubernetes.io/projected/4db4c33f-0c58-40f8-bf74-51b3eae3c9db-kube-api-access-q2kvz\") pod \"service-ca-operator-544d749c8c-hb49h\" (UID: \"4db4c33f-0c58-40f8-bf74-51b3eae3c9db\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-hb49h"
Oct 31 07:28:55.231314 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:55.231292    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v2zhk\" (UniqueName: \"kubernetes.io/projected/993cfed6-806d-44c5-bba8-c02203638fe7-kube-api-access-v2zhk\") pod \"image-registry-7886c7d6c6-qdc9f\" (UID: \"993cfed6-806d-44c5-bba8-c02203638fe7\") " pod="openshift-image-registry/image-registry-7886c7d6c6-qdc9f"
Oct 31 07:28:55.231488 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:55.231470    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xvv26\" (UniqueName: \"kubernetes.io/projected/28b59d93-b9ce-4ceb-9f3a-742525eb94c4-kube-api-access-xvv26\") pod \"image-registry-7955c67bcc-pkntv\" (UID: \"28b59d93-b9ce-4ceb-9f3a-742525eb94c4\") " pod="openshift-image-registry/image-registry-7955c67bcc-pkntv"
Oct 31 07:28:55.231898 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:55.231864    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wqmxh\" (UniqueName: \"kubernetes.io/projected/789f11d8-d951-4144-a589-a2bdbf6844e3-kube-api-access-wqmxh\") pod \"cluster-monitoring-operator-884576c57-6k8mn\" (UID: \"789f11d8-d951-4144-a589-a2bdbf6844e3\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-6k8mn"
Oct 31 07:28:55.232593 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:55.232572    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/993cfed6-806d-44c5-bba8-c02203638fe7-bound-sa-token\") pod \"image-registry-7886c7d6c6-qdc9f\" (UID: \"993cfed6-806d-44c5-bba8-c02203638fe7\") " pod="openshift-image-registry/image-registry-7886c7d6c6-qdc9f"
Oct 31 07:28:55.272097 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:55.272061    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-65chl"
Oct 31 07:28:55.274660 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:55.273548    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/a73024cb-f581-450d-a09c-0565838c5baa-config-volume\") pod \"dns-default-9cmfx\" (UID: \"a73024cb-f581-450d-a09c-0565838c5baa\") " pod="openshift-dns/dns-default-9cmfx"
Oct 31 07:28:55.274660 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:55.273626    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bfzv8\" (UniqueName: \"kubernetes.io/projected/ce8cac52-291b-47fb-a7b6-4cf7562449b9-kube-api-access-bfzv8\") pod \"network-check-source-6f78d59f99-fqhzf\" (UID: \"ce8cac52-291b-47fb-a7b6-4cf7562449b9\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-fqhzf"
Oct 31 07:28:55.275343 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:55.273655    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2536c1c8-8a46-4af3-bc6d-81454a908d3f-cert\") pod \"ingress-canary-kmcg5\" (UID: \"2536c1c8-8a46-4af3-bc6d-81454a908d3f\") " pod="openshift-ingress-canary/ingress-canary-kmcg5"
Oct 31 07:28:55.275432 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:55.275413    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/314c29aa-db6b-4f7f-86b4-c8952ce09c84-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-j94dz\" (UID: \"314c29aa-db6b-4f7f-86b4-c8952ce09c84\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-j94dz"
Oct 31 07:28:55.275468 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:55.275456    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-czqtq\" (UniqueName: \"kubernetes.io/projected/b1db0e3b-e85d-4769-8012-cf4b6f740d9d-kube-api-access-czqtq\") pod \"console-operator-7cd56b8dbc-4jszp\" (UID: \"b1db0e3b-e85d-4769-8012-cf4b6f740d9d\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-4jszp"
Oct 31 07:28:55.275515 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:55.275500    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/cb9e4575-9265-43b5-bf02-7843611d6563-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-p9z4r\" (UID: \"cb9e4575-9265-43b5-bf02-7843611d6563\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-p9z4r"
Oct 31 07:28:55.275558 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:55.275543    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/314c29aa-db6b-4f7f-86b4-c8952ce09c84-nginx-conf\") pod \"networking-console-plugin-86c478bb88-j94dz\" (UID: \"314c29aa-db6b-4f7f-86b4-c8952ce09c84\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-j94dz"
Oct 31 07:28:55.275593 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:55.275580    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/b1db0e3b-e85d-4769-8012-cf4b6f740d9d-serving-cert\") pod \"console-operator-7cd56b8dbc-4jszp\" (UID: \"b1db0e3b-e85d-4769-8012-cf4b6f740d9d\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-4jszp"
Oct 31 07:28:55.275624 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:55.275605    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g2r66\" (UniqueName: \"kubernetes.io/projected/2536c1c8-8a46-4af3-bc6d-81454a908d3f-kube-api-access-g2r66\") pod \"ingress-canary-kmcg5\" (UID: \"2536c1c8-8a46-4af3-bc6d-81454a908d3f\") " pod="openshift-ingress-canary/ingress-canary-kmcg5"
Oct 31 07:28:55.275659 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:55.275630    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m6l7n\" (UniqueName: \"kubernetes.io/projected/a73024cb-f581-450d-a09c-0565838c5baa-kube-api-access-m6l7n\") pod \"dns-default-9cmfx\" (UID: \"a73024cb-f581-450d-a09c-0565838c5baa\") " pod="openshift-dns/dns-default-9cmfx"
Oct 31 07:28:55.275688 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:55.275659    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a73024cb-f581-450d-a09c-0565838c5baa-tmp-dir\") pod \"dns-default-9cmfx\" (UID: \"a73024cb-f581-450d-a09c-0565838c5baa\") " pod="openshift-dns/dns-default-9cmfx"
Oct 31 07:28:55.275717 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:55.275684    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/cb9e4575-9265-43b5-bf02-7843611d6563-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-p9z4r\" (UID: \"cb9e4575-9265-43b5-bf02-7843611d6563\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-p9z4r"
Oct 31 07:28:55.275751 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:55.275732    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a73024cb-f581-450d-a09c-0565838c5baa-metrics-tls\") pod \"dns-default-9cmfx\" (UID: \"a73024cb-f581-450d-a09c-0565838c5baa\") " pod="openshift-dns/dns-default-9cmfx"
Oct 31 07:28:55.275782 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:55.275761    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kr9wm\" (UniqueName: \"kubernetes.io/projected/cb9e4575-9265-43b5-bf02-7843611d6563-kube-api-access-kr9wm\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-p9z4r\" (UID: \"cb9e4575-9265-43b5-bf02-7843611d6563\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-p9z4r"
Oct 31 07:28:55.275817 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:55.275785    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/b1db0e3b-e85d-4769-8012-cf4b6f740d9d-config\") pod \"console-operator-7cd56b8dbc-4jszp\" (UID: \"b1db0e3b-e85d-4769-8012-cf4b6f740d9d\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-4jszp"
Oct 31 07:28:55.275848 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:55.275815    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b1db0e3b-e85d-4769-8012-cf4b6f740d9d-trusted-ca\") pod \"console-operator-7cd56b8dbc-4jszp\" (UID: \"b1db0e3b-e85d-4769-8012-cf4b6f740d9d\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-4jszp"
Oct 31 07:28:55.277252 ip-10-0-139-83 kubenswrapper[2573]: E1031 07:28:55.276558    2573 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Oct 31 07:28:55.277252 ip-10-0-139-83 kubenswrapper[2573]: E1031 07:28:55.276630    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/314c29aa-db6b-4f7f-86b4-c8952ce09c84-networking-console-plugin-cert podName:314c29aa-db6b-4f7f-86b4-c8952ce09c84 nodeName:}" failed. No retries permitted until 2026-10-31 07:28:55.776611847 +0000 UTC m=+32.467382651 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/314c29aa-db6b-4f7f-86b4-c8952ce09c84-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-j94dz" (UID: "314c29aa-db6b-4f7f-86b4-c8952ce09c84") : secret "networking-console-plugin-cert" not found
Oct 31 07:28:55.277252 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:55.276688    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b1db0e3b-e85d-4769-8012-cf4b6f740d9d-trusted-ca\") pod \"console-operator-7cd56b8dbc-4jszp\" (UID: \"b1db0e3b-e85d-4769-8012-cf4b6f740d9d\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-4jszp"
Oct 31 07:28:55.277252 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:55.277217    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/cb9e4575-9265-43b5-bf02-7843611d6563-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-p9z4r\" (UID: \"cb9e4575-9265-43b5-bf02-7843611d6563\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-p9z4r"
Oct 31 07:28:55.279902 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:55.277957    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/314c29aa-db6b-4f7f-86b4-c8952ce09c84-nginx-conf\") pod \"networking-console-plugin-86c478bb88-j94dz\" (UID: \"314c29aa-db6b-4f7f-86b4-c8952ce09c84\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-j94dz"
Oct 31 07:28:55.279902 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:55.278444    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/b1db0e3b-e85d-4769-8012-cf4b6f740d9d-config\") pod \"console-operator-7cd56b8dbc-4jszp\" (UID: \"b1db0e3b-e85d-4769-8012-cf4b6f740d9d\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-4jszp"
Oct 31 07:28:55.279902 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:55.279734    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/cb9e4575-9265-43b5-bf02-7843611d6563-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-p9z4r\" (UID: \"cb9e4575-9265-43b5-bf02-7843611d6563\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-p9z4r"
Oct 31 07:28:55.279902 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:55.279868    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2536c1c8-8a46-4af3-bc6d-81454a908d3f-cert\") pod \"ingress-canary-kmcg5\" (UID: \"2536c1c8-8a46-4af3-bc6d-81454a908d3f\") " pod="openshift-ingress-canary/ingress-canary-kmcg5"
Oct 31 07:28:55.280751 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:55.280728    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/b1db0e3b-e85d-4769-8012-cf4b6f740d9d-serving-cert\") pod \"console-operator-7cd56b8dbc-4jszp\" (UID: \"b1db0e3b-e85d-4769-8012-cf4b6f740d9d\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-4jszp"
Oct 31 07:28:55.330543 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:55.330518    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bfzv8\" (UniqueName: \"kubernetes.io/projected/ce8cac52-291b-47fb-a7b6-4cf7562449b9-kube-api-access-bfzv8\") pod \"network-check-source-6f78d59f99-fqhzf\" (UID: \"ce8cac52-291b-47fb-a7b6-4cf7562449b9\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-fqhzf"
Oct 31 07:28:55.333584 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:55.333431    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-czqtq\" (UniqueName: \"kubernetes.io/projected/b1db0e3b-e85d-4769-8012-cf4b6f740d9d-kube-api-access-czqtq\") pod \"console-operator-7cd56b8dbc-4jszp\" (UID: \"b1db0e3b-e85d-4769-8012-cf4b6f740d9d\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-4jszp"
Oct 31 07:28:55.336278 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:55.336243    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g2r66\" (UniqueName: \"kubernetes.io/projected/2536c1c8-8a46-4af3-bc6d-81454a908d3f-kube-api-access-g2r66\") pod \"ingress-canary-kmcg5\" (UID: \"2536c1c8-8a46-4af3-bc6d-81454a908d3f\") " pod="openshift-ingress-canary/ingress-canary-kmcg5"
Oct 31 07:28:55.337418 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:55.337400    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kr9wm\" (UniqueName: \"kubernetes.io/projected/cb9e4575-9265-43b5-bf02-7843611d6563-kube-api-access-kr9wm\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-p9z4r\" (UID: \"cb9e4575-9265-43b5-bf02-7843611d6563\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-p9z4r"
Oct 31 07:28:55.365716 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:55.365692    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-hb49h"
Oct 31 07:28:55.377033 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:55.377012    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m6l7n\" (UniqueName: \"kubernetes.io/projected/a73024cb-f581-450d-a09c-0565838c5baa-kube-api-access-m6l7n\") pod \"dns-default-9cmfx\" (UID: \"a73024cb-f581-450d-a09c-0565838c5baa\") " pod="openshift-dns/dns-default-9cmfx"
Oct 31 07:28:55.377119 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:55.377041    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a73024cb-f581-450d-a09c-0565838c5baa-tmp-dir\") pod \"dns-default-9cmfx\" (UID: \"a73024cb-f581-450d-a09c-0565838c5baa\") " pod="openshift-dns/dns-default-9cmfx"
Oct 31 07:28:55.377119 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:55.377058    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a73024cb-f581-450d-a09c-0565838c5baa-metrics-tls\") pod \"dns-default-9cmfx\" (UID: \"a73024cb-f581-450d-a09c-0565838c5baa\") " pod="openshift-dns/dns-default-9cmfx"
Oct 31 07:28:55.377216 ip-10-0-139-83 kubenswrapper[2573]: E1031 07:28:55.377145    2573 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 31 07:28:55.377216 ip-10-0-139-83 kubenswrapper[2573]: E1031 07:28:55.377182    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a73024cb-f581-450d-a09c-0565838c5baa-metrics-tls podName:a73024cb-f581-450d-a09c-0565838c5baa nodeName:}" failed. No retries permitted until 2026-10-31 07:28:55.877170708 +0000 UTC m=+32.567941510 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/a73024cb-f581-450d-a09c-0565838c5baa-metrics-tls") pod "dns-default-9cmfx" (UID: "a73024cb-f581-450d-a09c-0565838c5baa") : secret "dns-default-metrics-tls" not found
Oct 31 07:28:55.377366 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:55.377230    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/a73024cb-f581-450d-a09c-0565838c5baa-config-volume\") pod \"dns-default-9cmfx\" (UID: \"a73024cb-f581-450d-a09c-0565838c5baa\") " pod="openshift-dns/dns-default-9cmfx"
Oct 31 07:28:55.377366 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:55.377309    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a73024cb-f581-450d-a09c-0565838c5baa-tmp-dir\") pod \"dns-default-9cmfx\" (UID: \"a73024cb-f581-450d-a09c-0565838c5baa\") " pod="openshift-dns/dns-default-9cmfx"
Oct 31 07:28:55.377715 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:55.377696    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/a73024cb-f581-450d-a09c-0565838c5baa-config-volume\") pod \"dns-default-9cmfx\" (UID: \"a73024cb-f581-450d-a09c-0565838c5baa\") " pod="openshift-dns/dns-default-9cmfx"
Oct 31 07:28:55.415495 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:55.415475    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-p9z4r"
Oct 31 07:28:55.434674 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:55.434655    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m6l7n\" (UniqueName: \"kubernetes.io/projected/a73024cb-f581-450d-a09c-0565838c5baa-kube-api-access-m6l7n\") pod \"dns-default-9cmfx\" (UID: \"a73024cb-f581-450d-a09c-0565838c5baa\") " pod="openshift-dns/dns-default-9cmfx"
Oct 31 07:28:55.436367 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:55.436349    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-7cd56b8dbc-4jszp"
Oct 31 07:28:55.474474 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:55.474443    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-fqhzf"
Oct 31 07:28:55.493030 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:55.493004    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-kmcg5"
Oct 31 07:28:55.576258 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:55.576231    2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-hxmpf"]
Oct 31 07:28:55.580402 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:55.579803    2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-xcrsn"]
Oct 31 07:28:55.580890 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:55.580749    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/ca31519e-aa90-4104-a175-f9af2b3bbeb4-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-6vkbv\" (UID: \"ca31519e-aa90-4104-a175-f9af2b3bbeb4\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-6vkbv"
Oct 31 07:28:55.580968 ip-10-0-139-83 kubenswrapper[2573]: E1031 07:28:55.580888    2573 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Oct 31 07:28:55.580968 ip-10-0-139-83 kubenswrapper[2573]: E1031 07:28:55.580944    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ca31519e-aa90-4104-a175-f9af2b3bbeb4-samples-operator-tls podName:ca31519e-aa90-4104-a175-f9af2b3bbeb4 nodeName:}" failed. No retries permitted until 2026-10-31 07:28:56.580927263 +0000 UTC m=+33.271698068 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/ca31519e-aa90-4104-a175-f9af2b3bbeb4-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-6vkbv" (UID: "ca31519e-aa90-4104-a175-f9af2b3bbeb4") : secret "samples-operator-tls" not found
Oct 31 07:28:55.582123 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:55.581185    2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-f9gtg"]
Oct 31 07:28:55.589699 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:55.589671    2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod91363236_08db_4f54_ae0a_e68604fcecc6.slice/crio-397e8b2b367abe48f39d3961ba7ce8bb1ebfd59819edf275333c202df43087e2 WatchSource:0}: Error finding container 397e8b2b367abe48f39d3961ba7ce8bb1ebfd59819edf275333c202df43087e2: Status 404 returned error can't find the container with id 397e8b2b367abe48f39d3961ba7ce8bb1ebfd59819edf275333c202df43087e2
Oct 31 07:28:55.631125 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:55.631040    2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-7c4986d449-dwc4p"]
Oct 31 07:28:55.683061 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:55.682195    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/993cfed6-806d-44c5-bba8-c02203638fe7-registry-tls\") pod \"image-registry-7886c7d6c6-qdc9f\" (UID: \"993cfed6-806d-44c5-bba8-c02203638fe7\") " pod="openshift-image-registry/image-registry-7886c7d6c6-qdc9f"
Oct 31 07:28:55.683061 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:55.682254    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b44ed0e0-a72e-4746-a7de-ed8610da845d-service-ca-bundle\") pod \"router-default-6c88c45d57-c2bgn\" (UID: \"b44ed0e0-a72e-4746-a7de-ed8610da845d\") " pod="openshift-ingress/router-default-6c88c45d57-c2bgn"
Oct 31 07:28:55.683061 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:55.682381    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/28b59d93-b9ce-4ceb-9f3a-742525eb94c4-registry-tls\") pod \"image-registry-7955c67bcc-pkntv\" (UID: \"28b59d93-b9ce-4ceb-9f3a-742525eb94c4\") " pod="openshift-image-registry/image-registry-7955c67bcc-pkntv"
Oct 31 07:28:55.683061 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:55.682412    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/789f11d8-d951-4144-a589-a2bdbf6844e3-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-6k8mn\" (UID: \"789f11d8-d951-4144-a589-a2bdbf6844e3\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-6k8mn"
Oct 31 07:28:55.683061 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:55.682454    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b44ed0e0-a72e-4746-a7de-ed8610da845d-metrics-certs\") pod \"router-default-6c88c45d57-c2bgn\" (UID: \"b44ed0e0-a72e-4746-a7de-ed8610da845d\") " pod="openshift-ingress/router-default-6c88c45d57-c2bgn"
Oct 31 07:28:55.683061 ip-10-0-139-83 kubenswrapper[2573]: E1031 07:28:55.682622    2573 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Oct 31 07:28:55.683061 ip-10-0-139-83 kubenswrapper[2573]: E1031 07:28:55.682675    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b44ed0e0-a72e-4746-a7de-ed8610da845d-metrics-certs podName:b44ed0e0-a72e-4746-a7de-ed8610da845d nodeName:}" failed. No retries permitted until 2026-10-31 07:28:56.682658231 +0000 UTC m=+33.373429034 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b44ed0e0-a72e-4746-a7de-ed8610da845d-metrics-certs") pod "router-default-6c88c45d57-c2bgn" (UID: "b44ed0e0-a72e-4746-a7de-ed8610da845d") : secret "router-metrics-certs-default" not found
Oct 31 07:28:55.683061 ip-10-0-139-83 kubenswrapper[2573]: E1031 07:28:55.682743    2573 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 31 07:28:55.683061 ip-10-0-139-83 kubenswrapper[2573]: E1031 07:28:55.682754    2573 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7886c7d6c6-qdc9f: secret "image-registry-tls" not found
Oct 31 07:28:55.683061 ip-10-0-139-83 kubenswrapper[2573]: E1031 07:28:55.682788    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/993cfed6-806d-44c5-bba8-c02203638fe7-registry-tls podName:993cfed6-806d-44c5-bba8-c02203638fe7 nodeName:}" failed. No retries permitted until 2026-10-31 07:28:56.682777245 +0000 UTC m=+33.373548050 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/993cfed6-806d-44c5-bba8-c02203638fe7-registry-tls") pod "image-registry-7886c7d6c6-qdc9f" (UID: "993cfed6-806d-44c5-bba8-c02203638fe7") : secret "image-registry-tls" not found
Oct 31 07:28:55.683061 ip-10-0-139-83 kubenswrapper[2573]: E1031 07:28:55.682860    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b44ed0e0-a72e-4746-a7de-ed8610da845d-service-ca-bundle podName:b44ed0e0-a72e-4746-a7de-ed8610da845d nodeName:}" failed. No retries permitted until 2026-10-31 07:28:56.68284723 +0000 UTC m=+33.373618034 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/b44ed0e0-a72e-4746-a7de-ed8610da845d-service-ca-bundle") pod "router-default-6c88c45d57-c2bgn" (UID: "b44ed0e0-a72e-4746-a7de-ed8610da845d") : configmap references non-existent config key: service-ca.crt
Oct 31 07:28:55.683061 ip-10-0-139-83 kubenswrapper[2573]: E1031 07:28:55.682916    2573 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 31 07:28:55.683061 ip-10-0-139-83 kubenswrapper[2573]: E1031 07:28:55.682927    2573 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7955c67bcc-pkntv: secret "image-registry-tls" not found
Oct 31 07:28:55.683061 ip-10-0-139-83 kubenswrapper[2573]: E1031 07:28:55.682954    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/28b59d93-b9ce-4ceb-9f3a-742525eb94c4-registry-tls podName:28b59d93-b9ce-4ceb-9f3a-742525eb94c4 nodeName:}" failed. No retries permitted until 2026-10-31 07:28:56.682944622 +0000 UTC m=+33.373715428 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/28b59d93-b9ce-4ceb-9f3a-742525eb94c4-registry-tls") pod "image-registry-7955c67bcc-pkntv" (UID: "28b59d93-b9ce-4ceb-9f3a-742525eb94c4") : secret "image-registry-tls" not found
Oct 31 07:28:55.683061 ip-10-0-139-83 kubenswrapper[2573]: E1031 07:28:55.683003    2573 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Oct 31 07:28:55.683061 ip-10-0-139-83 kubenswrapper[2573]: E1031 07:28:55.683033    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/789f11d8-d951-4144-a589-a2bdbf6844e3-cluster-monitoring-operator-tls podName:789f11d8-d951-4144-a589-a2bdbf6844e3 nodeName:}" failed. No retries permitted until 2026-10-31 07:28:56.683024356 +0000 UTC m=+33.373795159 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/789f11d8-d951-4144-a589-a2bdbf6844e3-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-6k8mn" (UID: "789f11d8-d951-4144-a589-a2bdbf6844e3") : secret "cluster-monitoring-operator-tls" not found
Oct 31 07:28:55.688200 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:55.688096    2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-65chl"]
Oct 31 07:28:55.700576 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:55.700390    2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podee4a4a2a_7439_4c78_8858_d6ba0085fdde.slice/crio-4b442c7c5f25e692aee5d663f2bd1e5c0c7168cd799aab781bfed2102e3740f1 WatchSource:0}: Error finding container 4b442c7c5f25e692aee5d663f2bd1e5c0c7168cd799aab781bfed2102e3740f1: Status 404 returned error can't find the container with id 4b442c7c5f25e692aee5d663f2bd1e5c0c7168cd799aab781bfed2102e3740f1
Oct 31 07:28:55.763875 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:55.763718    2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-hb49h"]
Oct 31 07:28:55.768358 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:55.768305    2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4db4c33f_0c58_40f8_bf74_51b3eae3c9db.slice/crio-90033c9a0ade2d06af05ffaab30cc49f21be3df3b48302c519fd8d263f9886af WatchSource:0}: Error finding container 90033c9a0ade2d06af05ffaab30cc49f21be3df3b48302c519fd8d263f9886af: Status 404 returned error can't find the container with id 90033c9a0ade2d06af05ffaab30cc49f21be3df3b48302c519fd8d263f9886af
Oct 31 07:28:55.783307 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:55.783285    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/314c29aa-db6b-4f7f-86b4-c8952ce09c84-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-j94dz\" (UID: \"314c29aa-db6b-4f7f-86b4-c8952ce09c84\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-j94dz"
Oct 31 07:28:55.783438 ip-10-0-139-83 kubenswrapper[2573]: E1031 07:28:55.783422    2573 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Oct 31 07:28:55.783485 ip-10-0-139-83 kubenswrapper[2573]: E1031 07:28:55.783477    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/314c29aa-db6b-4f7f-86b4-c8952ce09c84-networking-console-plugin-cert podName:314c29aa-db6b-4f7f-86b4-c8952ce09c84 nodeName:}" failed. No retries permitted until 2026-10-31 07:28:56.783463485 +0000 UTC m=+33.474234305 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/314c29aa-db6b-4f7f-86b4-c8952ce09c84-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-j94dz" (UID: "314c29aa-db6b-4f7f-86b4-c8952ce09c84") : secret "networking-console-plugin-cert" not found
Oct 31 07:28:55.809460 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:55.809436    2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-p9z4r"]
Oct 31 07:28:55.812691 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:55.812670    2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb9e4575_9265_43b5_bf02_7843611d6563.slice/crio-6a2e938ce4315b140940dad931d9f3c7bb967a6dfafd01ff57780c53f7df30a7 WatchSource:0}: Error finding container 6a2e938ce4315b140940dad931d9f3c7bb967a6dfafd01ff57780c53f7df30a7: Status 404 returned error can't find the container with id 6a2e938ce4315b140940dad931d9f3c7bb967a6dfafd01ff57780c53f7df30a7
Oct 31 07:28:55.859656 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:55.859628    2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb1db0e3b_e85d_4769_8012_cf4b6f740d9d.slice/crio-44c067aa8f06e75a4a32430911035f32efde1168203a54d7abd7aa733b407210 WatchSource:0}: Error finding container 44c067aa8f06e75a4a32430911035f32efde1168203a54d7abd7aa733b407210: Status 404 returned error can't find the container with id 44c067aa8f06e75a4a32430911035f32efde1168203a54d7abd7aa733b407210
Oct 31 07:28:55.860477 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:55.860456    2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-4jszp"]
Oct 31 07:28:55.884477 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:55.884455    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a73024cb-f581-450d-a09c-0565838c5baa-metrics-tls\") pod \"dns-default-9cmfx\" (UID: \"a73024cb-f581-450d-a09c-0565838c5baa\") " pod="openshift-dns/dns-default-9cmfx"
Oct 31 07:28:55.884576 ip-10-0-139-83 kubenswrapper[2573]: E1031 07:28:55.884554    2573 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 31 07:28:55.884618 ip-10-0-139-83 kubenswrapper[2573]: E1031 07:28:55.884596    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a73024cb-f581-450d-a09c-0565838c5baa-metrics-tls podName:a73024cb-f581-450d-a09c-0565838c5baa nodeName:}" failed. No retries permitted until 2026-10-31 07:28:56.88458297 +0000 UTC m=+33.575353773 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/a73024cb-f581-450d-a09c-0565838c5baa-metrics-tls") pod "dns-default-9cmfx" (UID: "a73024cb-f581-450d-a09c-0565838c5baa") : secret "dns-default-metrics-tls" not found
Oct 31 07:28:55.896426 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:55.896406    2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-fqhzf"]
Oct 31 07:28:55.900622 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:55.900602    2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce8cac52_291b_47fb_a7b6_4cf7562449b9.slice/crio-4b27a1e9df141368f9aec55f507c535d7c7a4e98dd0d847f22a930188408d21e WatchSource:0}: Error finding container 4b27a1e9df141368f9aec55f507c535d7c7a4e98dd0d847f22a930188408d21e: Status 404 returned error can't find the container with id 4b27a1e9df141368f9aec55f507c535d7c7a4e98dd0d847f22a930188408d21e
Oct 31 07:28:55.903809 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:55.903789    2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-kmcg5"]
Oct 31 07:28:55.906869 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:55.906844    2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2536c1c8_8a46_4af3_bc6d_81454a908d3f.slice/crio-26d38dd238ca56039d3f30825038e78a1478f38371827bb224e5be4d97b94b28 WatchSource:0}: Error finding container 26d38dd238ca56039d3f30825038e78a1478f38371827bb224e5be4d97b94b28: Status 404 returned error can't find the container with id 26d38dd238ca56039d3f30825038e78a1478f38371827bb224e5be4d97b94b28
Oct 31 07:28:56.074419 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:56.074391    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-fqhzf" event={"ID":"ce8cac52-291b-47fb-a7b6-4cf7562449b9","Type":"ContainerStarted","Data":"4b27a1e9df141368f9aec55f507c535d7c7a4e98dd0d847f22a930188408d21e"}
Oct 31 07:28:56.075589 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:56.075568    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-4jszp" event={"ID":"b1db0e3b-e85d-4769-8012-cf4b6f740d9d","Type":"ContainerStarted","Data":"44c067aa8f06e75a4a32430911035f32efde1168203a54d7abd7aa733b407210"}
Oct 31 07:28:56.076748 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:56.076723    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-p9z4r" event={"ID":"cb9e4575-9265-43b5-bf02-7843611d6563","Type":"ContainerStarted","Data":"6a2e938ce4315b140940dad931d9f3c7bb967a6dfafd01ff57780c53f7df30a7"}
Oct 31 07:28:56.077924 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:56.077902    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-hb49h" event={"ID":"4db4c33f-0c58-40f8-bf74-51b3eae3c9db","Type":"ContainerStarted","Data":"90033c9a0ade2d06af05ffaab30cc49f21be3df3b48302c519fd8d263f9886af"}
Oct 31 07:28:56.079451 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:56.079426    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-f9gtg" event={"ID":"631cff9e-a1f1-4dd2-bdc8-ef36519c7c1b","Type":"ContainerStarted","Data":"181d57cbc5eb0c027f165c71e663cd90ec96d38215abab7b2e126a4597eacbaa"}
Oct 31 07:28:56.079546 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:56.079466    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-f9gtg" event={"ID":"631cff9e-a1f1-4dd2-bdc8-ef36519c7c1b","Type":"ContainerStarted","Data":"24977ca031ccca9556605789709d856b16771b92e2cd05e39a8df96eb7e98438"}
Oct 31 07:28:56.080852 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:56.080829    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-hxmpf" event={"ID":"323c8334-9a56-4c4e-8a6e-ae80995810d0","Type":"ContainerStarted","Data":"0814cdcf8b9008862cb434f8481c45d48157ce334ad20b45b782caaf5658cb1b"}
Oct 31 07:28:56.080931 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:56.080865    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-hxmpf" event={"ID":"323c8334-9a56-4c4e-8a6e-ae80995810d0","Type":"ContainerStarted","Data":"6d2a6480d09c4deb2fe4f31aa76ca7365b7305ee8d41ba48d3831a072eab9b1d"}
Oct 31 07:28:56.082056 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:56.082039    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-kmcg5" event={"ID":"2536c1c8-8a46-4af3-bc6d-81454a908d3f","Type":"ContainerStarted","Data":"26d38dd238ca56039d3f30825038e78a1478f38371827bb224e5be4d97b94b28"}
Oct 31 07:28:56.083129 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:56.083109    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-dwc4p" event={"ID":"81829763-27b1-4a48-a568-cd5f6a39a953","Type":"ContainerStarted","Data":"7dab47a696030acdb1ddfc6fe11ed4c83ad1726d144ad583b503091bc2bb5a62"}
Oct 31 07:28:56.084168 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:56.084151    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-65chl" event={"ID":"ee4a4a2a-7439-4c78-8858-d6ba0085fdde","Type":"ContainerStarted","Data":"4b442c7c5f25e692aee5d663f2bd1e5c0c7168cd799aab781bfed2102e3740f1"}
Oct 31 07:28:56.085545 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:56.085526    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-xcrsn" event={"ID":"91363236-08db-4f54-ae0a-e68604fcecc6","Type":"ContainerStarted","Data":"364c4ae021254d020cb558429d62571af12df9a087a77a797708cdc1e7991306"}
Oct 31 07:28:56.085617 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:56.085563    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-xcrsn" event={"ID":"91363236-08db-4f54-ae0a-e68604fcecc6","Type":"ContainerStarted","Data":"397e8b2b367abe48f39d3961ba7ce8bb1ebfd59819edf275333c202df43087e2"}
Oct 31 07:28:56.589640 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:56.589612    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/ca31519e-aa90-4104-a175-f9af2b3bbeb4-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-6vkbv\" (UID: \"ca31519e-aa90-4104-a175-f9af2b3bbeb4\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-6vkbv"
Oct 31 07:28:56.590032 ip-10-0-139-83 kubenswrapper[2573]: E1031 07:28:56.589756    2573 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Oct 31 07:28:56.590032 ip-10-0-139-83 kubenswrapper[2573]: E1031 07:28:56.589811    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ca31519e-aa90-4104-a175-f9af2b3bbeb4-samples-operator-tls podName:ca31519e-aa90-4104-a175-f9af2b3bbeb4 nodeName:}" failed. No retries permitted until 2026-10-31 07:28:58.589794598 +0000 UTC m=+35.280565403 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/ca31519e-aa90-4104-a175-f9af2b3bbeb4-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-6vkbv" (UID: "ca31519e-aa90-4104-a175-f9af2b3bbeb4") : secret "samples-operator-tls" not found
Oct 31 07:28:56.690352 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:56.690304    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/28b59d93-b9ce-4ceb-9f3a-742525eb94c4-registry-tls\") pod \"image-registry-7955c67bcc-pkntv\" (UID: \"28b59d93-b9ce-4ceb-9f3a-742525eb94c4\") " pod="openshift-image-registry/image-registry-7955c67bcc-pkntv"
Oct 31 07:28:56.690508 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:56.690369    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/789f11d8-d951-4144-a589-a2bdbf6844e3-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-6k8mn\" (UID: \"789f11d8-d951-4144-a589-a2bdbf6844e3\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-6k8mn"
Oct 31 07:28:56.690508 ip-10-0-139-83 kubenswrapper[2573]: E1031 07:28:56.690441    2573 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 31 07:28:56.690508 ip-10-0-139-83 kubenswrapper[2573]: E1031 07:28:56.690457    2573 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Oct 31 07:28:56.690508 ip-10-0-139-83 kubenswrapper[2573]: E1031 07:28:56.690473    2573 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7955c67bcc-pkntv: secret "image-registry-tls" not found
Oct 31 07:28:56.690690 ip-10-0-139-83 kubenswrapper[2573]: E1031 07:28:56.690517    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/789f11d8-d951-4144-a589-a2bdbf6844e3-cluster-monitoring-operator-tls podName:789f11d8-d951-4144-a589-a2bdbf6844e3 nodeName:}" failed. No retries permitted until 2026-10-31 07:28:58.690499323 +0000 UTC m=+35.381270129 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/789f11d8-d951-4144-a589-a2bdbf6844e3-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-6k8mn" (UID: "789f11d8-d951-4144-a589-a2bdbf6844e3") : secret "cluster-monitoring-operator-tls" not found
Oct 31 07:28:56.690690 ip-10-0-139-83 kubenswrapper[2573]: E1031 07:28:56.690535    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/28b59d93-b9ce-4ceb-9f3a-742525eb94c4-registry-tls podName:28b59d93-b9ce-4ceb-9f3a-742525eb94c4 nodeName:}" failed. No retries permitted until 2026-10-31 07:28:58.690525637 +0000 UTC m=+35.381296439 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/28b59d93-b9ce-4ceb-9f3a-742525eb94c4-registry-tls") pod "image-registry-7955c67bcc-pkntv" (UID: "28b59d93-b9ce-4ceb-9f3a-742525eb94c4") : secret "image-registry-tls" not found
Oct 31 07:28:56.690690 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:56.690551    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b44ed0e0-a72e-4746-a7de-ed8610da845d-metrics-certs\") pod \"router-default-6c88c45d57-c2bgn\" (UID: \"b44ed0e0-a72e-4746-a7de-ed8610da845d\") " pod="openshift-ingress/router-default-6c88c45d57-c2bgn"
Oct 31 07:28:56.690690 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:56.690625    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/993cfed6-806d-44c5-bba8-c02203638fe7-registry-tls\") pod \"image-registry-7886c7d6c6-qdc9f\" (UID: \"993cfed6-806d-44c5-bba8-c02203638fe7\") " pod="openshift-image-registry/image-registry-7886c7d6c6-qdc9f"
Oct 31 07:28:56.690690 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:56.690669    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b44ed0e0-a72e-4746-a7de-ed8610da845d-service-ca-bundle\") pod \"router-default-6c88c45d57-c2bgn\" (UID: \"b44ed0e0-a72e-4746-a7de-ed8610da845d\") " pod="openshift-ingress/router-default-6c88c45d57-c2bgn"
Oct 31 07:28:56.690899 ip-10-0-139-83 kubenswrapper[2573]: E1031 07:28:56.690704    2573 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Oct 31 07:28:56.690899 ip-10-0-139-83 kubenswrapper[2573]: E1031 07:28:56.690735    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b44ed0e0-a72e-4746-a7de-ed8610da845d-metrics-certs podName:b44ed0e0-a72e-4746-a7de-ed8610da845d nodeName:}" failed. No retries permitted until 2026-10-31 07:28:58.690724485 +0000 UTC m=+35.381495287 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b44ed0e0-a72e-4746-a7de-ed8610da845d-metrics-certs") pod "router-default-6c88c45d57-c2bgn" (UID: "b44ed0e0-a72e-4746-a7de-ed8610da845d") : secret "router-metrics-certs-default" not found
Oct 31 07:28:56.690899 ip-10-0-139-83 kubenswrapper[2573]: E1031 07:28:56.690776    2573 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 31 07:28:56.690899 ip-10-0-139-83 kubenswrapper[2573]: E1031 07:28:56.690781    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b44ed0e0-a72e-4746-a7de-ed8610da845d-service-ca-bundle podName:b44ed0e0-a72e-4746-a7de-ed8610da845d nodeName:}" failed. No retries permitted until 2026-10-31 07:28:58.690767456 +0000 UTC m=+35.381538279 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/b44ed0e0-a72e-4746-a7de-ed8610da845d-service-ca-bundle") pod "router-default-6c88c45d57-c2bgn" (UID: "b44ed0e0-a72e-4746-a7de-ed8610da845d") : configmap references non-existent config key: service-ca.crt
Oct 31 07:28:56.690899 ip-10-0-139-83 kubenswrapper[2573]: E1031 07:28:56.690782    2573 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7886c7d6c6-qdc9f: secret "image-registry-tls" not found
Oct 31 07:28:56.690899 ip-10-0-139-83 kubenswrapper[2573]: E1031 07:28:56.690807    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/993cfed6-806d-44c5-bba8-c02203638fe7-registry-tls podName:993cfed6-806d-44c5-bba8-c02203638fe7 nodeName:}" failed. No retries permitted until 2026-10-31 07:28:58.690800585 +0000 UTC m=+35.381571388 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/993cfed6-806d-44c5-bba8-c02203638fe7-registry-tls") pod "image-registry-7886c7d6c6-qdc9f" (UID: "993cfed6-806d-44c5-bba8-c02203638fe7") : secret "image-registry-tls" not found
Oct 31 07:28:56.791319 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:56.791282    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/77256436-39ff-4c2a-8390-4e9fa7d68486-original-pull-secret\") pod \"global-pull-secret-syncer-kht4l\" (UID: \"77256436-39ff-4c2a-8390-4e9fa7d68486\") " pod="kube-system/global-pull-secret-syncer-kht4l"
Oct 31 07:28:56.791538 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:56.791419    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/314c29aa-db6b-4f7f-86b4-c8952ce09c84-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-j94dz\" (UID: \"314c29aa-db6b-4f7f-86b4-c8952ce09c84\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-j94dz"
Oct 31 07:28:56.791608 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:56.791589    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c3ad95fd-e997-459c-93fd-27c4c030ea0d-metrics-certs\") pod \"network-metrics-daemon-r2zbv\" (UID: \"c3ad95fd-e997-459c-93fd-27c4c030ea0d\") " pod="openshift-multus/network-metrics-daemon-r2zbv"
Oct 31 07:28:56.791783 ip-10-0-139-83 kubenswrapper[2573]: E1031 07:28:56.791766    2573 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Oct 31 07:28:56.791873 ip-10-0-139-83 kubenswrapper[2573]: E1031 07:28:56.791812    2573 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Oct 31 07:28:56.791873 ip-10-0-139-83 kubenswrapper[2573]: E1031 07:28:56.791824    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c3ad95fd-e997-459c-93fd-27c4c030ea0d-metrics-certs podName:c3ad95fd-e997-459c-93fd-27c4c030ea0d nodeName:}" failed. No retries permitted until 2026-10-31 07:29:28.79180705 +0000 UTC m=+65.482577856 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c3ad95fd-e997-459c-93fd-27c4c030ea0d-metrics-certs") pod "network-metrics-daemon-r2zbv" (UID: "c3ad95fd-e997-459c-93fd-27c4c030ea0d") : secret "metrics-daemon-secret" not found
Oct 31 07:28:56.791873 ip-10-0-139-83 kubenswrapper[2573]: E1031 07:28:56.791867    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/314c29aa-db6b-4f7f-86b4-c8952ce09c84-networking-console-plugin-cert podName:314c29aa-db6b-4f7f-86b4-c8952ce09c84 nodeName:}" failed. No retries permitted until 2026-10-31 07:28:58.791851611 +0000 UTC m=+35.482622419 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/314c29aa-db6b-4f7f-86b4-c8952ce09c84-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-j94dz" (UID: "314c29aa-db6b-4f7f-86b4-c8952ce09c84") : secret "networking-console-plugin-cert" not found
Oct 31 07:28:56.795794 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:56.795768    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/77256436-39ff-4c2a-8390-4e9fa7d68486-original-pull-secret\") pod \"global-pull-secret-syncer-kht4l\" (UID: \"77256436-39ff-4c2a-8390-4e9fa7d68486\") " pod="kube-system/global-pull-secret-syncer-kht4l"
Oct 31 07:28:56.880071 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:56.879961    2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-xcrsn" podStartSLOduration=415.87993696 podStartE2EDuration="6m55.87993696s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 07:22:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 07:28:56.879444301 +0000 UTC m=+33.570215124" watchObservedRunningTime="2026-10-31 07:28:56.87993696 +0000 UTC m=+33.570707786"
Oct 31 07:28:56.893676 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:56.892608    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a73024cb-f581-450d-a09c-0565838c5baa-metrics-tls\") pod \"dns-default-9cmfx\" (UID: \"a73024cb-f581-450d-a09c-0565838c5baa\") " pod="openshift-dns/dns-default-9cmfx"
Oct 31 07:28:56.893676 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:56.892730    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ghkb9\" (UniqueName: \"kubernetes.io/projected/f1ba68f6-d87c-4492-a5eb-5f2d3a60eb1c-kube-api-access-ghkb9\") pod \"network-check-target-9zp4l\" (UID: \"f1ba68f6-d87c-4492-a5eb-5f2d3a60eb1c\") " pod="openshift-network-diagnostics/network-check-target-9zp4l"
Oct 31 07:28:56.893676 ip-10-0-139-83 kubenswrapper[2573]: E1031 07:28:56.893542    2573 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 31 07:28:56.893676 ip-10-0-139-83 kubenswrapper[2573]: E1031 07:28:56.893607    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a73024cb-f581-450d-a09c-0565838c5baa-metrics-tls podName:a73024cb-f581-450d-a09c-0565838c5baa nodeName:}" failed. No retries permitted until 2026-10-31 07:28:58.893588886 +0000 UTC m=+35.584359693 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/a73024cb-f581-450d-a09c-0565838c5baa-metrics-tls") pod "dns-default-9cmfx" (UID: "a73024cb-f581-450d-a09c-0565838c5baa") : secret "dns-default-metrics-tls" not found
Oct 31 07:28:56.896500 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:56.896479    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ghkb9\" (UniqueName: \"kubernetes.io/projected/f1ba68f6-d87c-4492-a5eb-5f2d3a60eb1c-kube-api-access-ghkb9\") pod \"network-check-target-9zp4l\" (UID: \"f1ba68f6-d87c-4492-a5eb-5f2d3a60eb1c\") " pod="openshift-network-diagnostics/network-check-target-9zp4l"
Oct 31 07:28:56.993406 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:56.992875    2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-hxmpf" podStartSLOduration=415.992857636 podStartE2EDuration="6m55.992857636s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 07:22:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 07:28:56.992547898 +0000 UTC m=+33.683318723" watchObservedRunningTime="2026-10-31 07:28:56.992857636 +0000 UTC m=+33.683628461"
Oct 31 07:28:57.049818 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:57.049623    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kht4l"
Oct 31 07:28:57.095935 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:57.095321    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9zp4l"
Oct 31 07:28:57.107415 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:57.107362    2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-f9gtg" podStartSLOduration=416.107345299 podStartE2EDuration="6m56.107345299s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 07:22:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 07:28:57.10538254 +0000 UTC m=+33.796153365" watchObservedRunningTime="2026-10-31 07:28:57.107345299 +0000 UTC m=+33.798116119"
Oct 31 07:28:57.481903 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:57.481585    2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-kht4l"]
Oct 31 07:28:57.491169 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:57.491141    2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod77256436_39ff_4c2a_8390_4e9fa7d68486.slice/crio-7a306c5d41908aad98edab19712d66d4e434fca13d0fb3b8c8f78ed55689cab5 WatchSource:0}: Error finding container 7a306c5d41908aad98edab19712d66d4e434fca13d0fb3b8c8f78ed55689cab5: Status 404 returned error can't find the container with id 7a306c5d41908aad98edab19712d66d4e434fca13d0fb3b8c8f78ed55689cab5
Oct 31 07:28:57.523560 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:57.523539    2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-9zp4l"]
Oct 31 07:28:57.537913 ip-10-0-139-83 kubenswrapper[2573]: W1031 07:28:57.537887    2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf1ba68f6_d87c_4492_a5eb_5f2d3a60eb1c.slice/crio-7c6cf74c69f7fceea5860f8c2894b19081b8a4f44528c01cdf3a23bc22f567b5 WatchSource:0}: Error finding container 7c6cf74c69f7fceea5860f8c2894b19081b8a4f44528c01cdf3a23bc22f567b5: Status 404 returned error can't find the container with id 7c6cf74c69f7fceea5860f8c2894b19081b8a4f44528c01cdf3a23bc22f567b5
Oct 31 07:28:58.100646 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:58.100614    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-kht4l" event={"ID":"77256436-39ff-4c2a-8390-4e9fa7d68486","Type":"ContainerStarted","Data":"7a306c5d41908aad98edab19712d66d4e434fca13d0fb3b8c8f78ed55689cab5"}
Oct 31 07:28:58.104680 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:58.104648    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-9zp4l" event={"ID":"f1ba68f6-d87c-4492-a5eb-5f2d3a60eb1c","Type":"ContainerStarted","Data":"7c6cf74c69f7fceea5860f8c2894b19081b8a4f44528c01cdf3a23bc22f567b5"}
Oct 31 07:28:58.611608 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:58.611566    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/ca31519e-aa90-4104-a175-f9af2b3bbeb4-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-6vkbv\" (UID: \"ca31519e-aa90-4104-a175-f9af2b3bbeb4\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-6vkbv"
Oct 31 07:28:58.625696 ip-10-0-139-83 kubenswrapper[2573]: E1031 07:28:58.625657    2573 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Oct 31 07:28:58.625860 ip-10-0-139-83 kubenswrapper[2573]: E1031 07:28:58.625749    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ca31519e-aa90-4104-a175-f9af2b3bbeb4-samples-operator-tls podName:ca31519e-aa90-4104-a175-f9af2b3bbeb4 nodeName:}" failed. No retries permitted until 2026-10-31 07:29:02.625721921 +0000 UTC m=+39.316492727 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/ca31519e-aa90-4104-a175-f9af2b3bbeb4-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-6vkbv" (UID: "ca31519e-aa90-4104-a175-f9af2b3bbeb4") : secret "samples-operator-tls" not found
Oct 31 07:28:58.712411 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:58.712316    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/28b59d93-b9ce-4ceb-9f3a-742525eb94c4-registry-tls\") pod \"image-registry-7955c67bcc-pkntv\" (UID: \"28b59d93-b9ce-4ceb-9f3a-742525eb94c4\") " pod="openshift-image-registry/image-registry-7955c67bcc-pkntv"
Oct 31 07:28:58.712411 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:58.712392    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/789f11d8-d951-4144-a589-a2bdbf6844e3-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-6k8mn\" (UID: \"789f11d8-d951-4144-a589-a2bdbf6844e3\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-6k8mn"
Oct 31 07:28:58.712613 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:58.712423    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b44ed0e0-a72e-4746-a7de-ed8610da845d-metrics-certs\") pod \"router-default-6c88c45d57-c2bgn\" (UID: \"b44ed0e0-a72e-4746-a7de-ed8610da845d\") " pod="openshift-ingress/router-default-6c88c45d57-c2bgn"
Oct 31 07:28:58.712613 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:58.712488    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/993cfed6-806d-44c5-bba8-c02203638fe7-registry-tls\") pod \"image-registry-7886c7d6c6-qdc9f\" (UID: \"993cfed6-806d-44c5-bba8-c02203638fe7\") " pod="openshift-image-registry/image-registry-7886c7d6c6-qdc9f"
Oct 31 07:28:58.712613 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:58.712522    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b44ed0e0-a72e-4746-a7de-ed8610da845d-service-ca-bundle\") pod \"router-default-6c88c45d57-c2bgn\" (UID: \"b44ed0e0-a72e-4746-a7de-ed8610da845d\") " pod="openshift-ingress/router-default-6c88c45d57-c2bgn"
Oct 31 07:28:58.712763 ip-10-0-139-83 kubenswrapper[2573]: E1031 07:28:58.712662    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b44ed0e0-a72e-4746-a7de-ed8610da845d-service-ca-bundle podName:b44ed0e0-a72e-4746-a7de-ed8610da845d nodeName:}" failed. No retries permitted until 2026-10-31 07:29:02.712643399 +0000 UTC m=+39.403414208 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/b44ed0e0-a72e-4746-a7de-ed8610da845d-service-ca-bundle") pod "router-default-6c88c45d57-c2bgn" (UID: "b44ed0e0-a72e-4746-a7de-ed8610da845d") : configmap references non-existent config key: service-ca.crt
Oct 31 07:28:58.713184 ip-10-0-139-83 kubenswrapper[2573]: E1031 07:28:58.713162    2573 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 31 07:28:58.713184 ip-10-0-139-83 kubenswrapper[2573]: E1031 07:28:58.713185    2573 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7955c67bcc-pkntv: secret "image-registry-tls" not found
Oct 31 07:28:58.713384 ip-10-0-139-83 kubenswrapper[2573]: E1031 07:28:58.713287    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/28b59d93-b9ce-4ceb-9f3a-742525eb94c4-registry-tls podName:28b59d93-b9ce-4ceb-9f3a-742525eb94c4 nodeName:}" failed. No retries permitted until 2026-10-31 07:29:02.713214113 +0000 UTC m=+39.403984920 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/28b59d93-b9ce-4ceb-9f3a-742525eb94c4-registry-tls") pod "image-registry-7955c67bcc-pkntv" (UID: "28b59d93-b9ce-4ceb-9f3a-742525eb94c4") : secret "image-registry-tls" not found
Oct 31 07:28:58.713455 ip-10-0-139-83 kubenswrapper[2573]: E1031 07:28:58.713395    2573 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Oct 31 07:28:58.713455 ip-10-0-139-83 kubenswrapper[2573]: E1031 07:28:58.713452    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/789f11d8-d951-4144-a589-a2bdbf6844e3-cluster-monitoring-operator-tls podName:789f11d8-d951-4144-a589-a2bdbf6844e3 nodeName:}" failed. No retries permitted until 2026-10-31 07:29:02.713422029 +0000 UTC m=+39.404192832 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/789f11d8-d951-4144-a589-a2bdbf6844e3-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-6k8mn" (UID: "789f11d8-d951-4144-a589-a2bdbf6844e3") : secret "cluster-monitoring-operator-tls" not found
Oct 31 07:28:58.713559 ip-10-0-139-83 kubenswrapper[2573]: E1031 07:28:58.713507    2573 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Oct 31 07:28:58.713608 ip-10-0-139-83 kubenswrapper[2573]: E1031 07:28:58.713538    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b44ed0e0-a72e-4746-a7de-ed8610da845d-metrics-certs podName:b44ed0e0-a72e-4746-a7de-ed8610da845d nodeName:}" failed. No retries permitted until 2026-10-31 07:29:02.713529398 +0000 UTC m=+39.404300201 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b44ed0e0-a72e-4746-a7de-ed8610da845d-metrics-certs") pod "router-default-6c88c45d57-c2bgn" (UID: "b44ed0e0-a72e-4746-a7de-ed8610da845d") : secret "router-metrics-certs-default" not found
Oct 31 07:28:58.713663 ip-10-0-139-83 kubenswrapper[2573]: E1031 07:28:58.713626    2573 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 31 07:28:58.713663 ip-10-0-139-83 kubenswrapper[2573]: E1031 07:28:58.713637    2573 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7886c7d6c6-qdc9f: secret "image-registry-tls" not found
Oct 31 07:28:58.713777 ip-10-0-139-83 kubenswrapper[2573]: E1031 07:28:58.713666    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/993cfed6-806d-44c5-bba8-c02203638fe7-registry-tls podName:993cfed6-806d-44c5-bba8-c02203638fe7 nodeName:}" failed. No retries permitted until 2026-10-31 07:29:02.713656275 +0000 UTC m=+39.404427082 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/993cfed6-806d-44c5-bba8-c02203638fe7-registry-tls") pod "image-registry-7886c7d6c6-qdc9f" (UID: "993cfed6-806d-44c5-bba8-c02203638fe7") : secret "image-registry-tls" not found
Oct 31 07:28:58.814104 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:58.814047    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/314c29aa-db6b-4f7f-86b4-c8952ce09c84-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-j94dz\" (UID: \"314c29aa-db6b-4f7f-86b4-c8952ce09c84\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-j94dz"
Oct 31 07:28:58.814425 ip-10-0-139-83 kubenswrapper[2573]: E1031 07:28:58.814401    2573 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Oct 31 07:28:58.814507 ip-10-0-139-83 kubenswrapper[2573]: E1031 07:28:58.814481    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/314c29aa-db6b-4f7f-86b4-c8952ce09c84-networking-console-plugin-cert podName:314c29aa-db6b-4f7f-86b4-c8952ce09c84 nodeName:}" failed. No retries permitted until 2026-10-31 07:29:02.814462068 +0000 UTC m=+39.505232874 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/314c29aa-db6b-4f7f-86b4-c8952ce09c84-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-j94dz" (UID: "314c29aa-db6b-4f7f-86b4-c8952ce09c84") : secret "networking-console-plugin-cert" not found
Oct 31 07:28:58.914791 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:58.914753    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a73024cb-f581-450d-a09c-0565838c5baa-metrics-tls\") pod \"dns-default-9cmfx\" (UID: \"a73024cb-f581-450d-a09c-0565838c5baa\") " pod="openshift-dns/dns-default-9cmfx"
Oct 31 07:28:58.915073 ip-10-0-139-83 kubenswrapper[2573]: E1031 07:28:58.915051    2573 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 31 07:28:58.915176 ip-10-0-139-83 kubenswrapper[2573]: E1031 07:28:58.915118    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a73024cb-f581-450d-a09c-0565838c5baa-metrics-tls podName:a73024cb-f581-450d-a09c-0565838c5baa nodeName:}" failed. No retries permitted until 2026-10-31 07:29:02.915098522 +0000 UTC m=+39.605869347 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/a73024cb-f581-450d-a09c-0565838c5baa-metrics-tls") pod "dns-default-9cmfx" (UID: "a73024cb-f581-450d-a09c-0565838c5baa") : secret "dns-default-metrics-tls" not found
Oct 31 07:28:59.115148 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:59.115114    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-65chl" event={"ID":"ee4a4a2a-7439-4c78-8858-d6ba0085fdde","Type":"ContainerStarted","Data":"b7441184620ee2bed3f65cff62bee476361075968220dd5915fa8083e5923b7d"}
Oct 31 07:28:59.229691 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:59.229634    2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-65chl" podStartSLOduration=387.684255369 podStartE2EDuration="6m30.229536839s" totalImagesPullingTime="2.54528147s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 07:22:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 07:28:59.2280837 +0000 UTC m=+35.918854526" watchObservedRunningTime="2026-10-31 07:28:59.229536839 +0000 UTC m=+35.920307667"
Oct 31 07:28:59.267018 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:59.266957    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 07:28:59.511532 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:28:59.511432    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 07:29:02.129940 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:29:02.129747    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-hb49h" event={"ID":"4db4c33f-0c58-40f8-bf74-51b3eae3c9db","Type":"ContainerStarted","Data":"acc53d1d3f53d66239c0908e4ff0412ae73dbf42bd87be51c0d44b406a269631"}
Oct 31 07:29:02.133825 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:29:02.133794    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-dwc4p" event={"ID":"81829763-27b1-4a48-a568-cd5f6a39a953","Type":"ContainerStarted","Data":"44664141967c1ff5700a167e429d15365939c9433c32292fcd3583ff3e207b08"}
Oct 31 07:29:02.139203 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:29:02.138639    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-4jszp_b1db0e3b-e85d-4769-8012-cf4b6f740d9d/console-operator/0.log"
Oct 31 07:29:02.139203 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:29:02.138671    2573 generic.go:356] "Generic (PLEG): container finished" podID="b1db0e3b-e85d-4769-8012-cf4b6f740d9d" containerID="40bc998862f811cf65d3ac61aa449437d6521a1d37ba6e8f97920063c644bdbf" exitCode=255
Oct 31 07:29:02.139203 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:29:02.138733    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-4jszp" event={"ID":"b1db0e3b-e85d-4769-8012-cf4b6f740d9d","Type":"ContainerDied","Data":"40bc998862f811cf65d3ac61aa449437d6521a1d37ba6e8f97920063c644bdbf"}
Oct 31 07:29:02.139203 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:29:02.138937    2573 scope.go:122] "RemoveContainer" containerID="40bc998862f811cf65d3ac61aa449437d6521a1d37ba6e8f97920063c644bdbf"
Oct 31 07:29:02.142552 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:29:02.142508    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-p9z4r" event={"ID":"cb9e4575-9265-43b5-bf02-7843611d6563","Type":"ContainerStarted","Data":"e25341347c884d204f4de3bdc1eb8e6f14c7fb76f88609e30c77e2007599130e"}
Oct 31 07:29:02.246379 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:29:02.243756    2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-hb49h" podStartSLOduration=386.20570931 podStartE2EDuration="6m32.243741943s" totalImagesPullingTime="6.038032633s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 07:22:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 07:29:02.242594605 +0000 UTC m=+38.933365433" watchObservedRunningTime="2026-10-31 07:29:02.243741943 +0000 UTC m=+38.934512769"
Oct 31 07:29:02.476064 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:29:02.476004    2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/insights-operator-7c4986d449-dwc4p" podStartSLOduration=387.871475057 podStartE2EDuration="6m33.475984815s" totalImagesPullingTime="5.604509758s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 07:22:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 07:29:02.473802584 +0000 UTC m=+39.164573410" watchObservedRunningTime="2026-10-31 07:29:02.475984815 +0000 UTC m=+39.166755649"
Oct 31 07:29:02.587624 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:29:02.587567    2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-p9z4r" podStartSLOduration=385.612603556 podStartE2EDuration="6m31.587548157s" totalImagesPullingTime="5.974944601s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 07:22:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 07:29:02.586189675 +0000 UTC m=+39.276960500" watchObservedRunningTime="2026-10-31 07:29:02.587548157 +0000 UTC m=+39.278318983"
Oct 31 07:29:02.673743 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:29:02.673703    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/ca31519e-aa90-4104-a175-f9af2b3bbeb4-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-6vkbv\" (UID: \"ca31519e-aa90-4104-a175-f9af2b3bbeb4\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-6vkbv"
Oct 31 07:29:02.674119 ip-10-0-139-83 kubenswrapper[2573]: E1031 07:29:02.674099    2573 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Oct 31 07:29:02.674211 ip-10-0-139-83 kubenswrapper[2573]: E1031 07:29:02.674166    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ca31519e-aa90-4104-a175-f9af2b3bbeb4-samples-operator-tls podName:ca31519e-aa90-4104-a175-f9af2b3bbeb4 nodeName:}" failed. No retries permitted until 2026-10-31 07:29:10.67414603 +0000 UTC m=+47.364916847 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/ca31519e-aa90-4104-a175-f9af2b3bbeb4-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-6vkbv" (UID: "ca31519e-aa90-4104-a175-f9af2b3bbeb4") : secret "samples-operator-tls" not found
Oct 31 07:29:02.774631 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:29:02.774557    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/993cfed6-806d-44c5-bba8-c02203638fe7-registry-tls\") pod \"image-registry-7886c7d6c6-qdc9f\" (UID: \"993cfed6-806d-44c5-bba8-c02203638fe7\") " pod="openshift-image-registry/image-registry-7886c7d6c6-qdc9f"
Oct 31 07:29:02.774631 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:29:02.774605    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b44ed0e0-a72e-4746-a7de-ed8610da845d-service-ca-bundle\") pod \"router-default-6c88c45d57-c2bgn\" (UID: \"b44ed0e0-a72e-4746-a7de-ed8610da845d\") " pod="openshift-ingress/router-default-6c88c45d57-c2bgn"
Oct 31 07:29:02.774842 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:29:02.774672    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/28b59d93-b9ce-4ceb-9f3a-742525eb94c4-registry-tls\") pod \"image-registry-7955c67bcc-pkntv\" (UID: \"28b59d93-b9ce-4ceb-9f3a-742525eb94c4\") " pod="openshift-image-registry/image-registry-7955c67bcc-pkntv"
Oct 31 07:29:02.774842 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:29:02.774690    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/789f11d8-d951-4144-a589-a2bdbf6844e3-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-6k8mn\" (UID: \"789f11d8-d951-4144-a589-a2bdbf6844e3\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-6k8mn"
Oct 31 07:29:02.774842 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:29:02.774710    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b44ed0e0-a72e-4746-a7de-ed8610da845d-metrics-certs\") pod \"router-default-6c88c45d57-c2bgn\" (UID: \"b44ed0e0-a72e-4746-a7de-ed8610da845d\") " pod="openshift-ingress/router-default-6c88c45d57-c2bgn"
Oct 31 07:29:02.774842 ip-10-0-139-83 kubenswrapper[2573]: E1031 07:29:02.774729    2573 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 31 07:29:02.774842 ip-10-0-139-83 kubenswrapper[2573]: E1031 07:29:02.774753    2573 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7886c7d6c6-qdc9f: secret "image-registry-tls" not found
Oct 31 07:29:02.774842 ip-10-0-139-83 kubenswrapper[2573]: E1031 07:29:02.774805    2573 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Oct 31 07:29:02.774842 ip-10-0-139-83 kubenswrapper[2573]: E1031 07:29:02.774813    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/993cfed6-806d-44c5-bba8-c02203638fe7-registry-tls podName:993cfed6-806d-44c5-bba8-c02203638fe7 nodeName:}" failed. No retries permitted until 2026-10-31 07:29:10.77479391 +0000 UTC m=+47.465564719 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/993cfed6-806d-44c5-bba8-c02203638fe7-registry-tls") pod "image-registry-7886c7d6c6-qdc9f" (UID: "993cfed6-806d-44c5-bba8-c02203638fe7") : secret "image-registry-tls" not found
Oct 31 07:29:02.774842 ip-10-0-139-83 kubenswrapper[2573]: E1031 07:29:02.774845    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b44ed0e0-a72e-4746-a7de-ed8610da845d-metrics-certs podName:b44ed0e0-a72e-4746-a7de-ed8610da845d nodeName:}" failed. No retries permitted until 2026-10-31 07:29:10.774833042 +0000 UTC m=+47.465603850 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b44ed0e0-a72e-4746-a7de-ed8610da845d-metrics-certs") pod "router-default-6c88c45d57-c2bgn" (UID: "b44ed0e0-a72e-4746-a7de-ed8610da845d") : secret "router-metrics-certs-default" not found
Oct 31 07:29:02.775201 ip-10-0-139-83 kubenswrapper[2573]: E1031 07:29:02.774888    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b44ed0e0-a72e-4746-a7de-ed8610da845d-service-ca-bundle podName:b44ed0e0-a72e-4746-a7de-ed8610da845d nodeName:}" failed. No retries permitted until 2026-10-31 07:29:10.774882174 +0000 UTC m=+47.465652997 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/b44ed0e0-a72e-4746-a7de-ed8610da845d-service-ca-bundle") pod "router-default-6c88c45d57-c2bgn" (UID: "b44ed0e0-a72e-4746-a7de-ed8610da845d") : configmap references non-existent config key: service-ca.crt
Oct 31 07:29:02.775201 ip-10-0-139-83 kubenswrapper[2573]: E1031 07:29:02.774928    2573 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 31 07:29:02.775201 ip-10-0-139-83 kubenswrapper[2573]: E1031 07:29:02.774935    2573 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7955c67bcc-pkntv: secret "image-registry-tls" not found
Oct 31 07:29:02.775201 ip-10-0-139-83 kubenswrapper[2573]: E1031 07:29:02.774953    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/28b59d93-b9ce-4ceb-9f3a-742525eb94c4-registry-tls podName:28b59d93-b9ce-4ceb-9f3a-742525eb94c4 nodeName:}" failed. No retries permitted until 2026-10-31 07:29:10.774947305 +0000 UTC m=+47.465718146 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/28b59d93-b9ce-4ceb-9f3a-742525eb94c4-registry-tls") pod "image-registry-7955c67bcc-pkntv" (UID: "28b59d93-b9ce-4ceb-9f3a-742525eb94c4") : secret "image-registry-tls" not found
Oct 31 07:29:02.775201 ip-10-0-139-83 kubenswrapper[2573]: E1031 07:29:02.774988    2573 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Oct 31 07:29:02.775201 ip-10-0-139-83 kubenswrapper[2573]: E1031 07:29:02.775006    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/789f11d8-d951-4144-a589-a2bdbf6844e3-cluster-monitoring-operator-tls podName:789f11d8-d951-4144-a589-a2bdbf6844e3 nodeName:}" failed. No retries permitted until 2026-10-31 07:29:10.775000515 +0000 UTC m=+47.465771318 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/789f11d8-d951-4144-a589-a2bdbf6844e3-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-6k8mn" (UID: "789f11d8-d951-4144-a589-a2bdbf6844e3") : secret "cluster-monitoring-operator-tls" not found
Oct 31 07:29:02.846599 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:29:02.846566    2573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-8m768"]
Oct 31 07:29:02.864905 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:29:02.864853    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-8m768"
Oct 31 07:29:02.875273 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:29:02.875170    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cjt6r\" (UniqueName: \"kubernetes.io/projected/b1066ece-e6f3-4491-aa1f-a7b9a3aa248d-kube-api-access-cjt6r\") pod \"insights-runtime-extractor-8m768\" (UID: \"b1066ece-e6f3-4491-aa1f-a7b9a3aa248d\") " pod="openshift-insights/insights-runtime-extractor-8m768"
Oct 31 07:29:02.875273 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:29:02.875248    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b1066ece-e6f3-4491-aa1f-a7b9a3aa248d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-8m768\" (UID: \"b1066ece-e6f3-4491-aa1f-a7b9a3aa248d\") " pod="openshift-insights/insights-runtime-extractor-8m768"
Oct 31 07:29:02.875506 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:29:02.875384    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/b1066ece-e6f3-4491-aa1f-a7b9a3aa248d-crio-socket\") pod \"insights-runtime-extractor-8m768\" (UID: \"b1066ece-e6f3-4491-aa1f-a7b9a3aa248d\") " pod="openshift-insights/insights-runtime-extractor-8m768"
Oct 31 07:29:02.875506 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:29:02.875424    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b1066ece-e6f3-4491-aa1f-a7b9a3aa248d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-8m768\" (UID: \"b1066ece-e6f3-4491-aa1f-a7b9a3aa248d\") " pod="openshift-insights/insights-runtime-extractor-8m768"
Oct 31 07:29:02.875506 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:29:02.875462    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/b1066ece-e6f3-4491-aa1f-a7b9a3aa248d-data-volume\") pod \"insights-runtime-extractor-8m768\" (UID: \"b1066ece-e6f3-4491-aa1f-a7b9a3aa248d\") " pod="openshift-insights/insights-runtime-extractor-8m768"
Oct 31 07:29:02.875653 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:29:02.875525    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/314c29aa-db6b-4f7f-86b4-c8952ce09c84-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-j94dz\" (UID: \"314c29aa-db6b-4f7f-86b4-c8952ce09c84\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-j94dz"
Oct 31 07:29:02.875733 ip-10-0-139-83 kubenswrapper[2573]: E1031 07:29:02.875715    2573 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Oct 31 07:29:02.875797 ip-10-0-139-83 kubenswrapper[2573]: E1031 07:29:02.875784    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/314c29aa-db6b-4f7f-86b4-c8952ce09c84-networking-console-plugin-cert podName:314c29aa-db6b-4f7f-86b4-c8952ce09c84 nodeName:}" failed. No retries permitted until 2026-10-31 07:29:10.875763089 +0000 UTC m=+47.566533906 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/314c29aa-db6b-4f7f-86b4-c8952ce09c84-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-j94dz" (UID: "314c29aa-db6b-4f7f-86b4-c8952ce09c84") : secret "networking-console-plugin-cert" not found
Oct 31 07:29:02.909136 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:29:02.909045    2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-8m768"]
Oct 31 07:29:02.918932 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:29:02.918903    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Oct 31 07:29:02.919135 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:29:02.919120    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-g75wk\""
Oct 31 07:29:02.940703 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:29:02.940676    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Oct 31 07:29:02.976218 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:29:02.975947    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cjt6r\" (UniqueName: \"kubernetes.io/projected/b1066ece-e6f3-4491-aa1f-a7b9a3aa248d-kube-api-access-cjt6r\") pod \"insights-runtime-extractor-8m768\" (UID: \"b1066ece-e6f3-4491-aa1f-a7b9a3aa248d\") " pod="openshift-insights/insights-runtime-extractor-8m768"
Oct 31 07:29:02.976218 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:29:02.976022    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b1066ece-e6f3-4491-aa1f-a7b9a3aa248d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-8m768\" (UID: \"b1066ece-e6f3-4491-aa1f-a7b9a3aa248d\") " pod="openshift-insights/insights-runtime-extractor-8m768"
Oct 31 07:29:02.976218 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:29:02.976092    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/b1066ece-e6f3-4491-aa1f-a7b9a3aa248d-crio-socket\") pod \"insights-runtime-extractor-8m768\" (UID: \"b1066ece-e6f3-4491-aa1f-a7b9a3aa248d\") " pod="openshift-insights/insights-runtime-extractor-8m768"
Oct 31 07:29:02.976218 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:29:02.976122    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b1066ece-e6f3-4491-aa1f-a7b9a3aa248d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-8m768\" (UID: \"b1066ece-e6f3-4491-aa1f-a7b9a3aa248d\") " pod="openshift-insights/insights-runtime-extractor-8m768"
Oct 31 07:29:02.976218 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:29:02.976159    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/b1066ece-e6f3-4491-aa1f-a7b9a3aa248d-data-volume\") pod \"insights-runtime-extractor-8m768\" (UID: \"b1066ece-e6f3-4491-aa1f-a7b9a3aa248d\") " pod="openshift-insights/insights-runtime-extractor-8m768"
Oct 31 07:29:02.976649 ip-10-0-139-83 kubenswrapper[2573]: E1031 07:29:02.976225    2573 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Oct 31 07:29:02.976649 ip-10-0-139-83 kubenswrapper[2573]: E1031 07:29:02.976297    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b1066ece-e6f3-4491-aa1f-a7b9a3aa248d-kube-rbac-proxy-cm podName:b1066ece-e6f3-4491-aa1f-a7b9a3aa248d nodeName:}" failed. No retries permitted until 2026-10-31 07:29:03.476273925 +0000 UTC m=+40.167044742 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/b1066ece-e6f3-4491-aa1f-a7b9a3aa248d-kube-rbac-proxy-cm") pod "insights-runtime-extractor-8m768" (UID: "b1066ece-e6f3-4491-aa1f-a7b9a3aa248d") : configmap "kube-rbac-proxy" not found
Oct 31 07:29:02.976649 ip-10-0-139-83 kubenswrapper[2573]: E1031 07:29:02.976429    2573 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 31 07:29:02.976649 ip-10-0-139-83 kubenswrapper[2573]: E1031 07:29:02.976483    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b1066ece-e6f3-4491-aa1f-a7b9a3aa248d-insights-runtime-extractor-tls podName:b1066ece-e6f3-4491-aa1f-a7b9a3aa248d nodeName:}" failed. No retries permitted until 2026-10-31 07:29:03.476466122 +0000 UTC m=+40.167236946 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/b1066ece-e6f3-4491-aa1f-a7b9a3aa248d-insights-runtime-extractor-tls") pod "insights-runtime-extractor-8m768" (UID: "b1066ece-e6f3-4491-aa1f-a7b9a3aa248d") : secret "insights-runtime-extractor-tls" not found
Oct 31 07:29:02.976649 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:29:02.976560    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/b1066ece-e6f3-4491-aa1f-a7b9a3aa248d-crio-socket\") pod \"insights-runtime-extractor-8m768\" (UID: \"b1066ece-e6f3-4491-aa1f-a7b9a3aa248d\") " pod="openshift-insights/insights-runtime-extractor-8m768"
Oct 31 07:29:02.976949 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:29:02.976920    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/b1066ece-e6f3-4491-aa1f-a7b9a3aa248d-data-volume\") pod \"insights-runtime-extractor-8m768\" (UID: \"b1066ece-e6f3-4491-aa1f-a7b9a3aa248d\") " pod="openshift-insights/insights-runtime-extractor-8m768"
Oct 31 07:29:02.977046 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:29:02.977005    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a73024cb-f581-450d-a09c-0565838c5baa-metrics-tls\") pod \"dns-default-9cmfx\" (UID: \"a73024cb-f581-450d-a09c-0565838c5baa\") " pod="openshift-dns/dns-default-9cmfx"
Oct 31 07:29:02.977167 ip-10-0-139-83 kubenswrapper[2573]: E1031 07:29:02.977148    2573 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 31 07:29:02.977231 ip-10-0-139-83 kubenswrapper[2573]: E1031 07:29:02.977197    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a73024cb-f581-450d-a09c-0565838c5baa-metrics-tls podName:a73024cb-f581-450d-a09c-0565838c5baa nodeName:}" failed. No retries permitted until 2026-10-31 07:29:10.977182636 +0000 UTC m=+47.667953460 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/a73024cb-f581-450d-a09c-0565838c5baa-metrics-tls") pod "dns-default-9cmfx" (UID: "a73024cb-f581-450d-a09c-0565838c5baa") : secret "dns-default-metrics-tls" not found
Oct 31 07:29:03.042312 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:29:03.035994    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cjt6r\" (UniqueName: \"kubernetes.io/projected/b1066ece-e6f3-4491-aa1f-a7b9a3aa248d-kube-api-access-cjt6r\") pod \"insights-runtime-extractor-8m768\" (UID: \"b1066ece-e6f3-4491-aa1f-a7b9a3aa248d\") " pod="openshift-insights/insights-runtime-extractor-8m768"
Oct 31 07:29:03.140607 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:29:03.140409    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 07:29:03.148525 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:29:03.148498    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-kmcg5" event={"ID":"2536c1c8-8a46-4af3-bc6d-81454a908d3f","Type":"ContainerStarted","Data":"00e80aa14dcb2e0dbc47a7da2c823abc06a0bc0bc8be397de3ec4ff40530cceb"}
Oct 31 07:29:03.154611 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:29:03.154562    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-fqhzf" event={"ID":"ce8cac52-291b-47fb-a7b6-4cf7562449b9","Type":"ContainerStarted","Data":"b921edd94725dc6d1ef6e4474e087f7b23a687d171aba11277afe09380664342"}
Oct 31 07:29:03.156865 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:29:03.156846    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-4jszp_b1db0e3b-e85d-4769-8012-cf4b6f740d9d/console-operator/1.log"
Oct 31 07:29:03.158700 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:29:03.157241    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-4jszp_b1db0e3b-e85d-4769-8012-cf4b6f740d9d/console-operator/0.log"
Oct 31 07:29:03.158700 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:29:03.157264    2573 generic.go:356] "Generic (PLEG): container finished" podID="b1db0e3b-e85d-4769-8012-cf4b6f740d9d" containerID="c92430b7e7806bf2e27a6cbf56f84a5decbb26ad6ba153e76cb4e22794254de8" exitCode=255
Oct 31 07:29:03.158700 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:29:03.157845    2573 scope.go:122] "RemoveContainer" containerID="c92430b7e7806bf2e27a6cbf56f84a5decbb26ad6ba153e76cb4e22794254de8"
Oct 31 07:29:03.158700 ip-10-0-139-83 kubenswrapper[2573]: E1031 07:29:03.157974    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-4jszp_openshift-console-operator(b1db0e3b-e85d-4769-8012-cf4b6f740d9d)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-4jszp" podUID="b1db0e3b-e85d-4769-8012-cf4b6f740d9d"
Oct 31 07:29:03.158700 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:29:03.158017    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-4jszp" event={"ID":"b1db0e3b-e85d-4769-8012-cf4b6f740d9d","Type":"ContainerDied","Data":"c92430b7e7806bf2e27a6cbf56f84a5decbb26ad6ba153e76cb4e22794254de8"}
Oct 31 07:29:03.158700 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:29:03.158045    2573 scope.go:122] "RemoveContainer" containerID="40bc998862f811cf65d3ac61aa449437d6521a1d37ba6e8f97920063c644bdbf"
Oct 31 07:29:03.262974 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:29:03.262917    2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-kmcg5" podStartSLOduration=2.692339369 podStartE2EDuration="9.262898687s" totalImagesPullingTime="6.570559318s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 07:28:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 07:29:03.26258906 +0000 UTC m=+39.953359887" watchObservedRunningTime="2026-10-31 07:29:03.262898687 +0000 UTC m=+39.953669517"
Oct 31 07:29:03.313969 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:29:03.313536    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 07:29:03.471872 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:29:03.471844    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 07:29:03.482210 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:29:03.482170    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b1066ece-e6f3-4491-aa1f-a7b9a3aa248d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-8m768\" (UID: \"b1066ece-e6f3-4491-aa1f-a7b9a3aa248d\") " pod="openshift-insights/insights-runtime-extractor-8m768"
Oct 31 07:29:03.482383 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:29:03.482251    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b1066ece-e6f3-4491-aa1f-a7b9a3aa248d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-8m768\" (UID: \"b1066ece-e6f3-4491-aa1f-a7b9a3aa248d\") " pod="openshift-insights/insights-runtime-extractor-8m768"
Oct 31 07:29:03.482488 ip-10-0-139-83 kubenswrapper[2573]: E1031 07:29:03.482460    2573 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 31 07:29:03.482555 ip-10-0-139-83 kubenswrapper[2573]: E1031 07:29:03.482522    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b1066ece-e6f3-4491-aa1f-a7b9a3aa248d-insights-runtime-extractor-tls podName:b1066ece-e6f3-4491-aa1f-a7b9a3aa248d nodeName:}" failed. No retries permitted until 2026-10-31 07:29:04.482505042 +0000 UTC m=+41.173275849 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/b1066ece-e6f3-4491-aa1f-a7b9a3aa248d-insights-runtime-extractor-tls") pod "insights-runtime-extractor-8m768" (UID: "b1066ece-e6f3-4491-aa1f-a7b9a3aa248d") : secret "insights-runtime-extractor-tls" not found
Oct 31 07:29:03.482679 ip-10-0-139-83 kubenswrapper[2573]: E1031 07:29:03.482574    2573 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Oct 31 07:29:03.482679 ip-10-0-139-83 kubenswrapper[2573]: E1031 07:29:03.482611    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b1066ece-e6f3-4491-aa1f-a7b9a3aa248d-kube-rbac-proxy-cm podName:b1066ece-e6f3-4491-aa1f-a7b9a3aa248d nodeName:}" failed. No retries permitted until 2026-10-31 07:29:04.482599908 +0000 UTC m=+41.173370727 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/b1066ece-e6f3-4491-aa1f-a7b9a3aa248d-kube-rbac-proxy-cm") pod "insights-runtime-extractor-8m768" (UID: "b1066ece-e6f3-4491-aa1f-a7b9a3aa248d") : configmap "kube-rbac-proxy" not found
Oct 31 07:29:03.491984 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:29:03.491936    2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-fqhzf" podStartSLOduration=380.355674697 podStartE2EDuration="6m27.491924333s" totalImagesPullingTime="7.136249636s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 07:22:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 07:29:03.376988374 +0000 UTC m=+40.067759198" watchObservedRunningTime="2026-10-31 07:29:03.491924333 +0000 UTC m=+40.182695158"
Oct 31 07:29:03.637242 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:29:03.637202    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 07:29:03.770089 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:29:03.770060    2573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-5t4pm"]
Oct 31 07:29:03.784223 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:29:03.784189    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-5t4pm"
Oct 31 07:29:03.794368 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:29:03.794344    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 07:29:03.828679 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:29:03.828649    2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-5t4pm"]
Oct 31 07:29:03.837017 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:29:03.836992    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Oct 31 07:29:03.837214 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:29:03.837104    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-6g5bx\""
Oct 31 07:29:03.837214 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:29:03.837122    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Oct 31 07:29:03.886871 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:29:03.886832    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nhphg\" (UniqueName: \"kubernetes.io/projected/0b25f791-3520-481d-acbb-a3bef222fdac-kube-api-access-nhphg\") pod \"migrator-66df5d496b-5t4pm\" (UID: \"0b25f791-3520-481d-acbb-a3bef222fdac\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-5t4pm"
Oct 31 07:29:03.946410 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:29:03.946381    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 07:29:03.988125 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:29:03.988093    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nhphg\" (UniqueName: \"kubernetes.io/projected/0b25f791-3520-481d-acbb-a3bef222fdac-kube-api-access-nhphg\") pod \"migrator-66df5d496b-5t4pm\" (UID: \"0b25f791-3520-481d-acbb-a3bef222fdac\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-5t4pm"
Oct 31 07:29:04.046258 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:29:04.046231    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nhphg\" (UniqueName: \"kubernetes.io/projected/0b25f791-3520-481d-acbb-a3bef222fdac-kube-api-access-nhphg\") pod \"migrator-66df5d496b-5t4pm\" (UID: \"0b25f791-3520-481d-acbb-a3bef222fdac\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-5t4pm"
Oct 31 07:29:04.089377 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:29:04.089320    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 07:29:04.111386 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:29:04.111362    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-5t4pm"
Oct 31 07:29:04.164743 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:29:04.163881    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-9zp4l" event={"ID":"f1ba68f6-d87c-4492-a5eb-5f2d3a60eb1c","Type":"ContainerStarted","Data":"5ae0646a2a77e63099fac42a9911e2488a7d85f9bee5b6009f6e7fd9851b30b9"}
Oct 31 07:29:04.164743 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:29:04.164695    2573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-9zp4l"
Oct 31 07:29:04.166726 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:29:04.166709    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-4jszp_b1db0e3b-e85d-4769-8012-cf4b6f740d9d/console-operator/1.log"
Oct 31 07:29:04.167152 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:29:04.167132    2573 scope.go:122] "RemoveContainer" containerID="c92430b7e7806bf2e27a6cbf56f84a5decbb26ad6ba153e76cb4e22794254de8"
Oct 31 07:29:04.167380 ip-10-0-139-83 kubenswrapper[2573]: E1031 07:29:04.167318    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-4jszp_openshift-console-operator(b1db0e3b-e85d-4769-8012-cf4b6f740d9d)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-4jszp" podUID="b1db0e3b-e85d-4769-8012-cf4b6f740d9d"
Oct 31 07:29:04.279120 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:29:04.278908    2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-9zp4l" podStartSLOduration=32.907225752 podStartE2EDuration="39.278893107s" totalImagesPullingTime="6.371667355s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 07:28:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 07:29:04.276882671 +0000 UTC m=+40.967653496" watchObservedRunningTime="2026-10-31 07:29:04.278893107 +0000 UTC m=+40.969663932"
Oct 31 07:29:04.493121 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:29:04.493013    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b1066ece-e6f3-4491-aa1f-a7b9a3aa248d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-8m768\" (UID: \"b1066ece-e6f3-4491-aa1f-a7b9a3aa248d\") " pod="openshift-insights/insights-runtime-extractor-8m768"
Oct 31 07:29:04.493121 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:29:04.493090    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b1066ece-e6f3-4491-aa1f-a7b9a3aa248d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-8m768\" (UID: \"b1066ece-e6f3-4491-aa1f-a7b9a3aa248d\") " pod="openshift-insights/insights-runtime-extractor-8m768"
Oct 31 07:29:04.493364 ip-10-0-139-83 kubenswrapper[2573]: E1031 07:29:04.493223    2573 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 31 07:29:04.493364 ip-10-0-139-83 kubenswrapper[2573]: E1031 07:29:04.493284    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b1066ece-e6f3-4491-aa1f-a7b9a3aa248d-insights-runtime-extractor-tls podName:b1066ece-e6f3-4491-aa1f-a7b9a3aa248d nodeName:}" failed. No retries permitted until 2026-10-31 07:29:06.493267244 +0000 UTC m=+43.184038053 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/b1066ece-e6f3-4491-aa1f-a7b9a3aa248d-insights-runtime-extractor-tls") pod "insights-runtime-extractor-8m768" (UID: "b1066ece-e6f3-4491-aa1f-a7b9a3aa248d") : secret "insights-runtime-extractor-tls" not found
Oct 31 07:29:04.493672 ip-10-0-139-83 kubenswrapper[2573]: E1031 07:29:04.493651    2573 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Oct 31 07:29:04.493760 ip-10-0-139-83 kubenswrapper[2573]: E1031 07:29:04.493707    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b1066ece-e6f3-4491-aa1f-a7b9a3aa248d-kube-rbac-proxy-cm podName:b1066ece-e6f3-4491-aa1f-a7b9a3aa248d nodeName:}" failed. No retries permitted until 2026-10-31 07:29:06.493692468 +0000 UTC m=+43.184463273 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/b1066ece-e6f3-4491-aa1f-a7b9a3aa248d-kube-rbac-proxy-cm") pod "insights-runtime-extractor-8m768" (UID: "b1066ece-e6f3-4491-aa1f-a7b9a3aa248d") : configmap "kube-rbac-proxy" not found
Oct 31 07:29:04.510643 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:29:04.510619    2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-5t4pm"]
Oct 31 07:29:04.662157 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:29:04.662118    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 07:29:04.822883 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:29:04.822793    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 07:29:04.985821 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:29:04.985787    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 07:29:05.171257 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:29:05.171224    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-5t4pm" event={"ID":"0b25f791-3520-481d-acbb-a3bef222fdac","Type":"ContainerStarted","Data":"57c7cad37211feee315413dd85f61718ce6291d6c4be0727f145731331c93e71"}
Oct 31 07:29:05.271574 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:29:05.271544    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 07:29:05.437279 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:29:05.437254    2573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-4jszp"
Oct 31 07:29:05.437437 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:29:05.437346    2573 kubelet.go:2727] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-7cd56b8dbc-4jszp"
Oct 31 07:29:05.437708 ip-10-0-139-83 kubenswrapper[2573]: I1031 07:29:05.437695    2573 scope.go:122] "RemoveContainer" containerID="c92430b7e7806bf2e27a6cbf56f84a5decbb26ad6ba153e76cb4e22794254de8"
Oct 31 07:29:05.437886 ip-10-0-139-83 kubenswrapper[2573]: E1031 07:29:05.437867    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-4jszp_openshift-console-operator(b1db0e3b-e85d-4769-8012-cf4b6f740d9d)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-4jszp" podUID="b1db0e3b-e85d-4769-8012-cf4b6f740d9d"
Oct 31 07:29:05.784312 ip-10-0-139-83 systemd[1]: Stopping Kubernetes Kubelet...
Oct 31 07:29:05.797749 ip-10-0-139-83 systemd[1]: kubelet.service: Deactivated successfully.
Oct 31 07:29:05.797920 ip-10-0-139-83 systemd[1]: Stopped Kubernetes Kubelet.
Oct 31 07:29:05.798082 ip-10-0-139-83 systemd[1]: kubelet.service: Consumed 2.940s CPU time.
Oct 31 07:29:05.807973 ip-10-0-139-83 systemd[1]: Starting Kubernetes Kubelet...
Oct 31 07:29:05.912219 ip-10-0-139-83 kubenswrapper[4720]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 31 07:29:05.912219 ip-10-0-139-83 kubenswrapper[4720]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Oct 31 07:29:05.912219 ip-10-0-139-83 kubenswrapper[4720]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 31 07:29:05.912219 ip-10-0-139-83 kubenswrapper[4720]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 31 07:29:05.915367 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.915262    4720 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 31 07:29:05.915367 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.915357    4720 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 31 07:29:05.915367 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.915361    4720 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 31 07:29:05.915367 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.915367    4720 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 31 07:29:05.915367 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.915371    4720 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 31 07:29:05.915367 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.915374    4720 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 31 07:29:05.915367 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.915377    4720 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 31 07:29:05.915716 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.915380    4720 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 31 07:29:05.915716 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.915383    4720 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 31 07:29:05.915716 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.915385    4720 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 31 07:29:05.915716 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.915388    4720 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 31 07:29:05.915716 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.915390    4720 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 31 07:29:05.915716 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.915393    4720 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 31 07:29:05.915716 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.915395    4720 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 31 07:29:05.915716 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.915398    4720 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 31 07:29:05.915716 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.915400    4720 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 31 07:29:05.915716 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.915403    4720 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 31 07:29:05.915716 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.915405    4720 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 31 07:29:05.915716 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.915407    4720 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 31 07:29:05.915716 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.915410    4720 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 31 07:29:05.915716 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.915413    4720 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 31 07:29:05.915716 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.915415    4720 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 31 07:29:05.915716 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.915418    4720 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 31 07:29:05.915716 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.915420    4720 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 31 07:29:05.915716 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.915423    4720 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 31 07:29:05.915716 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.915425    4720 feature_gate.go:386] unrecognized feature gate: Example2
Oct 31 07:29:05.915716 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.915428    4720 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 31 07:29:05.916421 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.915430    4720 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 31 07:29:05.916421 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.915442    4720 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 31 07:29:05.916421 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.915445    4720 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 31 07:29:05.916421 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.915448    4720 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 31 07:29:05.916421 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.915450    4720 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 31 07:29:05.916421 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.915453    4720 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 31 07:29:05.916421 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.915458    4720 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 31 07:29:05.916421 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.915462    4720 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 31 07:29:05.916421 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.915468    4720 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 31 07:29:05.916421 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.915472    4720 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 31 07:29:05.916421 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.915476    4720 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 31 07:29:05.916421 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.915481    4720 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 31 07:29:05.916421 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.915486    4720 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 31 07:29:05.916421 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.915489    4720 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 31 07:29:05.916421 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.915493    4720 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 31 07:29:05.916421 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.915496    4720 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 31 07:29:05.916421 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.915499    4720 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 31 07:29:05.916421 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.915502    4720 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 31 07:29:05.916421 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.915505    4720 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 31 07:29:05.916421 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.915508    4720 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 31 07:29:05.917055 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.915510    4720 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 31 07:29:05.917055 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.915514    4720 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 31 07:29:05.917055 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.915519    4720 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 31 07:29:05.917055 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.915523    4720 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 31 07:29:05.917055 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.915528    4720 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 31 07:29:05.917055 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.915531    4720 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 31 07:29:05.917055 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.915534    4720 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 31 07:29:05.917055 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.915536    4720 feature_gate.go:386] unrecognized feature gate: Example
Oct 31 07:29:05.917055 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.915539    4720 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 31 07:29:05.917055 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.915541    4720 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 31 07:29:05.917055 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.915544    4720 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 31 07:29:05.917055 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.915547    4720 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 31 07:29:05.917055 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.915549    4720 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 31 07:29:05.917055 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.915552    4720 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 31 07:29:05.917055 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.915554    4720 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 31 07:29:05.917055 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.915558    4720 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 31 07:29:05.917055 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.915561    4720 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 31 07:29:05.917055 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.915563    4720 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 31 07:29:05.917055 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.915566    4720 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 31 07:29:05.917706 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.915570    4720 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 31 07:29:05.917706 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.915575    4720 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 31 07:29:05.917706 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.915579    4720 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 31 07:29:05.917706 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.915583    4720 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 31 07:29:05.917706 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.915585    4720 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 31 07:29:05.917706 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.915588    4720 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 31 07:29:05.917706 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.915591    4720 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 31 07:29:05.917706 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.915593    4720 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 31 07:29:05.917706 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.915596    4720 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 31 07:29:05.917706 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.915601    4720 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 31 07:29:05.917706 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.915605    4720 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 31 07:29:05.917706 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.915610    4720 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 31 07:29:05.917706 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.915614    4720 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 31 07:29:05.917706 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.915618    4720 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 31 07:29:05.917706 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.915623    4720 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 31 07:29:05.917706 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.915627    4720 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 31 07:29:05.917706 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.915631    4720 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 31 07:29:05.917706 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.915639    4720 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 31 07:29:05.917706 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.915645    4720 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 31 07:29:05.919227 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.915650    4720 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 31 07:29:05.919227 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.915655    4720 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 31 07:29:05.919227 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.915660    4720 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 31 07:29:05.919227 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.915665    4720 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 31 07:29:05.919227 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.915669    4720 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 31 07:29:05.919227 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.915672    4720 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 31 07:29:05.919227 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.915674    4720 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 31 07:29:05.919227 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.915677    4720 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 31 07:29:05.919227 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.915680    4720 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 31 07:29:05.919227 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.915682    4720 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 31 07:29:05.919227 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.915684    4720 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 31 07:29:05.919227 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.915687    4720 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 31 07:29:05.919227 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.915690    4720 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 31 07:29:05.919227 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.915692    4720 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 31 07:29:05.919227 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.915695    4720 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 31 07:29:05.919227 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.915697    4720 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 31 07:29:05.919227 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.915700    4720 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 31 07:29:05.919227 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.915702    4720 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 31 07:29:05.919227 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.915704    4720 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 31 07:29:05.919799 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.915708    4720 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 31 07:29:05.919799 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.915711    4720 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 31 07:29:05.919799 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.916704    4720 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 31 07:29:05.919799 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.916713    4720 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 31 07:29:05.919799 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.916716    4720 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 31 07:29:05.919799 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.916718    4720 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 31 07:29:05.919799 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.916721    4720 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 31 07:29:05.919799 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.916724    4720 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 31 07:29:05.919799 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.916728    4720 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 31 07:29:05.919799 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.916733    4720 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 31 07:29:05.919799 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.916737    4720 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 31 07:29:05.919799 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.916741    4720 feature_gate.go:386] unrecognized feature gate: Example2
Oct 31 07:29:05.919799 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.916745    4720 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 31 07:29:05.919799 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.916749    4720 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 31 07:29:05.919799 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.916753    4720 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 31 07:29:05.919799 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.916758    4720 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 31 07:29:05.919799 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.916762    4720 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 31 07:29:05.919799 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.916765    4720 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 31 07:29:05.919799 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.916768    4720 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 31 07:29:05.920365 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.916770    4720 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 31 07:29:05.920365 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.916773    4720 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 31 07:29:05.920365 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.916776    4720 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 31 07:29:05.920365 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.916778    4720 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 31 07:29:05.920365 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.916781    4720 feature_gate.go:386] unrecognized feature gate: Example
Oct 31 07:29:05.920365 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.916784    4720 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 31 07:29:05.920365 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.916786    4720 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 31 07:29:05.920365 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.916789    4720 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 31 07:29:05.920365 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.916791    4720 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 31 07:29:05.920365 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.916795    4720 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 31 07:29:05.920365 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.916800    4720 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 31 07:29:05.920365 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.916803    4720 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 31 07:29:05.920365 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.916806    4720 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 31 07:29:05.920365 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.916808    4720 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 31 07:29:05.920365 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.916811    4720 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 31 07:29:05.920365 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.916813    4720 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 31 07:29:05.920365 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.916817    4720 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 31 07:29:05.920365 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.916819    4720 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 31 07:29:05.920365 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.916822    4720 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 31 07:29:05.921010 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.916825    4720 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 31 07:29:05.921010 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.916827    4720 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 31 07:29:05.921010 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.916830    4720 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 31 07:29:05.921010 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.916832    4720 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 31 07:29:05.921010 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.916835    4720 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 31 07:29:05.921010 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.916838    4720 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 31 07:29:05.921010 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.916841    4720 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 31 07:29:05.921010 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.916843    4720 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 31 07:29:05.921010 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.916847    4720 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 31 07:29:05.921010 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.916851    4720 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 31 07:29:05.921010 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.916857    4720 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 31 07:29:05.921010 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.916863    4720 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 31 07:29:05.921010 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.916866    4720 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 31 07:29:05.921010 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.916871    4720 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 31 07:29:05.921010 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.916875    4720 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 31 07:29:05.921010 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.916879    4720 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 31 07:29:05.921010 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.916883    4720 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 31 07:29:05.921010 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.916886    4720 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 31 07:29:05.921010 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.916888    4720 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 31 07:29:05.921654 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.916892    4720 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 31 07:29:05.921654 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.916894    4720 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 31 07:29:05.921654 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.916897    4720 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 31 07:29:05.921654 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.916899    4720 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 31 07:29:05.921654 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.916902    4720 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 31 07:29:05.921654 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.916904    4720 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 31 07:29:05.921654 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.916907    4720 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 31 07:29:05.921654 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.916909    4720 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 31 07:29:05.921654 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.916912    4720 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 31 07:29:05.921654 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.916915    4720 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 31 07:29:05.921654 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.916917    4720 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 31 07:29:05.921654 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.916920    4720 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 31 07:29:05.921654 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.916922    4720 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 31 07:29:05.921654 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.916927    4720 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 31 07:29:05.921654 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.916931    4720 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 31 07:29:05.921654 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.916936    4720 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 31 07:29:05.921654 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.916940    4720 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 31 07:29:05.921654 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.916943    4720 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 31 07:29:05.921654 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.916947    4720 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 31 07:29:05.921654 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.916949    4720 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 31 07:29:05.922350 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.916952    4720 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 31 07:29:05.922350 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.916955    4720 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 31 07:29:05.922350 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.916957    4720 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 31 07:29:05.922350 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.916960    4720 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 31 07:29:05.922350 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.916963    4720 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 31 07:29:05.922350 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.916966    4720 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 31 07:29:05.922350 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.916968    4720 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 31 07:29:05.922350 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.916971    4720 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 31 07:29:05.922350 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.916974    4720 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 31 07:29:05.922350 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.916976    4720 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 31 07:29:05.922350 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.916979    4720 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 31 07:29:05.922350 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.916981    4720 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 31 07:29:05.922350 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.916984    4720 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 31 07:29:05.922350 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.916986    4720 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 31 07:29:05.922350 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.916989    4720 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 31 07:29:05.922350 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.916991    4720 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 31 07:29:05.922350 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.916994    4720 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 31 07:29:05.922350 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.916996    4720 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 31 07:29:05.922350 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.917000    4720 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 31 07:29:05.922350 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.917004    4720 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 31 07:29:05.922998 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.917008    4720 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 31 07:29:05.922998 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.917013    4720 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 31 07:29:05.922998 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.917018    4720 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 31 07:29:05.922998 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.917023    4720 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 31 07:29:05.922998 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.917027    4720 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 31 07:29:05.922998 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.917031    4720 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 31 07:29:05.922998 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.917036    4720 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 31 07:29:05.922998 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.917041    4720 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 31 07:29:05.922998 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.917045    4720 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 31 07:29:05.922998 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.917049    4720 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 31 07:29:05.922998 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.917053    4720 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 31 07:29:05.922998 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.917653    4720 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 31 07:29:05.922998 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.917660    4720 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 31 07:29:05.922998 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.917663    4720 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 31 07:29:05.922998 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.917666    4720 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 31 07:29:05.922998 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.917669    4720 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 31 07:29:05.922998 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.917672    4720 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 31 07:29:05.922998 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.917674    4720 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 31 07:29:05.922998 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.917678    4720 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 31 07:29:05.923632 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.917682    4720 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 31 07:29:05.923632 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.917687    4720 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 31 07:29:05.923632 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.917691    4720 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 31 07:29:05.923632 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.917695    4720 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 31 07:29:05.923632 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.917698    4720 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 31 07:29:05.923632 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.917702    4720 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 31 07:29:05.923632 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.917706    4720 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 31 07:29:05.923632 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.917711    4720 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 31 07:29:05.923632 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.917716    4720 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 31 07:29:05.923632 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.917720    4720 feature_gate.go:386] unrecognized feature gate: Example2
Oct 31 07:29:05.923632 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.917723    4720 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 31 07:29:05.923632 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.917726    4720 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 31 07:29:05.923632 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.917729    4720 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 31 07:29:05.923632 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.917734    4720 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 31 07:29:05.923632 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.917740    4720 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 31 07:29:05.923632 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.917747    4720 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 31 07:29:05.923632 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.917752    4720 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 31 07:29:05.923632 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.917756    4720 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 31 07:29:05.924150 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.917761    4720 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 31 07:29:05.924150 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.917765    4720 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 31 07:29:05.924150 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.917769    4720 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 31 07:29:05.924150 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.917774    4720 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 31 07:29:05.924150 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.917779    4720 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 31 07:29:05.924150 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.917783    4720 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 31 07:29:05.924150 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.917789    4720 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 31 07:29:05.924150 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.917793    4720 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 31 07:29:05.924150 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.917797    4720 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 31 07:29:05.924150 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.917801    4720 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 31 07:29:05.924150 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.917806    4720 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 31 07:29:05.924150 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.917810    4720 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 31 07:29:05.924150 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.917814    4720 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 31 07:29:05.924150 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.917819    4720 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 31 07:29:05.924150 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.917823    4720 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 31 07:29:05.924150 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.917827    4720 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 31 07:29:05.924150 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.917831    4720 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 31 07:29:05.924150 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.917835    4720 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 31 07:29:05.924150 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.917839    4720 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 31 07:29:05.924150 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.917842    4720 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 31 07:29:05.924809 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.917844    4720 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 31 07:29:05.924809 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.917847    4720 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 31 07:29:05.924809 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.917850    4720 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 31 07:29:05.924809 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.917852    4720 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 31 07:29:05.924809 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.917855    4720 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 31 07:29:05.924809 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.917858    4720 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 31 07:29:05.924809 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.917861    4720 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 31 07:29:05.924809 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.917863    4720 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 31 07:29:05.924809 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.917866    4720 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 31 07:29:05.924809 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.917868    4720 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 31 07:29:05.924809 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.917870    4720 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 31 07:29:05.924809 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.917873    4720 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 31 07:29:05.924809 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.917875    4720 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 31 07:29:05.924809 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.917879    4720 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 31 07:29:05.924809 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.917884    4720 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 31 07:29:05.924809 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.917888    4720 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 31 07:29:05.924809 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.917896    4720 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 31 07:29:05.924809 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.917901    4720 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 31 07:29:05.924809 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.917906    4720 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 31 07:29:05.924809 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.917912    4720 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 31 07:29:05.925450 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.917917    4720 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 31 07:29:05.925450 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.917921    4720 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 31 07:29:05.925450 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.917926    4720 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 31 07:29:05.925450 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.917930    4720 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 31 07:29:05.925450 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.917934    4720 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 31 07:29:05.925450 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.917938    4720 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 31 07:29:05.925450 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.917943    4720 feature_gate.go:386] unrecognized feature gate: Example
Oct 31 07:29:05.925450 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.917947    4720 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 31 07:29:05.925450 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.917952    4720 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 31 07:29:05.925450 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.917957    4720 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 31 07:29:05.925450 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.917961    4720 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 31 07:29:05.925450 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.917965    4720 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 31 07:29:05.925450 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.917969    4720 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 31 07:29:05.925450 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.917972    4720 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 31 07:29:05.925450 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.917975    4720 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 31 07:29:05.925450 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.917978    4720 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 31 07:29:05.925450 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.917980    4720 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 31 07:29:05.925450 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.917983    4720 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 31 07:29:05.925450 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.917985    4720 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 31 07:29:05.925450 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.917988    4720 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 31 07:29:05.925999 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.917990    4720 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 31 07:29:05.925999 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.917992    4720 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 31 07:29:05.925999 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.917995    4720 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 31 07:29:05.925999 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.917998    4720 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 31 07:29:05.925999 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.918001    4720 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 31 07:29:05.925999 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.918003    4720 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 31 07:29:05.925999 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.918006    4720 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 31 07:29:05.925999 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.918008    4720 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 31 07:29:05.925999 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.918012    4720 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 31 07:29:05.925999 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.918015    4720 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 31 07:29:05.925999 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.918018    4720 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 31 07:29:05.925999 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.918020    4720 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 31 07:29:05.925999 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.918023    4720 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 31 07:29:05.925999 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.918026    4720 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 31 07:29:05.925999 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.918029    4720 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 31 07:29:05.925999 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.918031    4720 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 31 07:29:05.925999 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.918035    4720 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 31 07:29:05.925999 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.918037    4720 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 31 07:29:05.925999 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.918040    4720 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 31 07:29:05.926538 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.918042    4720 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 31 07:29:05.926538 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:05.918163    4720 flags.go:64] FLAG: --address="0.0.0.0"
Oct 31 07:29:05.926538 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:05.918171    4720 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Oct 31 07:29:05.926538 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:05.918179    4720 flags.go:64] FLAG: --anonymous-auth="true"
Oct 31 07:29:05.926538 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:05.918183    4720 flags.go:64] FLAG: --application-metrics-count-limit="100"
Oct 31 07:29:05.926538 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:05.918188    4720 flags.go:64] FLAG: --authentication-token-webhook="false"
Oct 31 07:29:05.926538 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:05.918191    4720 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Oct 31 07:29:05.926538 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:05.918196    4720 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Oct 31 07:29:05.926538 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:05.918200    4720 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Oct 31 07:29:05.926538 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:05.918203    4720 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Oct 31 07:29:05.926538 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:05.918206    4720 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Oct 31 07:29:05.926538 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:05.918210    4720 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Oct 31 07:29:05.926538 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:05.918213    4720 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Oct 31 07:29:05.926538 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:05.918216    4720 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Oct 31 07:29:05.926538 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:05.918220    4720 flags.go:64] FLAG: --cgroup-root=""
Oct 31 07:29:05.926538 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:05.918228    4720 flags.go:64] FLAG: --cgroups-per-qos="true"
Oct 31 07:29:05.926538 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:05.918233    4720 flags.go:64] FLAG: --client-ca-file=""
Oct 31 07:29:05.926538 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:05.918238    4720 flags.go:64] FLAG: --cloud-provider="external"
Oct 31 07:29:05.926538 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:05.918241    4720 flags.go:64] FLAG: --cluster-dns="[]"
Oct 31 07:29:05.926538 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:05.918245    4720 flags.go:64] FLAG: --cluster-domain=""
Oct 31 07:29:05.926538 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:05.918248    4720 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Oct 31 07:29:05.926538 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:05.918252    4720 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Oct 31 07:29:05.926538 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:05.918255    4720 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Oct 31 07:29:05.926538 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:05.918258    4720 flags.go:64] FLAG: --container-log-max-files="5"
Oct 31 07:29:05.927248 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:05.918262    4720 flags.go:64] FLAG: --container-log-max-size="10Mi"
Oct 31 07:29:05.927248 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:05.918265    4720 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Oct 31 07:29:05.927248 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:05.918268    4720 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Oct 31 07:29:05.927248 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:05.918271    4720 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Oct 31 07:29:05.927248 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:05.918274    4720 flags.go:64] FLAG: --contention-profiling="false"
Oct 31 07:29:05.927248 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:05.918277    4720 flags.go:64] FLAG: --cpu-cfs-quota="true"
Oct 31 07:29:05.927248 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:05.918280    4720 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Oct 31 07:29:05.927248 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:05.918283    4720 flags.go:64] FLAG: --cpu-manager-policy="none"
Oct 31 07:29:05.927248 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:05.918288    4720 flags.go:64] FLAG: --cpu-manager-policy-options=""
Oct 31 07:29:05.927248 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:05.918295    4720 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Oct 31 07:29:05.927248 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:05.918299    4720 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Oct 31 07:29:05.927248 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:05.918302    4720 flags.go:64] FLAG: --enable-debugging-handlers="true"
Oct 31 07:29:05.927248 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:05.918305    4720 flags.go:64] FLAG: --enable-load-reader="false"
Oct 31 07:29:05.927248 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:05.918308    4720 flags.go:64] FLAG: --enable-server="true"
Oct 31 07:29:05.927248 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:05.918311    4720 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Oct 31 07:29:05.927248 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:05.918315    4720 flags.go:64] FLAG: --event-burst="100"
Oct 31 07:29:05.927248 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:05.918318    4720 flags.go:64] FLAG: --event-qps="50"
Oct 31 07:29:05.927248 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:05.918321    4720 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Oct 31 07:29:05.927248 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:05.918341    4720 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Oct 31 07:29:05.927248 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:05.918345    4720 flags.go:64] FLAG: --eviction-hard=""
Oct 31 07:29:05.927248 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:05.918350    4720 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Oct 31 07:29:05.927248 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:05.918352    4720 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Oct 31 07:29:05.927248 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:05.918355    4720 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Oct 31 07:29:05.927248 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:05.918361    4720 flags.go:64] FLAG: --eviction-soft=""
Oct 31 07:29:05.927248 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:05.918366    4720 flags.go:64] FLAG: --eviction-soft-grace-period=""
Oct 31 07:29:05.928138 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:05.918371    4720 flags.go:64] FLAG: --exit-on-lock-contention="false"
Oct 31 07:29:05.928138 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:05.918376    4720 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Oct 31 07:29:05.928138 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:05.918381    4720 flags.go:64] FLAG: --experimental-mounter-path=""
Oct 31 07:29:05.928138 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:05.918384    4720 flags.go:64] FLAG: --fail-cgroupv1="true"
Oct 31 07:29:05.928138 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:05.918388    4720 flags.go:64] FLAG: --fail-swap-on="true"
Oct 31 07:29:05.928138 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:05.918391    4720 flags.go:64] FLAG: --feature-gates=""
Oct 31 07:29:05.928138 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:05.918395    4720 flags.go:64] FLAG: --file-check-frequency="20s"
Oct 31 07:29:05.928138 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:05.918398    4720 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Oct 31 07:29:05.928138 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:05.918401    4720 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Oct 31 07:29:05.928138 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:05.918404    4720 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Oct 31 07:29:05.928138 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:05.918408    4720 flags.go:64] FLAG: --healthz-port="10248"
Oct 31 07:29:05.928138 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:05.918411    4720 flags.go:64] FLAG: --help="false"
Oct 31 07:29:05.928138 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:05.918414    4720 flags.go:64] FLAG: --hostname-override="ip-10-0-139-83.us-west-1.compute.internal"
Oct 31 07:29:05.928138 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:05.918419    4720 flags.go:64] FLAG: --housekeeping-interval="10s"
Oct 31 07:29:05.928138 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:05.918424    4720 flags.go:64] FLAG: --http-check-frequency="20s"
Oct 31 07:29:05.928138 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:05.918429    4720 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Oct 31 07:29:05.928138 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:05.918432    4720 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Oct 31 07:29:05.928138 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:05.918435    4720 flags.go:64] FLAG: --image-gc-high-threshold="85"
Oct 31 07:29:05.928138 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:05.918438    4720 flags.go:64] FLAG: --image-gc-low-threshold="80"
Oct 31 07:29:05.928138 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:05.918442    4720 flags.go:64] FLAG: --image-service-endpoint=""
Oct 31 07:29:05.928138 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:05.918446    4720 flags.go:64] FLAG: --kernel-memcg-notification="false"
Oct 31 07:29:05.928138 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:05.918451    4720 flags.go:64] FLAG: --kube-api-burst="100"
Oct 31 07:29:05.928138 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:05.918456    4720 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Oct 31 07:29:05.928965 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:05.918461    4720 flags.go:64] FLAG: --kube-api-qps="50"
Oct 31 07:29:05.928965 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:05.918466    4720 flags.go:64] FLAG: --kube-reserved=""
Oct 31 07:29:05.928965 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:05.918469    4720 flags.go:64] FLAG: --kube-reserved-cgroup=""
Oct 31 07:29:05.928965 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:05.918472    4720 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Oct 31 07:29:05.928965 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:05.918475    4720 flags.go:64] FLAG: --kubelet-cgroups=""
Oct 31 07:29:05.928965 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:05.918477    4720 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Oct 31 07:29:05.928965 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:05.918480    4720 flags.go:64] FLAG: --lock-file=""
Oct 31 07:29:05.928965 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:05.918485    4720 flags.go:64] FLAG: --log-cadvisor-usage="false"
Oct 31 07:29:05.928965 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:05.918488    4720 flags.go:64] FLAG: --log-flush-frequency="5s"
Oct 31 07:29:05.928965 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:05.918491    4720 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Oct 31 07:29:05.928965 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:05.918496    4720 flags.go:64] FLAG: --log-json-split-stream="false"
Oct 31 07:29:05.928965 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:05.918501    4720 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Oct 31 07:29:05.928965 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:05.918506    4720 flags.go:64] FLAG: --log-text-split-stream="false"
Oct 31 07:29:05.928965 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:05.918512    4720 flags.go:64] FLAG: --logging-format="text"
Oct 31 07:29:05.928965 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:05.918515    4720 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Oct 31 07:29:05.928965 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:05.918518    4720 flags.go:64] FLAG: --make-iptables-util-chains="true"
Oct 31 07:29:05.928965 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:05.918521    4720 flags.go:64] FLAG: --manifest-url=""
Oct 31 07:29:05.928965 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:05.918524    4720 flags.go:64] FLAG: --manifest-url-header=""
Oct 31 07:29:05.928965 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:05.918528    4720 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Oct 31 07:29:05.928965 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:05.918531    4720 flags.go:64] FLAG: --max-open-files="1000000"
Oct 31 07:29:05.928965 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:05.918539    4720 flags.go:64] FLAG: --max-pods="110"
Oct 31 07:29:05.928965 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:05.918542    4720 flags.go:64] FLAG: --maximum-dead-containers="-1"
Oct 31 07:29:05.928965 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:05.918546    4720 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Oct 31 07:29:05.928965 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:05.918548    4720 flags.go:64] FLAG: --memory-manager-policy="None"
Oct 31 07:29:05.928965 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:05.918551    4720 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Oct 31 07:29:05.929915 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:05.918554    4720 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Oct 31 07:29:05.929915 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:05.918557    4720 flags.go:64] FLAG: --node-ip="0.0.0.0"
Oct 31 07:29:05.929915 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:05.918560    4720 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Oct 31 07:29:05.929915 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:05.918567    4720 flags.go:64] FLAG: --node-status-max-images="50"
Oct 31 07:29:05.929915 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:05.918570    4720 flags.go:64] FLAG: --node-status-update-frequency="10s"
Oct 31 07:29:05.929915 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:05.918573    4720 flags.go:64] FLAG: --oom-score-adj="-999"
Oct 31 07:29:05.929915 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:05.918577    4720 flags.go:64] FLAG: --pod-cidr=""
Oct 31 07:29:05.929915 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:05.918582    4720 flags.go:64] FLAG: --pod-manifest-path=""
Oct 31 07:29:05.929915 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:05.918587    4720 flags.go:64] FLAG: --pod-max-pids="-1"
Oct 31 07:29:05.929915 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:05.918592    4720 flags.go:64] FLAG: --pods-per-core="0"
Oct 31 07:29:05.929915 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:05.918596    4720 flags.go:64] FLAG: --port="10250"
Oct 31 07:29:05.929915 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:05.918599    4720 flags.go:64] FLAG: --protect-kernel-defaults="false"
Oct 31 07:29:05.929915 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:05.918602    4720 flags.go:64] FLAG: --provider-id="aws:///us-west-1a/i-0c45e2b77a29ed30d"
Oct 31 07:29:05.929915 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:05.918605    4720 flags.go:64] FLAG: --qos-reserved=""
Oct 31 07:29:05.929915 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:05.918610    4720 flags.go:64] FLAG: --read-only-port="10255"
Oct 31 07:29:05.929915 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:05.918613    4720 flags.go:64] FLAG: --register-node="true"
Oct 31 07:29:05.929915 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:05.918616    4720 flags.go:64] FLAG: --register-with-taints=""
Oct 31 07:29:05.929915 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:05.918619    4720 flags.go:64] FLAG: --registry-burst="10"
Oct 31 07:29:05.929915 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:05.918622    4720 flags.go:64] FLAG: --registry-qps="5"
Oct 31 07:29:05.929915 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:05.918626    4720 flags.go:64] FLAG: --reserved-cpus=""
Oct 31 07:29:05.929915 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:05.918630    4720 flags.go:64] FLAG: --reserved-memory=""
Oct 31 07:29:05.929915 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:05.918634    4720 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Oct 31 07:29:05.929915 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:05.918637    4720 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Oct 31 07:29:05.929915 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:05.918639    4720 flags.go:64] FLAG: --rotate-certificates="false"
Oct 31 07:29:05.929915 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:05.918642    4720 flags.go:64] FLAG: --rotate-server-certificates="false"
Oct 31 07:29:05.929915 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:05.918645    4720 flags.go:64] FLAG: --runonce="false"
Oct 31 07:29:05.930922 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:05.918648    4720 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Oct 31 07:29:05.930922 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:05.918651    4720 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Oct 31 07:29:05.930922 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:05.918654    4720 flags.go:64] FLAG: --seccomp-default="false"
Oct 31 07:29:05.930922 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:05.918659    4720 flags.go:64] FLAG: --serialize-image-pulls="true"
Oct 31 07:29:05.930922 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:05.918663    4720 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Oct 31 07:29:05.930922 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:05.918668    4720 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Oct 31 07:29:05.930922 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:05.918673    4720 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Oct 31 07:29:05.930922 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:05.918676    4720 flags.go:64] FLAG: --storage-driver-password="root"
Oct 31 07:29:05.930922 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:05.918679    4720 flags.go:64] FLAG: --storage-driver-secure="false"
Oct 31 07:29:05.930922 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:05.918682    4720 flags.go:64] FLAG: --storage-driver-table="stats"
Oct 31 07:29:05.930922 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:05.918684    4720 flags.go:64] FLAG: --storage-driver-user="root"
Oct 31 07:29:05.930922 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:05.918687    4720 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Oct 31 07:29:05.930922 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:05.918690    4720 flags.go:64] FLAG: --sync-frequency="1m0s"
Oct 31 07:29:05.930922 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:05.918693    4720 flags.go:64] FLAG: --system-cgroups=""
Oct 31 07:29:05.930922 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:05.918696    4720 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Oct 31 07:29:05.930922 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:05.918701    4720 flags.go:64] FLAG: --system-reserved-cgroup=""
Oct 31 07:29:05.930922 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:05.918704    4720 flags.go:64] FLAG: --tls-cert-file=""
Oct 31 07:29:05.930922 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:05.918707    4720 flags.go:64] FLAG: --tls-cipher-suites="[]"
Oct 31 07:29:05.930922 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:05.918711    4720 flags.go:64] FLAG: --tls-min-version=""
Oct 31 07:29:05.930922 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:05.918714    4720 flags.go:64] FLAG: --tls-private-key-file=""
Oct 31 07:29:05.930922 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:05.918718    4720 flags.go:64] FLAG: --topology-manager-policy="none"
Oct 31 07:29:05.930922 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:05.918721    4720 flags.go:64] FLAG: --topology-manager-policy-options=""
Oct 31 07:29:05.930922 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:05.918724    4720 flags.go:64] FLAG: --topology-manager-scope="container"
Oct 31 07:29:05.930922 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:05.918727    4720 flags.go:64] FLAG: --v="2"
Oct 31 07:29:05.930922 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:05.918731    4720 flags.go:64] FLAG: --version="false"
Oct 31 07:29:05.931877 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:05.918735    4720 flags.go:64] FLAG: --vmodule=""
Oct 31 07:29:05.931877 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:05.918743    4720 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Oct 31 07:29:05.931877 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:05.918749    4720 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Oct 31 07:29:05.931877 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.918894    4720 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 31 07:29:05.931877 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.918899    4720 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 31 07:29:05.931877 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.918902    4720 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 31 07:29:05.931877 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.918905    4720 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 31 07:29:05.931877 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.918909    4720 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 31 07:29:05.931877 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.918913    4720 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 31 07:29:05.931877 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.918917    4720 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 31 07:29:05.931877 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.918923    4720 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 31 07:29:05.931877 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.918927    4720 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 31 07:29:05.931877 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.918931    4720 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 31 07:29:05.931877 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.918934    4720 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 31 07:29:05.931877 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.918939    4720 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 31 07:29:05.931877 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.918943    4720 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 31 07:29:05.931877 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.918946    4720 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 31 07:29:05.931877 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.918950    4720 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 31 07:29:05.931877 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.918954    4720 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 31 07:29:05.931877 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.918958    4720 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 31 07:29:05.932677 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.918962    4720 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 31 07:29:05.932677 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.918966    4720 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 31 07:29:05.932677 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.918969    4720 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 31 07:29:05.932677 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.918972    4720 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 31 07:29:05.932677 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.918975    4720 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 31 07:29:05.932677 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.918979    4720 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 31 07:29:05.932677 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.918983    4720 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 31 07:29:05.932677 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.918990    4720 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 31 07:29:05.932677 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.918994    4720 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 31 07:29:05.932677 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.918999    4720 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 31 07:29:05.932677 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.919003    4720 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 31 07:29:05.932677 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.919009    4720 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 31 07:29:05.932677 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.919013    4720 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 31 07:29:05.932677 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.919020    4720 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 31 07:29:05.932677 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.919024    4720 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 31 07:29:05.932677 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.919028    4720 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 31 07:29:05.932677 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.919031    4720 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 31 07:29:05.932677 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.919034    4720 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 31 07:29:05.932677 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.919036    4720 feature_gate.go:386] unrecognized feature gate: Example2
Oct 31 07:29:05.932677 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.919039    4720 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 31 07:29:05.933608 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.919041    4720 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 31 07:29:05.933608 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.919044    4720 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 31 07:29:05.933608 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.919047    4720 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 31 07:29:05.933608 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.919050    4720 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 31 07:29:05.933608 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.919052    4720 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 31 07:29:05.933608 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.919055    4720 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 31 07:29:05.933608 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.919058    4720 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 31 07:29:05.933608 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.919060    4720 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 31 07:29:05.933608 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.919063    4720 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 31 07:29:05.933608 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.919065    4720 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 31 07:29:05.933608 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.919068    4720 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 31 07:29:05.933608 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.919071    4720 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 31 07:29:05.933608 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.919073    4720 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 31 07:29:05.933608 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.919075    4720 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 31 07:29:05.933608 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.919078    4720 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 31 07:29:05.933608 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.919080    4720 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 31 07:29:05.933608 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.919083    4720 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 31 07:29:05.933608 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.919085    4720 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 31 07:29:05.933608 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.919088    4720 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 31 07:29:05.933608 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.919094    4720 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 31 07:29:05.934440 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.919099    4720 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 31 07:29:05.934440 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.919103    4720 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 31 07:29:05.934440 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.919107    4720 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 31 07:29:05.934440 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.919111    4720 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 31 07:29:05.934440 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.919113    4720 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 31 07:29:05.934440 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.919118    4720 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 31 07:29:05.934440 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.919121    4720 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 31 07:29:05.934440 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.919123    4720 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 31 07:29:05.934440 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.919126    4720 feature_gate.go:386] unrecognized feature gate: Example
Oct 31 07:29:05.934440 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.919128    4720 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 31 07:29:05.934440 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.919131    4720 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 31 07:29:05.934440 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.919134    4720 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 31 07:29:05.934440 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.919136    4720 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 31 07:29:05.934440 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.919139    4720 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 31 07:29:05.934440 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.919142    4720 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 31 07:29:05.934440 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.919144    4720 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 31 07:29:05.934440 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.919147    4720 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 31 07:29:05.934440 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.919150    4720 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 31 07:29:05.934440 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.919152    4720 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 31 07:29:05.934440 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.919155    4720 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 31 07:29:05.935241 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.919157    4720 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 31 07:29:05.935241 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.919160    4720 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 31 07:29:05.935241 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.919164    4720 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 31 07:29:05.935241 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.919167    4720 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 31 07:29:05.935241 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.919173    4720 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 31 07:29:05.935241 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.919179    4720 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 31 07:29:05.935241 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.919184    4720 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 31 07:29:05.935241 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.919188    4720 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 31 07:29:05.935241 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.919191    4720 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 31 07:29:05.935241 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.919194    4720 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 31 07:29:05.935241 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.919196    4720 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 31 07:29:05.935241 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.919200    4720 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 31 07:29:05.935241 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.919202    4720 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 31 07:29:05.935241 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.919205    4720 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 31 07:29:05.935241 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.919207    4720 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 31 07:29:05.935241 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.919210    4720 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 31 07:29:05.935241 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.919212    4720 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 31 07:29:05.935241 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.919216    4720 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 31 07:29:05.935241 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.919218    4720 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 31 07:29:05.935875 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.919221    4720 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 31 07:29:05.935875 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.919223    4720 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 31 07:29:05.935875 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.919226    4720 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 31 07:29:05.935875 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.919229    4720 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 31 07:29:05.935875 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.919232    4720 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 31 07:29:05.935875 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.919236    4720 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 31 07:29:05.935875 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.919240    4720 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 31 07:29:05.935875 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.919243    4720 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 31 07:29:05.935875 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.919248    4720 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 31 07:29:05.935875 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.919253    4720 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 31 07:29:05.935875 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:05.919299    4720 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 31 07:29:05.935875 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.919359    4720 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 31 07:29:05.935875 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.919364    4720 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 31 07:29:05.935875 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.919367    4720 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 31 07:29:05.935875 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.919370    4720 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 31 07:29:05.935875 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.919373    4720 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 31 07:29:05.936471 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.919375    4720 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 31 07:29:05.936471 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.919378    4720 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 31 07:29:05.936471 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.919380    4720 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 31 07:29:05.936471 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.919383    4720 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 31 07:29:05.936471 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.919385    4720 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 31 07:29:05.936471 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.919387    4720 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 31 07:29:05.936471 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.919390    4720 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 31 07:29:05.936471 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.919393    4720 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 31 07:29:05.936471 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.919397    4720 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 31 07:29:05.936471 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.919399    4720 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 31 07:29:05.936471 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.919401    4720 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 31 07:29:05.936471 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.919404    4720 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 31 07:29:05.936471 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.919407    4720 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 31 07:29:05.936471 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.919409    4720 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 31 07:29:05.936471 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.919414    4720 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 31 07:29:05.936471 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.919419    4720 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 31 07:29:05.936471 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.919424    4720 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 31 07:29:05.936471 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.919428    4720 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 31 07:29:05.936471 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.919431    4720 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 31 07:29:05.937014 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.919436    4720 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 31 07:29:05.937014 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.919440    4720 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 31 07:29:05.937014 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.919444    4720 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 31 07:29:05.937014 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.919449    4720 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 31 07:29:05.937014 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.919453    4720 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 31 07:29:05.937014 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.919458    4720 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 31 07:29:05.937014 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.919462    4720 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 31 07:29:05.937014 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.919466    4720 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 31 07:29:05.937014 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.919471    4720 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 31 07:29:05.937014 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.919475    4720 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 31 07:29:05.937014 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.919480    4720 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 31 07:29:05.937014 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.919483    4720 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 31 07:29:05.937014 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.919485    4720 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 31 07:29:05.937014 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.919488    4720 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 31 07:29:05.937014 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.919490    4720 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 31 07:29:05.937014 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.919493    4720 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 31 07:29:05.937014 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.919495    4720 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 31 07:29:05.937014 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.919498    4720 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 31 07:29:05.937014 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.919501    4720 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 31 07:29:05.937539 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.919505    4720 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 31 07:29:05.937539 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.919509    4720 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 31 07:29:05.937539 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.919516    4720 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 31 07:29:05.937539 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.919519    4720 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 31 07:29:05.937539 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.919522    4720 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 31 07:29:05.937539 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.919525    4720 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 31 07:29:05.937539 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.919527    4720 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 31 07:29:05.937539 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.919530    4720 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 31 07:29:05.937539 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.919532    4720 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 31 07:29:05.937539 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.919536    4720 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 31 07:29:05.937539 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.919539    4720 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 31 07:29:05.937539 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.919541    4720 feature_gate.go:386] unrecognized feature gate: Example2
Oct 31 07:29:05.937539 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.919544    4720 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 31 07:29:05.937539 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.919546    4720 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 31 07:29:05.937539 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.919549    4720 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 31 07:29:05.937539 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.919551    4720 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 31 07:29:05.937539 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.919554    4720 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 31 07:29:05.937539 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.919556    4720 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 31 07:29:05.937539 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.919560    4720 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 31 07:29:05.937539 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.919562    4720 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 31 07:29:05.938052 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.919565    4720 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 31 07:29:05.938052 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.919567    4720 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 31 07:29:05.938052 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.919570    4720 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 31 07:29:05.938052 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.919573    4720 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 31 07:29:05.938052 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.919576    4720 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 31 07:29:05.938052 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.919579    4720 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 31 07:29:05.938052 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.919583    4720 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 31 07:29:05.938052 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.919587    4720 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 31 07:29:05.938052 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.919592    4720 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 31 07:29:05.938052 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.919596    4720 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 31 07:29:05.938052 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.919599    4720 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 31 07:29:05.938052 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.919602    4720 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 31 07:29:05.938052 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.919604    4720 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 31 07:29:05.938052 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.919607    4720 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 31 07:29:05.938052 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.919611    4720 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 31 07:29:05.938052 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.919613    4720 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 31 07:29:05.938052 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.919616    4720 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 31 07:29:05.938052 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.919619    4720 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 31 07:29:05.938052 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.919621    4720 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 31 07:29:05.938674 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.919624    4720 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 31 07:29:05.938674 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.919626    4720 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 31 07:29:05.938674 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.919632    4720 feature_gate.go:386] unrecognized feature gate: Example
Oct 31 07:29:05.938674 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.919634    4720 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 31 07:29:05.938674 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.919638    4720 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 31 07:29:05.938674 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.919640    4720 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 31 07:29:05.938674 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.919643    4720 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 31 07:29:05.938674 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.919646    4720 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 31 07:29:05.938674 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.919648    4720 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 31 07:29:05.938674 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.919651    4720 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 31 07:29:05.938674 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.919653    4720 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 31 07:29:05.938674 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.919656    4720 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 31 07:29:05.938674 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.919659    4720 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 31 07:29:05.938674 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.919662    4720 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 31 07:29:05.938674 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.919666    4720 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 31 07:29:05.938674 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.919671    4720 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 31 07:29:05.938674 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.919678    4720 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 31 07:29:05.938674 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.919683    4720 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 31 07:29:05.938674 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.919686    4720 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 31 07:29:05.939273 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.919691    4720 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 31 07:29:05.939273 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.919694    4720 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 31 07:29:05.939273 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.919697    4720 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 31 07:29:05.939273 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.919700    4720 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 31 07:29:05.939273 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.919704    4720 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 31 07:29:05.939273 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:05.925586    4720 server.go:525] "Kubelet version" kubeletVersion="v1.35.9"
Oct 31 07:29:05.939273 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:05.925604    4720 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Oct 31 07:29:05.939273 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:05.925630    4720 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Oct 31 07:29:05.939273 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:05.925638    4720 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Oct 31 07:29:05.939273 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.925714    4720 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 31 07:29:05.939273 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.925724    4720 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 31 07:29:05.939273 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.925729    4720 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 31 07:29:05.939273 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.925734    4720 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 31 07:29:05.939273 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.925741    4720 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 31 07:29:05.939273 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.925746    4720 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 31 07:29:05.939273 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.925750    4720 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 31 07:29:05.939273 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.925755    4720 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 31 07:29:05.939273 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.925759    4720 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 31 07:29:05.939273 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.925762    4720 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 31 07:29:05.939273 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.925764    4720 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 31 07:29:05.940220 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.925767    4720 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 31 07:29:05.940220 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.925769    4720 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 31 07:29:05.940220 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.925772    4720 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 31 07:29:05.940220 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.925775    4720 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 31 07:29:05.940220 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.925777    4720 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 31 07:29:05.940220 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.925781    4720 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 31 07:29:05.940220 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.925788    4720 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 31 07:29:05.940220 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.925794    4720 feature_gate.go:386] unrecognized feature gate: Example
Oct 31 07:29:05.940220 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.925799    4720 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 31 07:29:05.940220 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.925805    4720 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 31 07:29:05.940220 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.925810    4720 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 31 07:29:05.940220 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.925814    4720 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 31 07:29:05.940220 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.925819    4720 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 31 07:29:05.940220 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.925824    4720 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 31 07:29:05.940220 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.925829    4720 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 31 07:29:05.940220 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.925833    4720 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 31 07:29:05.940220 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.925838    4720 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 31 07:29:05.940220 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.925841    4720 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 31 07:29:05.940220 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.925844    4720 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 31 07:29:05.941009 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.925847    4720 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 31 07:29:05.941009 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.925849    4720 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 31 07:29:05.941009 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.925852    4720 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 31 07:29:05.941009 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.925854    4720 feature_gate.go:386] unrecognized feature gate: Example2
Oct 31 07:29:05.941009 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.925857    4720 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 31 07:29:05.941009 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.925859    4720 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 31 07:29:05.941009 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.925862    4720 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 31 07:29:05.941009 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.925864    4720 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 31 07:29:05.941009 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.925867    4720 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 31 07:29:05.941009 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.925869    4720 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 31 07:29:05.941009 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.925872    4720 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 31 07:29:05.941009 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.925874    4720 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 31 07:29:05.941009 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.925877    4720 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 31 07:29:05.941009 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.925879    4720 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 31 07:29:05.941009 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.925882    4720 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 31 07:29:05.941009 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.925884    4720 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 31 07:29:05.941009 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.925888    4720 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 31 07:29:05.941009 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.925892    4720 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 31 07:29:05.941009 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.925895    4720 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 31 07:29:05.941009 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.925898    4720 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 31 07:29:05.941839 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.925900    4720 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 31 07:29:05.941839 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.925903    4720 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 31 07:29:05.941839 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.925905    4720 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 31 07:29:05.941839 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.925908    4720 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 31 07:29:05.941839 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.925912    4720 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 31 07:29:05.941839 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.925915    4720 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 31 07:29:05.941839 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.925918    4720 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 31 07:29:05.941839 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.925920    4720 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 31 07:29:05.941839 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.925923    4720 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 31 07:29:05.941839 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.925925    4720 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 31 07:29:05.941839 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.925928    4720 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 31 07:29:05.941839 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.925930    4720 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 31 07:29:05.941839 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.925933    4720 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 31 07:29:05.941839 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.925935    4720 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 31 07:29:05.941839 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.925937    4720 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 31 07:29:05.941839 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.925940    4720 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 31 07:29:05.941839 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.925942    4720 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 31 07:29:05.941839 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.925945    4720 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 31 07:29:05.941839 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.925947    4720 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 31 07:29:05.942634 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.925950    4720 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 31 07:29:05.942634 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.925952    4720 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 31 07:29:05.942634 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.925955    4720 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 31 07:29:05.942634 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.925957    4720 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 31 07:29:05.942634 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.925959    4720 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 31 07:29:05.942634 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.925962    4720 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 31 07:29:05.942634 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.925964    4720 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 31 07:29:05.942634 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.925967    4720 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 31 07:29:05.942634 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.925969    4720 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 31 07:29:05.942634 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.925971    4720 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 31 07:29:05.942634 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.925974    4720 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 31 07:29:05.942634 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.925976    4720 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 31 07:29:05.942634 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.925979    4720 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 31 07:29:05.942634 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.925981    4720 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 31 07:29:05.942634 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.925984    4720 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 31 07:29:05.942634 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.925986    4720 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 31 07:29:05.942634 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.925988    4720 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 31 07:29:05.942634 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.925991    4720 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 31 07:29:05.942634 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.925994    4720 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 31 07:29:05.942634 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.925996    4720 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 31 07:29:05.943522 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.926001    4720 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 31 07:29:05.943522 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.926005    4720 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 31 07:29:05.943522 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.926010    4720 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 31 07:29:05.943522 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.926013    4720 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 31 07:29:05.943522 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.926016    4720 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 31 07:29:05.943522 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.926020    4720 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 31 07:29:05.943522 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.926024    4720 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 31 07:29:05.943522 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.926029    4720 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 31 07:29:05.943522 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.926032    4720 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 31 07:29:05.943522 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.926034    4720 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 31 07:29:05.943522 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.926037    4720 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 31 07:29:05.943522 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.926039    4720 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 31 07:29:05.943522 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.926041    4720 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 31 07:29:05.943522 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.926044    4720 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 31 07:29:05.943522 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.926046    4720 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 31 07:29:05.943522 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.926049    4720 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 31 07:29:05.943522 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.926051    4720 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 31 07:29:05.944266 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:05.926080    4720 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 31 07:29:05.944266 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.926270    4720 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 31 07:29:05.944266 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.926276    4720 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 31 07:29:05.944266 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.926279    4720 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 31 07:29:05.944266 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.926282    4720 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 31 07:29:05.944266 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.926285    4720 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 31 07:29:05.944266 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.926288    4720 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 31 07:29:05.944266 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.926292    4720 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 31 07:29:05.944266 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.926298    4720 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 31 07:29:05.944266 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.926305    4720 feature_gate.go:386] unrecognized feature gate: Example2
Oct 31 07:29:05.944266 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.926310    4720 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 31 07:29:05.944266 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.926315    4720 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 31 07:29:05.944266 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.926319    4720 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 31 07:29:05.944266 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.926340    4720 feature_gate.go:386] unrecognized feature gate: Example
Oct 31 07:29:05.944266 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.926344    4720 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 31 07:29:05.944266 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.926349    4720 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 31 07:29:05.945013 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.926353    4720 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 31 07:29:05.945013 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.926358    4720 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 31 07:29:05.945013 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.926362    4720 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 31 07:29:05.945013 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.926367    4720 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 31 07:29:05.945013 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.926371    4720 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 31 07:29:05.945013 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.926375    4720 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 31 07:29:05.945013 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.926379    4720 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 31 07:29:05.945013 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.926383    4720 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 31 07:29:05.945013 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.926387    4720 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 31 07:29:05.945013 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.926390    4720 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 31 07:29:05.945013 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.926394    4720 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 31 07:29:05.945013 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.926399    4720 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 31 07:29:05.945013 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.926402    4720 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 31 07:29:05.945013 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.926404    4720 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 31 07:29:05.945013 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.926407    4720 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 31 07:29:05.945013 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.926409    4720 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 31 07:29:05.945013 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.926412    4720 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 31 07:29:05.945013 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.926415    4720 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 31 07:29:05.945013 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.926417    4720 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 31 07:29:05.945826 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.926420    4720 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 31 07:29:05.945826 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.926422    4720 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 31 07:29:05.945826 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.926425    4720 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 31 07:29:05.945826 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.926427    4720 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 31 07:29:05.945826 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.926430    4720 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 31 07:29:05.945826 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.926432    4720 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 31 07:29:05.945826 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.926435    4720 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 31 07:29:05.945826 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.926437    4720 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 31 07:29:05.945826 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.926440    4720 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 31 07:29:05.945826 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.926443    4720 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 31 07:29:05.945826 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.926445    4720 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 31 07:29:05.945826 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.926448    4720 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 31 07:29:05.945826 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.926451    4720 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 31 07:29:05.945826 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.926454    4720 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 31 07:29:05.945826 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.926456    4720 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 31 07:29:05.945826 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.926459    4720 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 31 07:29:05.945826 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.926461    4720 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 31 07:29:05.945826 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.926464    4720 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 31 07:29:05.945826 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.926466    4720 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 31 07:29:05.946647 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.926468    4720 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 31 07:29:05.946647 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.926471    4720 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 31 07:29:05.946647 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.926473    4720 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 31 07:29:05.946647 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.926476    4720 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 31 07:29:05.946647 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.926479    4720 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 31 07:29:05.946647 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.926481    4720 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 31 07:29:05.946647 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.926484    4720 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 31 07:29:05.946647 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.926486    4720 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 31 07:29:05.946647 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.926489    4720 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 31 07:29:05.946647 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.926491    4720 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 31 07:29:05.946647 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.926494    4720 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 31 07:29:05.946647 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.926497    4720 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 31 07:29:05.946647 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.926499    4720 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 31 07:29:05.946647 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.926502    4720 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 31 07:29:05.946647 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.926505    4720 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 31 07:29:05.946647 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.926509    4720 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 31 07:29:05.946647 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.926512    4720 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 31 07:29:05.946647 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.926515    4720 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 31 07:29:05.946647 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.926517    4720 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 31 07:29:05.947497 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.926520    4720 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 31 07:29:05.947497 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.926522    4720 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 31 07:29:05.947497 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.926524    4720 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 31 07:29:05.947497 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.926527    4720 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 31 07:29:05.947497 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.926529    4720 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 31 07:29:05.947497 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.926532    4720 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 31 07:29:05.947497 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.926535    4720 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 31 07:29:05.947497 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.926538    4720 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 31 07:29:05.947497 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.926540    4720 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 31 07:29:05.947497 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.926543    4720 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 31 07:29:05.947497 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.926545    4720 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 31 07:29:05.947497 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.926548    4720 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 31 07:29:05.947497 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.926550    4720 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 31 07:29:05.947497 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.926553    4720 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 31 07:29:05.947497 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.926556    4720 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 31 07:29:05.947497 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.926560    4720 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 31 07:29:05.947497 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.926565    4720 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 31 07:29:05.947497 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.926569    4720 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 31 07:29:05.947497 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.926573    4720 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 31 07:29:05.947497 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.926577    4720 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 31 07:29:05.948484 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.926581    4720 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 31 07:29:05.948484 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.926585    4720 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 31 07:29:05.948484 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.926607    4720 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 31 07:29:05.948484 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.926612    4720 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 31 07:29:05.948484 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.926615    4720 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 31 07:29:05.948484 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.926619    4720 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 31 07:29:05.948484 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.926624    4720 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 31 07:29:05.948484 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.926628    4720 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 31 07:29:05.948484 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.926633    4720 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 31 07:29:05.948484 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.926638    4720 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 31 07:29:05.948484 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.926642    4720 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 31 07:29:05.948484 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.926645    4720 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 31 07:29:05.948484 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.926648    4720 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 31 07:29:05.948484 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.926651    4720 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 31 07:29:05.948484 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:05.926687    4720 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 31 07:29:05.948484 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.926726    4720 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 31 07:29:05.949347 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.926730    4720 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 31 07:29:05.949347 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.926733    4720 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 31 07:29:05.949347 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.926736    4720 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 31 07:29:05.949347 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.926739    4720 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 31 07:29:05.949347 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.926742    4720 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 31 07:29:05.949347 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.926747    4720 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 31 07:29:05.949347 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.926751    4720 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 31 07:29:05.949347 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.926755    4720 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 31 07:29:05.949347 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.926759    4720 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 31 07:29:05.949347 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.926762    4720 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 31 07:29:05.949347 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.926764    4720 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 31 07:29:05.949347 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.926767    4720 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 31 07:29:05.949347 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.926771    4720 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 31 07:29:05.949347 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.926775    4720 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 31 07:29:05.949347 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.926779    4720 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 31 07:29:05.949347 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.926783    4720 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 31 07:29:05.949347 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.926790    4720 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 31 07:29:05.949347 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.926795    4720 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 31 07:29:05.949347 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.926798    4720 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 31 07:29:05.950268 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.926800    4720 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 31 07:29:05.950268 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.926803    4720 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 31 07:29:05.950268 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.926805    4720 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 31 07:29:05.950268 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.926807    4720 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 31 07:29:05.950268 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.926810    4720 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 31 07:29:05.950268 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.926812    4720 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 31 07:29:05.950268 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.926815    4720 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 31 07:29:05.950268 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.926817    4720 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 31 07:29:05.950268 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.926820    4720 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 31 07:29:05.950268 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.926824    4720 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 31 07:29:05.950268 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.926827    4720 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 31 07:29:05.950268 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.926832    4720 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 31 07:29:05.950268 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.926836    4720 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 31 07:29:05.950268 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.926840    4720 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 31 07:29:05.950268 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.926844    4720 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 31 07:29:05.950268 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.926847    4720 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 31 07:29:05.950268 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.926850    4720 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 31 07:29:05.950268 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.926853    4720 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 31 07:29:05.950268 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.926856    4720 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 31 07:29:05.950268 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.926858    4720 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 31 07:29:05.951217 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.926861    4720 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 31 07:29:05.951217 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.926863    4720 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 31 07:29:05.951217 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.926865    4720 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 31 07:29:05.951217 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.926868    4720 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 31 07:29:05.951217 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.926870    4720 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 31 07:29:05.951217 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.926873    4720 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 31 07:29:05.951217 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.926875    4720 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 31 07:29:05.951217 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.926879    4720 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 31 07:29:05.951217 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.926883    4720 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 31 07:29:05.951217 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.926896    4720 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 31 07:29:05.951217 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.926900    4720 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 31 07:29:05.951217 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.926903    4720 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 31 07:29:05.951217 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.926907    4720 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 31 07:29:05.951217 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.926912    4720 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 31 07:29:05.951217 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.926916    4720 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 31 07:29:05.951217 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.926921    4720 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 31 07:29:05.951217 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.926925    4720 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 31 07:29:05.951217 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.926927    4720 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 31 07:29:05.951217 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.926930    4720 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 31 07:29:05.951217 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.926933    4720 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 31 07:29:05.952159 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.926935    4720 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 31 07:29:05.952159 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.926938    4720 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 31 07:29:05.952159 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.926940    4720 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 31 07:29:05.952159 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.926944    4720 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 31 07:29:05.952159 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.926948    4720 feature_gate.go:386] unrecognized feature gate: Example2
Oct 31 07:29:05.952159 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.926952    4720 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 31 07:29:05.952159 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.926957    4720 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 31 07:29:05.952159 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.926962    4720 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 31 07:29:05.952159 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.926966    4720 feature_gate.go:386] unrecognized feature gate: Example
Oct 31 07:29:05.952159 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.926970    4720 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 31 07:29:05.952159 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.926975    4720 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 31 07:29:05.952159 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.926978    4720 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 31 07:29:05.952159 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.926981    4720 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 31 07:29:05.952159 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.926984    4720 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 31 07:29:05.952159 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.926987    4720 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 31 07:29:05.952159 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.926989    4720 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 31 07:29:05.952159 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.926992    4720 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 31 07:29:05.952159 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.926994    4720 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 31 07:29:05.952159 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.926997    4720 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 31 07:29:05.952159 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.926999    4720 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 31 07:29:05.952999 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.927002    4720 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 31 07:29:05.952999 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.927004    4720 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 31 07:29:05.952999 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.927007    4720 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 31 07:29:05.952999 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.927011    4720 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 31 07:29:05.952999 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.927015    4720 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 31 07:29:05.952999 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.927019    4720 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 31 07:29:05.952999 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.927023    4720 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 31 07:29:05.952999 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.927025    4720 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 31 07:29:05.952999 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.927028    4720 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 31 07:29:05.952999 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.927031    4720 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 31 07:29:05.952999 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.927033    4720 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 31 07:29:05.952999 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.927036    4720 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 31 07:29:05.952999 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.927038    4720 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 31 07:29:05.952999 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.927041    4720 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 31 07:29:05.952999 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.927043    4720 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 31 07:29:05.952999 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.927045    4720 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 31 07:29:05.952999 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.927048    4720 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 31 07:29:05.952999 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.927050    4720 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 31 07:29:05.952999 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.927052    4720 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 31 07:29:05.953709 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.927055    4720 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 31 07:29:05.953709 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.927057    4720 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 31 07:29:05.953709 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.927060    4720 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 31 07:29:05.953709 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.927063    4720 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 31 07:29:05.953709 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.927067    4720 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 31 07:29:05.953709 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.927070    4720 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 31 07:29:05.953709 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:05.927075    4720 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 31 07:29:05.953709 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:05.927131    4720 server.go:957] "Client rotation is on, will bootstrap in background"
Oct 31 07:29:05.953709 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:05.928281    4720 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary"
Oct 31 07:29:05.953709 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:05.928358    4720 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem"
Oct 31 07:29:05.953709 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:05.928557    4720 server.go:1014] "Starting client certificate rotation"
Oct 31 07:29:05.953709 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:05.928701    4720 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Oct 31 07:29:05.953709 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:05.928759    4720 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-30 07:23:24 +0000 UTC" deadline="2028-07-12 21:59:43.085224834 +0000 UTC"
Oct 31 07:29:05.953709 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:05.928797    4720 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14894h30m37.156431027s"
Oct 31 07:29:05.953709 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:05.929116    4720 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 31 07:29:05.954182 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:05.930586    4720 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 31 07:29:05.954182 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:05.935834    4720 log.go:25] "Validated CRI v1 runtime API"
Oct 31 07:29:05.954182 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:05.937936    4720 log.go:25] "Validated CRI v1 image API"
Oct 31 07:29:05.954182 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:05.938734    4720 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Oct 31 07:29:05.954182 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:05.941472    4720 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 bc678eae-8b6f-494f-90bb-737d90244670:/dev/nvme0n1p3 eec75f1d-0616-4685-bf4f-0a76eaa0b8d4:/dev/nvme0n1p4]
Oct 31 07:29:05.954405 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:05.941502    4720 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/24977ca031ccca9556605789709d856b16771b92e2cd05e39a8df96eb7e98438/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/24977ca031ccca9556605789709d856b16771b92e2cd05e39a8df96eb7e98438/userdata/shm major:0 minor:217 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/26d38dd238ca56039d3f30825038e78a1478f38371827bb224e5be4d97b94b28/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/26d38dd238ca56039d3f30825038e78a1478f38371827bb224e5be4d97b94b28/userdata/shm major:0 minor:246 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/37019c645339bd463a7dee60255920849f38a55d3c31eda8755469f8ada74ba0/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/37019c645339bd463a7dee60255920849f38a55d3c31eda8755469f8ada74ba0/userdata/shm major:0 minor:61 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/397e8b2b367abe48f39d3961ba7ce8bb1ebfd59819edf275333c202df43087e2/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/397e8b2b367abe48f39d3961ba7ce8bb1ebfd59819edf275333c202df43087e2/userdata/shm major:0 minor:206 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/44c067aa8f06e75a4a32430911035f32efde1168203a54d7abd7aa733b407210/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/44c067aa8f06e75a4a32430911035f32efde1168203a54d7abd7aa733b407210/userdata/shm major:0 minor:242 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/4b27a1e9df141368f9aec55f507c535d7c7a4e98dd0d847f22a930188408d21e/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/4b27a1e9df141368f9aec55f507c535d7c7a4e98dd0d847f22a930188408d21e/userdata/shm major:0 minor:244 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/4b442c7c5f25e692aee5d663f2bd1e5c0c7168cd799aab781bfed2102e3740f1/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/4b442c7c5f25e692aee5d663f2bd1e5c0c7168cd799aab781bfed2102e3740f1/userdata/shm major:0 minor:231 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/5292ef84738cab81213f85cb2c9188469f27c00aa7758fb95524fa5c53ddac15/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/5292ef84738cab81213f85cb2c9188469f27c00aa7758fb95524fa5c53ddac15/userdata/shm major:0 minor:74 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/57c7cad37211feee315413dd85f61718ce6291d6c4be0727f145731331c93e71/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/57c7cad37211feee315413dd85f61718ce6291d6c4be0727f145731331c93e71/userdata/shm major:0 minor:340 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/69600ee490f34bf474850cf25f80679e6519843dc40baf37c5d57342b0cec097/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/69600ee490f34bf474850cf25f80679e6519843dc40baf37c5d57342b0cec097/userdata/shm major:0 minor:80 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/6a2e938ce4315b140940dad931d9f3c7bb967a6dfafd01ff57780c53f7df30a7/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/6a2e938ce4315b140940dad931d9f3c7bb967a6dfafd01ff57780c53f7df30a7/userdata/shm major:0 minor:240 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/6d2a6480d09c4deb2fe4f31aa76ca7365b7305ee8d41ba48d3831a072eab9b1d/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/6d2a6480d09c4deb2fe4f31aa76ca7365b7305ee8d41ba48d3831a072eab9b1d/userdata/shm major:0 minor:215 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/7a306c5d41908aad98edab19712d66d4e434fca13d0fb3b8c8f78ed55689cab5/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/7a306c5d41908aad98edab19712d66d4e434fca13d0fb3b8c8f78ed55689cab5/userdata/shm major:0 minor:288 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/7c6cf74c69f7fceea5860f8c2894b19081b8a4f44528c01cdf3a23bc22f567b5/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/7c6cf74c69f7fceea5860f8c2894b19081b8a4f44528c01cdf3a23bc22f567b5/userdata/shm major:0 minor:289 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/7dab47a696030acdb1ddfc6fe11ed4c83ad1726d144ad583b503091bc2bb5a62/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/7dab47a696030acdb1ddfc6fe11ed4c83ad1726d144ad583b503091bc2bb5a62/userdata/shm major:0 minor:219 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/7df3a2737e0ecbaf7eb754ef7539c98a3a1aac973f357f73d3c8d852b7d2312f/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/7df3a2737e0ecbaf7eb754ef7539c98a3a1aac973f357f73d3c8d852b7d2312f/userdata/shm major:0 minor:88 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/82a11c7ec21dcf611cf9180470eb718c94c4332da4e8f2b47dcf174499bada7a/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/82a11c7ec21dcf611cf9180470eb718c94c4332da4e8f2b47dcf174499bada7a/userdata/shm major:0 minor:72 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/90033c9a0ade2d06af05ffaab30cc49f21be3df3b48302c519fd8d263f9886af/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/90033c9a0ade2d06af05ffaab30cc49f21be3df3b48302c519fd8d263f9886af/userdata/shm major:0 minor:237 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/9f2589536197b37032285af9622b146bd14cd852452e464e8eb73c0a610e5078/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/9f2589536197b37032285af9622b146bd14cd852452e464e8eb73c0a610e5078/userdata/shm major:0 minor:60 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/c1ffc96f411310ca3c9bbb972b3c1df1fe0467e4564c58e844cd9b804d40194c/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/c1ffc96f411310ca3c9bbb972b3c1df1fe0467e4564c58e844cd9b804d40194c/userdata/shm major:0 minor:98 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/c8d95e23ad7960db9871e1b75389bf92563cfef5ab50bfece8472084ee3133ba/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/c8d95e23ad7960db9871e1b75389bf92563cfef5ab50bfece8472084ee3133ba/userdata/shm major:0 minor:94 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/e5d2dc100f3cacdb69fc74b690c1bbe741b4298bd41697263a403392aa95b8b8/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/e5d2dc100f3cacdb69fc74b690c1bbe741b4298bd41697263a403392aa95b8b8/userdata/shm major:0 minor:62 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/ecb08d77ee39577a396f19d406280850735fe9412808b0951b5d432f6275f230/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/ecb08d77ee39577a396f19d406280850735fe9412808b0951b5d432f6275f230/userdata/shm major:0 minor:84 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/08198bc1-314f-4881-833c-e633bac15e4a/volumes/kubernetes.io~projected/kube-api-access-xs9lt:{mountpoint:/var/lib/kubelet/pods/08198bc1-314f-4881-833c-e633bac15e4a/volumes/kubernetes.io~projected/kube-api-access-xs9lt major:0 minor:54 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0b25f791-3520-481d-acbb-a3bef222fdac/volumes/kubernetes.io~projected/kube-api-access-nhphg:{mountpoint:/var/lib/kubelet/pods/0b25f791-3520-481d-acbb-a3bef222fdac/volumes/kubernetes.io~projected/kube-api-access-nhphg major:0 minor:335 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2536c1c8-8a46-4af3-bc6d-81454a908d3f/volumes/kubernetes.io~projected/kube-api-access-g2r66:{mountpoint:/var/lib/kubelet/pods/2536c1c8-8a46-4af3-bc6d-81454a908d3f/volumes/kubernetes.io~projected/kube-api-access-g2r66 major:0 minor:235 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2536c1c8-8a46-4af3-bc6d-81454a908d3f/volumes/kubernetes.io~secret/cert:{mountpoint:/var/lib/kubelet/pods/2536c1c8-8a46-4af3-bc6d-81454a908d3f/volumes/kubernetes.io~secret/cert major:0 minor:228 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/26659955-3909-4171-90b8-21b0ad6eef64/volumes/kubernetes.io~projected/kube-api-access-jbdqx:{mountpoint:/var/lib/kubelet/pods/26659955-3909-4171-90b8-21b0ad6eef64/volumes/kubernetes.io~projected/kube-api-access-jbdqx major:0 minor:57 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/28b59d93-b9ce-4ceb-9f3a-742525eb94c4/volumes/kubernetes.io~projected/bound-sa-token:{mountpoint:/var/lib/kubelet/pods/28b59d93-b9ce-4ceb-9f3a-742525eb94c4/volumes/kubernetes.io~projected/bound-sa-token major:0 minor:221 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/28b59d93-b9ce-4ceb-9f3a-742525eb94c4/volumes/kubernetes.io~projected/kube-api-access-xvv26:{mountpoint:/var/lib/kubelet/pods/28b59d93-b9ce-4ceb-9f3a-742525eb94c4/volumes/kubernetes.io~projected/kube-api-access-xvv26 major:0 minor:226 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/28b59d93-b9ce-4ceb-9f3a-742525eb94c4/volumes/kubernetes.io~secret/image-registry-private-configuration:{mountpoint:/var/lib/kubelet/pods/28b59d93-b9ce-4ceb-9f3a-742525eb94c4/volumes/kubernetes.io~secret/image-registry-private-configuration major:0 minor:214 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/28b59d93-b9ce-4ceb-9f3a-742525eb94c4/volumes/kubernetes.io~secret/installation-pull-secrets:{mountpoint:/var/lib/kubelet/pods/28b59d93-b9ce-4ceb-9f3a-742525eb94c4/volumes/kubernetes.io~secret/installation-pull-secrets major:0 minor:212 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/volumes/kubernetes.io~secret/agent-certs major:0 minor:47 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/323c8334-9a56-4c4e-8a6e-ae80995810d0/volumes/kubernetes.io~projected/kube-api-access-l4nfd:{mountpoint:/var/lib/kubelet/pods/323c8334-9a56-4c4e-8a6e-ae80995810d0/volumes/kubernetes.io~projected/kube-api-access-l4nfd major:0 minor:202 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3fa078c7-4bf2-4e76-9692-ae460e8fadfa/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/3fa078c7-4bf2-4e76-9692-ae460e8fadfa/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:50 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3fa078c7-4bf2-4e76-9692-ae460e8fadfa/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/3fa078c7-4bf2-4e76-9692-ae460e8fadfa/volumes/kubernetes.io~empty-dir/tmp major:0 minor:48 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3fa078c7-4bf2-4e76-9692-ae460e8fadfa/volumes/kubernetes.io~projected/kube-api-access-cq7rv:{mountpoint:/var/lib/kubelet/pods/3fa078c7-4bf2-4e76-9692-ae460e8fadfa/volumes/kubernetes.io~projected/kube-api-access-cq7rv major:0 minor:51 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4bfbebfd-2be7-438f-b88d-273f24751f8c/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/4bfbebfd-2be7-438f-b88d-273f24751f8c/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4bfbebfd-2be7-438f-b88d-273f24751f8c/volumes/kubernetes.io~projected/kube-api-access-cz8tm:{mountpoint:/var/lib/kubelet/pods/4bfbebfd-2be7-438f-b88d-273f24751f8c/volumes/kubernetes.io~projected/kube-api-access-cz8tm major:0 minor:56 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4bfbebfd-2be7-438f-b88d-273f24751f8c/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/4bfbebfd-2be7-438f-b88d-273f24751f8c/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:49 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4db4c33f-0c58-40f8-bf74-51b3eae3c9db/volumes/kubernetes.io~projected/kube-api-access-q2kvz:{mountpoint:/var/lib/kubelet/pods/4db4c33f-0c58-40f8-bf74-51b3eae3c9db/volumes/kubernetes.io~projected/kube-api-access-q2kvz major:0 minor:223 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4db4c33f-0c58-40f8-bf74-51b3eae3c9db/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/4db4c33f-0c58-40f8-bf74-51b3eae3c9db/volumes/kubernetes.io~secret/serving-cert major:0 minor:213 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/631cff9e-a1f1-4dd2-bdc8-ef36519c7c1b/volumes/kubernetes.io~projected/kube-api-access-dx598:{mountpoint:/var/lib/kubelet/pods/631cff9e-a1f1-4dd2-bdc8-ef36519c7c1b/volumes/kubernetes.io~projected/kube-api-access-dx598 major:0 minor:204 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/77256436-39ff-4c2a-8390-4e9fa7d68486/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/77256436-39ff-4c2a-8390-4e9fa7d68486/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:286 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/789f11d8-d951-4144-a589-a2bdbf6844e3/volumes/kubernetes.io~projected/kube-api-access-wqmxh:{mountpoint:/var/lib/kubelet/pods/789f11d8-d951-4144-a589-a2bdbf6844e3/volumes/kubernetes.io~projected/kube-api-access-wqmxh major:0 minor:225 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/81829763-27b1-4a48-a568-cd5f6a39a953/volumes/kubernetes.io~projected/kube-api-access-k6576:{mountpoint:/var/lib/kubelet/pods/81829763-27b1-4a48-a568-cd5f6a39a953/volumes/kubernetes.io~projected/kube-api-access-k6576 major:0 minor:205 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/81829763-27b1-4a48-a568-cd5f6a39a953/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/81829763-27b1-4a48-a568-cd5f6a39a953/volumes/kubernetes.io~secret/serving-cert major:0 minor:45 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/91363236-08db-4f54-ae0a-e68604fcecc6/volumes/kubernetes.io~projected/kube-api-access-sn2lx:{mountpoint:/var/lib/kubelet/pods/91363236-08db-4f54-ae0a-e68604fcecc6/volumes/kubernetes.io~projected/kube-api-access-sn2lx major:0 minor:201 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/993cfed6-806d-44c5-bba8-c02203638fe7/volumes/kubernetes.io~projected/bound-sa-token:{mountpoint:/var/lib/kubelet/pods/993cfed6-806d-44c5-bba8-c02203638fe7/volumes/kubernetes.io~projected/bound-sa-token major:0 minor:227 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/993cfed6-806d-44c5-bba8-c02203638fe7/volumes/kubernetes.io~projected/kube-api-access-v2zhk:{mountpoint:/var/lib/kubelet/pods/993cfed6-806d-44c5-bba8-c02203638fe7/volumes/kubernetes.io~projected/kube-api-access-v2zhk major:0 minor:224 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/993cfed6-806d-44c5-bba8-c02203638fe7/volumes/kubernetes.io~secret/image-registry-private-configuration:{mountpoint:/var/lib/kubelet/pods/993cfed6-806d-44c5-bba8-c02203638fe7/volumes/kubernetes.io~secret/image-registry-private-configuration major:0 minor:209 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/993cfed6-806d-44c5-bba8-c02203638fe7/volumes/kubernetes.io~secret/installation-pull-secrets:{mountpoint:/var/lib/kubelet/pods/993cfed6-806d-44c5-bba8-c02203638fe7/volumes/kubernetes.io~secret/installation-pull-secrets major:0 minor:208 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9c72279a-708f-46df-8915-9995c5aea14d/volumes/kubernetes.io~projected/kube-api-access-h4gzh:{mountpoint:/var/lib/kubelet/pods/9c72279a-708f-46df-8915-9995c5aea14d/volumes/kubernetes.io~projected/kube-api-access-h4gzh major:0 minor:58 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9dad8dd1-cfc5-412d-bf9a-bac3c9162af1/volumes/kubernetes.io~projected/kube-api-access-5n5lv:{mountpoint:/var/lib/kubelet/pods/9dad8dd1-cfc5-412d-bf9a-bac3c9162af1/volumes/kubernetes.io~projected/kube-api-access-5n5lv major:0 minor:53 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a73024cb-f581-450d-a09c-0565838c5baa/volumes/kubernetes.io~projected/kube-api-access-m6l7n:{mountpoint:/var/lib/kubelet/pods/a73024cb-f581-450d-a09c-0565838c5baa/volumes/kubernetes.io~projected/kube-api-access-m6l7n major:0 minor:239 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b1066ece-e6f3-4491-aa1f-a7b9a3aa248d/volumes/kubernetes.io~projected/kube-api-access-cjt6r:{mountpoint:/var/lib/kubelet/pods/b1066ece-e6f3-4491-aa1f-a7b9a3aa248d/volumes/kubernetes.io~projected/kube-api-access-cjt6r major:0 minor:322 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b1db0e3b-e85d-4769-8012-cf4b6f740d9d/volumes/kubernetes.io~projected/kube-api-access-czqtq:{mountpoint:/var/lib/kubelet/pods/b1db0e3b-e85d-4769-8012-cf4b6f740d9d/volumes/kubernetes.io~projected/kube-api-access-czqtq major:0 minor:234 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b1db0e3b-e85d-4769-8012-cf4b6f740d9d/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/b1db0e3b-e85d-4769-8012-cf4b6f740d9d/volumes/kubernetes.io~secret/serving-cert major:0 minor:230 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b44ed0e0-a72e-4746-a7de-ed8610da845d/volumes/kubernetes.io~projected/kube-api-access-9pvp2:{mountpoint:/var/lib/kubelet/pods/b44ed0e0-a72e-4746-a7de-ed8610da845d/volumes/kubernetes.io~projected/kube-api-access-9pvp2 major:0 minor:222 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b44ed0e0-a72e-4746-a7de-ed8610da845d/volumes/kubernetes.io~secret/default-certificate:{mountpoint:/var/lib/kubelet/pods/b44ed0e0-a72e-4746-a7de-ed8610da845d/volumes/kubernetes.io~secret/default-certificate major:0 minor:210 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b44ed0e0-a72e-4746-a7de-ed8610da845d/volumes/kubernetes.io~secret/stats-auth:{mountpoint:/var/lib/kubelet/pods/b44ed0e0-a72e-4746-a7de-ed8610da845d/volumes/kubernetes.io~secret/stats-auth major:0 minor:211 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c3ad95fd-e997-459c-93fd-27c4c030ea0d/volumes/kubernetes.io~projected/kube-api-access-kxwb9:{mountpoint:/var/lib/kubelet/pods/c3ad95fd-e997-459c-93fd-27c4c030ea0d/volumes/kubernetes.io~projected/kube-api-access-kxwb9 major:0 minor:59 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c4233c69-4213-43be-8809-76612b6b6f3e/volumes/kubernetes.io~projected/kube-api-access-5hqbq:{mountpoint:/var/lib/kubelet/pods/c4233c69-4213-43be-8809-76612b6b6f3e/volumes/kubernetes.io~projected/kube-api-access-5hqbq major:0 minor:55 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ca31519e-aa90-4104-a175-f9af2b3bbeb4/volumes/kubernetes.io~projected/kube-api-access-n6p25:{mountpoint:/var/lib/kubelet/pods/ca31519e-aa90-4104-a175-f9af2b3bbeb4/volumes/kubernetes.io~projected/kube-api-access-n6p25 major:0 minor:197 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/cb9e4575-9265-43b5-bf02-7843611d6563/volumes/kubernetes.io~projected/kube-api-access-kr9wm:{mountpoint:/var/lib/kubelet/pods/cb9e4575-9265-43b5-bf02-7843611d6563/volumes/kubernetes.io~projected/kube-api-access-kr9wm major:0 minor:236 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/cb9e4575-9265-43b5-bf02-7843611d6563/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/cb9e4575-9265-43b5-bf02-7843611d6563/volumes/kubernetes.io~secret/serving-cert major:0 minor:229 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ce8cac52-291b-47fb-a7b6-4cf7562449b9/volumes/kubernetes.io~projected/kube-api-access-bfzv8:{mountpoint:/var/lib/kubelet/pods/ce8cac52-291b-47fb-a7b6-4cf7562449b9/volumes/kubernetes.io~projected/kube-api-access-bfzv8 major:0 minor:233 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ee4a4a2a-7439-4c78-8858-d6ba0085fdde/volumes/kubernetes.io~projected/kube-api-access-q7vwq:{mountpoint:/var/lib/kubelet/pods/ee4a4a2a-7439-4c78-8858-d6ba0085fdde/volumes/kubernetes.io~projected/kube-api-access-q7vwq major:0 minor:203 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f1ba68f6-d87c-4492-a5eb-5f2d3a60eb1c/volumes/kubernetes.io~projected/kube-api-access-ghkb9:{mountpoint:/var/lib/kubelet/pods/f1ba68f6-d87c-4492-a5eb-5f2d3a60eb1c/volumes/kubernetes.io~projected/kube-api-access-ghkb9 major:0 minor:287 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/fc64d31a-f48a-4293-a4de-262045d0b7e8/volumes/kubernetes.io~projected/kube-api-access-kk4nh:{mountpoint:/var/lib/kubelet/pods/fc64d31a-f48a-4293-a4de-262045d0b7e8/volumes/kubernetes.io~projected/kube-api-access-kk4nh major:0 minor:52 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-100:{mountpoint:/var/lib/containers/storage/overlay/b020245d8b1332dd46a564bb721de0142d69c42d4feb290d24fef96664418273/merged major:0 minor:100 fsType:overlay blockSize:0} overlay_0-102:{mountpoint:/var/lib/containers/storage/overlay/bb07e79628246a351f4ea867fd612f7879ef2ccd79957316623734a55a7614d9/merged major:0 minor:102 fsType:overlay blockSize:0} overlay_0-104:{mountpoint:/var/lib/containers/storage/overlay/1be0e571273e78a030cd1f59f1579cea5ff1245a9c11696084a9b5c978d0e203/merged major:0 minor:104 fsType:overlay blockSize:0} overlay_0-106:{mountpoint:/var/lib/containers/storage/overlay/d223dd71f97c6538fb253712d3c3c0d8340a5ac340b0cc6fa20018254a657d00/merged major:0 minor:106 fsType:overlay blockSize:0} overlay_0-114:{mountpoint:/var/lib/containers/storage/overlay/b97e54753db169445d11776ed8866d9c469dced9d61fb96ce73ac835b0e07ec3/merged major:0 minor:114 fsType:overlay blockSize:0} overlay_0-119:{mountpoint:/var/lib/containers/storage/overlay/814793b29c417423d939eda2c9e24c9c91e5c6304f98e67883b005db1c5e1ea5/merged major:0 minor:119 fsType:overlay blockSize:0} overlay_0-124:{mountpoint:/var/lib/containers/storage/overlay/35f8f07142c88cef3a694808bbf62356ffac27fdb0b6044ba2355ca83d4eeece/merged major:0 minor:124 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/cce2edc2a6e1cc105313fdda9824399cc8b61482cc8741b1f20869927773030e/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-129:{mountpoint:/var/lib/containers/storage/overlay/87089b83e64274bded9a280cd6eddea8cf4cfd6733ca797eadc37dfc993e6ce2/merged major:0 minor:129 fsType:overlay blockSize:0} overlay_0-137:{mountpoint:/var/lib/containers/storage/overlay/499f3ce0c8f3e9f03d92bf24c52d3314ba9da8c8f23792d9efbaab8282343d5c/merged major:0 minor:137 fsType:overlay blockSize:0} overlay_0-139:{mountpoint:/var/lib/containers/storage/overlay/8f3eb8e2c85b45840823e0210a1e83564436163250e5a1ca14fae010b8bc3cbf/merged major:0 minor:139 fsType:overlay blockSize:0} overlay_0-147:{mountpoint:/var/lib/containers/storage/overlay/f7900ef7a7b1984a7a042d550e4f4493f531bc7ed3e0e0f60b8ccc98fb60c4d0/merged major:0 minor:147 fsType:overlay blockSize:0} overlay_0-152:{mountpoint:/var/lib/containers/storage/overlay/89e38c4be17bf356da3852ba01896733daa5aaef73a6e31a67acaa034e8c7a6f/merged major:0 minor:152 fsType:overlay blockSize:0} overlay_0-157:{mountpoint:/var/lib/containers/storage/overlay/331d51d9dd5d43a8b4617f608637fc44aca0095ffe26c8c19ee723ad5601de27/merged major:0 minor:157 fsType:overlay blockSize:0} overlay_0-162:{mountpoint:/var/lib/containers/storage/overlay/e5e1319a438652164d250c971aeaf5b3f57cfbd22ddac9da3b2552264b6e5cad/merged major:0 minor:162 fsType:overlay blockSize:0} overlay_0-167:{mountpoint:/var/lib/containers/storage/overlay/3f610bcdb99905371447abe3dbde29e85f0bf9233f48826fc0c598951d528aac/merged major:0 minor:167 fsType:overlay blockSize:0} overlay_0-172:{mountpoint:/var/lib/containers/storage/overlay/764f60864330bb0fc2f4db0cae0aef2cb07d1b46821b7487dbabdfd756107b42/merged major:0 minor:172 fsType:overlay blockSize:0} overlay_0-177:{mountpoint:/var/lib/containers/storage/overlay/11092ad11cf8577938398bbed653bc569b2143725bf74c221edc87bf8e6740c3/merged major:0 minor:177 fsType:overlay blockSize:0} overlay_0-182:{mountpoint:/var/lib/containers/storage/overlay/2ff0e4c9fc5c2435ee5afa60ade245cc6a9bbce376809fb126b0c92e0b784c36/merged major:0 minor:182 fsType:overlay blockSize:0} overlay_0-187:{mountpoint:/var/lib/containers/storage/overlay/fb7f7b9aca86469f7fbb52142cd36fd8f235da6b69ed384d4a6e761f65a6c69d/merged major:0 minor:187 fsType:overlay blockSize:0} overlay_0-192:{mountpoint:/var/lib/containers/storage/overlay/f97569e74381efe595126fcdcf002b2ebc00d8231d617fc252ad7132c9e0f947/merged major:0 minor:192 fsType:overlay blockSize:0} overlay_0-248:{mountpoint:/var/lib/containers/storage/overlay/b0fd5bc1a979da19877cc59acfe976d34aa28794aef770c8abbc828e0c13684b/merged major:0 minor:248 fsType:overlay blockSize:0} overlay_0-250:{mountpoint:/var/lib/containers/storage/overlay/30c38bbd3d02fbe815a1e2c7042fbb779b020022adfacdc8288048ab151e3b95/merged major:0 minor:250 fsType:overlay blockSize:0} overlay_0-252:{mountpoint:/var/lib/containers/storage/overlay/1c3a63d8e1d4b2847c975366662f5cf952576bb06e2f221791077c2d21a92c32/merged major:0 minor:252 fsType:overlay blockSize:0} overlay_0-254:{mountpoint:/var/lib/containers/storage/overlay/c5c9ee5969222ed2fe0221b1fc5270ed96bcfdcc8973f232449b95f6f2c8ecb8/merged major:0 minor:254 fsType:overlay blockSize:0} overlay_0-256:{mountpoint:/var/lib/containers/storage/overlay/8f160ee85d95a62159082338cca2f137981843dba406ea4e39f0b47115d8f64d/merged major:0 minor:256 fsType:overlay blockSize:0} overlay_0-258:{mountpoint:/var/lib/containers/storage/overlay/c91116890f4ee21dd710412850892c2702fcb05c58dfd1c20acbde5b8ddc0bd5/merged major:0 minor:258 fsType:overlay blockSize:0} overlay_0-260:{mountpoint:/var/lib/containers/storage/overlay/78a869a5e032b34f840efd426122924838731a72ff61f566bfa976a16fb26d9f/merged major:0 minor:260 fsType:overlay blockSize:0} overlay_0-262:{mountpoint:/var/lib/containers/storage/overlay/26cb2a4f20986dc9f636a5fed2c8385494c69e6b874647b5e5cd0b03fd032cfd/merged major:0 minor:262 fsType:overlay blockSize:0} overlay_0-264:{mountpoint:/var/lib/containers/storage/overlay/33690c5aeb0f16e3bdb764b79126bc17e38e8a2f4327d02bd6471bd83b6645c8/merged major:0 minor:264 fsType:overlay blockSize:0} overlay_0-266:{mountpoint:/var/lib/containers/storage/overlay/cf92419e27be7dec75b6739593b0feb11ccd7557311e8dc39d4d3dd63d625ba9/merged major:0 minor:266 fsType:overlay blockSize:0} overlay_0-280:{mountpoint:/var/lib/containers/storage/overlay/33b97f16f1b3ef692b788b05e9562c5dd1fd65e6063e7d0c35170626e8ff85a3/merged major:0 minor:280 fsType:overlay blockSize:0} overlay_0-282:{mountpoint:/var/lib/containers/storage/overlay/29b3860a84bcbf5c10e43e7d66b78b2d727ecb9b4fd4042a20b95ad3b210714e/merged major:0 minor:282 fsType:overlay blockSize:0} overlay_0-284:{mountpoint:/var/lib/containers/storage/overlay/aab58df97d8ed48b35e7817abc68935d8d58045dccd874babaa7b31689186417/merged major:0 minor:284 fsType:overlay blockSize:0} overlay_0-292:{mountpoint:/var/lib/containers/storage/overlay/ba5958e326ae5c11a154656358bda0c3367843366fbc3ae54b66c062b04ef2f2/merged major:0 minor:292 fsType:overlay blockSize:0} overlay_0-294:{mountpoint:/var/lib/containers/storage/overlay/b7a43a12e1ede581db8ea222bb5d3b12e4b78d8ed66d7a28e43b3050bc76140b/merged major:0 minor:294 fsType:overlay blockSize:0} overlay_0-296:{mountpoint:/var/lib/containers/storage/overlay/c0f9e4a0f7cc8eb6c88c6a49ffd78e9add6629e7360014e24304351d0e3e7444/merged major:0 minor:296 fsType:overlay blockSize:0} overlay_0-298:{mountpoint:/var/lib/containers/storage/overlay/26275ea0a5b2eb556a9c6231856639244b8942036602c436dd7b36b2d4f9e73d/merged major:0 minor:298 fsType:overlay blockSize:0} overlay_0-304:{mountpoint:/var/lib/containers/storage/overlay/5411913f4de187e8955d8c83d7199003c39402edb77d32b2bf21d2c8e253f955/merged major:0 minor:304 fsType:overlay blockSize:0} overlay_0-306:{mountpoint:/var/lib/containers/storage/overlay/ddb0a0ea3d7594330cd94ba0ed0254d36f18cb027b31700c51d38a19982b873d/merged major:0 minor:306 fsType:overlay blockSize:0} overlay_0-312:{mountpoint:/var/lib/containers/storage/overlay/dceb34da317778cc3e01199dd880e881f8aa7a8215f1679b05b13a45d5c65cb8/merged major:0 minor:312 fsType:overlay blockSize:0} overlay_0-327:{mountpoint:/var/lib/containers/storage/overlay/99dd83d2b96e5c499a97279f244e42273cd306e689c327ebc2c96643d07fec0c/merged major:0 minor:327 fsType:overlay blockSize:0} overlay_0-329:{mountpoint:/var/lib/containers/storage/overlay/21170f66673590a5042583e8620ff44c86bc1083822046a64927ee61003f75d9/merged major:0 minor:329 fsType:overlay blockSize:0} overlay_0-342:{mountpoint:/var/lib/containers/storage/overlay/e21ff67bb290b665b1ed24aece96958f7b1643dab4b6df3e73ff601a811c3f06/merged major:0 minor:342 fsType:overlay blockSize:0} overlay_0-344:{mountpoint:/var/lib/containers/storage/overlay/3da00bbdbd114d4b7423c89e8e4446220a6bf6b8c1f1430397a4af9576b86566/merged major:0 minor:344 fsType:overlay blockSize:0} overlay_0-66:{mountpoint:/var/lib/containers/storage/overlay/c062b83276b57be1bcbe1745aca64b08369b8039f586ebc65a896bb6c1072398/merged major:0 minor:66 fsType:overlay blockSize:0} overlay_0-68:{mountpoint:/var/lib/containers/storage/overlay/3e7abf51243144b98da2684bdd449b3599fb438c7d23f7b4f3a4e8e4832a4d3f/merged major:0 minor:68 fsType:overlay blockSize:0} overlay_0-70:{mountpoint:/var/lib/containers/storage/overlay/bb01107d10817ab1bba6d53c4d173fa420ae335b0f092182b1b1332094dd9f45/merged major:0 minor:70 fsType:overlay blockSize:0} overlay_0-75:{mountpoint:/var/lib/containers/storage/overlay/95285325f30d526f297a952e7e6bc6619fb535a231cec569f8918c7c4f8b8638/merged major:0 minor:75 fsType:overlay blockSize:0} overlay_0-78:{mountpoint:/var/lib/containers/storage/overlay/4e8382d6adc3d01ebb86df094cfa964b998a4f9b568ca05af65c0d5360c5c916/merged major:0 minor:78 fsType:overlay blockSize:0} overlay_0-82:{mountpoint:/var/lib/containers/storage/overlay/fa155312cf8a8765b8453ab7fa672737c280bdd56402280cd3e80ef720c35e46/merged major:0 minor:82 fsType:overlay blockSize:0} overlay_0-86:{mountpoint:/var/lib/containers/storage/overlay/8ffa9a39167337eb6cb7d9f4d26d4fb59f48ae5d80558f188292a88b1703f8d4/merged major:0 minor:86 fsType:overlay blockSize:0} overlay_0-90:{mountpoint:/var/lib/containers/storage/overlay/6cfc8dc5e9c33ce373469ec5776994f6184aaf27f6efde9ec5eb8ce2353830ae/merged major:0 minor:90 fsType:overlay blockSize:0} overlay_0-92:{mountpoint:/var/lib/containers/storage/overlay/b599a3c0760308f63cd715f63fd3ea766d0dffd13f337a66deeeae2a19839b30/merged major:0 minor:92 fsType:overlay blockSize:0} overlay_0-96:{mountpoint:/var/lib/containers/storage/overlay/8db1edb6f78a6ad75f5cc13ab26a8183fadd520729fd314c767d74e4560a30ad/merged major:0 minor:96 fsType:overlay blockSize:0}]
Oct 31 07:29:05.954729 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:05.952525    4720 manager.go:217] Machine: {Timestamp:2026-10-31 07:29:05.951676421 +0000 UTC m=+0.082744487 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3099758 MemoryCapacity:32812007424 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2e19ef47dea3198edfb2a67ef8bd76 SystemUUID:ec2e19ef-47de-a319-8edf-b2a67ef8bd76 BootID:faed6606-cdad-45f4-a07a-b407dea0d5e9 Filesystems:[{Device:/var/lib/kubelet/pods/28b59d93-b9ce-4ceb-9f3a-742525eb94c4/volumes/kubernetes.io~projected/bound-sa-token DeviceMajor:0 DeviceMinor:221 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/4b442c7c5f25e692aee5d663f2bd1e5c0c7168cd799aab781bfed2102e3740f1/userdata/shm DeviceMajor:0 DeviceMinor:231 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-250 DeviceMajor:0 DeviceMinor:250 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-82 DeviceMajor:0 DeviceMinor:82 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-102 DeviceMajor:0 DeviceMinor:102 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-172 DeviceMajor:0 DeviceMinor:172 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/7a306c5d41908aad98edab19712d66d4e434fca13d0fb3b8c8f78ed55689cab5/userdata/shm DeviceMajor:0 DeviceMinor:288 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/ecb08d77ee39577a396f19d406280850735fe9412808b0951b5d432f6275f230/userdata/shm DeviceMajor:0 DeviceMinor:84 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-139 DeviceMajor:0 DeviceMinor:139 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/a73024cb-f581-450d-a09c-0565838c5baa/volumes/kubernetes.io~projected/kube-api-access-m6l7n DeviceMajor:0 DeviceMinor:239 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-266 DeviceMajor:0 DeviceMinor:266 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-284 DeviceMajor:0 DeviceMinor:284 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-327 DeviceMajor:0 DeviceMinor:327 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/9f2589536197b37032285af9622b146bd14cd852452e464e8eb73c0a610e5078/userdata/shm DeviceMajor:0 DeviceMinor:60 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-344 DeviceMajor:0 DeviceMinor:344 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-162 DeviceMajor:0 DeviceMinor:162 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:overlay_0-75 DeviceMajor:0 DeviceMinor:75 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/7dab47a696030acdb1ddfc6fe11ed4c83ad1726d144ad583b503091bc2bb5a62/userdata/shm DeviceMajor:0 DeviceMinor:219 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/b1db0e3b-e85d-4769-8012-cf4b6f740d9d/volumes/kubernetes.io~projected/kube-api-access-czqtq DeviceMajor:0 DeviceMinor:234 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/cb9e4575-9265-43b5-bf02-7843611d6563/volumes/kubernetes.io~projected/kube-api-access-kr9wm DeviceMajor:0 DeviceMinor:236 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-254 DeviceMajor:0 DeviceMinor:254 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/f1ba68f6-d87c-4492-a5eb-5f2d3a60eb1c/volumes/kubernetes.io~projected/kube-api-access-ghkb9 DeviceMajor:0 DeviceMinor:287 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/7c6cf74c69f7fceea5860f8c2894b19081b8a4f44528c01cdf3a23bc22f567b5/userdata/shm DeviceMajor:0 DeviceMinor:289 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:47 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-182 DeviceMajor:0 DeviceMinor:182 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/4db4c33f-0c58-40f8-bf74-51b3eae3c9db/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:213 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/28b59d93-b9ce-4ceb-9f3a-742525eb94c4/volumes/kubernetes.io~projected/kube-api-access-xvv26 DeviceMajor:0 DeviceMinor:226 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/6a2e938ce4315b140940dad931d9f3c7bb967a6dfafd01ff57780c53f7df30a7/userdata/shm DeviceMajor:0 DeviceMinor:240 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/c3ad95fd-e997-459c-93fd-27c4c030ea0d/volumes/kubernetes.io~projected/kube-api-access-kxwb9 DeviceMajor:0 DeviceMinor:59 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-167 DeviceMajor:0 DeviceMinor:167 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/81829763-27b1-4a48-a568-cd5f6a39a953/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:45 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/ee4a4a2a-7439-4c78-8858-d6ba0085fdde/volumes/kubernetes.io~projected/kube-api-access-q7vwq DeviceMajor:0 DeviceMinor:203 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-306 DeviceMajor:0 DeviceMinor:306 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-329 DeviceMajor:0 DeviceMinor:329 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-114 DeviceMajor:0 DeviceMinor:114 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/993cfed6-806d-44c5-bba8-c02203638fe7/volumes/kubernetes.io~secret/image-registry-private-configuration DeviceMajor:0 DeviceMinor:209 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-342 DeviceMajor:0 DeviceMinor:342 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/44c067aa8f06e75a4a32430911035f32efde1168203a54d7abd7aa733b407210/userdata/shm DeviceMajor:0 DeviceMinor:242 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-298 DeviceMajor:0 DeviceMinor:298 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/9dad8dd1-cfc5-412d-bf9a-bac3c9162af1/volumes/kubernetes.io~projected/kube-api-access-5n5lv DeviceMajor:0 DeviceMinor:53 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/c1ffc96f411310ca3c9bbb972b3c1df1fe0467e4564c58e844cd9b804d40194c/userdata/shm DeviceMajor:0 DeviceMinor:98 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-177 DeviceMajor:0 DeviceMinor:177 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/cb9e4575-9265-43b5-bf02-7843611d6563/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:229 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-129 DeviceMajor:0 DeviceMinor:129 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/789f11d8-d951-4144-a589-a2bdbf6844e3/volumes/kubernetes.io~projected/kube-api-access-wqmxh DeviceMajor:0 DeviceMinor:225 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-256 DeviceMajor:0 DeviceMinor:256 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-262 DeviceMajor:0 DeviceMinor:262 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/3fa078c7-4bf2-4e76-9692-ae460e8fadfa/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:48 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/08198bc1-314f-4881-833c-e633bac15e4a/volumes/kubernetes.io~projected/kube-api-access-xs9lt DeviceMajor:0 DeviceMinor:54 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/e5d2dc100f3cacdb69fc74b690c1bbe741b4298bd41697263a403392aa95b8b8/userdata/shm DeviceMajor:0 DeviceMinor:62 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-78 DeviceMajor:0 DeviceMinor:78 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/993cfed6-806d-44c5-bba8-c02203638fe7/volumes/kubernetes.io~projected/bound-sa-token DeviceMajor:0 DeviceMinor:227 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/4b27a1e9df141368f9aec55f507c535d7c7a4e98dd0d847f22a930188408d21e/userdata/shm DeviceMajor:0 DeviceMinor:244 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/0b25f791-3520-481d-acbb-a3bef222fdac/volumes/kubernetes.io~projected/kube-api-access-nhphg DeviceMajor:0 DeviceMinor:335 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/3fa078c7-4bf2-4e76-9692-ae460e8fadfa/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:50 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/57c7cad37211feee315413dd85f61718ce6291d6c4be0727f145731331c93e71/userdata/shm DeviceMajor:0 DeviceMinor:340 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:overlay_0-90 DeviceMajor:0 DeviceMinor:90 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-100 DeviceMajor:0 DeviceMinor:100 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-187 DeviceMajor:0 DeviceMinor:187 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/b1066ece-e6f3-4491-aa1f-a7b9a3aa248d/volumes/kubernetes.io~projected/kube-api-access-cjt6r DeviceMajor:0 DeviceMinor:322 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6562402304 Type:vfs Inodes:819200 HasInodes:true} {Device:/var/lib/kubelet/pods/3fa078c7-4bf2-4e76-9692-ae460e8fadfa/volumes/kubernetes.io~projected/kube-api-access-cq7rv DeviceMajor:0 DeviceMinor:51 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/4bfbebfd-2be7-438f-b88d-273f24751f8c/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6562402304 Type:vfs Inodes:819200 HasInodes:true} {Device:/var/lib/kubelet/pods/993cfed6-806d-44c5-bba8-c02203638fe7/volumes/kubernetes.io~projected/kube-api-access-v2zhk DeviceMajor:0 DeviceMinor:224 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-252 DeviceMajor:0 DeviceMinor:252 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/26659955-3909-4171-90b8-21b0ad6eef64/volumes/kubernetes.io~projected/kube-api-access-jbdqx DeviceMajor:0 DeviceMinor:57 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/82a11c7ec21dcf611cf9180470eb718c94c4332da4e8f2b47dcf174499bada7a/userdata/shm DeviceMajor:0 DeviceMinor:72 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/91363236-08db-4f54-ae0a-e68604fcecc6/volumes/kubernetes.io~projected/kube-api-access-sn2lx DeviceMajor:0 DeviceMinor:201 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/993cfed6-806d-44c5-bba8-c02203638fe7/volumes/kubernetes.io~secret/installation-pull-secrets DeviceMajor:0 DeviceMinor:208 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/b1db0e3b-e85d-4769-8012-cf4b6f740d9d/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:230 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-304 DeviceMajor:0 DeviceMinor:304 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-106 DeviceMajor:0 DeviceMinor:106 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/b44ed0e0-a72e-4746-a7de-ed8610da845d/volumes/kubernetes.io~secret/default-certificate DeviceMajor:0 DeviceMinor:210 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/28b59d93-b9ce-4ceb-9f3a-742525eb94c4/volumes/kubernetes.io~secret/image-registry-private-configuration DeviceMajor:0 DeviceMinor:214 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/4bfbebfd-2be7-438f-b88d-273f24751f8c/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:49 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/c4233c69-4213-43be-8809-76612b6b6f3e/volumes/kubernetes.io~projected/kube-api-access-5hqbq DeviceMajor:0 DeviceMinor:55 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/631cff9e-a1f1-4dd2-bdc8-ef36519c7c1b/volumes/kubernetes.io~projected/kube-api-access-dx598 DeviceMajor:0 DeviceMinor:204 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/b44ed0e0-a72e-4746-a7de-ed8610da845d/volumes/kubernetes.io~secret/stats-auth DeviceMajor:0 DeviceMinor:211 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/4db4c33f-0c58-40f8-bf74-51b3eae3c9db/volumes/kubernetes.io~projected/kube-api-access-q2kvz DeviceMajor:0 DeviceMinor:223 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/81829763-27b1-4a48-a568-cd5f6a39a953/volumes/kubernetes.io~projected/kube-api-access-k6576 DeviceMajor:0 DeviceMinor:205 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/28b59d93-b9ce-4ceb-9f3a-742525eb94c4/volumes/kubernetes.io~secret/installation-pull-secrets DeviceMajor:0 DeviceMinor:212 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/2536c1c8-8a46-4af3-bc6d-81454a908d3f/volumes/kubernetes.io~secret/cert DeviceMajor:0 DeviceMinor:228 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-264 DeviceMajor:0 DeviceMinor:264 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-292 DeviceMajor:0 DeviceMinor:292 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-66 DeviceMajor:0 DeviceMinor:66 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/ce8cac52-291b-47fb-a7b6-4cf7562449b9/volumes/kubernetes.io~projected/kube-api-access-bfzv8 DeviceMajor:0 DeviceMinor:233 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/2536c1c8-8a46-4af3-bc6d-81454a908d3f/volumes/kubernetes.io~projected/kube-api-access-g2r66 DeviceMajor:0 DeviceMinor:235 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-280 DeviceMajor:0 DeviceMinor:280 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-296 DeviceMajor:0 DeviceMinor:296 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16406003712 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-70 DeviceMajor:0 DeviceMinor:70 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-96 DeviceMajor:0 DeviceMinor:96 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/24977ca031ccca9556605789709d856b16771b92e2cd05e39a8df96eb7e98438/userdata/shm DeviceMajor:0 DeviceMinor:217 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-260 DeviceMajor:0 DeviceMinor:260 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/5292ef84738cab81213f85cb2c9188469f27c00aa7758fb95524fa5c53ddac15/userdata/shm DeviceMajor:0 DeviceMinor:74 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-86 DeviceMajor:0 DeviceMinor:86 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/7df3a2737e0ecbaf7eb754ef7539c98a3a1aac973f357f73d3c8d852b7d2312f/userdata/shm DeviceMajor:0 DeviceMinor:88 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-192 DeviceMajor:0 DeviceMinor:192 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/26d38dd238ca56039d3f30825038e78a1478f38371827bb224e5be4d97b94b28/userdata/shm DeviceMajor:0 DeviceMinor:246 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-248 DeviceMajor:0 DeviceMinor:248 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-282 DeviceMajor:0 DeviceMinor:282 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/77256436-39ff-4c2a-8390-4e9fa7d68486/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:286 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-312 DeviceMajor:0 DeviceMinor:312 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/69600ee490f34bf474850cf25f80679e6519843dc40baf37c5d57342b0cec097/userdata/shm DeviceMajor:0 DeviceMinor:80 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/c8d95e23ad7960db9871e1b75389bf92563cfef5ab50bfece8472084ee3133ba/userdata/shm DeviceMajor:0 DeviceMinor:94 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-119 DeviceMajor:0 DeviceMinor:119 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/323c8334-9a56-4c4e-8a6e-ae80995810d0/volumes/kubernetes.io~projected/kube-api-access-l4nfd DeviceMajor:0 DeviceMinor:202 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-258 DeviceMajor:0 DeviceMinor:258 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/37019c645339bd463a7dee60255920849f38a55d3c31eda8755469f8ada74ba0/userdata/shm DeviceMajor:0 DeviceMinor:61 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-137 DeviceMajor:0 DeviceMinor:137 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-147 DeviceMajor:0 DeviceMinor:147 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-152 DeviceMajor:0 DeviceMinor:152 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/397e8b2b367abe48f39d3961ba7ce8bb1ebfd59819edf275333c202df43087e2/userdata/shm DeviceMajor:0 DeviceMinor:206 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/90033c9a0ade2d06af05ffaab30cc49f21be3df3b48302c519fd8d263f9886af/userdata/shm DeviceMajor:0 DeviceMinor:237 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-294 DeviceMajor:0 DeviceMinor:294 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/ca31519e-aa90-4104-a175-f9af2b3bbeb4/volumes/kubernetes.io~projected/kube-api-access-n6p25 DeviceMajor:0 DeviceMinor:197 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/fc64d31a-f48a-4293-a4de-262045d0b7e8/volumes/kubernetes.io~projected/kube-api-access-kk4nh DeviceMajor:0 DeviceMinor:52 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/4bfbebfd-2be7-438f-b88d-273f24751f8c/volumes/kubernetes.io~projected/kube-api-access-cz8tm DeviceMajor:0 DeviceMinor:56 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-68 DeviceMajor:0 DeviceMinor:68 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-92 DeviceMajor:0 DeviceMinor:92 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-124 DeviceMajor:0 DeviceMinor:124 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-157 DeviceMajor:0 DeviceMinor:157 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16406003712 Type:vfs Inodes:1048576 HasInodes:true} {Device:/var/lib/kubelet/pods/9c72279a-708f-46df-8915-9995c5aea14d/volumes/kubernetes.io~projected/kube-api-access-h4gzh DeviceMajor:0 DeviceMinor:58 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-104 DeviceMajor:0 DeviceMinor:104 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/6d2a6480d09c4deb2fe4f31aa76ca7365b7305ee8d41ba48d3831a072eab9b1d/userdata/shm DeviceMajor:0 DeviceMinor:215 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/b44ed0e0-a72e-4746-a7de-ed8610da845d/volumes/kubernetes.io~projected/kube-api-access-9pvp2 DeviceMajor:0 DeviceMinor:222 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:24977ca031ccca9 MacAddress:82:db:20:5c:75:df Speed:10000 Mtu:8901} {Name:26d38dd238ca560 MacAddress:0e:d6:0c:f2:e6:d2 Speed:10000 Mtu:8901} {Name:397e8b2b367abe4 MacAddress:ee:a9:36:28:25:61 Speed:10000 Mtu:8901} {Name:44c067aa8f06e75 MacAddress:1a:a4:78:a0:33:89 Speed:10000 Mtu:8901} {Name:4b27a1e9df14136 MacAddress:3a:59:15:76:e2:5f Speed:10000 Mtu:8901} {Name:4b442c7c5f25e69 MacAddress:76:4e:85:71:aa:a3 Speed:10000 Mtu:8901} {Name:57c7cad37211fee MacAddress:fa:c2:33:2c:58:2b Speed:10000 Mtu:8901} {Name:6a2e938ce4315b1 MacAddress:16:83:ee:b6:cb:a8 Speed:10000 Mtu:8901} {Name:6d2a6480d09c4de MacAddress:82:b8:07:9e:51:da Speed:10000 Mtu:8901} {Name:7a306c5d41908aa MacAddress:ea:13:db:4e:9f:96 Speed:10000 Mtu:8901} {Name:7c6cf74c69f7fce MacAddress:0a:4f:f6:2e:d9:17 Speed:10000 Mtu:8901} {Name:7dab47a696030ac MacAddress:f6:81:39:10:95:e6 Speed:10000 Mtu:8901} {Name:90033c9a0ade2d0 MacAddress:92:0d:50:c1:ff:b6 Speed:10000 Mtu:8901} {Name:br-ex MacAddress:02:28:50:23:a4:5b Speed:0 Mtu:9001} {Name:br-int MacAddress:da:0d:38:fd:fb:40 Speed:0 Mtu:8901} {Name:ens5 MacAddress:02:28:50:23:a4:5b Speed:0 Mtu:9001} {Name:genev_sys_6081 MacAddress:c2:e6:45:7a:d3:b9 Speed:0 Mtu:65000} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:84:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:8e:74:45:93:74:17 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812007424 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Oct 31 07:29:05.954729 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:05.953120    4720 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Oct 31 07:29:05.954729 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:05.953252    4720 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Oct 31 07:29:05.954729 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:05.953610    4720 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Oct 31 07:29:05.954729 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:05.953627    4720 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-139-83.us-west-1.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Oct 31 07:29:05.954729 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:05.953759    4720 topology_manager.go:143] "Creating topology manager with none policy"
Oct 31 07:29:05.954729 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:05.953770    4720 container_manager_linux.go:311] "Creating device plugin manager"
Oct 31 07:29:05.954729 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:05.953788    4720 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 31 07:29:05.954729 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:05.953811    4720 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 31 07:29:05.954729 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:05.953873    4720 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager"
Oct 31 07:29:05.954729 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:05.954055    4720 state_mem.go:41] "Initialized" logger="CPUManager state memory"
Oct 31 07:29:05.954729 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:05.954179    4720 server.go:1235] "Using root directory" path="/var/lib/kubelet"
Oct 31 07:29:05.954729 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:05.954217    4720 kubelet.go:493] "Attempting to sync node with API server"
Oct 31 07:29:05.954729 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:05.954231    4720 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests"
Oct 31 07:29:05.954729 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:05.954241    4720 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Oct 31 07:29:05.954729 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:05.954253    4720 kubelet.go:405] "Adding apiserver pod source"
Oct 31 07:29:05.954729 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:05.954265    4720 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Oct 31 07:29:05.955783 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:05.955108    4720 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint"
Oct 31 07:29:05.955783 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:05.955120    4720 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 31 07:29:05.955783 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:05.955655    4720 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1"
Oct 31 07:29:05.971923 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:05.971900    4720 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint"
Oct 31 07:29:05.972026 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:05.971929    4720 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 31 07:29:05.972099 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:05.972082    4720 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem"
Oct 31 07:29:05.972440 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:05.972403    4720 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Oct 31 07:29:05.972535 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:05.972459    4720 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Oct 31 07:29:05.972588 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:05.972575    4720 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Oct 31 07:29:05.972636 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:05.972591    4720 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Oct 31 07:29:05.972636 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:05.972601    4720 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Oct 31 07:29:05.972636 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:05.972616    4720 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Oct 31 07:29:05.972636 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:05.972624    4720 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Oct 31 07:29:05.972636 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:05.972634    4720 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Oct 31 07:29:05.972844 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:05.972643    4720 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Oct 31 07:29:05.972844 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:05.972652    4720 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Oct 31 07:29:05.972844 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:05.972689    4720 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Oct 31 07:29:05.972844 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:05.972699    4720 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Oct 31 07:29:05.972844 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:05.972711    4720 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Oct 31 07:29:05.972844 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:05.972727    4720 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Oct 31 07:29:05.972844 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:05.972758    4720 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Oct 31 07:29:05.972844 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:05.972767    4720 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Oct 31 07:29:05.977058 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:05.977041    4720 server.go:1263] "Started kubelet"
Oct 31 07:29:05.977181 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:05.977140    4720 server.go:182] "Starting to listen" address="0.0.0.0" port=10250
Oct 31 07:29:05.977267 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:05.977126    4720 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Oct 31 07:29:05.977359 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:05.977300    4720 server_v1.go:49] "podresources" method="list" useActivePods=true
Oct 31 07:29:05.977798 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:05.977773    4720 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Oct 31 07:29:05.978036 ip-10-0-139-83 systemd[1]: Started Kubernetes Kubelet.
Oct 31 07:29:05.982364 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:05.980671    4720 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer"
Oct 31 07:29:05.982364 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:05.982092    4720 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Oct 31 07:29:05.983490 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:05.983403    4720 volume_manager.go:309] "The desired_state_of_world populator starts"
Oct 31 07:29:05.983599 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:05.983514    4720 volume_manager.go:311] "Starting Kubelet Volume Manager"
Oct 31 07:29:05.984121 ip-10-0-139-83 kubenswrapper[4720]: E1031 07:29:05.984101    4720 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-139-83.us-west-1.compute.internal\" not found"
Oct 31 07:29:05.984219 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:05.984181    4720 server.go:317] "Adding debug handlers to kubelet server"
Oct 31 07:29:05.984298 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:05.984281    4720 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Oct 31 07:29:05.985060 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:05.984211    4720 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-30 07:23:25 +0000 UTC" deadline="2028-05-09 13:21:24.521336871 +0000 UTC"
Oct 31 07:29:05.985153 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:05.985065    4720 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13349h52m18.536278845s"
Oct 31 07:29:05.986937 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:05.986900    4720 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ee4a4a2a-7439-4c78-8858-d6ba0085fdde" volumeName="kubernetes.io/projected/ee4a4a2a-7439-4c78-8858-d6ba0085fdde-kube-api-access-q7vwq" seLinuxMountContext=""
Oct 31 07:29:05.987014 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:05.986942    4720 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f1ba68f6-d87c-4492-a5eb-5f2d3a60eb1c" volumeName="kubernetes.io/projected/f1ba68f6-d87c-4492-a5eb-5f2d3a60eb1c-kube-api-access-ghkb9" seLinuxMountContext=""
Oct 31 07:29:05.987014 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:05.986957    4720 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="28b59d93-b9ce-4ceb-9f3a-742525eb94c4" volumeName="kubernetes.io/configmap/28b59d93-b9ce-4ceb-9f3a-742525eb94c4-registry-certificates" seLinuxMountContext=""
Oct 31 07:29:05.987014 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:05.986973    4720 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4db4c33f-0c58-40f8-bf74-51b3eae3c9db" volumeName="kubernetes.io/projected/4db4c33f-0c58-40f8-bf74-51b3eae3c9db-kube-api-access-q2kvz" seLinuxMountContext=""
Oct 31 07:29:05.987014 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:05.986984    4720 factory.go:55] Registering systemd factory
Oct 31 07:29:05.987014 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:05.986997    4720 factory.go:223] Registration of the systemd container factory successfully
Oct 31 07:29:05.987188 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:05.986986    4720 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="81829763-27b1-4a48-a568-cd5f6a39a953" volumeName="kubernetes.io/empty-dir/81829763-27b1-4a48-a568-cd5f6a39a953-snapshots" seLinuxMountContext=""
Oct 31 07:29:05.987188 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:05.987051    4720 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="993cfed6-806d-44c5-bba8-c02203638fe7" volumeName="kubernetes.io/configmap/993cfed6-806d-44c5-bba8-c02203638fe7-trusted-ca" seLinuxMountContext=""
Oct 31 07:29:05.987188 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:05.987070    4720 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b1066ece-e6f3-4491-aa1f-a7b9a3aa248d" volumeName="kubernetes.io/projected/b1066ece-e6f3-4491-aa1f-a7b9a3aa248d-kube-api-access-cjt6r" seLinuxMountContext=""
Oct 31 07:29:05.987188 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:05.987085    4720 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b44ed0e0-a72e-4746-a7de-ed8610da845d" volumeName="kubernetes.io/projected/b44ed0e0-a72e-4746-a7de-ed8610da845d-kube-api-access-9pvp2" seLinuxMountContext=""
Oct 31 07:29:05.987188 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:05.987098    4720 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b44ed0e0-a72e-4746-a7de-ed8610da845d" volumeName="kubernetes.io/secret/b44ed0e0-a72e-4746-a7de-ed8610da845d-default-certificate" seLinuxMountContext=""
Oct 31 07:29:05.987188 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:05.987108    4720 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c4233c69-4213-43be-8809-76612b6b6f3e" volumeName="kubernetes.io/empty-dir/c4233c69-4213-43be-8809-76612b6b6f3e-tmp-dir" seLinuxMountContext=""
Oct 31 07:29:05.987188 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:05.987148    4720 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3fa078c7-4bf2-4e76-9692-ae460e8fadfa" volumeName="kubernetes.io/projected/3fa078c7-4bf2-4e76-9692-ae460e8fadfa-kube-api-access-cq7rv" seLinuxMountContext=""
Oct 31 07:29:05.989062 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:05.989027    4720 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c4233c69-4213-43be-8809-76612b6b6f3e" volumeName="kubernetes.io/projected/c4233c69-4213-43be-8809-76612b6b6f3e-kube-api-access-5hqbq" seLinuxMountContext=""
Oct 31 07:29:05.989140 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:05.989061    4720 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cb9e4575-9265-43b5-bf02-7843611d6563" volumeName="kubernetes.io/secret/cb9e4575-9265-43b5-bf02-7843611d6563-serving-cert" seLinuxMountContext=""
Oct 31 07:29:05.989140 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:05.989100    4720 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4bfbebfd-2be7-438f-b88d-273f24751f8c" volumeName="kubernetes.io/configmap/4bfbebfd-2be7-438f-b88d-273f24751f8c-ovnkube-config" seLinuxMountContext=""
Oct 31 07:29:05.989140 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:05.989121    4720 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="631cff9e-a1f1-4dd2-bdc8-ef36519c7c1b" volumeName="kubernetes.io/projected/631cff9e-a1f1-4dd2-bdc8-ef36519c7c1b-kube-api-access-dx598" seLinuxMountContext=""
Oct 31 07:29:05.989234 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:05.989169    4720 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ca31519e-aa90-4104-a175-f9af2b3bbeb4" volumeName="kubernetes.io/configmap/ca31519e-aa90-4104-a175-f9af2b3bbeb4-samples-operator-config" seLinuxMountContext=""
Oct 31 07:29:05.989234 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:05.989184    4720 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cb9e4575-9265-43b5-bf02-7843611d6563" volumeName="kubernetes.io/projected/cb9e4575-9265-43b5-bf02-7843611d6563-kube-api-access-kr9wm" seLinuxMountContext=""
Oct 31 07:29:05.989234 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:05.989200    4720 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="08198bc1-314f-4881-833c-e633bac15e4a" volumeName="kubernetes.io/configmap/08198bc1-314f-4881-833c-e633bac15e4a-cni-binary-copy" seLinuxMountContext=""
Oct 31 07:29:05.989234 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:05.989220    4720 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="81829763-27b1-4a48-a568-cd5f6a39a953" volumeName="kubernetes.io/empty-dir/81829763-27b1-4a48-a568-cd5f6a39a953-tmp" seLinuxMountContext=""
Oct 31 07:29:05.989451 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:05.989233    4720 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="91363236-08db-4f54-ae0a-e68604fcecc6" volumeName="kubernetes.io/projected/91363236-08db-4f54-ae0a-e68604fcecc6-kube-api-access-sn2lx" seLinuxMountContext=""
Oct 31 07:29:05.989451 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:05.989253    4720 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9c72279a-708f-46df-8915-9995c5aea14d" volumeName="kubernetes.io/configmap/9c72279a-708f-46df-8915-9995c5aea14d-cni-sysctl-allowlist" seLinuxMountContext=""
Oct 31 07:29:05.989451 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:05.989266    4720 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="08198bc1-314f-4881-833c-e633bac15e4a" volumeName="kubernetes.io/configmap/08198bc1-314f-4881-833c-e633bac15e4a-multus-daemon-config" seLinuxMountContext=""
Oct 31 07:29:05.989451 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:05.989279    4720 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="28b59d93-b9ce-4ceb-9f3a-742525eb94c4" volumeName="kubernetes.io/configmap/28b59d93-b9ce-4ceb-9f3a-742525eb94c4-trusted-ca" seLinuxMountContext=""
Oct 31 07:29:05.989451 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:05.989305    4720 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9c72279a-708f-46df-8915-9995c5aea14d" volumeName="kubernetes.io/configmap/9c72279a-708f-46df-8915-9995c5aea14d-whereabouts-flatfile-configmap" seLinuxMountContext=""
Oct 31 07:29:05.989662 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:05.989646    4720 factory.go:153] Registering CRI-O factory
Oct 31 07:29:05.989757 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:05.989664    4720 factory.go:223] Registration of the crio container factory successfully
Oct 31 07:29:05.989757 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:05.989743    4720 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Oct 31 07:29:05.989826 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:05.989762    4720 factory.go:103] Registering Raw factory
Oct 31 07:29:05.989857 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:05.989840    4720 manager.go:1199] Started watching for new ooms in manager
Oct 31 07:29:05.990510 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:05.990495    4720 manager.go:319] Starting recovery of all containers
Oct 31 07:29:05.993354 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:05.989322    4720 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b1db0e3b-e85d-4769-8012-cf4b6f740d9d" volumeName="kubernetes.io/secret/b1db0e3b-e85d-4769-8012-cf4b6f740d9d-serving-cert" seLinuxMountContext=""
Oct 31 07:29:05.993354 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:05.992845    4720 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cb9e4575-9265-43b5-bf02-7843611d6563" volumeName="kubernetes.io/configmap/cb9e4575-9265-43b5-bf02-7843611d6563-config" seLinuxMountContext=""
Oct 31 07:29:05.994095 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:05.993882    4720 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="314c29aa-db6b-4f7f-86b4-c8952ce09c84" volumeName="kubernetes.io/configmap/314c29aa-db6b-4f7f-86b4-c8952ce09c84-nginx-conf" seLinuxMountContext=""
Oct 31 07:29:05.994095 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:05.993915    4720 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4bfbebfd-2be7-438f-b88d-273f24751f8c" volumeName="kubernetes.io/configmap/4bfbebfd-2be7-438f-b88d-273f24751f8c-ovnkube-script-lib" seLinuxMountContext=""
Oct 31 07:29:05.994095 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:05.993937    4720 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4db4c33f-0c58-40f8-bf74-51b3eae3c9db" volumeName="kubernetes.io/configmap/4db4c33f-0c58-40f8-bf74-51b3eae3c9db-config" seLinuxMountContext=""
Oct 31 07:29:05.994095 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:05.993949    4720 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="81829763-27b1-4a48-a568-cd5f6a39a953" volumeName="kubernetes.io/configmap/81829763-27b1-4a48-a568-cd5f6a39a953-trusted-ca-bundle" seLinuxMountContext=""
Oct 31 07:29:05.994095 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:05.993963    4720 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="81829763-27b1-4a48-a568-cd5f6a39a953" volumeName="kubernetes.io/projected/81829763-27b1-4a48-a568-cd5f6a39a953-kube-api-access-k6576" seLinuxMountContext=""
Oct 31 07:29:05.994095 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:05.993974    4720 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="993cfed6-806d-44c5-bba8-c02203638fe7" volumeName="kubernetes.io/secret/993cfed6-806d-44c5-bba8-c02203638fe7-installation-pull-secrets" seLinuxMountContext=""
Oct 31 07:29:05.994095 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:05.993989    4720 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="81829763-27b1-4a48-a568-cd5f6a39a953" volumeName="kubernetes.io/configmap/81829763-27b1-4a48-a568-cd5f6a39a953-service-ca-bundle" seLinuxMountContext=""
Oct 31 07:29:05.994095 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:05.994001    4720 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a73024cb-f581-450d-a09c-0565838c5baa" volumeName="kubernetes.io/configmap/a73024cb-f581-450d-a09c-0565838c5baa-config-volume" seLinuxMountContext=""
Oct 31 07:29:05.994095 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:05.994011    4720 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b1db0e3b-e85d-4769-8012-cf4b6f740d9d" volumeName="kubernetes.io/configmap/b1db0e3b-e85d-4769-8012-cf4b6f740d9d-trusted-ca" seLinuxMountContext=""
Oct 31 07:29:05.994095 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:05.994024    4720 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3fa078c7-4bf2-4e76-9692-ae460e8fadfa" volumeName="kubernetes.io/empty-dir/3fa078c7-4bf2-4e76-9692-ae460e8fadfa-tmp" seLinuxMountContext=""
Oct 31 07:29:05.994095 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:05.994037    4720 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="77256436-39ff-4c2a-8390-4e9fa7d68486" volumeName="kubernetes.io/secret/77256436-39ff-4c2a-8390-4e9fa7d68486-original-pull-secret" seLinuxMountContext=""
Oct 31 07:29:05.994095 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:05.994048    4720 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="789f11d8-d951-4144-a589-a2bdbf6844e3" volumeName="kubernetes.io/configmap/789f11d8-d951-4144-a589-a2bdbf6844e3-telemetry-config" seLinuxMountContext=""
Oct 31 07:29:05.994095 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:05.994061    4720 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9c72279a-708f-46df-8915-9995c5aea14d" volumeName="kubernetes.io/configmap/9c72279a-708f-46df-8915-9995c5aea14d-cni-binary-copy" seLinuxMountContext=""
Oct 31 07:29:05.994095 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:05.994073    4720 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b1db0e3b-e85d-4769-8012-cf4b6f740d9d" volumeName="kubernetes.io/configmap/b1db0e3b-e85d-4769-8012-cf4b6f740d9d-config" seLinuxMountContext=""
Oct 31 07:29:05.994095 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:05.994085    4720 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="26659955-3909-4171-90b8-21b0ad6eef64" volumeName="kubernetes.io/configmap/26659955-3909-4171-90b8-21b0ad6eef64-iptables-alerter-script" seLinuxMountContext=""
Oct 31 07:29:05.994095 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:05.994096    4720 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="28b59d93-b9ce-4ceb-9f3a-742525eb94c4" volumeName="kubernetes.io/empty-dir/28b59d93-b9ce-4ceb-9f3a-742525eb94c4-ca-trust-extracted" seLinuxMountContext=""
Oct 31 07:29:05.994095 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:05.994108    4720 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9c72279a-708f-46df-8915-9995c5aea14d" volumeName="kubernetes.io/projected/9c72279a-708f-46df-8915-9995c5aea14d-kube-api-access-h4gzh" seLinuxMountContext=""
Oct 31 07:29:05.994095 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:05.994119    4720 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a73024cb-f581-450d-a09c-0565838c5baa" volumeName="kubernetes.io/empty-dir/a73024cb-f581-450d-a09c-0565838c5baa-tmp-dir" seLinuxMountContext=""
Oct 31 07:29:05.994095 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:05.994132    4720 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b1066ece-e6f3-4491-aa1f-a7b9a3aa248d" volumeName="kubernetes.io/empty-dir/b1066ece-e6f3-4491-aa1f-a7b9a3aa248d-data-volume" seLinuxMountContext=""
Oct 31 07:29:05.994095 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:05.994144    4720 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b1db0e3b-e85d-4769-8012-cf4b6f740d9d" volumeName="kubernetes.io/projected/b1db0e3b-e85d-4769-8012-cf4b6f740d9d-kube-api-access-czqtq" seLinuxMountContext=""
Oct 31 07:29:05.994095 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:05.994155    4720 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fc64d31a-f48a-4293-a4de-262045d0b7e8" volumeName="kubernetes.io/projected/fc64d31a-f48a-4293-a4de-262045d0b7e8-kube-api-access-kk4nh" seLinuxMountContext=""
Oct 31 07:29:05.994095 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:05.994169    4720 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="08198bc1-314f-4881-833c-e633bac15e4a" volumeName="kubernetes.io/projected/08198bc1-314f-4881-833c-e633bac15e4a-kube-api-access-xs9lt" seLinuxMountContext=""
Oct 31 07:29:05.995229 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:05.994180    4720 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3fa078c7-4bf2-4e76-9692-ae460e8fadfa" volumeName="kubernetes.io/empty-dir/3fa078c7-4bf2-4e76-9692-ae460e8fadfa-etc-tuned" seLinuxMountContext=""
Oct 31 07:29:05.995229 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:05.994192    4720 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="993cfed6-806d-44c5-bba8-c02203638fe7" volumeName="kubernetes.io/configmap/993cfed6-806d-44c5-bba8-c02203638fe7-registry-certificates" seLinuxMountContext=""
Oct 31 07:29:05.995229 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:05.994218    4720 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9dad8dd1-cfc5-412d-bf9a-bac3c9162af1" volumeName="kubernetes.io/projected/9dad8dd1-cfc5-412d-bf9a-bac3c9162af1-kube-api-access-5n5lv" seLinuxMountContext=""
Oct 31 07:29:05.995229 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:05.994231    4720 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ce8cac52-291b-47fb-a7b6-4cf7562449b9" volumeName="kubernetes.io/projected/ce8cac52-291b-47fb-a7b6-4cf7562449b9-kube-api-access-bfzv8" seLinuxMountContext=""
Oct 31 07:29:05.995229 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:05.994244    4720 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fc64d31a-f48a-4293-a4de-262045d0b7e8" volumeName="kubernetes.io/configmap/fc64d31a-f48a-4293-a4de-262045d0b7e8-serviceca" seLinuxMountContext=""
Oct 31 07:29:05.995229 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:05.994256    4720 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2536c1c8-8a46-4af3-bc6d-81454a908d3f" volumeName="kubernetes.io/projected/2536c1c8-8a46-4af3-bc6d-81454a908d3f-kube-api-access-g2r66" seLinuxMountContext=""
Oct 31 07:29:05.995229 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:05.994269    4720 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="26659955-3909-4171-90b8-21b0ad6eef64" volumeName="kubernetes.io/projected/26659955-3909-4171-90b8-21b0ad6eef64-kube-api-access-jbdqx" seLinuxMountContext=""
Oct 31 07:29:05.995229 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:05.994281    4720 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5" volumeName="kubernetes.io/configmap/2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5-konnectivity-ca" seLinuxMountContext=""
Oct 31 07:29:05.995229 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:05.994302    4720 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="789f11d8-d951-4144-a589-a2bdbf6844e3" volumeName="kubernetes.io/projected/789f11d8-d951-4144-a589-a2bdbf6844e3-kube-api-access-wqmxh" seLinuxMountContext=""
Oct 31 07:29:05.995229 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:05.994313    4720 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c3ad95fd-e997-459c-93fd-27c4c030ea0d" volumeName="kubernetes.io/projected/c3ad95fd-e997-459c-93fd-27c4c030ea0d-kube-api-access-kxwb9" seLinuxMountContext=""
Oct 31 07:29:05.995229 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:05.994350    4720 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0b25f791-3520-481d-acbb-a3bef222fdac" volumeName="kubernetes.io/projected/0b25f791-3520-481d-acbb-a3bef222fdac-kube-api-access-nhphg" seLinuxMountContext=""
Oct 31 07:29:05.995229 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:05.994363    4720 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="28b59d93-b9ce-4ceb-9f3a-742525eb94c4" volumeName="kubernetes.io/projected/28b59d93-b9ce-4ceb-9f3a-742525eb94c4-bound-sa-token" seLinuxMountContext=""
Oct 31 07:29:05.995229 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:05.994375    4720 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4bfbebfd-2be7-438f-b88d-273f24751f8c" volumeName="kubernetes.io/configmap/4bfbebfd-2be7-438f-b88d-273f24751f8c-env-overrides" seLinuxMountContext=""
Oct 31 07:29:05.995229 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:05.994386    4720 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="993cfed6-806d-44c5-bba8-c02203638fe7" volumeName="kubernetes.io/projected/993cfed6-806d-44c5-bba8-c02203638fe7-bound-sa-token" seLinuxMountContext=""
Oct 31 07:29:05.995229 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:05.994398    4720 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="28b59d93-b9ce-4ceb-9f3a-742525eb94c4" volumeName="kubernetes.io/secret/28b59d93-b9ce-4ceb-9f3a-742525eb94c4-image-registry-private-configuration" seLinuxMountContext=""
Oct 31 07:29:05.995229 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:05.994410    4720 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5" volumeName="kubernetes.io/secret/2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5-agent-certs" seLinuxMountContext=""
Oct 31 07:29:05.995229 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:05.994422    4720 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4bfbebfd-2be7-438f-b88d-273f24751f8c" volumeName="kubernetes.io/projected/4bfbebfd-2be7-438f-b88d-273f24751f8c-kube-api-access-cz8tm" seLinuxMountContext=""
Oct 31 07:29:05.995229 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:05.994432    4720 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4db4c33f-0c58-40f8-bf74-51b3eae3c9db" volumeName="kubernetes.io/secret/4db4c33f-0c58-40f8-bf74-51b3eae3c9db-serving-cert" seLinuxMountContext=""
Oct 31 07:29:05.995229 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:05.994443    4720 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="993cfed6-806d-44c5-bba8-c02203638fe7" volumeName="kubernetes.io/empty-dir/993cfed6-806d-44c5-bba8-c02203638fe7-ca-trust-extracted" seLinuxMountContext=""
Oct 31 07:29:05.995229 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:05.994454    4720 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4bfbebfd-2be7-438f-b88d-273f24751f8c" volumeName="kubernetes.io/secret/4bfbebfd-2be7-438f-b88d-273f24751f8c-ovn-node-metrics-cert" seLinuxMountContext=""
Oct 31 07:29:05.995229 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:05.994467    4720 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="28b59d93-b9ce-4ceb-9f3a-742525eb94c4" volumeName="kubernetes.io/projected/28b59d93-b9ce-4ceb-9f3a-742525eb94c4-kube-api-access-xvv26" seLinuxMountContext=""
Oct 31 07:29:05.995229 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:05.994479    4720 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="28b59d93-b9ce-4ceb-9f3a-742525eb94c4" volumeName="kubernetes.io/secret/28b59d93-b9ce-4ceb-9f3a-742525eb94c4-installation-pull-secrets" seLinuxMountContext=""
Oct 31 07:29:05.995229 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:05.994489    4720 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="323c8334-9a56-4c4e-8a6e-ae80995810d0" volumeName="kubernetes.io/projected/323c8334-9a56-4c4e-8a6e-ae80995810d0-kube-api-access-l4nfd" seLinuxMountContext=""
Oct 31 07:29:05.995229 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:05.994500    4720 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="81829763-27b1-4a48-a568-cd5f6a39a953" volumeName="kubernetes.io/secret/81829763-27b1-4a48-a568-cd5f6a39a953-serving-cert" seLinuxMountContext=""
Oct 31 07:29:05.995229 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:05.994510    4720 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="993cfed6-806d-44c5-bba8-c02203638fe7" volumeName="kubernetes.io/projected/993cfed6-806d-44c5-bba8-c02203638fe7-kube-api-access-v2zhk" seLinuxMountContext=""
Oct 31 07:29:05.995229 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:05.994525    4720 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="993cfed6-806d-44c5-bba8-c02203638fe7" volumeName="kubernetes.io/secret/993cfed6-806d-44c5-bba8-c02203638fe7-image-registry-private-configuration" seLinuxMountContext=""
Oct 31 07:29:05.995229 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:05.994536    4720 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b44ed0e0-a72e-4746-a7de-ed8610da845d" volumeName="kubernetes.io/secret/b44ed0e0-a72e-4746-a7de-ed8610da845d-stats-auth" seLinuxMountContext=""
Oct 31 07:29:05.995229 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:05.994547    4720 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ca31519e-aa90-4104-a175-f9af2b3bbeb4" volumeName="kubernetes.io/projected/ca31519e-aa90-4104-a175-f9af2b3bbeb4-kube-api-access-n6p25" seLinuxMountContext=""
Oct 31 07:29:05.995229 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:05.994558    4720 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2536c1c8-8a46-4af3-bc6d-81454a908d3f" volumeName="kubernetes.io/secret/2536c1c8-8a46-4af3-bc6d-81454a908d3f-cert" seLinuxMountContext=""
Oct 31 07:29:05.995229 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:05.994571    4720 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a73024cb-f581-450d-a09c-0565838c5baa" volumeName="kubernetes.io/projected/a73024cb-f581-450d-a09c-0565838c5baa-kube-api-access-m6l7n" seLinuxMountContext=""
Oct 31 07:29:05.995229 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:05.994582    4720 reconstruct.go:97] "Volume reconstruction finished"
Oct 31 07:29:05.995229 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:05.994588    4720 reconciler.go:29] "Reconciler: start to sync state"
Oct 31 07:29:05.997800 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:05.997779    4720 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-4jszp_b1db0e3b-e85d-4769-8012-cf4b6f740d9d/console-operator/1.log"
Oct 31 07:29:05.999163 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:05.999009    4720 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-4jszp_b1db0e3b-e85d-4769-8012-cf4b6f740d9d/console-operator/1.log"
Oct 31 07:29:06.017442 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:06.017405    4720 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Oct 31 07:29:06.019015 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:06.018994    4720 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Oct 31 07:29:06.019101 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:06.019021    4720 status_manager.go:249] "Starting to sync pod status with apiserver"
Oct 31 07:29:06.019101 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:06.019037    4720 kubelet.go:2521] "Starting kubelet main sync loop"
Oct 31 07:29:06.019210 ip-10-0-139-83 kubenswrapper[4720]: E1031 07:29:06.019101    4720 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]"
Oct 31 07:29:06.033037 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:06.033010    4720 generic.go:356] "Generic (PLEG): container finished" podID="8dfeb47fa079b05125409320442d08bf" containerID="e3679e2424b009b37ae06bb097193ef296d3a36862993a093b27fbf5d970d554" exitCode=0
Oct 31 07:29:06.057025 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:06.056972    4720 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-4jszp_b1db0e3b-e85d-4769-8012-cf4b6f740d9d/console-operator/1.log"
Oct 31 07:29:06.057025 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:06.057003    4720 generic.go:356] "Generic (PLEG): container finished" podID="b1db0e3b-e85d-4769-8012-cf4b6f740d9d" containerID="c92430b7e7806bf2e27a6cbf56f84a5decbb26ad6ba153e76cb4e22794254de8" exitCode=255
Oct 31 07:29:06.079209 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:06.079075    4720 generic.go:356] "Generic (PLEG): container finished" podID="9c72279a-708f-46df-8915-9995c5aea14d" containerID="5983783fae630464092023d00a7fd5341a04199f277281a2a7f800c2c9041fe5" exitCode=0
Oct 31 07:29:06.079209 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:06.079207    4720 generic.go:356] "Generic (PLEG): container finished" podID="9c72279a-708f-46df-8915-9995c5aea14d" containerID="3f7eeca7807c5d5ea10ecc01516192cae358f4c8a3ce534d2ec240a880a25abc" exitCode=0
Oct 31 07:29:06.079384 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:06.079214    4720 generic.go:356] "Generic (PLEG): container finished" podID="9c72279a-708f-46df-8915-9995c5aea14d" containerID="bfe4d1e06bbdacaeac725652d553ddb60e20fa727a3734cdbfe038de3598eb2f" exitCode=0
Oct 31 07:29:06.079384 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:06.079219    4720 generic.go:356] "Generic (PLEG): container finished" podID="9c72279a-708f-46df-8915-9995c5aea14d" containerID="acf8ef1de7748f37a3862bc7e78950c0776180ce342a1c0eb6c64b0fdb516e73" exitCode=0
Oct 31 07:29:06.079384 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:06.079224    4720 generic.go:356] "Generic (PLEG): container finished" podID="9c72279a-708f-46df-8915-9995c5aea14d" containerID="1984fc27154b51abf200eca7b767ac5b05187e6919cc510f5d47ba5be76a1a54" exitCode=0
Oct 31 07:29:06.079384 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:06.079229    4720 generic.go:356] "Generic (PLEG): container finished" podID="9c72279a-708f-46df-8915-9995c5aea14d" containerID="9969a8fe850f4420a949de006c685300fd4d2047c9eefef4e247bf296bc04038" exitCode=0
Oct 31 07:29:06.084250 ip-10-0-139-83 kubenswrapper[4720]: E1031 07:29:06.084230    4720 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-139-83.us-west-1.compute.internal\" not found"
Oct 31 07:29:06.088231 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:06.088214    4720 manager.go:324] Recovery completed
Oct 31 07:29:06.116446 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:06.116424    4720 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 31 07:29:06.118125 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:06.118112    4720 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 31 07:29:06.119237 ip-10-0-139-83 kubenswrapper[4720]: E1031 07:29:06.119219    4720 kubelet.go:2545] "Skipping pod synchronization" err="container runtime status check may not have completed yet"
Oct 31 07:29:06.119307 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:06.119261    4720 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-83.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Oct 31 07:29:06.119307 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:06.119287    4720 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-83.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Oct 31 07:29:06.119307 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:06.119298    4720 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-83.us-west-1.compute.internal" event="NodeHasSufficientPID"
Oct 31 07:29:06.121241 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:06.121225    4720 cpu_manager.go:226] "Starting" policy="none"
Oct 31 07:29:06.121241 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:06.121238    4720 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s"
Oct 31 07:29:06.121398 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:06.121244    4720 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 31 07:29:06.121398 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:06.121257    4720 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory"
Oct 31 07:29:06.121491 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:06.121446    4720 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet=""
Oct 31 07:29:06.121491 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:06.121461    4720 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={}
Oct 31 07:29:06.121637 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:06.121497    4720 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint"
Oct 31 07:29:06.121637 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:06.121505    4720 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet=""
Oct 31 07:29:06.121637 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:06.121513    4720 policy_none.go:50] "Start"
Oct 31 07:29:06.121637 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:06.121525    4720 memory_manager.go:187] "Starting memorymanager" policy="None"
Oct 31 07:29:06.121637 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:06.121538    4720 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Oct 31 07:29:06.122345 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:06.122305    4720 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-83.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Oct 31 07:29:06.122390 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:06.122350    4720 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-83.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Oct 31 07:29:06.122390 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:06.122362    4720 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-83.us-west-1.compute.internal" event="NodeHasSufficientPID"
Oct 31 07:29:06.128228 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:06.128211    4720 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint"
Oct 31 07:29:06.128307 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:06.128232    4720 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint"
Oct 31 07:29:06.128307 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:06.128241    4720 policy_none.go:44] "Start"
Oct 31 07:29:06.132909 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:06.132887    4720 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"}
Oct 31 07:29:06.135321 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:06.135308    4720 manager.go:341] "Starting Device Plugin manager"
Oct 31 07:29:06.135442 ip-10-0-139-83 kubenswrapper[4720]: E1031 07:29:06.135427    4720 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Oct 31 07:29:06.135507 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:06.135443    4720 server.go:87] "Starting device plugin registration server"
Oct 31 07:29:06.135680 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:06.135667    4720 eviction_manager.go:194] "Eviction manager: starting control loop"
Oct 31 07:29:06.136140 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:06.135680    4720 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Oct 31 07:29:06.136211 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:06.136166    4720 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Oct 31 07:29:06.136289 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:06.136259    4720 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Oct 31 07:29:06.136342 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:06.136297    4720 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts"
Oct 31 07:29:06.136342 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:06.136304    4720 plugin_manager.go:121] "Starting Kubelet Plugin Manager"
Oct 31 07:29:06.136444 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:06.136366    4720 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-31T07:29:06.136288403Z","UUID":"fad7eb5c-5ec9-4c4d-9231-dcf38836fe50","Handler":null,"Name":"","Endpoint":""}
Oct 31 07:29:06.137385 ip-10-0-139-83 kubenswrapper[4720]: E1031 07:29:06.137363    4720 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Oct 31 07:29:06.137470 ip-10-0-139-83 kubenswrapper[4720]: E1031 07:29:06.137409    4720 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-139-83.us-west-1.compute.internal\" not found"
Oct 31 07:29:06.137761 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:06.137745    4720 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Oct 31 07:29:06.137827 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:06.137768    4720 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Oct 31 07:29:06.164941 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:06.164917    4720 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 31 07:29:06.215281 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:06.215263    4720 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 31 07:29:06.215980 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:06.215410    4720 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 31 07:29:06.216345 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:06.216302    4720 reconstruct.go:205] "DevicePaths of reconstructed volumes updated"
Oct 31 07:29:06.236695 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:06.236679    4720 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 31 07:29:06.237777 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:06.237761    4720 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-83.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Oct 31 07:29:06.237854 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:06.237791    4720 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-83.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Oct 31 07:29:06.237854 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:06.237806    4720 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-83.us-west-1.compute.internal" event="NodeHasSufficientPID"
Oct 31 07:29:06.237854 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:06.237855    4720 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-139-83.us-west-1.compute.internal"
Oct 31 07:29:06.319686 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:06.319579    4720 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["kube-system/kube-apiserver-proxy-ip-10-0-139-83.us-west-1.compute.internal","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-83.us-west-1.compute.internal"]
Oct 31 07:29:06.320200 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:06.320167    4720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-83.us-west-1.compute.internal" event={"ID":"8dfeb47fa079b05125409320442d08bf","Type":"ContainerStarted","Data":"7147ad859942ca65bf1b1ede84537503a2044ebcd89d053724d075b9b70bda3f"}
Oct 31 07:29:06.320299 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:06.320209    4720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-83.us-west-1.compute.internal" event={"ID":"8dfeb47fa079b05125409320442d08bf","Type":"ContainerDied","Data":"e3679e2424b009b37ae06bb097193ef296d3a36862993a093b27fbf5d970d554"}
Oct 31 07:29:06.320299 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:06.320219    4720 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-83.us-west-1.compute.internal"
Oct 31 07:29:06.320299 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:06.320229    4720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-83.us-west-1.compute.internal" event={"ID":"8dfeb47fa079b05125409320442d08bf","Type":"ContainerStarted","Data":"9f2589536197b37032285af9622b146bd14cd852452e464e8eb73c0a610e5078"}
Oct 31 07:29:06.320299 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:06.320285    4720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-83.us-west-1.compute.internal" event={"ID":"2183ffc19200ec816330ede9883f3bf6","Type":"ContainerStarted","Data":"cd2467be2b735e8292bf12bac2cb19105c276629b61450f504271e37fb9c7e7b"}
Oct 31 07:29:06.320503 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:06.320302    4720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-83.us-west-1.compute.internal" event={"ID":"2183ffc19200ec816330ede9883f3bf6","Type":"ContainerStarted","Data":"e5d2dc100f3cacdb69fc74b690c1bbe741b4298bd41697263a403392aa95b8b8"}
Oct 31 07:29:06.320503 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:06.320456    4720 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-83.us-west-1.compute.internal"
Oct 31 07:29:06.345499 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:06.345480    4720 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-139-83.us-west-1.compute.internal"
Oct 31 07:29:06.345568 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:06.345531    4720 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-139-83.us-west-1.compute.internal"
Oct 31 07:29:06.379675 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:06.379654    4720 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 31 07:29:06.379762 ip-10-0-139-83 kubenswrapper[4720]: E1031 07:29:06.379691    4720 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-139-83.us-west-1.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-83.us-west-1.compute.internal"
Oct 31 07:29:06.380349 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:06.380311    4720 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 31 07:29:06.380415 ip-10-0-139-83 kubenswrapper[4720]: E1031 07:29:06.380366    4720 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-139-83.us-west-1.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-83.us-west-1.compute.internal"
Oct 31 07:29:06.417832 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:06.417809    4720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/2183ffc19200ec816330ede9883f3bf6-config\") pod \"kube-apiserver-proxy-ip-10-0-139-83.us-west-1.compute.internal\" (UID: \"2183ffc19200ec816330ede9883f3bf6\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-83.us-west-1.compute.internal"
Oct 31 07:29:06.417911 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:06.417839    4720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/8dfeb47fa079b05125409320442d08bf-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-83.us-west-1.compute.internal\" (UID: \"8dfeb47fa079b05125409320442d08bf\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-83.us-west-1.compute.internal"
Oct 31 07:29:06.417949 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:06.417925    4720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8dfeb47fa079b05125409320442d08bf-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-83.us-west-1.compute.internal\" (UID: \"8dfeb47fa079b05125409320442d08bf\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-83.us-west-1.compute.internal"
Oct 31 07:29:06.518459 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:06.518428    4720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/2183ffc19200ec816330ede9883f3bf6-config\") pod \"kube-apiserver-proxy-ip-10-0-139-83.us-west-1.compute.internal\" (UID: \"2183ffc19200ec816330ede9883f3bf6\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-83.us-west-1.compute.internal"
Oct 31 07:29:06.518588 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:06.518461    4720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/2183ffc19200ec816330ede9883f3bf6-config\") pod \"kube-apiserver-proxy-ip-10-0-139-83.us-west-1.compute.internal\" (UID: \"2183ffc19200ec816330ede9883f3bf6\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-83.us-west-1.compute.internal"
Oct 31 07:29:06.518588 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:06.518500    4720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/8dfeb47fa079b05125409320442d08bf-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-83.us-west-1.compute.internal\" (UID: \"8dfeb47fa079b05125409320442d08bf\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-83.us-west-1.compute.internal"
Oct 31 07:29:06.518588 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:06.518518    4720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8dfeb47fa079b05125409320442d08bf-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-83.us-west-1.compute.internal\" (UID: \"8dfeb47fa079b05125409320442d08bf\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-83.us-west-1.compute.internal"
Oct 31 07:29:06.518588 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:06.518554    4720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8dfeb47fa079b05125409320442d08bf-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-83.us-west-1.compute.internal\" (UID: \"8dfeb47fa079b05125409320442d08bf\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-83.us-west-1.compute.internal"
Oct 31 07:29:06.518705 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:06.518580    4720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/8dfeb47fa079b05125409320442d08bf-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-83.us-west-1.compute.internal\" (UID: \"8dfeb47fa079b05125409320442d08bf\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-83.us-west-1.compute.internal"
Oct 31 07:29:06.954643 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:06.954616    4720 apiserver.go:52] "Watching apiserver"
Oct 31 07:29:07.085454 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:07.085432    4720 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-83.us-west-1.compute.internal"
Oct 31 07:29:07.107612 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:07.107591    4720 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Oct 31 07:29:07.108247 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:07.108226    4720 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ovn-kubernetes/ovnkube-node-5p7xw","openshift-multus/network-metrics-daemon-r2zbv","openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-6vkbv","openshift-insights/insights-operator-7c4986d449-dwc4p","openshift-multus/multus-additional-cni-plugins-dh6w8","openshift-image-registry/node-ca-th7x7","openshift-network-diagnostics/network-check-target-9zp4l","openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-65chl","openshift-ingress-canary/ingress-canary-kmcg5","openshift-service-ca-operator/service-ca-operator-544d749c8c-hb49h","kube-system/kas-connection-checker-685895c7c9-f9gtg","kube-system/kas-connection-checker-685895c7c9-xcrsn","openshift-monitoring/cluster-monitoring-operator-884576c57-6k8mn","openshift-network-diagnostics/network-check-source-6f78d59f99-fqhzf","openshift-ingress/router-default-6c88c45d57-c2bgn","openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-p9z4r","openshift-network-operator/iptables-alerter-65hzp","openshift-insights/insights-runtime-extractor-8m768","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-797gj","openshift-dns/dns-default-9cmfx","openshift-dns/node-resolver-tnw4j","openshift-image-registry/image-registry-7886c7d6c6-qdc9f","openshift-kube-storage-version-migrator/migrator-66df5d496b-5t4pm","openshift-console-operator/console-operator-7cd56b8dbc-4jszp","openshift-image-registry/image-registry-7955c67bcc-pkntv","openshift-multus/multus-pmnqx","kube-system/kube-apiserver-proxy-ip-10-0-139-83.us-west-1.compute.internal","kube-system/global-pull-secret-syncer-kht4l","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-83.us-west-1.compute.internal","openshift-cluster-node-tuning-operator/tuned-cpn4m","openshift-network-console/networking-console-plugin-86c478bb88-j94dz","kube-system/kas-connection-checker-685895c7c9-hxmpf","kube-system/konnectivity-agent-h4k2m"]
Oct 31 07:29:07.109560 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:07.109532    4720 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-6c88c45d57-c2bgn"
Oct 31 07:29:07.109902 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:07.109885    4720 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7886c7d6c6-qdc9f"
Oct 31 07:29:07.110836 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:07.110818    4720 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-6vkbv"
Oct 31 07:29:07.111210 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:07.111193    4720 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-6k8mn"
Oct 31 07:29:07.113239 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:07.113221    4720 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-86c478bb88-j94dz"
Oct 31 07:29:07.118064 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:07.118046    4720 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r2zbv"
Oct 31 07:29:07.121808 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:07.121790    4720 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7955c67bcc-pkntv"
Oct 31 07:29:07.122290 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:07.122268    4720 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-9cmfx"
Oct 31 07:29:07.123264 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:07.123246    4720 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-8m768"
Oct 31 07:29:07.140772 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:07.140752    4720 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 31 07:29:07.140865 ip-10-0-139-83 kubenswrapper[4720]: E1031 07:29:07.140801    4720 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-139-83.us-west-1.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-83.us-west-1.compute.internal"
Oct 31 07:29:07.173820 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:07.173796    4720 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\""
Oct 31 07:29:07.173820 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:07.173799    4720 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\""
Oct 31 07:29:07.173937 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:07.173845    4720 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Oct 31 07:29:07.174017 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:07.174002    4720 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\""
Oct 31 07:29:07.174066 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:07.174050    4720 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Oct 31 07:29:07.174106 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:07.174064    4720 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Oct 31 07:29:07.174212 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:07.174199    4720 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\""
Oct 31 07:29:07.174487 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:07.174474    4720 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\""
Oct 31 07:29:07.174556 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:07.174491    4720 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Oct 31 07:29:07.174556 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:07.174503    4720 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\""
Oct 31 07:29:07.174556 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:07.174545    4720 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"kube-root-ca.crt\""
Oct 31 07:29:07.174755 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:07.174738    4720 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Oct 31 07:29:07.174796 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:07.174755    4720 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Oct 31 07:29:07.174827 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:07.174800    4720 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\""
Oct 31 07:29:07.174960 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:07.174945    4720 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Oct 31 07:29:07.174960 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:07.174956    4720 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"openshift-service-ca.crt\""
Oct 31 07:29:07.175074 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:07.174974    4720 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-x96s4\""
Oct 31 07:29:07.175074 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:07.174995    4720 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\""
Oct 31 07:29:07.175074 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:07.175002    4720 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\""
Oct 31 07:29:07.175074 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:07.175008    4720 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\""
Oct 31 07:29:07.175074 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:07.174995    4720 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-config\""
Oct 31 07:29:07.175357 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:07.175194    4720 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\""
Oct 31 07:29:07.175445 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:07.175428    4720 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Oct 31 07:29:07.175495 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:07.175433    4720 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\""
Oct 31 07:29:07.175495 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:07.175457    4720 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\""
Oct 31 07:29:07.175495 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:07.175476    4720 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\""
Oct 31 07:29:07.175495 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:07.175482    4720 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-4d4mq\""
Oct 31 07:29:07.175643 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:07.175458    4720 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\""
Oct 31 07:29:07.175643 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:07.175524    4720 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\""
Oct 31 07:29:07.175643 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:07.175526    4720 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Oct 31 07:29:07.175643 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:07.175538    4720 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\""
Oct 31 07:29:07.175643 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:07.175555    4720 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Oct 31 07:29:07.175643 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:07.175549    4720 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\""
Oct 31 07:29:07.175643 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:07.175596    4720 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\""
Oct 31 07:29:07.175822 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:07.175800    4720 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\""
Oct 31 07:29:07.175822 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:07.175814    4720 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\""
Oct 31 07:29:07.175881 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:07.175840    4720 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\""
Oct 31 07:29:07.175881 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:07.175857    4720 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\""
Oct 31 07:29:07.175999 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:07.175985    4720 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Oct 31 07:29:07.176110 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:07.176095    4720 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Oct 31 07:29:07.176166 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:07.176155    4720 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\""
Oct 31 07:29:07.176471 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:07.176458    4720 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Oct 31 07:29:07.186985 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:07.186972    4720 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Oct 31 07:29:07.221925 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:07.221879    4720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/4bfbebfd-2be7-438f-b88d-273f24751f8c-host-run-netns\") pod \"ovnkube-node-5p7xw\" (UID: \"4bfbebfd-2be7-438f-b88d-273f24751f8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-5p7xw"
Oct 31 07:29:07.221925 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:07.221906    4720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cjt6r\" (UniqueName: \"kubernetes.io/projected/b1066ece-e6f3-4491-aa1f-a7b9a3aa248d-kube-api-access-cjt6r\") pod \"insights-runtime-extractor-8m768\" (UID: \"b1066ece-e6f3-4491-aa1f-a7b9a3aa248d\") " pod="openshift-insights/insights-runtime-extractor-8m768"
Oct 31 07:29:07.222027 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:07.221933    4720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/993cfed6-806d-44c5-bba8-c02203638fe7-registry-certificates\") pod \"image-registry-7886c7d6c6-qdc9f\" (UID: \"993cfed6-806d-44c5-bba8-c02203638fe7\") " pod="openshift-image-registry/image-registry-7886c7d6c6-qdc9f"
Oct 31 07:29:07.222027 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:07.221970    4720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/9c72279a-708f-46df-8915-9995c5aea14d-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-dh6w8\" (UID: \"9c72279a-708f-46df-8915-9995c5aea14d\") " pod="openshift-multus/multus-additional-cni-plugins-dh6w8"
Oct 31 07:29:07.222027 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:07.222005    4720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/b1db0e3b-e85d-4769-8012-cf4b6f740d9d-serving-cert\") pod \"console-operator-7cd56b8dbc-4jszp\" (UID: \"b1db0e3b-e85d-4769-8012-cf4b6f740d9d\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-4jszp"
Oct 31 07:29:07.222123 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:07.222072    4720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-czqtq\" (UniqueName: \"kubernetes.io/projected/b1db0e3b-e85d-4769-8012-cf4b6f740d9d-kube-api-access-czqtq\") pod \"console-operator-7cd56b8dbc-4jszp\" (UID: \"b1db0e3b-e85d-4769-8012-cf4b6f740d9d\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-4jszp"
Oct 31 07:29:07.222163 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:07.222119    4720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4bfbebfd-2be7-438f-b88d-273f24751f8c-host-run-ovn-kubernetes\") pod \"ovnkube-node-5p7xw\" (UID: \"4bfbebfd-2be7-438f-b88d-273f24751f8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-5p7xw"
Oct 31 07:29:07.222163 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:07.222146    4720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bfzv8\" (UniqueName: \"kubernetes.io/projected/ce8cac52-291b-47fb-a7b6-4cf7562449b9-kube-api-access-bfzv8\") pod \"network-check-source-6f78d59f99-fqhzf\" (UID: \"ce8cac52-291b-47fb-a7b6-4cf7562449b9\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-fqhzf"
Oct 31 07:29:07.222234 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:07.222173    4720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/993cfed6-806d-44c5-bba8-c02203638fe7-registry-certificates\") pod \"image-registry-7886c7d6c6-qdc9f\" (UID: \"993cfed6-806d-44c5-bba8-c02203638fe7\") " pod="openshift-image-registry/image-registry-7886c7d6c6-qdc9f"
Oct 31 07:29:07.222234 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:07.222177    4720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/77256436-39ff-4c2a-8390-4e9fa7d68486-kubelet-config\") pod \"global-pull-secret-syncer-kht4l\" (UID: \"77256436-39ff-4c2a-8390-4e9fa7d68486\") " pod="kube-system/global-pull-secret-syncer-kht4l"
Oct 31 07:29:07.222344 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:07.222238    4720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kk4nh\" (UniqueName: \"kubernetes.io/projected/fc64d31a-f48a-4293-a4de-262045d0b7e8-kube-api-access-kk4nh\") pod \"node-ca-th7x7\" (UID: \"fc64d31a-f48a-4293-a4de-262045d0b7e8\") " pod="openshift-image-registry/node-ca-th7x7"
Oct 31 07:29:07.222344 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:07.222264    4720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/314c29aa-db6b-4f7f-86b4-c8952ce09c84-nginx-conf\") pod \"networking-console-plugin-86c478bb88-j94dz\" (UID: \"314c29aa-db6b-4f7f-86b4-c8952ce09c84\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-j94dz"
Oct 31 07:29:07.222344 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:07.222307    4720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/3fa078c7-4bf2-4e76-9692-ae460e8fadfa-etc-sysctl-conf\") pod \"tuned-cpn4m\" (UID: \"3fa078c7-4bf2-4e76-9692-ae460e8fadfa\") " pod="openshift-cluster-node-tuning-operator/tuned-cpn4m"
Oct 31 07:29:07.222477 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:07.222372    4720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/b1db0e3b-e85d-4769-8012-cf4b6f740d9d-serving-cert\") pod \"console-operator-7cd56b8dbc-4jszp\" (UID: \"b1db0e3b-e85d-4769-8012-cf4b6f740d9d\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-4jszp"
Oct 31 07:29:07.222545 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:07.222419    4720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/08198bc1-314f-4881-833c-e633bac15e4a-etc-kubernetes\") pod \"multus-pmnqx\" (UID: \"08198bc1-314f-4881-833c-e633bac15e4a\") " pod="openshift-multus/multus-pmnqx"
Oct 31 07:29:07.222593 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:07.222558    4720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/4db4c33f-0c58-40f8-bf74-51b3eae3c9db-serving-cert\") pod \"service-ca-operator-544d749c8c-hb49h\" (UID: \"4db4c33f-0c58-40f8-bf74-51b3eae3c9db\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-hb49h"
Oct 31 07:29:07.222593 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:07.222574    4720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/9dad8dd1-cfc5-412d-bf9a-bac3c9162af1-kubelet-dir\") pod \"aws-ebs-csi-driver-node-797gj\" (UID: \"9dad8dd1-cfc5-412d-bf9a-bac3c9162af1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-797gj"
Oct 31 07:29:07.222593 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:07.222590    4720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/26659955-3909-4171-90b8-21b0ad6eef64-iptables-alerter-script\") pod \"iptables-alerter-65hzp\" (UID: \"26659955-3909-4171-90b8-21b0ad6eef64\") " pod="openshift-network-operator/iptables-alerter-65hzp"
Oct 31 07:29:07.222738 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:07.222604    4720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4bfbebfd-2be7-438f-b88d-273f24751f8c-var-lib-openvswitch\") pod \"ovnkube-node-5p7xw\" (UID: \"4bfbebfd-2be7-438f-b88d-273f24751f8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-5p7xw"
Oct 31 07:29:07.222738 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:07.222704    4720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/cb9e4575-9265-43b5-bf02-7843611d6563-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-p9z4r\" (UID: \"cb9e4575-9265-43b5-bf02-7843611d6563\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-p9z4r"
Oct 31 07:29:07.222738 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:07.222720    4720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/28b59d93-b9ce-4ceb-9f3a-742525eb94c4-registry-tls\") pod \"image-registry-7955c67bcc-pkntv\" (UID: \"28b59d93-b9ce-4ceb-9f3a-742525eb94c4\") " pod="openshift-image-registry/image-registry-7955c67bcc-pkntv"
Oct 31 07:29:07.222738 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:07.222735    4720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/81829763-27b1-4a48-a568-cd5f6a39a953-serving-cert\") pod \"insights-operator-7c4986d449-dwc4p\" (UID: \"81829763-27b1-4a48-a568-cd5f6a39a953\") " pod="openshift-insights/insights-operator-7c4986d449-dwc4p"
Oct 31 07:29:07.222921 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:07.222858    4720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/4bfbebfd-2be7-438f-b88d-273f24751f8c-run-systemd\") pod \"ovnkube-node-5p7xw\" (UID: \"4bfbebfd-2be7-438f-b88d-273f24751f8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-5p7xw"
Oct 31 07:29:07.222921 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:07.222903    4720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3fa078c7-4bf2-4e76-9692-ae460e8fadfa-tmp\") pod \"tuned-cpn4m\" (UID: \"3fa078c7-4bf2-4e76-9692-ae460e8fadfa\") " pod="openshift-cluster-node-tuning-operator/tuned-cpn4m"
Oct 31 07:29:07.223014 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:07.222934    4720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/ca31519e-aa90-4104-a175-f9af2b3bbeb4-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-6vkbv\" (UID: \"ca31519e-aa90-4104-a175-f9af2b3bbeb4\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-6vkbv"
Oct 31 07:29:07.223014 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:07.222949    4720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/4db4c33f-0c58-40f8-bf74-51b3eae3c9db-serving-cert\") pod \"service-ca-operator-544d749c8c-hb49h\" (UID: \"4db4c33f-0c58-40f8-bf74-51b3eae3c9db\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-hb49h"
Oct 31 07:29:07.223014 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:07.222964    4720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9c72279a-708f-46df-8915-9995c5aea14d-tuning-conf-dir\") pod \"multus-additional-cni-plugins-dh6w8\" (UID: \"9c72279a-708f-46df-8915-9995c5aea14d\") " pod="openshift-multus/multus-additional-cni-plugins-dh6w8"
Oct 31 07:29:07.223014 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:07.222965    4720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/cb9e4575-9265-43b5-bf02-7843611d6563-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-p9z4r\" (UID: \"cb9e4575-9265-43b5-bf02-7843611d6563\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-p9z4r"
Oct 31 07:29:07.223014 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:07.222965    4720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/81829763-27b1-4a48-a568-cd5f6a39a953-serving-cert\") pod \"insights-operator-7c4986d449-dwc4p\" (UID: \"81829763-27b1-4a48-a568-cd5f6a39a953\") " pod="openshift-insights/insights-operator-7c4986d449-dwc4p"
Oct 31 07:29:07.223014 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:07.222987    4720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/4bfbebfd-2be7-438f-b88d-273f24751f8c-log-socket\") pod \"ovnkube-node-5p7xw\" (UID: \"4bfbebfd-2be7-438f-b88d-273f24751f8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-5p7xw"
Oct 31 07:29:07.223262 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:07.223023    4720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/4bfbebfd-2be7-438f-b88d-273f24751f8c-host-kubelet\") pod \"ovnkube-node-5p7xw\" (UID: \"4bfbebfd-2be7-438f-b88d-273f24751f8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-5p7xw"
Oct 31 07:29:07.223262 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:07.223021    4720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3fa078c7-4bf2-4e76-9692-ae460e8fadfa-tmp\") pod \"tuned-cpn4m\" (UID: \"3fa078c7-4bf2-4e76-9692-ae460e8fadfa\") " pod="openshift-cluster-node-tuning-operator/tuned-cpn4m"
Oct 31 07:29:07.223262 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:07.223081    4720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cz8tm\" (UniqueName: \"kubernetes.io/projected/4bfbebfd-2be7-438f-b88d-273f24751f8c-kube-api-access-cz8tm\") pod \"ovnkube-node-5p7xw\" (UID: \"4bfbebfd-2be7-438f-b88d-273f24751f8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-5p7xw"
Oct 31 07:29:07.223262 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:07.223124    4720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/08198bc1-314f-4881-833c-e633bac15e4a-multus-conf-dir\") pod \"multus-pmnqx\" (UID: \"08198bc1-314f-4881-833c-e633bac15e4a\") " pod="openshift-multus/multus-pmnqx"
Oct 31 07:29:07.223262 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:07.223248    4720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/08198bc1-314f-4881-833c-e633bac15e4a-multus-daemon-config\") pod \"multus-pmnqx\" (UID: \"08198bc1-314f-4881-833c-e633bac15e4a\") " pod="openshift-multus/multus-pmnqx"
Oct 31 07:29:07.223481 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:07.223315    4720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/9dad8dd1-cfc5-412d-bf9a-bac3c9162af1-sys-fs\") pod \"aws-ebs-csi-driver-node-797gj\" (UID: \"9dad8dd1-cfc5-412d-bf9a-bac3c9162af1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-797gj"
Oct 31 07:29:07.223481 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:07.223426    4720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4bfbebfd-2be7-438f-b88d-273f24751f8c-run-openvswitch\") pod \"ovnkube-node-5p7xw\" (UID: \"4bfbebfd-2be7-438f-b88d-273f24751f8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-5p7xw"
Oct 31 07:29:07.223481 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:07.223450    4720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/08198bc1-314f-4881-833c-e633bac15e4a-host-var-lib-cni-bin\") pod \"multus-pmnqx\" (UID: \"08198bc1-314f-4881-833c-e633bac15e4a\") " pod="openshift-multus/multus-pmnqx"
Oct 31 07:29:07.223618 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:07.223472    4720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/4bfbebfd-2be7-438f-b88d-273f24751f8c-ovnkube-script-lib\") pod \"ovnkube-node-5p7xw\" (UID: \"4bfbebfd-2be7-438f-b88d-273f24751f8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-5p7xw"
Oct 31 07:29:07.223618 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:07.223555    4720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/993cfed6-806d-44c5-bba8-c02203638fe7-installation-pull-secrets\") pod \"image-registry-7886c7d6c6-qdc9f\" (UID: \"993cfed6-806d-44c5-bba8-c02203638fe7\") " pod="openshift-image-registry/image-registry-7886c7d6c6-qdc9f"
Oct 31 07:29:07.223618 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:07.223600    4720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/ca31519e-aa90-4104-a175-f9af2b3bbeb4-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-6vkbv\" (UID: \"ca31519e-aa90-4104-a175-f9af2b3bbeb4\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-6vkbv"
Oct 31 07:29:07.223753 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:07.223631    4720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n6p25\" (UniqueName: \"kubernetes.io/projected/ca31519e-aa90-4104-a175-f9af2b3bbeb4-kube-api-access-n6p25\") pod \"cluster-samples-operator-6ddb6c8b94-6vkbv\" (UID: \"ca31519e-aa90-4104-a175-f9af2b3bbeb4\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-6vkbv"
Oct 31 07:29:07.223753 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:07.223658    4720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/9dad8dd1-cfc5-412d-bf9a-bac3c9162af1-etc-selinux\") pod \"aws-ebs-csi-driver-node-797gj\" (UID: \"9dad8dd1-cfc5-412d-bf9a-bac3c9162af1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-797gj"
Oct 31 07:29:07.223753 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:07.223683    4720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5n5lv\" (UniqueName: \"kubernetes.io/projected/9dad8dd1-cfc5-412d-bf9a-bac3c9162af1-kube-api-access-5n5lv\") pod \"aws-ebs-csi-driver-node-797gj\" (UID: \"9dad8dd1-cfc5-412d-bf9a-bac3c9162af1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-797gj"
Oct 31 07:29:07.223753 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:07.223729    4720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9c72279a-708f-46df-8915-9995c5aea14d-cnibin\") pod \"multus-additional-cni-plugins-dh6w8\" (UID: \"9c72279a-708f-46df-8915-9995c5aea14d\") " pod="openshift-multus/multus-additional-cni-plugins-dh6w8"
Oct 31 07:29:07.223939 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:07.223755    4720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/81829763-27b1-4a48-a568-cd5f6a39a953-service-ca-bundle\") pod \"insights-operator-7c4986d449-dwc4p\" (UID: \"81829763-27b1-4a48-a568-cd5f6a39a953\") " pod="openshift-insights/insights-operator-7c4986d449-dwc4p"
Oct 31 07:29:07.223939 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:07.223844    4720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/ca31519e-aa90-4104-a175-f9af2b3bbeb4-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-6vkbv\" (UID: \"ca31519e-aa90-4104-a175-f9af2b3bbeb4\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-6vkbv"
Oct 31 07:29:07.224022 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:07.223954    4720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/993cfed6-806d-44c5-bba8-c02203638fe7-installation-pull-secrets\") pod \"image-registry-7886c7d6c6-qdc9f\" (UID: \"993cfed6-806d-44c5-bba8-c02203638fe7\") " pod="openshift-image-registry/image-registry-7886c7d6c6-qdc9f"
Oct 31 07:29:07.224022 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:07.223995    4720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/4bfbebfd-2be7-438f-b88d-273f24751f8c-host-cni-netd\") pod \"ovnkube-node-5p7xw\" (UID: \"4bfbebfd-2be7-438f-b88d-273f24751f8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-5p7xw"
Oct 31 07:29:07.224022 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:07.224016    4720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/81829763-27b1-4a48-a568-cd5f6a39a953-service-ca-bundle\") pod \"insights-operator-7c4986d449-dwc4p\" (UID: \"81829763-27b1-4a48-a568-cd5f6a39a953\") " pod="openshift-insights/insights-operator-7c4986d449-dwc4p"
Oct 31 07:29:07.224160 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:07.224045    4720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/08198bc1-314f-4881-833c-e633bac15e4a-host-run-netns\") pod \"multus-pmnqx\" (UID: \"08198bc1-314f-4881-833c-e633bac15e4a\") " pod="openshift-multus/multus-pmnqx"
Oct 31 07:29:07.224160 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:07.224074    4720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kxwb9\" (UniqueName: \"kubernetes.io/projected/c3ad95fd-e997-459c-93fd-27c4c030ea0d-kube-api-access-kxwb9\") pod \"network-metrics-daemon-r2zbv\" (UID: \"c3ad95fd-e997-459c-93fd-27c4c030ea0d\") " pod="openshift-multus/network-metrics-daemon-r2zbv"
Oct 31 07:29:07.224160 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:07.224090    4720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nhphg\" (UniqueName: \"kubernetes.io/projected/0b25f791-3520-481d-acbb-a3bef222fdac-kube-api-access-nhphg\") pod \"migrator-66df5d496b-5t4pm\" (UID: \"0b25f791-3520-481d-acbb-a3bef222fdac\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-5t4pm"
Oct 31 07:29:07.224160 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:07.224130    4720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b1db0e3b-e85d-4769-8012-cf4b6f740d9d-trusted-ca\") pod \"console-operator-7cd56b8dbc-4jszp\" (UID: \"b1db0e3b-e85d-4769-8012-cf4b6f740d9d\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-4jszp"
Oct 31 07:29:07.224364 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:07.224206    4720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4bfbebfd-2be7-438f-b88d-273f24751f8c-etc-openvswitch\") pod \"ovnkube-node-5p7xw\" (UID: \"4bfbebfd-2be7-438f-b88d-273f24751f8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-5p7xw"
Oct 31 07:29:07.224364 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:07.224257    4720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b1066ece-e6f3-4491-aa1f-a7b9a3aa248d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-8m768\" (UID: \"b1066ece-e6f3-4491-aa1f-a7b9a3aa248d\") " pod="openshift-insights/insights-runtime-extractor-8m768"
Oct 31 07:29:07.224364 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:07.224302    4720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/9dad8dd1-cfc5-412d-bf9a-bac3c9162af1-registration-dir\") pod \"aws-ebs-csi-driver-node-797gj\" (UID: \"9dad8dd1-cfc5-412d-bf9a-bac3c9162af1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-797gj"
Oct 31 07:29:07.224521 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:07.224363    4720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/81829763-27b1-4a48-a568-cd5f6a39a953-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-dwc4p\" (UID: \"81829763-27b1-4a48-a568-cd5f6a39a953\") " pod="openshift-insights/insights-operator-7c4986d449-dwc4p"
Oct 31 07:29:07.224521 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:07.224391    4720 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\""
Oct 31 07:29:07.224521 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:07.224422    4720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/314c29aa-db6b-4f7f-86b4-c8952ce09c84-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-j94dz\" (UID: \"314c29aa-db6b-4f7f-86b4-c8952ce09c84\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-j94dz"
Oct 31 07:29:07.224521 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:07.224489    4720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/3fa078c7-4bf2-4e76-9692-ae460e8fadfa-lib-modules\") pod \"tuned-cpn4m\" (UID: \"3fa078c7-4bf2-4e76-9692-ae460e8fadfa\") " pod="openshift-cluster-node-tuning-operator/tuned-cpn4m"
Oct 31 07:29:07.224672 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:07.224529    4720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/28b59d93-b9ce-4ceb-9f3a-742525eb94c4-ca-trust-extracted\") pod \"image-registry-7955c67bcc-pkntv\" (UID: \"28b59d93-b9ce-4ceb-9f3a-742525eb94c4\") " pod="openshift-image-registry/image-registry-7955c67bcc-pkntv"
Oct 31 07:29:07.224672 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:07.224570    4720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/993cfed6-806d-44c5-bba8-c02203638fe7-registry-tls\") pod \"image-registry-7886c7d6c6-qdc9f\" (UID: \"993cfed6-806d-44c5-bba8-c02203638fe7\") " pod="openshift-image-registry/image-registry-7886c7d6c6-qdc9f"
Oct 31 07:29:07.224672 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:07.224603    4720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2536c1c8-8a46-4af3-bc6d-81454a908d3f-cert\") pod \"ingress-canary-kmcg5\" (UID: \"2536c1c8-8a46-4af3-bc6d-81454a908d3f\") " pod="openshift-ingress-canary/ingress-canary-kmcg5"
Oct 31 07:29:07.224672 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:07.224618    4720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/28b59d93-b9ce-4ceb-9f3a-742525eb94c4-ca-trust-extracted\") pod \"image-registry-7955c67bcc-pkntv\" (UID: \"28b59d93-b9ce-4ceb-9f3a-742525eb94c4\") " pod="openshift-image-registry/image-registry-7955c67bcc-pkntv"
Oct 31 07:29:07.224847 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:07.224739    4720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/3fa078c7-4bf2-4e76-9692-ae460e8fadfa-etc-systemd\") pod \"tuned-cpn4m\" (UID: \"3fa078c7-4bf2-4e76-9692-ae460e8fadfa\") " pod="openshift-cluster-node-tuning-operator/tuned-cpn4m"
Oct 31 07:29:07.224847 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:07.224777    4720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q2kvz\" (UniqueName: \"kubernetes.io/projected/4db4c33f-0c58-40f8-bf74-51b3eae3c9db-kube-api-access-q2kvz\") pod \"service-ca-operator-544d749c8c-hb49h\" (UID: \"4db4c33f-0c58-40f8-bf74-51b3eae3c9db\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-hb49h"
Oct 31 07:29:07.224847 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:07.224839    4720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h4gzh\" (UniqueName: \"kubernetes.io/projected/9c72279a-708f-46df-8915-9995c5aea14d-kube-api-access-h4gzh\") pod \"multus-additional-cni-plugins-dh6w8\" (UID: \"9c72279a-708f-46df-8915-9995c5aea14d\") " pod="openshift-multus/multus-additional-cni-plugins-dh6w8"
Oct 31 07:29:07.240963 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:07.240919    4720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sn2lx\" (UniqueName: \"kubernetes.io/projected/91363236-08db-4f54-ae0a-e68604fcecc6-kube-api-access-sn2lx\") pod \"kas-connection-checker-685895c7c9-xcrsn\" (UID: \"91363236-08db-4f54-ae0a-e68604fcecc6\") " pod="kube-system/kas-connection-checker-685895c7c9-xcrsn"
Oct 31 07:29:07.241074 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:07.240982    4720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4bfbebfd-2be7-438f-b88d-273f24751f8c-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-5p7xw\" (UID: \"4bfbebfd-2be7-438f-b88d-273f24751f8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-5p7xw"
Oct 31 07:29:07.241074 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:07.241014    4720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/3fa078c7-4bf2-4e76-9692-ae460e8fadfa-etc-sysconfig\") pod \"tuned-cpn4m\" (UID: \"3fa078c7-4bf2-4e76-9692-ae460e8fadfa\") " pod="openshift-cluster-node-tuning-operator/tuned-cpn4m"
Oct 31 07:29:07.241074 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:07.241050    4720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/b1066ece-e6f3-4491-aa1f-a7b9a3aa248d-data-volume\") pod \"insights-runtime-extractor-8m768\" (UID: \"b1066ece-e6f3-4491-aa1f-a7b9a3aa248d\") " pod="openshift-insights/insights-runtime-extractor-8m768"
Oct 31 07:29:07.241242 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:07.241084    4720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/28b59d93-b9ce-4ceb-9f3a-742525eb94c4-registry-certificates\") pod \"image-registry-7955c67bcc-pkntv\" (UID: \"28b59d93-b9ce-4ceb-9f3a-742525eb94c4\") " pod="openshift-image-registry/image-registry-7955c67bcc-pkntv"
Oct 31 07:29:07.241242 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:07.241173    4720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/b1066ece-e6f3-4491-aa1f-a7b9a3aa248d-data-volume\") pod \"insights-runtime-extractor-8m768\" (UID: \"b1066ece-e6f3-4491-aa1f-a7b9a3aa248d\") " pod="openshift-insights/insights-runtime-extractor-8m768"
Oct 31 07:29:07.241377 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:07.241243    4720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cq7rv\" (UniqueName: \"kubernetes.io/projected/3fa078c7-4bf2-4e76-9692-ae460e8fadfa-kube-api-access-cq7rv\") pod \"tuned-cpn4m\" (UID: \"3fa078c7-4bf2-4e76-9692-ae460e8fadfa\") " pod="openshift-cluster-node-tuning-operator/tuned-cpn4m"
Oct 31 07:29:07.241377 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:07.241282    4720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g2r66\" (UniqueName: \"kubernetes.io/projected/2536c1c8-8a46-4af3-bc6d-81454a908d3f-kube-api-access-g2r66\") pod \"ingress-canary-kmcg5\" (UID: \"2536c1c8-8a46-4af3-bc6d-81454a908d3f\") " pod="openshift-ingress-canary/ingress-canary-kmcg5"
Oct 31 07:29:07.241465 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:07.241378    4720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9pvp2\" (UniqueName: \"kubernetes.io/projected/b44ed0e0-a72e-4746-a7de-ed8610da845d-kube-api-access-9pvp2\") pod \"router-default-6c88c45d57-c2bgn\" (UID: \"b44ed0e0-a72e-4746-a7de-ed8610da845d\") " pod="openshift-ingress/router-default-6c88c45d57-c2bgn"
Oct 31 07:29:07.241465 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:07.241401    4720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/28b59d93-b9ce-4ceb-9f3a-742525eb94c4-registry-certificates\") pod \"image-registry-7955c67bcc-pkntv\" (UID: \"28b59d93-b9ce-4ceb-9f3a-742525eb94c4\") " pod="openshift-image-registry/image-registry-7955c67bcc-pkntv"
Oct 31 07:29:07.242166 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:07.242138    4720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/08198bc1-314f-4881-833c-e633bac15e4a-multus-socket-dir-parent\") pod \"multus-pmnqx\" (UID: \"08198bc1-314f-4881-833c-e633bac15e4a\") " pod="openshift-multus/multus-pmnqx"
Oct 31 07:29:07.242266 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:07.242193    4720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b1066ece-e6f3-4491-aa1f-a7b9a3aa248d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-8m768\" (UID: \"b1066ece-e6f3-4491-aa1f-a7b9a3aa248d\") " pod="openshift-insights/insights-runtime-extractor-8m768"
Oct 31 07:29:07.242266 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:07.242235    4720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/28b59d93-b9ce-4ceb-9f3a-742525eb94c4-image-registry-private-configuration\") pod \"image-registry-7955c67bcc-pkntv\" (UID: \"28b59d93-b9ce-4ceb-9f3a-742525eb94c4\") " pod="openshift-image-registry/image-registry-7955c67bcc-pkntv"
Oct 31 07:29:07.242523 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:07.242499    4720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xs9lt\" (UniqueName: \"kubernetes.io/projected/08198bc1-314f-4881-833c-e633bac15e4a-kube-api-access-xs9lt\") pod \"multus-pmnqx\" (UID: \"08198bc1-314f-4881-833c-e633bac15e4a\") " pod="openshift-multus/multus-pmnqx"
Oct 31 07:29:07.242603 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:07.242541    4720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5-konnectivity-ca\") pod \"konnectivity-agent-h4k2m\" (UID: \"2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5\") " pod="kube-system/konnectivity-agent-h4k2m"
Oct 31 07:29:07.242603 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:07.242577    4720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c3ad95fd-e997-459c-93fd-27c4c030ea0d-metrics-certs\") pod \"network-metrics-daemon-r2zbv\" (UID: \"c3ad95fd-e997-459c-93fd-27c4c030ea0d\") " pod="openshift-multus/network-metrics-daemon-r2zbv"
Oct 31 07:29:07.242697 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:07.242611    4720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/fc64d31a-f48a-4293-a4de-262045d0b7e8-host\") pod \"node-ca-th7x7\" (UID: \"fc64d31a-f48a-4293-a4de-262045d0b7e8\") " pod="openshift-image-registry/node-ca-th7x7"
Oct 31 07:29:07.242697 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:07.242636    4720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/28b59d93-b9ce-4ceb-9f3a-742525eb94c4-image-registry-private-configuration\") pod \"image-registry-7955c67bcc-pkntv\" (UID: \"28b59d93-b9ce-4ceb-9f3a-742525eb94c4\") " pod="openshift-image-registry/image-registry-7955c67bcc-pkntv"
Oct 31 07:29:07.242697 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:07.242650    4720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3fa078c7-4bf2-4e76-9692-ae460e8fadfa-var-lib-kubelet\") pod \"tuned-cpn4m\" (UID: \"3fa078c7-4bf2-4e76-9692-ae460e8fadfa\") " pod="openshift-cluster-node-tuning-operator/tuned-cpn4m"
Oct 31 07:29:07.242824 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:07.242717    4720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/08198bc1-314f-4881-833c-e633bac15e4a-cnibin\") pod \"multus-pmnqx\" (UID: \"08198bc1-314f-4881-833c-e633bac15e4a\") " pod="openshift-multus/multus-pmnqx"
Oct 31 07:29:07.242824 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:07.242755    4720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/cb9e4575-9265-43b5-bf02-7843611d6563-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-p9z4r\" (UID: \"cb9e4575-9265-43b5-bf02-7843611d6563\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-p9z4r"
Oct 31 07:29:07.242915 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:07.242803    4720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/28b59d93-b9ce-4ceb-9f3a-742525eb94c4-trusted-ca\") pod \"image-registry-7955c67bcc-pkntv\" (UID: \"28b59d93-b9ce-4ceb-9f3a-742525eb94c4\") " pod="openshift-image-registry/image-registry-7955c67bcc-pkntv"
Oct 31 07:29:07.242964 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:07.242932    4720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a73024cb-f581-450d-a09c-0565838c5baa-metrics-tls\") pod \"dns-default-9cmfx\" (UID: \"a73024cb-f581-450d-a09c-0565838c5baa\") " pod="openshift-dns/dns-default-9cmfx"
Oct 31 07:29:07.243007 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:07.242968    4720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m6l7n\" (UniqueName: \"kubernetes.io/projected/a73024cb-f581-450d-a09c-0565838c5baa-kube-api-access-m6l7n\") pod \"dns-default-9cmfx\" (UID: \"a73024cb-f581-450d-a09c-0565838c5baa\") " pod="openshift-dns/dns-default-9cmfx"
Oct 31 07:29:07.243007 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:07.242972    4720 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Oct 31 07:29:07.243007 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:07.242999    4720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/b1db0e3b-e85d-4769-8012-cf4b6f740d9d-config\") pod \"console-operator-7cd56b8dbc-4jszp\" (UID: \"b1db0e3b-e85d-4769-8012-cf4b6f740d9d\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-4jszp"
Oct 31 07:29:07.243174 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:07.243044    4720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/4bfbebfd-2be7-438f-b88d-273f24751f8c-run-ovn\") pod \"ovnkube-node-5p7xw\" (UID: \"4bfbebfd-2be7-438f-b88d-273f24751f8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-5p7xw"
Oct 31 07:29:07.243174 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:07.243084    4720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/08198bc1-314f-4881-833c-e633bac15e4a-multus-cni-dir\") pod \"multus-pmnqx\" (UID: \"08198bc1-314f-4881-833c-e633bac15e4a\") " pod="openshift-multus/multus-pmnqx"
Oct 31 07:29:07.243174 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:07.243134    4720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/993cfed6-806d-44c5-bba8-c02203638fe7-trusted-ca\") pod \"image-registry-7886c7d6c6-qdc9f\" (UID: \"993cfed6-806d-44c5-bba8-c02203638fe7\") " pod="openshift-image-registry/image-registry-7886c7d6c6-qdc9f"
Oct 31 07:29:07.243319 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:07.243171    4720 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-46dh7\""
Oct 31 07:29:07.243319 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:07.243186    4720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/cb9e4575-9265-43b5-bf02-7843611d6563-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-p9z4r\" (UID: \"cb9e4575-9265-43b5-bf02-7843611d6563\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-p9z4r"
Oct 31 07:29:07.243319 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:07.243201    4720 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Oct 31 07:29:07.243476 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:07.243342    4720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/789f11d8-d951-4144-a589-a2bdbf6844e3-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-6k8mn\" (UID: \"789f11d8-d951-4144-a589-a2bdbf6844e3\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-6k8mn"
Oct 31 07:29:07.243476 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:07.243367    4720 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Oct 31 07:29:07.243476 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:07.243390    4720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/4bfbebfd-2be7-438f-b88d-273f24751f8c-systemd-units\") pod \"ovnkube-node-5p7xw\" (UID: \"4bfbebfd-2be7-438f-b88d-273f24751f8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-5p7xw"
Oct 31 07:29:07.243476 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:07.243419    4720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4bfbebfd-2be7-438f-b88d-273f24751f8c-host-slash\") pod \"ovnkube-node-5p7xw\" (UID: \"4bfbebfd-2be7-438f-b88d-273f24751f8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-5p7xw"
Oct 31 07:29:07.243476 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:07.243456    4720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/3fa078c7-4bf2-4e76-9692-ae460e8fadfa-host\") pod \"tuned-cpn4m\" (UID: \"3fa078c7-4bf2-4e76-9692-ae460e8fadfa\") " pod="openshift-cluster-node-tuning-operator/tuned-cpn4m"
Oct 31 07:29:07.243694 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:07.243512    4720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a73024cb-f581-450d-a09c-0565838c5baa-tmp-dir\") pod \"dns-default-9cmfx\" (UID: \"a73024cb-f581-450d-a09c-0565838c5baa\") " pod="openshift-dns/dns-default-9cmfx"
Oct 31 07:29:07.243694 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:07.243626    4720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a73024cb-f581-450d-a09c-0565838c5baa-tmp-dir\") pod \"dns-default-9cmfx\" (UID: \"a73024cb-f581-450d-a09c-0565838c5baa\") " pod="openshift-dns/dns-default-9cmfx"
Oct 31 07:29:07.243791 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:07.243757    4720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ghkb9\" (UniqueName: \"kubernetes.io/projected/f1ba68f6-d87c-4492-a5eb-5f2d3a60eb1c-kube-api-access-ghkb9\") pod \"network-check-target-9zp4l\" (UID: \"f1ba68f6-d87c-4492-a5eb-5f2d3a60eb1c\") " pod="openshift-network-diagnostics/network-check-target-9zp4l"
Oct 31 07:29:07.243848 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:07.243811    4720 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-kwsqb\""
Oct 31 07:29:07.243925 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:07.243904    4720 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Oct 31 07:29:07.243980 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:07.243957    4720 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Oct 31 07:29:07.244033 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:07.243994    4720 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Oct 31 07:29:07.244033 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:07.244016    4720 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Oct 31 07:29:07.244241 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:07.244224    4720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k6576\" (UniqueName: \"kubernetes.io/projected/81829763-27b1-4a48-a568-cd5f6a39a953-kube-api-access-k6576\") pod \"insights-operator-7c4986d449-dwc4p\" (UID: \"81829763-27b1-4a48-a568-cd5f6a39a953\") " pod="openshift-insights/insights-operator-7c4986d449-dwc4p"
Oct 31 07:29:07.244299 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:07.244243    4720 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-dockercfg-d69tc\""
Oct 31 07:29:07.244299 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:07.243917    4720 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Oct 31 07:29:07.244299 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:07.244267    4720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/08198bc1-314f-4881-833c-e633bac15e4a-cni-binary-copy\") pod \"multus-pmnqx\" (UID: \"08198bc1-314f-4881-833c-e633bac15e4a\") " pod="openshift-multus/multus-pmnqx"
Oct 31 07:29:07.244438 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:07.244301    4720 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\""
Oct 31 07:29:07.244438 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:07.244314    4720 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Oct 31 07:29:07.244438 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:07.244399    4720 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-w8pvl\""
Oct 31 07:29:07.244438 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:07.244416    4720 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\""
Oct 31 07:29:07.244618 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:07.244440    4720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/08198bc1-314f-4881-833c-e633bac15e4a-host-var-lib-kubelet\") pod \"multus-pmnqx\" (UID: \"08198bc1-314f-4881-833c-e633bac15e4a\") " pod="openshift-multus/multus-pmnqx"
Oct 31 07:29:07.244618 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:07.244470    4720 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Oct 31 07:29:07.244618 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:07.244475    4720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xvv26\" (UniqueName: \"kubernetes.io/projected/28b59d93-b9ce-4ceb-9f3a-742525eb94c4-kube-api-access-xvv26\") pod \"image-registry-7955c67bcc-pkntv\" (UID: \"28b59d93-b9ce-4ceb-9f3a-742525eb94c4\") " pod="openshift-image-registry/image-registry-7955c67bcc-pkntv"
Oct 31 07:29:07.244618 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:07.244514    4720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/9c72279a-708f-46df-8915-9995c5aea14d-cni-binary-copy\") pod \"multus-additional-cni-plugins-dh6w8\" (UID: \"9c72279a-708f-46df-8915-9995c5aea14d\") " pod="openshift-multus/multus-additional-cni-plugins-dh6w8"
Oct 31 07:29:07.244618 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:07.244551    4720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dx598\" (UniqueName: \"kubernetes.io/projected/631cff9e-a1f1-4dd2-bdc8-ef36519c7c1b-kube-api-access-dx598\") pod \"kas-connection-checker-685895c7c9-f9gtg\" (UID: \"631cff9e-a1f1-4dd2-bdc8-ef36519c7c1b\") " pod="kube-system/kas-connection-checker-685895c7c9-f9gtg"
Oct 31 07:29:07.244618 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:07.244557    4720 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Oct 31 07:29:07.244618 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:07.244586    4720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/b44ed0e0-a72e-4746-a7de-ed8610da845d-stats-auth\") pod \"router-default-6c88c45d57-c2bgn\" (UID: \"b44ed0e0-a72e-4746-a7de-ed8610da845d\") " pod="openshift-ingress/router-default-6c88c45d57-c2bgn"
Oct 31 07:29:07.244907 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:07.244622    4720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/77256436-39ff-4c2a-8390-4e9fa7d68486-original-pull-secret\") pod \"global-pull-secret-syncer-kht4l\" (UID: \"77256436-39ff-4c2a-8390-4e9fa7d68486\") " pod="kube-system/global-pull-secret-syncer-kht4l"
Oct 31 07:29:07.244907 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:07.244638    4720 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-bpwql\""
Oct 31 07:29:07.244907 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:07.244651    4720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/4bfbebfd-2be7-438f-b88d-273f24751f8c-host-cni-bin\") pod \"ovnkube-node-5p7xw\" (UID: \"4bfbebfd-2be7-438f-b88d-273f24751f8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-5p7xw"
Oct 31 07:29:07.244907 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:07.244682    4720 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Oct 31 07:29:07.244907 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:07.244445    4720 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Oct 31 07:29:07.244907 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:07.244897    4720 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Oct 31 07:29:07.245206 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:07.245189    4720 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Oct 31 07:29:07.245264 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:07.244683    4720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3fa078c7-4bf2-4e76-9692-ae460e8fadfa-etc-kubernetes\") pod \"tuned-cpn4m\" (UID: \"3fa078c7-4bf2-4e76-9692-ae460e8fadfa\") " pod="openshift-cluster-node-tuning-operator/tuned-cpn4m"
Oct 31 07:29:07.245264 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:07.245258    4720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jbdqx\" (UniqueName: \"kubernetes.io/projected/26659955-3909-4171-90b8-21b0ad6eef64-kube-api-access-jbdqx\") pod \"iptables-alerter-65hzp\" (UID: \"26659955-3909-4171-90b8-21b0ad6eef64\") " pod="openshift-network-operator/iptables-alerter-65hzp"
Oct 31 07:29:07.245416 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:07.245269    4720 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Oct 31 07:29:07.245416 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:07.245286    4720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/b44ed0e0-a72e-4746-a7de-ed8610da845d-stats-auth\") pod \"router-default-6c88c45d57-c2bgn\" (UID: \"b44ed0e0-a72e-4746-a7de-ed8610da845d\") " pod="openshift-ingress/router-default-6c88c45d57-c2bgn"
Oct 31 07:29:07.245416 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:07.245311    4720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/fc64d31a-f48a-4293-a4de-262045d0b7e8-serviceca\") pod \"node-ca-th7x7\" (UID: \"fc64d31a-f48a-4293-a4de-262045d0b7e8\") " pod="openshift-image-registry/node-ca-th7x7"
Oct 31 07:29:07.245602 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:07.245440    4720 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Oct 31 07:29:07.245602 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:07.245491    4720 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Oct 31 07:29:07.245602 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:07.243902    4720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/b1db0e3b-e85d-4769-8012-cf4b6f740d9d-config\") pod \"console-operator-7cd56b8dbc-4jszp\" (UID: \"b1db0e3b-e85d-4769-8012-cf4b6f740d9d\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-4jszp"
Oct 31 07:29:07.245735 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:07.245709    4720 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Oct 31 07:29:07.245785 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:07.245771    4720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/4bfbebfd-2be7-438f-b88d-273f24751f8c-ovn-node-metrics-cert\") pod \"ovnkube-node-5p7xw\" (UID: \"4bfbebfd-2be7-438f-b88d-273f24751f8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-5p7xw"
Oct 31 07:29:07.245833 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:07.245811    4720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/08198bc1-314f-4881-833c-e633bac15e4a-host-run-k8s-cni-cncf-io\") pod \"multus-pmnqx\" (UID: \"08198bc1-314f-4881-833c-e633bac15e4a\") " pod="openshift-multus/multus-pmnqx"
Oct 31 07:29:07.246274 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:07.245947    4720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/fc64d31a-f48a-4293-a4de-262045d0b7e8-serviceca\") pod \"node-ca-th7x7\" (UID: \"fc64d31a-f48a-4293-a4de-262045d0b7e8\") " pod="openshift-image-registry/node-ca-th7x7"
Oct 31 07:29:07.246274 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:07.245851    4720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kr9wm\" (UniqueName: \"kubernetes.io/projected/cb9e4575-9265-43b5-bf02-7843611d6563-kube-api-access-kr9wm\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-p9z4r\" (UID: \"cb9e4575-9265-43b5-bf02-7843611d6563\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-p9z4r"
Oct 31 07:29:07.246274 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:07.246204    4720 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-gttml\""
Oct 31 07:29:07.246530 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:07.246479    4720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/314c29aa-db6b-4f7f-86b4-c8952ce09c84-nginx-conf\") pod \"networking-console-plugin-86c478bb88-j94dz\" (UID: \"314c29aa-db6b-4f7f-86b4-c8952ce09c84\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-j94dz"
Oct 31 07:29:07.246601 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:07.246577    4720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/4bfbebfd-2be7-438f-b88d-273f24751f8c-ovn-node-metrics-cert\") pod \"ovnkube-node-5p7xw\" (UID: \"4bfbebfd-2be7-438f-b88d-273f24751f8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-5p7xw"
Oct 31 07:29:07.246703 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:07.246145    4720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/9dad8dd1-cfc5-412d-bf9a-bac3c9162af1-device-dir\") pod \"aws-ebs-csi-driver-node-797gj\" (UID: \"9dad8dd1-cfc5-412d-bf9a-bac3c9162af1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-797gj"
Oct 31 07:29:07.246703 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:07.246643    4720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/b44ed0e0-a72e-4746-a7de-ed8610da845d-default-certificate\") pod \"router-default-6c88c45d57-c2bgn\" (UID: \"b44ed0e0-a72e-4746-a7de-ed8610da845d\") " pod="openshift-ingress/router-default-6c88c45d57-c2bgn"
Oct 31 07:29:07.246703 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:07.246684    4720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/81829763-27b1-4a48-a568-cd5f6a39a953-tmp\") pod \"insights-operator-7c4986d449-dwc4p\" (UID: \"81829763-27b1-4a48-a568-cd5f6a39a953\") " pod="openshift-insights/insights-operator-7c4986d449-dwc4p"
Oct 31 07:29:07.246820 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:07.246723    4720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/08198bc1-314f-4881-833c-e633bac15e4a-host-run-multus-certs\") pod \"multus-pmnqx\" (UID: \"08198bc1-314f-4881-833c-e633bac15e4a\") " pod="openshift-multus/multus-pmnqx"
Oct 31 07:29:07.246820 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:07.246773    4720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/28b59d93-b9ce-4ceb-9f3a-742525eb94c4-bound-sa-token\") pod \"image-registry-7955c67bcc-pkntv\" (UID: \"28b59d93-b9ce-4ceb-9f3a-742525eb94c4\") " pod="openshift-image-registry/image-registry-7955c67bcc-pkntv"
Oct 31 07:29:07.247320 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:07.247297    4720 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\""
Oct 31 07:29:07.247430 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:07.247312    4720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/26659955-3909-4171-90b8-21b0ad6eef64-iptables-alerter-script\") pod \"iptables-alerter-65hzp\" (UID: \"26659955-3909-4171-90b8-21b0ad6eef64\") " pod="openshift-network-operator/iptables-alerter-65hzp"
Oct 31 07:29:07.247430 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:07.247304    4720 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\""
Oct 31 07:29:07.247522 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:07.247491    4720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/81829763-27b1-4a48-a568-cd5f6a39a953-tmp\") pod \"insights-operator-7c4986d449-dwc4p\" (UID: \"81829763-27b1-4a48-a568-cd5f6a39a953\") " pod="openshift-insights/insights-operator-7c4986d449-dwc4p"
Oct 31 07:29:07.247522 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:07.247511    4720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5-agent-certs\") pod \"konnectivity-agent-h4k2m\" (UID: \"2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5\") " pod="kube-system/konnectivity-agent-h4k2m"
Oct 31 07:29:07.247616 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:07.247542    4720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c4233c69-4213-43be-8809-76612b6b6f3e-tmp-dir\") pod \"node-resolver-tnw4j\" (UID: \"c4233c69-4213-43be-8809-76612b6b6f3e\") " pod="openshift-dns/node-resolver-tnw4j"
Oct 31 07:29:07.247616 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:07.247553    4720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/b44ed0e0-a72e-4746-a7de-ed8610da845d-default-certificate\") pod \"router-default-6c88c45d57-c2bgn\" (UID: \"b44ed0e0-a72e-4746-a7de-ed8610da845d\") " pod="openshift-ingress/router-default-6c88c45d57-c2bgn"
Oct 31 07:29:07.247616 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:07.247567    4720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/4bfbebfd-2be7-438f-b88d-273f24751f8c-node-log\") pod \"ovnkube-node-5p7xw\" (UID: \"4bfbebfd-2be7-438f-b88d-273f24751f8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-5p7xw"
Oct 31 07:29:07.247616 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:07.247584    4720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/08198bc1-314f-4881-833c-e633bac15e4a-system-cni-dir\") pod \"multus-pmnqx\" (UID: \"08198bc1-314f-4881-833c-e633bac15e4a\") " pod="openshift-multus/multus-pmnqx"
Oct 31 07:29:07.247616 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:07.247601    4720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b44ed0e0-a72e-4746-a7de-ed8610da845d-service-ca-bundle\") pod \"router-default-6c88c45d57-c2bgn\" (UID: \"b44ed0e0-a72e-4746-a7de-ed8610da845d\") " pod="openshift-ingress/router-default-6c88c45d57-c2bgn"
Oct 31 07:29:07.247841 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:07.247628    4720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wqmxh\" (UniqueName: \"kubernetes.io/projected/789f11d8-d951-4144-a589-a2bdbf6844e3-kube-api-access-wqmxh\") pod \"cluster-monitoring-operator-884576c57-6k8mn\" (UID: \"789f11d8-d951-4144-a589-a2bdbf6844e3\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-6k8mn"
Oct 31 07:29:07.247841 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:07.247669    4720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/26659955-3909-4171-90b8-21b0ad6eef64-host-slash\") pod \"iptables-alerter-65hzp\" (UID: \"26659955-3909-4171-90b8-21b0ad6eef64\") " pod="openshift-network-operator/iptables-alerter-65hzp"
Oct 31 07:29:07.247841 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:07.247729    4720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c4233c69-4213-43be-8809-76612b6b6f3e-tmp-dir\") pod \"node-resolver-tnw4j\" (UID: \"c4233c69-4213-43be-8809-76612b6b6f3e\") " pod="openshift-dns/node-resolver-tnw4j"
Oct 31 07:29:07.247841 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:07.247761    4720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/08198bc1-314f-4881-833c-e633bac15e4a-host-var-lib-cni-multus\") pod \"multus-pmnqx\" (UID: \"08198bc1-314f-4881-833c-e633bac15e4a\") " pod="openshift-multus/multus-pmnqx"
Oct 31 07:29:07.247841 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:07.247786    4720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5hqbq\" (UniqueName: \"kubernetes.io/projected/c4233c69-4213-43be-8809-76612b6b6f3e-kube-api-access-5hqbq\") pod \"node-resolver-tnw4j\" (UID: \"c4233c69-4213-43be-8809-76612b6b6f3e\") " pod="openshift-dns/node-resolver-tnw4j"
Oct 31 07:29:07.247841 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:07.247803    4720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/3fa078c7-4bf2-4e76-9692-ae460e8fadfa-etc-modprobe-d\") pod \"tuned-cpn4m\" (UID: \"3fa078c7-4bf2-4e76-9692-ae460e8fadfa\") " pod="openshift-cluster-node-tuning-operator/tuned-cpn4m"
Oct 31 07:29:07.248110 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:07.247863    4720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/3fa078c7-4bf2-4e76-9692-ae460e8fadfa-etc-tuned\") pod \"tuned-cpn4m\" (UID: \"3fa078c7-4bf2-4e76-9692-ae460e8fadfa\") " pod="openshift-cluster-node-tuning-operator/tuned-cpn4m"
Oct 31 07:29:07.248110 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:07.247965    4720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/3fa078c7-4bf2-4e76-9692-ae460e8fadfa-etc-tuned\") pod \"tuned-cpn4m\" (UID: \"3fa078c7-4bf2-4e76-9692-ae460e8fadfa\") " pod="openshift-cluster-node-tuning-operator/tuned-cpn4m"
Oct 31 07:29:07.248110 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:07.247985    4720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/28b59d93-b9ce-4ceb-9f3a-742525eb94c4-installation-pull-secrets\") pod \"image-registry-7955c67bcc-pkntv\" (UID: \"28b59d93-b9ce-4ceb-9f3a-742525eb94c4\") " pod="openshift-image-registry/image-registry-7955c67bcc-pkntv"
Oct 31 07:29:07.248110 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:07.248015    4720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/9dad8dd1-cfc5-412d-bf9a-bac3c9162af1-socket-dir\") pod \"aws-ebs-csi-driver-node-797gj\" (UID: \"9dad8dd1-cfc5-412d-bf9a-bac3c9162af1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-797gj"
Oct 31 07:29:07.248110 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:07.248041    4720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9c72279a-708f-46df-8915-9995c5aea14d-system-cni-dir\") pod \"multus-additional-cni-plugins-dh6w8\" (UID: \"9c72279a-708f-46df-8915-9995c5aea14d\") " pod="openshift-multus/multus-additional-cni-plugins-dh6w8"
Oct 31 07:29:07.248110 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:07.248085    4720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/789f11d8-d951-4144-a589-a2bdbf6844e3-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-6k8mn\" (UID: \"789f11d8-d951-4144-a589-a2bdbf6844e3\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-6k8mn"
Oct 31 07:29:07.248110 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:07.248106    4720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/3fa078c7-4bf2-4e76-9692-ae460e8fadfa-run\") pod \"tuned-cpn4m\" (UID: \"3fa078c7-4bf2-4e76-9692-ae460e8fadfa\") " pod="openshift-cluster-node-tuning-operator/tuned-cpn4m"
Oct 31 07:29:07.248450 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:07.248131    4720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3fa078c7-4bf2-4e76-9692-ae460e8fadfa-sys\") pod \"tuned-cpn4m\" (UID: \"3fa078c7-4bf2-4e76-9692-ae460e8fadfa\") " pod="openshift-cluster-node-tuning-operator/tuned-cpn4m"
Oct 31 07:29:07.248450 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:07.248158    4720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/4db4c33f-0c58-40f8-bf74-51b3eae3c9db-config\") pod \"service-ca-operator-544d749c8c-hb49h\" (UID: \"4db4c33f-0c58-40f8-bf74-51b3eae3c9db\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-hb49h"
Oct 31 07:29:07.248450 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:07.248215    4720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/9c72279a-708f-46df-8915-9995c5aea14d-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-dh6w8\" (UID: \"9c72279a-708f-46df-8915-9995c5aea14d\") " pod="openshift-multus/multus-additional-cni-plugins-dh6w8"
Oct 31 07:29:07.248450 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:07.248249    4720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/a73024cb-f581-450d-a09c-0565838c5baa-config-volume\") pod \"dns-default-9cmfx\" (UID: \"a73024cb-f581-450d-a09c-0565838c5baa\") " pod="openshift-dns/dns-default-9cmfx"
Oct 31 07:29:07.248450 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:07.248267    4720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/28b59d93-b9ce-4ceb-9f3a-742525eb94c4-installation-pull-secrets\") pod \"image-registry-7955c67bcc-pkntv\" (UID: \"28b59d93-b9ce-4ceb-9f3a-742525eb94c4\") " pod="openshift-image-registry/image-registry-7955c67bcc-pkntv"
Oct 31 07:29:07.248450 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:07.248278    4720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/77256436-39ff-4c2a-8390-4e9fa7d68486-dbus\") pod \"global-pull-secret-syncer-kht4l\" (UID: \"77256436-39ff-4c2a-8390-4e9fa7d68486\") " pod="kube-system/global-pull-secret-syncer-kht4l"
Oct 31 07:29:07.248450 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:07.248317    4720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/c4233c69-4213-43be-8809-76612b6b6f3e-hosts-file\") pod \"node-resolver-tnw4j\" (UID: \"c4233c69-4213-43be-8809-76612b6b6f3e\") " pod="openshift-dns/node-resolver-tnw4j"
Oct 31 07:29:07.248450 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:07.248377    4720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l4nfd\" (UniqueName: \"kubernetes.io/projected/323c8334-9a56-4c4e-8a6e-ae80995810d0-kube-api-access-l4nfd\") pod \"kas-connection-checker-685895c7c9-hxmpf\" (UID: \"323c8334-9a56-4c4e-8a6e-ae80995810d0\") " pod="kube-system/kas-connection-checker-685895c7c9-hxmpf"
Oct 31 07:29:07.248450 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:07.248379    4720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/789f11d8-d951-4144-a589-a2bdbf6844e3-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-6k8mn\" (UID: \"789f11d8-d951-4144-a589-a2bdbf6844e3\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-6k8mn"
Oct 31 07:29:07.248450 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:07.248405    4720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/4db4c33f-0c58-40f8-bf74-51b3eae3c9db-config\") pod \"service-ca-operator-544d749c8c-hb49h\" (UID: \"4db4c33f-0c58-40f8-bf74-51b3eae3c9db\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-hb49h"
Oct 31 07:29:07.248450 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:07.248435    4720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/3fa078c7-4bf2-4e76-9692-ae460e8fadfa-etc-sysctl-d\") pod \"tuned-cpn4m\" (UID: \"3fa078c7-4bf2-4e76-9692-ae460e8fadfa\") " pod="openshift-cluster-node-tuning-operator/tuned-cpn4m"
Oct 31 07:29:07.248904 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:07.248525    4720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/08198bc1-314f-4881-833c-e633bac15e4a-hostroot\") pod \"multus-pmnqx\" (UID: \"08198bc1-314f-4881-833c-e633bac15e4a\") " pod="openshift-multus/multus-pmnqx"
Oct 31 07:29:07.248904 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:07.248553    4720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/993cfed6-806d-44c5-bba8-c02203638fe7-bound-sa-token\") pod \"image-registry-7886c7d6c6-qdc9f\" (UID: \"993cfed6-806d-44c5-bba8-c02203638fe7\") " pod="openshift-image-registry/image-registry-7886c7d6c6-qdc9f"
Oct 31 07:29:07.248904 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:07.248571    4720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v2zhk\" (UniqueName: \"kubernetes.io/projected/993cfed6-806d-44c5-bba8-c02203638fe7-kube-api-access-v2zhk\") pod \"image-registry-7886c7d6c6-qdc9f\" (UID: \"993cfed6-806d-44c5-bba8-c02203638fe7\") " pod="openshift-image-registry/image-registry-7886c7d6c6-qdc9f"
Oct 31 07:29:07.248904 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:07.248588    4720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/4bfbebfd-2be7-438f-b88d-273f24751f8c-env-overrides\") pod \"ovnkube-node-5p7xw\" (UID: \"4bfbebfd-2be7-438f-b88d-273f24751f8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-5p7xw"
Oct 31 07:29:07.248904 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:07.248608    4720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/b1066ece-e6f3-4491-aa1f-a7b9a3aa248d-crio-socket\") pod \"insights-runtime-extractor-8m768\" (UID: \"b1066ece-e6f3-4491-aa1f-a7b9a3aa248d\") " pod="openshift-insights/insights-runtime-extractor-8m768"
Oct 31 07:29:07.248904 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:07.248708    4720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/993cfed6-806d-44c5-bba8-c02203638fe7-image-registry-private-configuration\") pod \"image-registry-7886c7d6c6-qdc9f\" (UID: \"993cfed6-806d-44c5-bba8-c02203638fe7\") " pod="openshift-image-registry/image-registry-7886c7d6c6-qdc9f"
Oct 31 07:29:07.248904 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:07.248718    4720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/4bfbebfd-2be7-438f-b88d-273f24751f8c-env-overrides\") pod \"ovnkube-node-5p7xw\" (UID: \"4bfbebfd-2be7-438f-b88d-273f24751f8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-5p7xw"
Oct 31 07:29:07.248904 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:07.248751    4720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/993cfed6-806d-44c5-bba8-c02203638fe7-ca-trust-extracted\") pod \"image-registry-7886c7d6c6-qdc9f\" (UID: \"993cfed6-806d-44c5-bba8-c02203638fe7\") " pod="openshift-image-registry/image-registry-7886c7d6c6-qdc9f"
Oct 31 07:29:07.248904 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:07.248780    4720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b44ed0e0-a72e-4746-a7de-ed8610da845d-metrics-certs\") pod \"router-default-6c88c45d57-c2bgn\" (UID: \"b44ed0e0-a72e-4746-a7de-ed8610da845d\") " pod="openshift-ingress/router-default-6c88c45d57-c2bgn"
Oct 31 07:29:07.248904 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:07.248812    4720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/81829763-27b1-4a48-a568-cd5f6a39a953-snapshots\") pod \"insights-operator-7c4986d449-dwc4p\" (UID: \"81829763-27b1-4a48-a568-cd5f6a39a953\") " pod="openshift-insights/insights-operator-7c4986d449-dwc4p"
Oct 31 07:29:07.248904 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:07.248825    4720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/993cfed6-806d-44c5-bba8-c02203638fe7-image-registry-private-configuration\") pod \"image-registry-7886c7d6c6-qdc9f\" (UID: \"993cfed6-806d-44c5-bba8-c02203638fe7\") " pod="openshift-image-registry/image-registry-7886c7d6c6-qdc9f"
Oct 31 07:29:07.248904 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:07.248827    4720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/993cfed6-806d-44c5-bba8-c02203638fe7-ca-trust-extracted\") pod \"image-registry-7886c7d6c6-qdc9f\" (UID: \"993cfed6-806d-44c5-bba8-c02203638fe7\") " pod="openshift-image-registry/image-registry-7886c7d6c6-qdc9f"
Oct 31 07:29:07.248904 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:07.248828    4720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/4bfbebfd-2be7-438f-b88d-273f24751f8c-ovnkube-config\") pod \"ovnkube-node-5p7xw\" (UID: \"4bfbebfd-2be7-438f-b88d-273f24751f8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-5p7xw"
Oct 31 07:29:07.248904 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:07.248876    4720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q7vwq\" (UniqueName: \"kubernetes.io/projected/ee4a4a2a-7439-4c78-8858-d6ba0085fdde-kube-api-access-q7vwq\") pod \"volume-data-source-validator-58c9f5f46d-65chl\" (UID: \"ee4a4a2a-7439-4c78-8858-d6ba0085fdde\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-65chl"
Oct 31 07:29:07.249443 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:07.248987    4720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/4bfbebfd-2be7-438f-b88d-273f24751f8c-ovnkube-config\") pod \"ovnkube-node-5p7xw\" (UID: \"4bfbebfd-2be7-438f-b88d-273f24751f8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-5p7xw"
Oct 31 07:29:07.249443 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:07.249128    4720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/81829763-27b1-4a48-a568-cd5f6a39a953-snapshots\") pod \"insights-operator-7c4986d449-dwc4p\" (UID: \"81829763-27b1-4a48-a568-cd5f6a39a953\") " pod="openshift-insights/insights-operator-7c4986d449-dwc4p"
Oct 31 07:29:07.254174 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:07.254149    4720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/4bfbebfd-2be7-438f-b88d-273f24751f8c-ovnkube-script-lib\") pod \"ovnkube-node-5p7xw\" (UID: \"4bfbebfd-2be7-438f-b88d-273f24751f8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-5p7xw"
Oct 31 07:29:07.254306 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:07.254288    4720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/993cfed6-806d-44c5-bba8-c02203638fe7-trusted-ca\") pod \"image-registry-7886c7d6c6-qdc9f\" (UID: \"993cfed6-806d-44c5-bba8-c02203638fe7\") " pod="openshift-image-registry/image-registry-7886c7d6c6-qdc9f"
Oct 31 07:29:07.254410 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:07.254351    4720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/28b59d93-b9ce-4ceb-9f3a-742525eb94c4-trusted-ca\") pod \"image-registry-7955c67bcc-pkntv\" (UID: \"28b59d93-b9ce-4ceb-9f3a-742525eb94c4\") " pod="openshift-image-registry/image-registry-7955c67bcc-pkntv"
Oct 31 07:29:07.254596 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:07.254577    4720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b1db0e3b-e85d-4769-8012-cf4b6f740d9d-trusted-ca\") pod \"console-operator-7cd56b8dbc-4jszp\" (UID: \"b1db0e3b-e85d-4769-8012-cf4b6f740d9d\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-4jszp"
Oct 31 07:29:07.254855 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:07.254835    4720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/81829763-27b1-4a48-a568-cd5f6a39a953-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-dwc4p\" (UID: \"81829763-27b1-4a48-a568-cd5f6a39a953\") " pod="openshift-insights/insights-operator-7c4986d449-dwc4p"
Oct 31 07:29:07.275184 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:07.275135    4720 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Oct 31 07:29:07.275303 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:07.275193    4720 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Oct 31 07:29:07.275303 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:07.275255    4720 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Oct 31 07:29:07.275303 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:07.275258    4720 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Oct 31 07:29:07.275499 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:07.275305    4720 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Oct 31 07:29:07.275499 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:07.275342    4720 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Oct 31 07:29:07.275499 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:07.275346    4720 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-798mv\""
Oct 31 07:29:07.275499 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:07.275360    4720 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Oct 31 07:29:07.275499 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:07.275457    4720 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Oct 31 07:29:07.275499 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:07.275463    4720 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Oct 31 07:29:07.275682 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:07.275560    4720 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Oct 31 07:29:07.275715 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:07.275680    4720 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Oct 31 07:29:07.275715 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:07.275686    4720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/9c72279a-708f-46df-8915-9995c5aea14d-cni-binary-copy\") pod \"multus-additional-cni-plugins-dh6w8\" (UID: \"9c72279a-708f-46df-8915-9995c5aea14d\") " pod="openshift-multus/multus-additional-cni-plugins-dh6w8"
Oct 31 07:29:07.275808 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:07.275731    4720 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Oct 31 07:29:07.275808 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:07.275756    4720 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-ldrdc\""
Oct 31 07:29:07.275808 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:07.275755    4720 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-pvmcp\""
Oct 31 07:29:07.275808 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:07.275770    4720 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Oct 31 07:29:07.275974 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:07.275961    4720 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-5qqnh\""
Oct 31 07:29:07.276023 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:07.276010    4720 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-xb4h7\""
Oct 31 07:29:07.276069 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:07.276018    4720 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Oct 31 07:29:07.276116 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:07.276077    4720 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-g75wk\""
Oct 31 07:29:07.276116 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:07.276082    4720 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-m5kd5\""
Oct 31 07:29:07.276116 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:07.276037    4720 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-w5nxl\""
Oct 31 07:29:07.276229 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:07.276174    4720 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Oct 31 07:29:07.276229 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:07.276184    4720 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-jtbjn\""
Oct 31 07:29:07.276229 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:07.276209    4720 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-7fk4f\""
Oct 31 07:29:07.276229 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:07.276221    4720 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Oct 31 07:29:07.276229 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:07.276226    4720 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-6g5bx\""
Oct 31 07:29:07.276498 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:07.276239    4720 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-crxcw\""
Oct 31 07:29:07.276498 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:07.276242    4720 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Oct 31 07:29:07.276498 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:07.276282    4720 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Oct 31 07:29:07.278228 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:07.278210    4720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5-agent-certs\") pod \"konnectivity-agent-h4k2m\" (UID: \"2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5\") " pod="kube-system/konnectivity-agent-h4k2m"
Oct 31 07:29:07.278488 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:07.278472    4720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cjt6r\" (UniqueName: \"kubernetes.io/projected/b1066ece-e6f3-4491-aa1f-a7b9a3aa248d-kube-api-access-cjt6r\") pod \"insights-runtime-extractor-8m768\" (UID: \"b1066ece-e6f3-4491-aa1f-a7b9a3aa248d\") " pod="openshift-insights/insights-runtime-extractor-8m768"
Oct 31 07:29:07.278654 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:07.278639    4720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/9c72279a-708f-46df-8915-9995c5aea14d-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-dh6w8\" (UID: \"9c72279a-708f-46df-8915-9995c5aea14d\") " pod="openshift-multus/multus-additional-cni-plugins-dh6w8"
Oct 31 07:29:07.278703 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:07.278656    4720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/a73024cb-f581-450d-a09c-0565838c5baa-config-volume\") pod \"dns-default-9cmfx\" (UID: \"a73024cb-f581-450d-a09c-0565838c5baa\") " pod="openshift-dns/dns-default-9cmfx"
Oct 31 07:29:07.280685 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:07.280667    4720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n6p25\" (UniqueName: \"kubernetes.io/projected/ca31519e-aa90-4104-a175-f9af2b3bbeb4-kube-api-access-n6p25\") pod \"cluster-samples-operator-6ddb6c8b94-6vkbv\" (UID: \"ca31519e-aa90-4104-a175-f9af2b3bbeb4\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-6vkbv"
Oct 31 07:29:07.280783 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:07.280767    4720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kxwb9\" (UniqueName: \"kubernetes.io/projected/c3ad95fd-e997-459c-93fd-27c4c030ea0d-kube-api-access-kxwb9\") pod \"network-metrics-daemon-r2zbv\" (UID: \"c3ad95fd-e997-459c-93fd-27c4c030ea0d\") " pod="openshift-multus/network-metrics-daemon-r2zbv"
Oct 31 07:29:07.282449 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:07.282421    4720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/9c72279a-708f-46df-8915-9995c5aea14d-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-dh6w8\" (UID: \"9c72279a-708f-46df-8915-9995c5aea14d\") " pod="openshift-multus/multus-additional-cni-plugins-dh6w8"
Oct 31 07:29:07.282692 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:07.282673    4720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nhphg\" (UniqueName: \"kubernetes.io/projected/0b25f791-3520-481d-acbb-a3bef222fdac-kube-api-access-nhphg\") pod \"migrator-66df5d496b-5t4pm\" (UID: \"0b25f791-3520-481d-acbb-a3bef222fdac\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-5t4pm"
Oct 31 07:29:07.283129 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:07.283102    4720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kk4nh\" (UniqueName: \"kubernetes.io/projected/fc64d31a-f48a-4293-a4de-262045d0b7e8-kube-api-access-kk4nh\") pod \"node-ca-th7x7\" (UID: \"fc64d31a-f48a-4293-a4de-262045d0b7e8\") " pod="openshift-image-registry/node-ca-th7x7"
Oct 31 07:29:07.283186 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:07.283151    4720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-czqtq\" (UniqueName: \"kubernetes.io/projected/b1db0e3b-e85d-4769-8012-cf4b6f740d9d-kube-api-access-czqtq\") pod \"console-operator-7cd56b8dbc-4jszp\" (UID: \"b1db0e3b-e85d-4769-8012-cf4b6f740d9d\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-4jszp"
Oct 31 07:29:07.283186 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:07.283172    4720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5n5lv\" (UniqueName: \"kubernetes.io/projected/9dad8dd1-cfc5-412d-bf9a-bac3c9162af1-kube-api-access-5n5lv\") pod \"aws-ebs-csi-driver-node-797gj\" (UID: \"9dad8dd1-cfc5-412d-bf9a-bac3c9162af1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-797gj"
Oct 31 07:29:07.283735 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:07.283717    4720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5-konnectivity-ca\") pod \"konnectivity-agent-h4k2m\" (UID: \"2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5\") " pod="kube-system/konnectivity-agent-h4k2m"
Oct 31 07:29:07.283785 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:07.283775    4720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/08198bc1-314f-4881-833c-e633bac15e4a-multus-daemon-config\") pod \"multus-pmnqx\" (UID: \"08198bc1-314f-4881-833c-e633bac15e4a\") " pod="openshift-multus/multus-pmnqx"
Oct 31 07:29:07.284062 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:07.284047    4720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bfzv8\" (UniqueName: \"kubernetes.io/projected/ce8cac52-291b-47fb-a7b6-4cf7562449b9-kube-api-access-bfzv8\") pod \"network-check-source-6f78d59f99-fqhzf\" (UID: \"ce8cac52-291b-47fb-a7b6-4cf7562449b9\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-fqhzf"
Oct 31 07:29:07.285049 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:07.285035    4720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/08198bc1-314f-4881-833c-e633bac15e4a-cni-binary-copy\") pod \"multus-pmnqx\" (UID: \"08198bc1-314f-4881-833c-e633bac15e4a\") " pod="openshift-multus/multus-pmnqx"
Oct 31 07:29:07.285252 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:07.285233    4720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2536c1c8-8a46-4af3-bc6d-81454a908d3f-cert\") pod \"ingress-canary-kmcg5\" (UID: \"2536c1c8-8a46-4af3-bc6d-81454a908d3f\") " pod="openshift-ingress-canary/ingress-canary-kmcg5"
Oct 31 07:29:07.285301 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:07.285239    4720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/77256436-39ff-4c2a-8390-4e9fa7d68486-original-pull-secret\") pod \"global-pull-secret-syncer-kht4l\" (UID: \"77256436-39ff-4c2a-8390-4e9fa7d68486\") " pod="kube-system/global-pull-secret-syncer-kht4l"
Oct 31 07:29:07.285436 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:07.285423    4720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cz8tm\" (UniqueName: \"kubernetes.io/projected/4bfbebfd-2be7-438f-b88d-273f24751f8c-kube-api-access-cz8tm\") pod \"ovnkube-node-5p7xw\" (UID: \"4bfbebfd-2be7-438f-b88d-273f24751f8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-5p7xw"
Oct 31 07:29:07.297217 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:07.297196    4720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q2kvz\" (UniqueName: \"kubernetes.io/projected/4db4c33f-0c58-40f8-bf74-51b3eae3c9db-kube-api-access-q2kvz\") pod \"service-ca-operator-544d749c8c-hb49h\" (UID: \"4db4c33f-0c58-40f8-bf74-51b3eae3c9db\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-hb49h"
Oct 31 07:29:07.297579 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:07.297562    4720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sn2lx\" (UniqueName: \"kubernetes.io/projected/91363236-08db-4f54-ae0a-e68604fcecc6-kube-api-access-sn2lx\") pod \"kas-connection-checker-685895c7c9-xcrsn\" (UID: \"91363236-08db-4f54-ae0a-e68604fcecc6\") " pod="kube-system/kas-connection-checker-685895c7c9-xcrsn"
Oct 31 07:29:07.298937 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:07.298914    4720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h4gzh\" (UniqueName: \"kubernetes.io/projected/9c72279a-708f-46df-8915-9995c5aea14d-kube-api-access-h4gzh\") pod \"multus-additional-cni-plugins-dh6w8\" (UID: \"9c72279a-708f-46df-8915-9995c5aea14d\") " pod="openshift-multus/multus-additional-cni-plugins-dh6w8"
Oct 31 07:29:07.299103 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:07.299087    4720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cq7rv\" (UniqueName: \"kubernetes.io/projected/3fa078c7-4bf2-4e76-9692-ae460e8fadfa-kube-api-access-cq7rv\") pod \"tuned-cpn4m\" (UID: \"3fa078c7-4bf2-4e76-9692-ae460e8fadfa\") " pod="openshift-cluster-node-tuning-operator/tuned-cpn4m"
Oct 31 07:29:07.299284 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:07.299268    4720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9pvp2\" (UniqueName: \"kubernetes.io/projected/b44ed0e0-a72e-4746-a7de-ed8610da845d-kube-api-access-9pvp2\") pod \"router-default-6c88c45d57-c2bgn\" (UID: \"b44ed0e0-a72e-4746-a7de-ed8610da845d\") " pod="openshift-ingress/router-default-6c88c45d57-c2bgn"
Oct 31 07:29:07.299371 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:07.299351    4720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g2r66\" (UniqueName: \"kubernetes.io/projected/2536c1c8-8a46-4af3-bc6d-81454a908d3f-kube-api-access-g2r66\") pod \"ingress-canary-kmcg5\" (UID: \"2536c1c8-8a46-4af3-bc6d-81454a908d3f\") " pod="openshift-ingress-canary/ingress-canary-kmcg5"
Oct 31 07:29:07.302433 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:07.302415    4720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m6l7n\" (UniqueName: \"kubernetes.io/projected/a73024cb-f581-450d-a09c-0565838c5baa-kube-api-access-m6l7n\") pod \"dns-default-9cmfx\" (UID: \"a73024cb-f581-450d-a09c-0565838c5baa\") " pod="openshift-dns/dns-default-9cmfx"
Oct 31 07:29:07.302506 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:07.302490    4720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xs9lt\" (UniqueName: \"kubernetes.io/projected/08198bc1-314f-4881-833c-e633bac15e4a-kube-api-access-xs9lt\") pod \"multus-pmnqx\" (UID: \"08198bc1-314f-4881-833c-e633bac15e4a\") " pod="openshift-multus/multus-pmnqx"
Oct 31 07:29:07.302984 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:07.302969    4720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xvv26\" (UniqueName: \"kubernetes.io/projected/28b59d93-b9ce-4ceb-9f3a-742525eb94c4-kube-api-access-xvv26\") pod \"image-registry-7955c67bcc-pkntv\" (UID: \"28b59d93-b9ce-4ceb-9f3a-742525eb94c4\") " pod="openshift-image-registry/image-registry-7955c67bcc-pkntv"
Oct 31 07:29:07.303528 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:07.303508    4720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kr9wm\" (UniqueName: \"kubernetes.io/projected/cb9e4575-9265-43b5-bf02-7843611d6563-kube-api-access-kr9wm\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-p9z4r\" (UID: \"cb9e4575-9265-43b5-bf02-7843611d6563\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-p9z4r"
Oct 31 07:29:07.303805 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:07.303791    4720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k6576\" (UniqueName: \"kubernetes.io/projected/81829763-27b1-4a48-a568-cd5f6a39a953-kube-api-access-k6576\") pod \"insights-operator-7c4986d449-dwc4p\" (UID: \"81829763-27b1-4a48-a568-cd5f6a39a953\") " pod="openshift-insights/insights-operator-7c4986d449-dwc4p"
Oct 31 07:29:07.304342 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:07.304312    4720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jbdqx\" (UniqueName: \"kubernetes.io/projected/26659955-3909-4171-90b8-21b0ad6eef64-kube-api-access-jbdqx\") pod \"iptables-alerter-65hzp\" (UID: \"26659955-3909-4171-90b8-21b0ad6eef64\") " pod="openshift-network-operator/iptables-alerter-65hzp"
Oct 31 07:29:07.305981 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:07.305966    4720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/993cfed6-806d-44c5-bba8-c02203638fe7-bound-sa-token\") pod \"image-registry-7886c7d6c6-qdc9f\" (UID: \"993cfed6-806d-44c5-bba8-c02203638fe7\") " pod="openshift-image-registry/image-registry-7886c7d6c6-qdc9f"
Oct 31 07:29:07.306686 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:07.306670    4720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v2zhk\" (UniqueName: \"kubernetes.io/projected/993cfed6-806d-44c5-bba8-c02203638fe7-kube-api-access-v2zhk\") pod \"image-registry-7886c7d6c6-qdc9f\" (UID: \"993cfed6-806d-44c5-bba8-c02203638fe7\") " pod="openshift-image-registry/image-registry-7886c7d6c6-qdc9f"
Oct 31 07:29:07.306810 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:07.306795    4720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q7vwq\" (UniqueName: \"kubernetes.io/projected/ee4a4a2a-7439-4c78-8858-d6ba0085fdde-kube-api-access-q7vwq\") pod \"volume-data-source-validator-58c9f5f46d-65chl\" (UID: \"ee4a4a2a-7439-4c78-8858-d6ba0085fdde\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-65chl"
Oct 31 07:29:07.307532 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:07.307517    4720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ghkb9\" (UniqueName: \"kubernetes.io/projected/f1ba68f6-d87c-4492-a5eb-5f2d3a60eb1c-kube-api-access-ghkb9\") pod \"network-check-target-9zp4l\" (UID: \"f1ba68f6-d87c-4492-a5eb-5f2d3a60eb1c\") " pod="openshift-network-diagnostics/network-check-target-9zp4l"
Oct 31 07:29:07.307565 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:07.307551    4720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5hqbq\" (UniqueName: \"kubernetes.io/projected/c4233c69-4213-43be-8809-76612b6b6f3e-kube-api-access-5hqbq\") pod \"node-resolver-tnw4j\" (UID: \"c4233c69-4213-43be-8809-76612b6b6f3e\") " pod="openshift-dns/node-resolver-tnw4j"
Oct 31 07:29:07.308412 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:07.308383    4720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l4nfd\" (UniqueName: \"kubernetes.io/projected/323c8334-9a56-4c4e-8a6e-ae80995810d0-kube-api-access-l4nfd\") pod \"kas-connection-checker-685895c7c9-hxmpf\" (UID: \"323c8334-9a56-4c4e-8a6e-ae80995810d0\") " pod="kube-system/kas-connection-checker-685895c7c9-hxmpf"
Oct 31 07:29:07.308691 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:07.308677    4720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dx598\" (UniqueName: \"kubernetes.io/projected/631cff9e-a1f1-4dd2-bdc8-ef36519c7c1b-kube-api-access-dx598\") pod \"kas-connection-checker-685895c7c9-f9gtg\" (UID: \"631cff9e-a1f1-4dd2-bdc8-ef36519c7c1b\") " pod="kube-system/kas-connection-checker-685895c7c9-f9gtg"
Oct 31 07:29:07.308809 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:07.308795    4720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/28b59d93-b9ce-4ceb-9f3a-742525eb94c4-bound-sa-token\") pod \"image-registry-7955c67bcc-pkntv\" (UID: \"28b59d93-b9ce-4ceb-9f3a-742525eb94c4\") " pod="openshift-image-registry/image-registry-7955c67bcc-pkntv"
Oct 31 07:29:07.309183 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:07.309168    4720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wqmxh\" (UniqueName: \"kubernetes.io/projected/789f11d8-d951-4144-a589-a2bdbf6844e3-kube-api-access-wqmxh\") pod \"cluster-monitoring-operator-884576c57-6k8mn\" (UID: \"789f11d8-d951-4144-a589-a2bdbf6844e3\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-6k8mn"
Oct 31 07:29:07.350076 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:07.350055    4720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/08198bc1-314f-4881-833c-e633bac15e4a-multus-socket-dir-parent\") pod \"multus-pmnqx\" (UID: \"08198bc1-314f-4881-833c-e633bac15e4a\") " pod="openshift-multus/multus-pmnqx"
Oct 31 07:29:07.350169 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:07.350081    4720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b1066ece-e6f3-4491-aa1f-a7b9a3aa248d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-8m768\" (UID: \"b1066ece-e6f3-4491-aa1f-a7b9a3aa248d\") " pod="openshift-insights/insights-runtime-extractor-8m768"
Oct 31 07:29:07.350169 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:07.350097    4720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c3ad95fd-e997-459c-93fd-27c4c030ea0d-metrics-certs\") pod \"network-metrics-daemon-r2zbv\" (UID: \"c3ad95fd-e997-459c-93fd-27c4c030ea0d\") " pod="openshift-multus/network-metrics-daemon-r2zbv"
Oct 31 07:29:07.350169 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:07.350111    4720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/fc64d31a-f48a-4293-a4de-262045d0b7e8-host\") pod \"node-ca-th7x7\" (UID: \"fc64d31a-f48a-4293-a4de-262045d0b7e8\") " pod="openshift-image-registry/node-ca-th7x7"
Oct 31 07:29:07.350169 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:07.350125    4720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3fa078c7-4bf2-4e76-9692-ae460e8fadfa-var-lib-kubelet\") pod \"tuned-cpn4m\" (UID: \"3fa078c7-4bf2-4e76-9692-ae460e8fadfa\") " pod="openshift-cluster-node-tuning-operator/tuned-cpn4m"
Oct 31 07:29:07.350169 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:07.350139    4720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/08198bc1-314f-4881-833c-e633bac15e4a-cnibin\") pod \"multus-pmnqx\" (UID: \"08198bc1-314f-4881-833c-e633bac15e4a\") " pod="openshift-multus/multus-pmnqx"
Oct 31 07:29:07.350169 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:07.350162    4720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a73024cb-f581-450d-a09c-0565838c5baa-metrics-tls\") pod \"dns-default-9cmfx\" (UID: \"a73024cb-f581-450d-a09c-0565838c5baa\") " pod="openshift-dns/dns-default-9cmfx"
Oct 31 07:29:07.350480 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:07.350184    4720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/4bfbebfd-2be7-438f-b88d-273f24751f8c-run-ovn\") pod \"ovnkube-node-5p7xw\" (UID: \"4bfbebfd-2be7-438f-b88d-273f24751f8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-5p7xw"
Oct 31 07:29:07.350480 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:07.350184    4720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/08198bc1-314f-4881-833c-e633bac15e4a-multus-socket-dir-parent\") pod \"multus-pmnqx\" (UID: \"08198bc1-314f-4881-833c-e633bac15e4a\") " pod="openshift-multus/multus-pmnqx"
Oct 31 07:29:07.350480 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:07.350191    4720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/fc64d31a-f48a-4293-a4de-262045d0b7e8-host\") pod \"node-ca-th7x7\" (UID: \"fc64d31a-f48a-4293-a4de-262045d0b7e8\") " pod="openshift-image-registry/node-ca-th7x7"
Oct 31 07:29:07.350480 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:07.350206    4720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/08198bc1-314f-4881-833c-e633bac15e4a-multus-cni-dir\") pod \"multus-pmnqx\" (UID: \"08198bc1-314f-4881-833c-e633bac15e4a\") " pod="openshift-multus/multus-pmnqx"
Oct 31 07:29:07.350480 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:07.350233    4720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3fa078c7-4bf2-4e76-9692-ae460e8fadfa-var-lib-kubelet\") pod \"tuned-cpn4m\" (UID: \"3fa078c7-4bf2-4e76-9692-ae460e8fadfa\") " pod="openshift-cluster-node-tuning-operator/tuned-cpn4m"
Oct 31 07:29:07.350480 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:07.350243    4720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/4bfbebfd-2be7-438f-b88d-273f24751f8c-run-ovn\") pod \"ovnkube-node-5p7xw\" (UID: \"4bfbebfd-2be7-438f-b88d-273f24751f8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-5p7xw"
Oct 31 07:29:07.350480 ip-10-0-139-83 kubenswrapper[4720]: E1031 07:29:07.350251    4720 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Oct 31 07:29:07.350480 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:07.350236    4720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/789f11d8-d951-4144-a589-a2bdbf6844e3-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-6k8mn\" (UID: \"789f11d8-d951-4144-a589-a2bdbf6844e3\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-6k8mn"
Oct 31 07:29:07.350480 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:07.350237    4720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/08198bc1-314f-4881-833c-e633bac15e4a-cnibin\") pod \"multus-pmnqx\" (UID: \"08198bc1-314f-4881-833c-e633bac15e4a\") " pod="openshift-multus/multus-pmnqx"
Oct 31 07:29:07.350480 ip-10-0-139-83 kubenswrapper[4720]: E1031 07:29:07.350297    4720 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 31 07:29:07.350480 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:07.350306    4720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/08198bc1-314f-4881-833c-e633bac15e4a-multus-cni-dir\") pod \"multus-pmnqx\" (UID: \"08198bc1-314f-4881-833c-e633bac15e4a\") " pod="openshift-multus/multus-pmnqx"
Oct 31 07:29:07.350480 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:07.350283    4720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/4bfbebfd-2be7-438f-b88d-273f24751f8c-systemd-units\") pod \"ovnkube-node-5p7xw\" (UID: \"4bfbebfd-2be7-438f-b88d-273f24751f8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-5p7xw"
Oct 31 07:29:07.350480 ip-10-0-139-83 kubenswrapper[4720]: E1031 07:29:07.350314    4720 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Oct 31 07:29:07.350480 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:07.350352    4720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/4bfbebfd-2be7-438f-b88d-273f24751f8c-systemd-units\") pod \"ovnkube-node-5p7xw\" (UID: \"4bfbebfd-2be7-438f-b88d-273f24751f8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-5p7xw"
Oct 31 07:29:07.350480 ip-10-0-139-83 kubenswrapper[4720]: E1031 07:29:07.350372    4720 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c3ad95fd-e997-459c-93fd-27c4c030ea0d-metrics-certs podName:c3ad95fd-e997-459c-93fd-27c4c030ea0d nodeName:}" failed. No retries permitted until 2026-10-31 07:29:07.850356542 +0000 UTC m=+1.981424608 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c3ad95fd-e997-459c-93fd-27c4c030ea0d-metrics-certs") pod "network-metrics-daemon-r2zbv" (UID: "c3ad95fd-e997-459c-93fd-27c4c030ea0d") : secret "metrics-daemon-secret" not found
Oct 31 07:29:07.350480 ip-10-0-139-83 kubenswrapper[4720]: E1031 07:29:07.350402    4720 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/789f11d8-d951-4144-a589-a2bdbf6844e3-cluster-monitoring-operator-tls podName:789f11d8-d951-4144-a589-a2bdbf6844e3 nodeName:}" failed. No retries permitted until 2026-10-31 07:29:07.850390289 +0000 UTC m=+1.981458338 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/789f11d8-d951-4144-a589-a2bdbf6844e3-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-6k8mn" (UID: "789f11d8-d951-4144-a589-a2bdbf6844e3") : secret "cluster-monitoring-operator-tls" not found
Oct 31 07:29:07.350480 ip-10-0-139-83 kubenswrapper[4720]: E1031 07:29:07.350413    4720 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a73024cb-f581-450d-a09c-0565838c5baa-metrics-tls podName:a73024cb-f581-450d-a09c-0565838c5baa nodeName:}" failed. No retries permitted until 2026-10-31 07:29:07.850408053 +0000 UTC m=+1.981476102 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/a73024cb-f581-450d-a09c-0565838c5baa-metrics-tls") pod "dns-default-9cmfx" (UID: "a73024cb-f581-450d-a09c-0565838c5baa") : secret "dns-default-metrics-tls" not found
Oct 31 07:29:07.350480 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:07.350430    4720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4bfbebfd-2be7-438f-b88d-273f24751f8c-host-slash\") pod \"ovnkube-node-5p7xw\" (UID: \"4bfbebfd-2be7-438f-b88d-273f24751f8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-5p7xw"
Oct 31 07:29:07.350480 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:07.350458    4720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/3fa078c7-4bf2-4e76-9692-ae460e8fadfa-host\") pod \"tuned-cpn4m\" (UID: \"3fa078c7-4bf2-4e76-9692-ae460e8fadfa\") " pod="openshift-cluster-node-tuning-operator/tuned-cpn4m"
Oct 31 07:29:07.350480 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:07.350476    4720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/08198bc1-314f-4881-833c-e633bac15e4a-host-var-lib-kubelet\") pod \"multus-pmnqx\" (UID: \"08198bc1-314f-4881-833c-e633bac15e4a\") " pod="openshift-multus/multus-pmnqx"
Oct 31 07:29:07.350480 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:07.350494    4720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/4bfbebfd-2be7-438f-b88d-273f24751f8c-host-cni-bin\") pod \"ovnkube-node-5p7xw\" (UID: \"4bfbebfd-2be7-438f-b88d-273f24751f8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-5p7xw"
Oct 31 07:29:07.351343 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:07.350516    4720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3fa078c7-4bf2-4e76-9692-ae460e8fadfa-etc-kubernetes\") pod \"tuned-cpn4m\" (UID: \"3fa078c7-4bf2-4e76-9692-ae460e8fadfa\") " pod="openshift-cluster-node-tuning-operator/tuned-cpn4m"
Oct 31 07:29:07.351343 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:07.350521    4720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/08198bc1-314f-4881-833c-e633bac15e4a-host-var-lib-kubelet\") pod \"multus-pmnqx\" (UID: \"08198bc1-314f-4881-833c-e633bac15e4a\") " pod="openshift-multus/multus-pmnqx"
Oct 31 07:29:07.351343 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:07.350530    4720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/3fa078c7-4bf2-4e76-9692-ae460e8fadfa-host\") pod \"tuned-cpn4m\" (UID: \"3fa078c7-4bf2-4e76-9692-ae460e8fadfa\") " pod="openshift-cluster-node-tuning-operator/tuned-cpn4m"
Oct 31 07:29:07.351343 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:07.350537    4720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/08198bc1-314f-4881-833c-e633bac15e4a-host-run-k8s-cni-cncf-io\") pod \"multus-pmnqx\" (UID: \"08198bc1-314f-4881-833c-e633bac15e4a\") " pod="openshift-multus/multus-pmnqx"
Oct 31 07:29:07.351343 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:07.350550    4720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/4bfbebfd-2be7-438f-b88d-273f24751f8c-host-cni-bin\") pod \"ovnkube-node-5p7xw\" (UID: \"4bfbebfd-2be7-438f-b88d-273f24751f8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-5p7xw"
Oct 31 07:29:07.351343 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:07.350559    4720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/08198bc1-314f-4881-833c-e633bac15e4a-host-run-k8s-cni-cncf-io\") pod \"multus-pmnqx\" (UID: \"08198bc1-314f-4881-833c-e633bac15e4a\") " pod="openshift-multus/multus-pmnqx"
Oct 31 07:29:07.351343 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:07.350558    4720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4bfbebfd-2be7-438f-b88d-273f24751f8c-host-slash\") pod \"ovnkube-node-5p7xw\" (UID: \"4bfbebfd-2be7-438f-b88d-273f24751f8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-5p7xw"
Oct 31 07:29:07.351343 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:07.350579    4720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/9dad8dd1-cfc5-412d-bf9a-bac3c9162af1-device-dir\") pod \"aws-ebs-csi-driver-node-797gj\" (UID: \"9dad8dd1-cfc5-412d-bf9a-bac3c9162af1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-797gj"
Oct 31 07:29:07.351343 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:07.350604    4720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3fa078c7-4bf2-4e76-9692-ae460e8fadfa-etc-kubernetes\") pod \"tuned-cpn4m\" (UID: \"3fa078c7-4bf2-4e76-9692-ae460e8fadfa\") " pod="openshift-cluster-node-tuning-operator/tuned-cpn4m"
Oct 31 07:29:07.351343 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:07.350596    4720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/08198bc1-314f-4881-833c-e633bac15e4a-host-run-multus-certs\") pod \"multus-pmnqx\" (UID: \"08198bc1-314f-4881-833c-e633bac15e4a\") " pod="openshift-multus/multus-pmnqx"
Oct 31 07:29:07.351343 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:07.350639    4720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/08198bc1-314f-4881-833c-e633bac15e4a-host-run-multus-certs\") pod \"multus-pmnqx\" (UID: \"08198bc1-314f-4881-833c-e633bac15e4a\") " pod="openshift-multus/multus-pmnqx"
Oct 31 07:29:07.351343 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:07.350638    4720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/4bfbebfd-2be7-438f-b88d-273f24751f8c-node-log\") pod \"ovnkube-node-5p7xw\" (UID: \"4bfbebfd-2be7-438f-b88d-273f24751f8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-5p7xw"
Oct 31 07:29:07.351343 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:07.350641    4720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/9dad8dd1-cfc5-412d-bf9a-bac3c9162af1-device-dir\") pod \"aws-ebs-csi-driver-node-797gj\" (UID: \"9dad8dd1-cfc5-412d-bf9a-bac3c9162af1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-797gj"
Oct 31 07:29:07.351343 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:07.350654    4720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b1066ece-e6f3-4491-aa1f-a7b9a3aa248d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-8m768\" (UID: \"b1066ece-e6f3-4491-aa1f-a7b9a3aa248d\") " pod="openshift-insights/insights-runtime-extractor-8m768"
Oct 31 07:29:07.351343 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:07.350661    4720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/4bfbebfd-2be7-438f-b88d-273f24751f8c-node-log\") pod \"ovnkube-node-5p7xw\" (UID: \"4bfbebfd-2be7-438f-b88d-273f24751f8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-5p7xw"
Oct 31 07:29:07.351343 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:07.350689    4720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/08198bc1-314f-4881-833c-e633bac15e4a-system-cni-dir\") pod \"multus-pmnqx\" (UID: \"08198bc1-314f-4881-833c-e633bac15e4a\") " pod="openshift-multus/multus-pmnqx"
Oct 31 07:29:07.351343 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:07.350663    4720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/08198bc1-314f-4881-833c-e633bac15e4a-system-cni-dir\") pod \"multus-pmnqx\" (UID: \"08198bc1-314f-4881-833c-e633bac15e4a\") " pod="openshift-multus/multus-pmnqx"
Oct 31 07:29:07.351343 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:07.350717    4720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b44ed0e0-a72e-4746-a7de-ed8610da845d-service-ca-bundle\") pod \"router-default-6c88c45d57-c2bgn\" (UID: \"b44ed0e0-a72e-4746-a7de-ed8610da845d\") " pod="openshift-ingress/router-default-6c88c45d57-c2bgn"
Oct 31 07:29:07.351343 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:07.350732    4720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/26659955-3909-4171-90b8-21b0ad6eef64-host-slash\") pod \"iptables-alerter-65hzp\" (UID: \"26659955-3909-4171-90b8-21b0ad6eef64\") " pod="openshift-network-operator/iptables-alerter-65hzp"
Oct 31 07:29:07.351343 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:07.350756    4720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/08198bc1-314f-4881-833c-e633bac15e4a-host-var-lib-cni-multus\") pod \"multus-pmnqx\" (UID: \"08198bc1-314f-4881-833c-e633bac15e4a\") " pod="openshift-multus/multus-pmnqx"
Oct 31 07:29:07.351343 ip-10-0-139-83 kubenswrapper[4720]: E1031 07:29:07.350777    4720 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b44ed0e0-a72e-4746-a7de-ed8610da845d-service-ca-bundle podName:b44ed0e0-a72e-4746-a7de-ed8610da845d nodeName:}" failed. No retries permitted until 2026-10-31 07:29:07.850767236 +0000 UTC m=+1.981835320 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/b44ed0e0-a72e-4746-a7de-ed8610da845d-service-ca-bundle") pod "router-default-6c88c45d57-c2bgn" (UID: "b44ed0e0-a72e-4746-a7de-ed8610da845d") : configmap references non-existent config key: service-ca.crt
Oct 31 07:29:07.351343 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:07.350790    4720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/3fa078c7-4bf2-4e76-9692-ae460e8fadfa-etc-modprobe-d\") pod \"tuned-cpn4m\" (UID: \"3fa078c7-4bf2-4e76-9692-ae460e8fadfa\") " pod="openshift-cluster-node-tuning-operator/tuned-cpn4m"
Oct 31 07:29:07.351343 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:07.350796    4720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/08198bc1-314f-4881-833c-e633bac15e4a-host-var-lib-cni-multus\") pod \"multus-pmnqx\" (UID: \"08198bc1-314f-4881-833c-e633bac15e4a\") " pod="openshift-multus/multus-pmnqx"
Oct 31 07:29:07.351343 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:07.350808    4720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/9dad8dd1-cfc5-412d-bf9a-bac3c9162af1-socket-dir\") pod \"aws-ebs-csi-driver-node-797gj\" (UID: \"9dad8dd1-cfc5-412d-bf9a-bac3c9162af1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-797gj"
Oct 31 07:29:07.351343 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:07.350775    4720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/26659955-3909-4171-90b8-21b0ad6eef64-host-slash\") pod \"iptables-alerter-65hzp\" (UID: \"26659955-3909-4171-90b8-21b0ad6eef64\") " pod="openshift-network-operator/iptables-alerter-65hzp"
Oct 31 07:29:07.351343 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:07.350824    4720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9c72279a-708f-46df-8915-9995c5aea14d-system-cni-dir\") pod \"multus-additional-cni-plugins-dh6w8\" (UID: \"9c72279a-708f-46df-8915-9995c5aea14d\") " pod="openshift-multus/multus-additional-cni-plugins-dh6w8"
Oct 31 07:29:07.351343 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:07.350839    4720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/3fa078c7-4bf2-4e76-9692-ae460e8fadfa-run\") pod \"tuned-cpn4m\" (UID: \"3fa078c7-4bf2-4e76-9692-ae460e8fadfa\") " pod="openshift-cluster-node-tuning-operator/tuned-cpn4m"
Oct 31 07:29:07.351343 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:07.350851    4720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3fa078c7-4bf2-4e76-9692-ae460e8fadfa-sys\") pod \"tuned-cpn4m\" (UID: \"3fa078c7-4bf2-4e76-9692-ae460e8fadfa\") " pod="openshift-cluster-node-tuning-operator/tuned-cpn4m"
Oct 31 07:29:07.351343 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:07.350866    4720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/77256436-39ff-4c2a-8390-4e9fa7d68486-dbus\") pod \"global-pull-secret-syncer-kht4l\" (UID: \"77256436-39ff-4c2a-8390-4e9fa7d68486\") " pod="kube-system/global-pull-secret-syncer-kht4l"
Oct 31 07:29:07.351343 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:07.350879    4720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/c4233c69-4213-43be-8809-76612b6b6f3e-hosts-file\") pod \"node-resolver-tnw4j\" (UID: \"c4233c69-4213-43be-8809-76612b6b6f3e\") " pod="openshift-dns/node-resolver-tnw4j"
Oct 31 07:29:07.351343 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:07.350889    4720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9c72279a-708f-46df-8915-9995c5aea14d-system-cni-dir\") pod \"multus-additional-cni-plugins-dh6w8\" (UID: \"9c72279a-708f-46df-8915-9995c5aea14d\") " pod="openshift-multus/multus-additional-cni-plugins-dh6w8"
Oct 31 07:29:07.351343 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:07.350889    4720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/9dad8dd1-cfc5-412d-bf9a-bac3c9162af1-socket-dir\") pod \"aws-ebs-csi-driver-node-797gj\" (UID: \"9dad8dd1-cfc5-412d-bf9a-bac3c9162af1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-797gj"
Oct 31 07:29:07.351343 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:07.350901    4720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/3fa078c7-4bf2-4e76-9692-ae460e8fadfa-etc-sysctl-d\") pod \"tuned-cpn4m\" (UID: \"3fa078c7-4bf2-4e76-9692-ae460e8fadfa\") " pod="openshift-cluster-node-tuning-operator/tuned-cpn4m"
Oct 31 07:29:07.351343 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:07.350910    4720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/3fa078c7-4bf2-4e76-9692-ae460e8fadfa-etc-modprobe-d\") pod \"tuned-cpn4m\" (UID: \"3fa078c7-4bf2-4e76-9692-ae460e8fadfa\") " pod="openshift-cluster-node-tuning-operator/tuned-cpn4m"
Oct 31 07:29:07.351343 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:07.350921    4720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3fa078c7-4bf2-4e76-9692-ae460e8fadfa-sys\") pod \"tuned-cpn4m\" (UID: \"3fa078c7-4bf2-4e76-9692-ae460e8fadfa\") " pod="openshift-cluster-node-tuning-operator/tuned-cpn4m"
Oct 31 07:29:07.351343 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:07.350909    4720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/3fa078c7-4bf2-4e76-9692-ae460e8fadfa-run\") pod \"tuned-cpn4m\" (UID: \"3fa078c7-4bf2-4e76-9692-ae460e8fadfa\") " pod="openshift-cluster-node-tuning-operator/tuned-cpn4m"
Oct 31 07:29:07.351343 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:07.350930    4720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/08198bc1-314f-4881-833c-e633bac15e4a-hostroot\") pod \"multus-pmnqx\" (UID: \"08198bc1-314f-4881-833c-e633bac15e4a\") " pod="openshift-multus/multus-pmnqx"
Oct 31 07:29:07.351343 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:07.350934    4720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/c4233c69-4213-43be-8809-76612b6b6f3e-hosts-file\") pod \"node-resolver-tnw4j\" (UID: \"c4233c69-4213-43be-8809-76612b6b6f3e\") " pod="openshift-dns/node-resolver-tnw4j"
Oct 31 07:29:07.351343 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:07.350947    4720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/77256436-39ff-4c2a-8390-4e9fa7d68486-dbus\") pod \"global-pull-secret-syncer-kht4l\" (UID: \"77256436-39ff-4c2a-8390-4e9fa7d68486\") " pod="kube-system/global-pull-secret-syncer-kht4l"
Oct 31 07:29:07.351343 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:07.350958    4720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/b1066ece-e6f3-4491-aa1f-a7b9a3aa248d-crio-socket\") pod \"insights-runtime-extractor-8m768\" (UID: \"b1066ece-e6f3-4491-aa1f-a7b9a3aa248d\") " pod="openshift-insights/insights-runtime-extractor-8m768"
Oct 31 07:29:07.351343 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:07.350985    4720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/b1066ece-e6f3-4491-aa1f-a7b9a3aa248d-crio-socket\") pod \"insights-runtime-extractor-8m768\" (UID: \"b1066ece-e6f3-4491-aa1f-a7b9a3aa248d\") " pod="openshift-insights/insights-runtime-extractor-8m768"
Oct 31 07:29:07.351343 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:07.350985    4720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b44ed0e0-a72e-4746-a7de-ed8610da845d-metrics-certs\") pod \"router-default-6c88c45d57-c2bgn\" (UID: \"b44ed0e0-a72e-4746-a7de-ed8610da845d\") " pod="openshift-ingress/router-default-6c88c45d57-c2bgn"
Oct 31 07:29:07.351343 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:07.350990    4720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/3fa078c7-4bf2-4e76-9692-ae460e8fadfa-etc-sysctl-d\") pod \"tuned-cpn4m\" (UID: \"3fa078c7-4bf2-4e76-9692-ae460e8fadfa\") " pod="openshift-cluster-node-tuning-operator/tuned-cpn4m"
Oct 31 07:29:07.351343 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:07.351012    4720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/08198bc1-314f-4881-833c-e633bac15e4a-hostroot\") pod \"multus-pmnqx\" (UID: \"08198bc1-314f-4881-833c-e633bac15e4a\") " pod="openshift-multus/multus-pmnqx"
Oct 31 07:29:07.351343 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:07.351012    4720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/4bfbebfd-2be7-438f-b88d-273f24751f8c-host-run-netns\") pod \"ovnkube-node-5p7xw\" (UID: \"4bfbebfd-2be7-438f-b88d-273f24751f8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-5p7xw"
Oct 31 07:29:07.351343 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:07.351036    4720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/4bfbebfd-2be7-438f-b88d-273f24751f8c-host-run-netns\") pod \"ovnkube-node-5p7xw\" (UID: \"4bfbebfd-2be7-438f-b88d-273f24751f8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-5p7xw"
Oct 31 07:29:07.351343 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:07.351041    4720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4bfbebfd-2be7-438f-b88d-273f24751f8c-host-run-ovn-kubernetes\") pod \"ovnkube-node-5p7xw\" (UID: \"4bfbebfd-2be7-438f-b88d-273f24751f8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-5p7xw"
Oct 31 07:29:07.351343 ip-10-0-139-83 kubenswrapper[4720]: E1031 07:29:07.351050    4720 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Oct 31 07:29:07.351343 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:07.351058    4720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/77256436-39ff-4c2a-8390-4e9fa7d68486-kubelet-config\") pod \"global-pull-secret-syncer-kht4l\" (UID: \"77256436-39ff-4c2a-8390-4e9fa7d68486\") " pod="kube-system/global-pull-secret-syncer-kht4l"
Oct 31 07:29:07.351343 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:07.351081    4720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/3fa078c7-4bf2-4e76-9692-ae460e8fadfa-etc-sysctl-conf\") pod \"tuned-cpn4m\" (UID: \"3fa078c7-4bf2-4e76-9692-ae460e8fadfa\") " pod="openshift-cluster-node-tuning-operator/tuned-cpn4m"
Oct 31 07:29:07.351343 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:07.351104    4720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4bfbebfd-2be7-438f-b88d-273f24751f8c-host-run-ovn-kubernetes\") pod \"ovnkube-node-5p7xw\" (UID: \"4bfbebfd-2be7-438f-b88d-273f24751f8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-5p7xw"
Oct 31 07:29:07.351343 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:07.351112    4720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/77256436-39ff-4c2a-8390-4e9fa7d68486-kubelet-config\") pod \"global-pull-secret-syncer-kht4l\" (UID: \"77256436-39ff-4c2a-8390-4e9fa7d68486\") " pod="kube-system/global-pull-secret-syncer-kht4l"
Oct 31 07:29:07.351343 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:07.351107    4720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/08198bc1-314f-4881-833c-e633bac15e4a-etc-kubernetes\") pod \"multus-pmnqx\" (UID: \"08198bc1-314f-4881-833c-e633bac15e4a\") " pod="openshift-multus/multus-pmnqx"
Oct 31 07:29:07.351343 ip-10-0-139-83 kubenswrapper[4720]: E1031 07:29:07.351129    4720 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b44ed0e0-a72e-4746-a7de-ed8610da845d-metrics-certs podName:b44ed0e0-a72e-4746-a7de-ed8610da845d nodeName:}" failed. No retries permitted until 2026-10-31 07:29:07.851114485 +0000 UTC m=+1.982182548 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b44ed0e0-a72e-4746-a7de-ed8610da845d-metrics-certs") pod "router-default-6c88c45d57-c2bgn" (UID: "b44ed0e0-a72e-4746-a7de-ed8610da845d") : secret "router-metrics-certs-default" not found
Oct 31 07:29:07.351343 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:07.351152    4720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/9dad8dd1-cfc5-412d-bf9a-bac3c9162af1-kubelet-dir\") pod \"aws-ebs-csi-driver-node-797gj\" (UID: \"9dad8dd1-cfc5-412d-bf9a-bac3c9162af1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-797gj"
Oct 31 07:29:07.351343 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:07.351158    4720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/08198bc1-314f-4881-833c-e633bac15e4a-etc-kubernetes\") pod \"multus-pmnqx\" (UID: \"08198bc1-314f-4881-833c-e633bac15e4a\") " pod="openshift-multus/multus-pmnqx"
Oct 31 07:29:07.351343 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:07.351168    4720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4bfbebfd-2be7-438f-b88d-273f24751f8c-var-lib-openvswitch\") pod \"ovnkube-node-5p7xw\" (UID: \"4bfbebfd-2be7-438f-b88d-273f24751f8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-5p7xw"
Oct 31 07:29:07.351343 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:07.351187    4720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/28b59d93-b9ce-4ceb-9f3a-742525eb94c4-registry-tls\") pod \"image-registry-7955c67bcc-pkntv\" (UID: \"28b59d93-b9ce-4ceb-9f3a-742525eb94c4\") " pod="openshift-image-registry/image-registry-7955c67bcc-pkntv"
Oct 31 07:29:07.351343 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:07.351196    4720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/9dad8dd1-cfc5-412d-bf9a-bac3c9162af1-kubelet-dir\") pod \"aws-ebs-csi-driver-node-797gj\" (UID: \"9dad8dd1-cfc5-412d-bf9a-bac3c9162af1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-797gj"
Oct 31 07:29:07.351343 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:07.351204    4720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/4bfbebfd-2be7-438f-b88d-273f24751f8c-run-systemd\") pod \"ovnkube-node-5p7xw\" (UID: \"4bfbebfd-2be7-438f-b88d-273f24751f8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-5p7xw"
Oct 31 07:29:07.351343 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:07.351167    4720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/3fa078c7-4bf2-4e76-9692-ae460e8fadfa-etc-sysctl-conf\") pod \"tuned-cpn4m\" (UID: \"3fa078c7-4bf2-4e76-9692-ae460e8fadfa\") " pod="openshift-cluster-node-tuning-operator/tuned-cpn4m"
Oct 31 07:29:07.351343 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:07.351208    4720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4bfbebfd-2be7-438f-b88d-273f24751f8c-var-lib-openvswitch\") pod \"ovnkube-node-5p7xw\" (UID: \"4bfbebfd-2be7-438f-b88d-273f24751f8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-5p7xw"
Oct 31 07:29:07.351343 ip-10-0-139-83 kubenswrapper[4720]: E1031 07:29:07.351243    4720 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 31 07:29:07.351343 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:07.351252    4720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/4bfbebfd-2be7-438f-b88d-273f24751f8c-run-systemd\") pod \"ovnkube-node-5p7xw\" (UID: \"4bfbebfd-2be7-438f-b88d-273f24751f8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-5p7xw"
Oct 31 07:29:07.351343 ip-10-0-139-83 kubenswrapper[4720]: E1031 07:29:07.351256    4720 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7955c67bcc-pkntv: secret "image-registry-tls" not found
Oct 31 07:29:07.351343 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:07.351269    4720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/ca31519e-aa90-4104-a175-f9af2b3bbeb4-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-6vkbv\" (UID: \"ca31519e-aa90-4104-a175-f9af2b3bbeb4\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-6vkbv"
Oct 31 07:29:07.351343 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:07.351287    4720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9c72279a-708f-46df-8915-9995c5aea14d-tuning-conf-dir\") pod \"multus-additional-cni-plugins-dh6w8\" (UID: \"9c72279a-708f-46df-8915-9995c5aea14d\") " pod="openshift-multus/multus-additional-cni-plugins-dh6w8"
Oct 31 07:29:07.351343 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:07.351300    4720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/4bfbebfd-2be7-438f-b88d-273f24751f8c-log-socket\") pod \"ovnkube-node-5p7xw\" (UID: \"4bfbebfd-2be7-438f-b88d-273f24751f8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-5p7xw"
Oct 31 07:29:07.351343 ip-10-0-139-83 kubenswrapper[4720]: E1031 07:29:07.351312    4720 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/28b59d93-b9ce-4ceb-9f3a-742525eb94c4-registry-tls podName:28b59d93-b9ce-4ceb-9f3a-742525eb94c4 nodeName:}" failed. No retries permitted until 2026-10-31 07:29:07.851299685 +0000 UTC m=+1.982367734 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/28b59d93-b9ce-4ceb-9f3a-742525eb94c4-registry-tls") pod "image-registry-7955c67bcc-pkntv" (UID: "28b59d93-b9ce-4ceb-9f3a-742525eb94c4") : secret "image-registry-tls" not found
Oct 31 07:29:07.351343 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:07.351320    4720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/4bfbebfd-2be7-438f-b88d-273f24751f8c-log-socket\") pod \"ovnkube-node-5p7xw\" (UID: \"4bfbebfd-2be7-438f-b88d-273f24751f8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-5p7xw"
Oct 31 07:29:07.351343 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:07.351356    4720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/4bfbebfd-2be7-438f-b88d-273f24751f8c-host-kubelet\") pod \"ovnkube-node-5p7xw\" (UID: \"4bfbebfd-2be7-438f-b88d-273f24751f8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-5p7xw"
Oct 31 07:29:07.351343 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:07.351365    4720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9c72279a-708f-46df-8915-9995c5aea14d-tuning-conf-dir\") pod \"multus-additional-cni-plugins-dh6w8\" (UID: \"9c72279a-708f-46df-8915-9995c5aea14d\") " pod="openshift-multus/multus-additional-cni-plugins-dh6w8"
Oct 31 07:29:07.351343 ip-10-0-139-83 kubenswrapper[4720]: E1031 07:29:07.351377    4720 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Oct 31 07:29:07.351343 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:07.351382    4720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/08198bc1-314f-4881-833c-e633bac15e4a-multus-conf-dir\") pod \"multus-pmnqx\" (UID: \"08198bc1-314f-4881-833c-e633bac15e4a\") " pod="openshift-multus/multus-pmnqx"
Oct 31 07:29:07.351343 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:07.351389    4720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/4bfbebfd-2be7-438f-b88d-273f24751f8c-host-kubelet\") pod \"ovnkube-node-5p7xw\" (UID: \"4bfbebfd-2be7-438f-b88d-273f24751f8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-5p7xw"
Oct 31 07:29:07.353466 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:07.351409    4720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/9dad8dd1-cfc5-412d-bf9a-bac3c9162af1-sys-fs\") pod \"aws-ebs-csi-driver-node-797gj\" (UID: \"9dad8dd1-cfc5-412d-bf9a-bac3c9162af1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-797gj"
Oct 31 07:29:07.353466 ip-10-0-139-83 kubenswrapper[4720]: E1031 07:29:07.351424    4720 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ca31519e-aa90-4104-a175-f9af2b3bbeb4-samples-operator-tls podName:ca31519e-aa90-4104-a175-f9af2b3bbeb4 nodeName:}" failed. No retries permitted until 2026-10-31 07:29:07.851412014 +0000 UTC m=+1.982480102 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/ca31519e-aa90-4104-a175-f9af2b3bbeb4-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-6vkbv" (UID: "ca31519e-aa90-4104-a175-f9af2b3bbeb4") : secret "samples-operator-tls" not found
Oct 31 07:29:07.353466 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:07.351440    4720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/08198bc1-314f-4881-833c-e633bac15e4a-multus-conf-dir\") pod \"multus-pmnqx\" (UID: \"08198bc1-314f-4881-833c-e633bac15e4a\") " pod="openshift-multus/multus-pmnqx"
Oct 31 07:29:07.353466 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:07.351448    4720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4bfbebfd-2be7-438f-b88d-273f24751f8c-run-openvswitch\") pod \"ovnkube-node-5p7xw\" (UID: \"4bfbebfd-2be7-438f-b88d-273f24751f8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-5p7xw"
Oct 31 07:29:07.353466 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:07.351462    4720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/9dad8dd1-cfc5-412d-bf9a-bac3c9162af1-sys-fs\") pod \"aws-ebs-csi-driver-node-797gj\" (UID: \"9dad8dd1-cfc5-412d-bf9a-bac3c9162af1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-797gj"
Oct 31 07:29:07.353466 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:07.351474    4720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/08198bc1-314f-4881-833c-e633bac15e4a-host-var-lib-cni-bin\") pod \"multus-pmnqx\" (UID: \"08198bc1-314f-4881-833c-e633bac15e4a\") " pod="openshift-multus/multus-pmnqx"
Oct 31 07:29:07.353466 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:07.351498    4720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/9dad8dd1-cfc5-412d-bf9a-bac3c9162af1-etc-selinux\") pod \"aws-ebs-csi-driver-node-797gj\" (UID: \"9dad8dd1-cfc5-412d-bf9a-bac3c9162af1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-797gj"
Oct 31 07:29:07.353466 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:07.351513    4720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9c72279a-708f-46df-8915-9995c5aea14d-cnibin\") pod \"multus-additional-cni-plugins-dh6w8\" (UID: \"9c72279a-708f-46df-8915-9995c5aea14d\") " pod="openshift-multus/multus-additional-cni-plugins-dh6w8"
Oct 31 07:29:07.353466 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:07.351517    4720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4bfbebfd-2be7-438f-b88d-273f24751f8c-run-openvswitch\") pod \"ovnkube-node-5p7xw\" (UID: \"4bfbebfd-2be7-438f-b88d-273f24751f8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-5p7xw"
Oct 31 07:29:07.353466 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:07.351531    4720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/08198bc1-314f-4881-833c-e633bac15e4a-host-var-lib-cni-bin\") pod \"multus-pmnqx\" (UID: \"08198bc1-314f-4881-833c-e633bac15e4a\") " pod="openshift-multus/multus-pmnqx"
Oct 31 07:29:07.353466 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:07.351535    4720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/4bfbebfd-2be7-438f-b88d-273f24751f8c-host-cni-netd\") pod \"ovnkube-node-5p7xw\" (UID: \"4bfbebfd-2be7-438f-b88d-273f24751f8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-5p7xw"
Oct 31 07:29:07.353466 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:07.351545    4720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/9dad8dd1-cfc5-412d-bf9a-bac3c9162af1-etc-selinux\") pod \"aws-ebs-csi-driver-node-797gj\" (UID: \"9dad8dd1-cfc5-412d-bf9a-bac3c9162af1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-797gj"
Oct 31 07:29:07.353466 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:07.351561    4720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/08198bc1-314f-4881-833c-e633bac15e4a-host-run-netns\") pod \"multus-pmnqx\" (UID: \"08198bc1-314f-4881-833c-e633bac15e4a\") " pod="openshift-multus/multus-pmnqx"
Oct 31 07:29:07.353466 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:07.351562    4720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9c72279a-708f-46df-8915-9995c5aea14d-cnibin\") pod \"multus-additional-cni-plugins-dh6w8\" (UID: \"9c72279a-708f-46df-8915-9995c5aea14d\") " pod="openshift-multus/multus-additional-cni-plugins-dh6w8"
Oct 31 07:29:07.353466 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:07.351569    4720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/4bfbebfd-2be7-438f-b88d-273f24751f8c-host-cni-netd\") pod \"ovnkube-node-5p7xw\" (UID: \"4bfbebfd-2be7-438f-b88d-273f24751f8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-5p7xw"
Oct 31 07:29:07.353466 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:07.351579    4720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4bfbebfd-2be7-438f-b88d-273f24751f8c-etc-openvswitch\") pod \"ovnkube-node-5p7xw\" (UID: \"4bfbebfd-2be7-438f-b88d-273f24751f8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-5p7xw"
Oct 31 07:29:07.353466 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:07.351595    4720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b1066ece-e6f3-4491-aa1f-a7b9a3aa248d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-8m768\" (UID: \"b1066ece-e6f3-4491-aa1f-a7b9a3aa248d\") " pod="openshift-insights/insights-runtime-extractor-8m768"
Oct 31 07:29:07.353466 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:07.351622    4720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/9dad8dd1-cfc5-412d-bf9a-bac3c9162af1-registration-dir\") pod \"aws-ebs-csi-driver-node-797gj\" (UID: \"9dad8dd1-cfc5-412d-bf9a-bac3c9162af1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-797gj"
Oct 31 07:29:07.353466 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:07.351638    4720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4bfbebfd-2be7-438f-b88d-273f24751f8c-etc-openvswitch\") pod \"ovnkube-node-5p7xw\" (UID: \"4bfbebfd-2be7-438f-b88d-273f24751f8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-5p7xw"
Oct 31 07:29:07.353466 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:07.351649    4720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/314c29aa-db6b-4f7f-86b4-c8952ce09c84-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-j94dz\" (UID: \"314c29aa-db6b-4f7f-86b4-c8952ce09c84\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-j94dz"
Oct 31 07:29:07.353466 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:07.351643    4720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/08198bc1-314f-4881-833c-e633bac15e4a-host-run-netns\") pod \"multus-pmnqx\" (UID: \"08198bc1-314f-4881-833c-e633bac15e4a\") " pod="openshift-multus/multus-pmnqx"
Oct 31 07:29:07.353466 ip-10-0-139-83 kubenswrapper[4720]: E1031 07:29:07.351668    4720 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 31 07:29:07.353466 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:07.351687    4720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/9dad8dd1-cfc5-412d-bf9a-bac3c9162af1-registration-dir\") pod \"aws-ebs-csi-driver-node-797gj\" (UID: \"9dad8dd1-cfc5-412d-bf9a-bac3c9162af1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-797gj"
Oct 31 07:29:07.353466 ip-10-0-139-83 kubenswrapper[4720]: E1031 07:29:07.351702    4720 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Oct 31 07:29:07.353466 ip-10-0-139-83 kubenswrapper[4720]: E1031 07:29:07.351700    4720 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b1066ece-e6f3-4491-aa1f-a7b9a3aa248d-insights-runtime-extractor-tls podName:b1066ece-e6f3-4491-aa1f-a7b9a3aa248d nodeName:}" failed. No retries permitted until 2026-10-31 07:29:07.851690918 +0000 UTC m=+1.982758967 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/b1066ece-e6f3-4491-aa1f-a7b9a3aa248d-insights-runtime-extractor-tls") pod "insights-runtime-extractor-8m768" (UID: "b1066ece-e6f3-4491-aa1f-a7b9a3aa248d") : secret "insights-runtime-extractor-tls" not found
Oct 31 07:29:07.353466 ip-10-0-139-83 kubenswrapper[4720]: E1031 07:29:07.351736    4720 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/314c29aa-db6b-4f7f-86b4-c8952ce09c84-networking-console-plugin-cert podName:314c29aa-db6b-4f7f-86b4-c8952ce09c84 nodeName:}" failed. No retries permitted until 2026-10-31 07:29:07.851723453 +0000 UTC m=+1.982791501 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/314c29aa-db6b-4f7f-86b4-c8952ce09c84-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-j94dz" (UID: "314c29aa-db6b-4f7f-86b4-c8952ce09c84") : secret "networking-console-plugin-cert" not found
Oct 31 07:29:07.353466 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:07.351756    4720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/3fa078c7-4bf2-4e76-9692-ae460e8fadfa-lib-modules\") pod \"tuned-cpn4m\" (UID: \"3fa078c7-4bf2-4e76-9692-ae460e8fadfa\") " pod="openshift-cluster-node-tuning-operator/tuned-cpn4m"
Oct 31 07:29:07.353466 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:07.351773    4720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/993cfed6-806d-44c5-bba8-c02203638fe7-registry-tls\") pod \"image-registry-7886c7d6c6-qdc9f\" (UID: \"993cfed6-806d-44c5-bba8-c02203638fe7\") " pod="openshift-image-registry/image-registry-7886c7d6c6-qdc9f"
Oct 31 07:29:07.353466 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:07.351794    4720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/3fa078c7-4bf2-4e76-9692-ae460e8fadfa-etc-systemd\") pod \"tuned-cpn4m\" (UID: \"3fa078c7-4bf2-4e76-9692-ae460e8fadfa\") " pod="openshift-cluster-node-tuning-operator/tuned-cpn4m"
Oct 31 07:29:07.353466 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:07.351812    4720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4bfbebfd-2be7-438f-b88d-273f24751f8c-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-5p7xw\" (UID: \"4bfbebfd-2be7-438f-b88d-273f24751f8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-5p7xw"
Oct 31 07:29:07.353466 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:07.351827    4720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/3fa078c7-4bf2-4e76-9692-ae460e8fadfa-etc-sysconfig\") pod \"tuned-cpn4m\" (UID: \"3fa078c7-4bf2-4e76-9692-ae460e8fadfa\") " pod="openshift-cluster-node-tuning-operator/tuned-cpn4m"
Oct 31 07:29:07.353466 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:07.351825    4720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/3fa078c7-4bf2-4e76-9692-ae460e8fadfa-lib-modules\") pod \"tuned-cpn4m\" (UID: \"3fa078c7-4bf2-4e76-9692-ae460e8fadfa\") " pod="openshift-cluster-node-tuning-operator/tuned-cpn4m"
Oct 31 07:29:07.353466 ip-10-0-139-83 kubenswrapper[4720]: E1031 07:29:07.351840    4720 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 31 07:29:07.353466 ip-10-0-139-83 kubenswrapper[4720]: E1031 07:29:07.351854    4720 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7886c7d6c6-qdc9f: secret "image-registry-tls" not found
Oct 31 07:29:07.353466 ip-10-0-139-83 kubenswrapper[4720]: E1031 07:29:07.351900    4720 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/993cfed6-806d-44c5-bba8-c02203638fe7-registry-tls podName:993cfed6-806d-44c5-bba8-c02203638fe7 nodeName:}" failed. No retries permitted until 2026-10-31 07:29:07.851892733 +0000 UTC m=+1.982960794 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/993cfed6-806d-44c5-bba8-c02203638fe7-registry-tls") pod "image-registry-7886c7d6c6-qdc9f" (UID: "993cfed6-806d-44c5-bba8-c02203638fe7") : secret "image-registry-tls" not found
Oct 31 07:29:07.353466 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:07.351920    4720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4bfbebfd-2be7-438f-b88d-273f24751f8c-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-5p7xw\" (UID: \"4bfbebfd-2be7-438f-b88d-273f24751f8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-5p7xw"
Oct 31 07:29:07.353466 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:07.351944    4720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/3fa078c7-4bf2-4e76-9692-ae460e8fadfa-etc-sysconfig\") pod \"tuned-cpn4m\" (UID: \"3fa078c7-4bf2-4e76-9692-ae460e8fadfa\") " pod="openshift-cluster-node-tuning-operator/tuned-cpn4m"
Oct 31 07:29:07.353466 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:07.351962    4720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/3fa078c7-4bf2-4e76-9692-ae460e8fadfa-etc-systemd\") pod \"tuned-cpn4m\" (UID: \"3fa078c7-4bf2-4e76-9692-ae460e8fadfa\") " pod="openshift-cluster-node-tuning-operator/tuned-cpn4m"
Oct 31 07:29:07.412779 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:07.412762    4720 scope.go:122] "RemoveContainer" containerID="c92430b7e7806bf2e27a6cbf56f84a5decbb26ad6ba153e76cb4e22794254de8"
Oct 31 07:29:07.425215 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:07.425199    4720 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 31 07:29:07.828280 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:07.828204    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 07:29:07.856127 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:07.856096    4720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c3ad95fd-e997-459c-93fd-27c4c030ea0d-metrics-certs\") pod \"network-metrics-daemon-r2zbv\" (UID: \"c3ad95fd-e997-459c-93fd-27c4c030ea0d\") " pod="openshift-multus/network-metrics-daemon-r2zbv"
Oct 31 07:29:07.856127 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:07.856128    4720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a73024cb-f581-450d-a09c-0565838c5baa-metrics-tls\") pod \"dns-default-9cmfx\" (UID: \"a73024cb-f581-450d-a09c-0565838c5baa\") " pod="openshift-dns/dns-default-9cmfx"
Oct 31 07:29:07.856316 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:07.856146    4720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/789f11d8-d951-4144-a589-a2bdbf6844e3-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-6k8mn\" (UID: \"789f11d8-d951-4144-a589-a2bdbf6844e3\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-6k8mn"
Oct 31 07:29:07.856316 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:07.856175    4720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b44ed0e0-a72e-4746-a7de-ed8610da845d-service-ca-bundle\") pod \"router-default-6c88c45d57-c2bgn\" (UID: \"b44ed0e0-a72e-4746-a7de-ed8610da845d\") " pod="openshift-ingress/router-default-6c88c45d57-c2bgn"
Oct 31 07:29:07.856316 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:07.856218    4720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b44ed0e0-a72e-4746-a7de-ed8610da845d-metrics-certs\") pod \"router-default-6c88c45d57-c2bgn\" (UID: \"b44ed0e0-a72e-4746-a7de-ed8610da845d\") " pod="openshift-ingress/router-default-6c88c45d57-c2bgn"
Oct 31 07:29:07.856316 ip-10-0-139-83 kubenswrapper[4720]: E1031 07:29:07.856233    4720 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Oct 31 07:29:07.856316 ip-10-0-139-83 kubenswrapper[4720]: E1031 07:29:07.856236    4720 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 31 07:29:07.856316 ip-10-0-139-83 kubenswrapper[4720]: E1031 07:29:07.856257    4720 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Oct 31 07:29:07.856316 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:07.856256    4720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/28b59d93-b9ce-4ceb-9f3a-742525eb94c4-registry-tls\") pod \"image-registry-7955c67bcc-pkntv\" (UID: \"28b59d93-b9ce-4ceb-9f3a-742525eb94c4\") " pod="openshift-image-registry/image-registry-7955c67bcc-pkntv"
Oct 31 07:29:07.856316 ip-10-0-139-83 kubenswrapper[4720]: E1031 07:29:07.856292    4720 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a73024cb-f581-450d-a09c-0565838c5baa-metrics-tls podName:a73024cb-f581-450d-a09c-0565838c5baa nodeName:}" failed. No retries permitted until 2026-10-31 07:29:08.856274257 +0000 UTC m=+2.987342321 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/a73024cb-f581-450d-a09c-0565838c5baa-metrics-tls") pod "dns-default-9cmfx" (UID: "a73024cb-f581-450d-a09c-0565838c5baa") : secret "dns-default-metrics-tls" not found
Oct 31 07:29:07.856316 ip-10-0-139-83 kubenswrapper[4720]: E1031 07:29:07.856313    4720 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c3ad95fd-e997-459c-93fd-27c4c030ea0d-metrics-certs podName:c3ad95fd-e997-459c-93fd-27c4c030ea0d nodeName:}" failed. No retries permitted until 2026-10-31 07:29:08.856305207 +0000 UTC m=+2.987373256 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c3ad95fd-e997-459c-93fd-27c4c030ea0d-metrics-certs") pod "network-metrics-daemon-r2zbv" (UID: "c3ad95fd-e997-459c-93fd-27c4c030ea0d") : secret "metrics-daemon-secret" not found
Oct 31 07:29:07.856316 ip-10-0-139-83 kubenswrapper[4720]: E1031 07:29:07.856318    4720 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 31 07:29:07.856836 ip-10-0-139-83 kubenswrapper[4720]: E1031 07:29:07.856373    4720 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7955c67bcc-pkntv: secret "image-registry-tls" not found
Oct 31 07:29:07.856836 ip-10-0-139-83 kubenswrapper[4720]: E1031 07:29:07.856374    4720 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Oct 31 07:29:07.856836 ip-10-0-139-83 kubenswrapper[4720]: E1031 07:29:07.856348    4720 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/789f11d8-d951-4144-a589-a2bdbf6844e3-cluster-monitoring-operator-tls podName:789f11d8-d951-4144-a589-a2bdbf6844e3 nodeName:}" failed. No retries permitted until 2026-10-31 07:29:08.856321628 +0000 UTC m=+2.987389677 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/789f11d8-d951-4144-a589-a2bdbf6844e3-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-6k8mn" (UID: "789f11d8-d951-4144-a589-a2bdbf6844e3") : secret "cluster-monitoring-operator-tls" not found
Oct 31 07:29:07.856836 ip-10-0-139-83 kubenswrapper[4720]: E1031 07:29:07.856456    4720 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b44ed0e0-a72e-4746-a7de-ed8610da845d-service-ca-bundle podName:b44ed0e0-a72e-4746-a7de-ed8610da845d nodeName:}" failed. No retries permitted until 2026-10-31 07:29:08.856439877 +0000 UTC m=+2.987507942 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/b44ed0e0-a72e-4746-a7de-ed8610da845d-service-ca-bundle") pod "router-default-6c88c45d57-c2bgn" (UID: "b44ed0e0-a72e-4746-a7de-ed8610da845d") : configmap references non-existent config key: service-ca.crt
Oct 31 07:29:07.856836 ip-10-0-139-83 kubenswrapper[4720]: E1031 07:29:07.856475    4720 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/28b59d93-b9ce-4ceb-9f3a-742525eb94c4-registry-tls podName:28b59d93-b9ce-4ceb-9f3a-742525eb94c4 nodeName:}" failed. No retries permitted until 2026-10-31 07:29:08.856465323 +0000 UTC m=+2.987533378 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/28b59d93-b9ce-4ceb-9f3a-742525eb94c4-registry-tls") pod "image-registry-7955c67bcc-pkntv" (UID: "28b59d93-b9ce-4ceb-9f3a-742525eb94c4") : secret "image-registry-tls" not found
Oct 31 07:29:07.856836 ip-10-0-139-83 kubenswrapper[4720]: E1031 07:29:07.856490    4720 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b44ed0e0-a72e-4746-a7de-ed8610da845d-metrics-certs podName:b44ed0e0-a72e-4746-a7de-ed8610da845d nodeName:}" failed. No retries permitted until 2026-10-31 07:29:08.856482269 +0000 UTC m=+2.987550322 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b44ed0e0-a72e-4746-a7de-ed8610da845d-metrics-certs") pod "router-default-6c88c45d57-c2bgn" (UID: "b44ed0e0-a72e-4746-a7de-ed8610da845d") : secret "router-metrics-certs-default" not found
Oct 31 07:29:07.856836 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:07.856519    4720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/ca31519e-aa90-4104-a175-f9af2b3bbeb4-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-6vkbv\" (UID: \"ca31519e-aa90-4104-a175-f9af2b3bbeb4\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-6vkbv"
Oct 31 07:29:07.856836 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:07.856571    4720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b1066ece-e6f3-4491-aa1f-a7b9a3aa248d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-8m768\" (UID: \"b1066ece-e6f3-4491-aa1f-a7b9a3aa248d\") " pod="openshift-insights/insights-runtime-extractor-8m768"
Oct 31 07:29:07.856836 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:07.856611    4720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/314c29aa-db6b-4f7f-86b4-c8952ce09c84-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-j94dz\" (UID: \"314c29aa-db6b-4f7f-86b4-c8952ce09c84\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-j94dz"
Oct 31 07:29:07.856836 ip-10-0-139-83 kubenswrapper[4720]: E1031 07:29:07.856624    4720 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Oct 31 07:29:07.856836 ip-10-0-139-83 kubenswrapper[4720]: E1031 07:29:07.856624    4720 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 31 07:29:07.856836 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:07.856639    4720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/993cfed6-806d-44c5-bba8-c02203638fe7-registry-tls\") pod \"image-registry-7886c7d6c6-qdc9f\" (UID: \"993cfed6-806d-44c5-bba8-c02203638fe7\") " pod="openshift-image-registry/image-registry-7886c7d6c6-qdc9f"
Oct 31 07:29:07.856836 ip-10-0-139-83 kubenswrapper[4720]: E1031 07:29:07.856655    4720 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ca31519e-aa90-4104-a175-f9af2b3bbeb4-samples-operator-tls podName:ca31519e-aa90-4104-a175-f9af2b3bbeb4 nodeName:}" failed. No retries permitted until 2026-10-31 07:29:08.856643311 +0000 UTC m=+2.987711393 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/ca31519e-aa90-4104-a175-f9af2b3bbeb4-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-6vkbv" (UID: "ca31519e-aa90-4104-a175-f9af2b3bbeb4") : secret "samples-operator-tls" not found
Oct 31 07:29:07.856836 ip-10-0-139-83 kubenswrapper[4720]: E1031 07:29:07.856687    4720 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b1066ece-e6f3-4491-aa1f-a7b9a3aa248d-insights-runtime-extractor-tls podName:b1066ece-e6f3-4491-aa1f-a7b9a3aa248d nodeName:}" failed. No retries permitted until 2026-10-31 07:29:08.856679715 +0000 UTC m=+2.987747769 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/b1066ece-e6f3-4491-aa1f-a7b9a3aa248d-insights-runtime-extractor-tls") pod "insights-runtime-extractor-8m768" (UID: "b1066ece-e6f3-4491-aa1f-a7b9a3aa248d") : secret "insights-runtime-extractor-tls" not found
Oct 31 07:29:07.856836 ip-10-0-139-83 kubenswrapper[4720]: E1031 07:29:07.856689    4720 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Oct 31 07:29:07.856836 ip-10-0-139-83 kubenswrapper[4720]: E1031 07:29:07.856708    4720 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 31 07:29:07.856836 ip-10-0-139-83 kubenswrapper[4720]: E1031 07:29:07.856717    4720 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/314c29aa-db6b-4f7f-86b4-c8952ce09c84-networking-console-plugin-cert podName:314c29aa-db6b-4f7f-86b4-c8952ce09c84 nodeName:}" failed. No retries permitted until 2026-10-31 07:29:08.856710633 +0000 UTC m=+2.987778714 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/314c29aa-db6b-4f7f-86b4-c8952ce09c84-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-j94dz" (UID: "314c29aa-db6b-4f7f-86b4-c8952ce09c84") : secret "networking-console-plugin-cert" not found
Oct 31 07:29:07.856836 ip-10-0-139-83 kubenswrapper[4720]: E1031 07:29:07.856719    4720 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7886c7d6c6-qdc9f: secret "image-registry-tls" not found
Oct 31 07:29:07.856836 ip-10-0-139-83 kubenswrapper[4720]: E1031 07:29:07.856757    4720 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/993cfed6-806d-44c5-bba8-c02203638fe7-registry-tls podName:993cfed6-806d-44c5-bba8-c02203638fe7 nodeName:}" failed. No retries permitted until 2026-10-31 07:29:08.856745428 +0000 UTC m=+2.987813506 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/993cfed6-806d-44c5-bba8-c02203638fe7-registry-tls") pod "image-registry-7886c7d6c6-qdc9f" (UID: "993cfed6-806d-44c5-bba8-c02203638fe7") : secret "image-registry-tls" not found
Oct 31 07:29:07.899865 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:07.899848    4720 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-5p7xw"
Oct 31 07:29:08.020880 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:08.020847    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 07:29:08.054180 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:08.054150    4720 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-9zp4l"
Oct 31 07:29:08.090051 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:08.090027    4720 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-4jszp_b1db0e3b-e85d-4769-8012-cf4b6f740d9d/console-operator/2.log"
Oct 31 07:29:08.090471 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:08.090455    4720 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-4jszp_b1db0e3b-e85d-4769-8012-cf4b6f740d9d/console-operator/1.log"
Oct 31 07:29:08.090574 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:08.090492    4720 generic.go:356] "Generic (PLEG): container finished" podID="b1db0e3b-e85d-4769-8012-cf4b6f740d9d" containerID="c65532fa5602117781458aaf71d70fac8484fefc6f0c87e31a745cca02403548" exitCode=255
Oct 31 07:29:08.090574 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:08.090537    4720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-4jszp" event={"ID":"b1db0e3b-e85d-4769-8012-cf4b6f740d9d","Type":"ContainerDied","Data":"c65532fa5602117781458aaf71d70fac8484fefc6f0c87e31a745cca02403548"}
Oct 31 07:29:08.090672 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:08.090584    4720 scope.go:122] "RemoveContainer" containerID="c92430b7e7806bf2e27a6cbf56f84a5decbb26ad6ba153e76cb4e22794254de8"
Oct 31 07:29:08.091037 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:08.091019    4720 scope.go:122] "RemoveContainer" containerID="c65532fa5602117781458aaf71d70fac8484fefc6f0c87e31a745cca02403548"
Oct 31 07:29:08.091235 ip-10-0-139-83 kubenswrapper[4720]: E1031 07:29:08.091214    4720 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-4jszp_openshift-console-operator(b1db0e3b-e85d-4769-8012-cf4b6f740d9d)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-4jszp" podUID="b1db0e3b-e85d-4769-8012-cf4b6f740d9d"
Oct 31 07:29:08.434901 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:08.434836    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 07:29:08.573236 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:08.573210    4720 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-4jszp"
Oct 31 07:29:08.601139 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:08.601108    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 07:29:08.865026 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:08.864800    4720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b44ed0e0-a72e-4746-a7de-ed8610da845d-metrics-certs\") pod \"router-default-6c88c45d57-c2bgn\" (UID: \"b44ed0e0-a72e-4746-a7de-ed8610da845d\") " pod="openshift-ingress/router-default-6c88c45d57-c2bgn"
Oct 31 07:29:08.865184 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:08.865077    4720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/28b59d93-b9ce-4ceb-9f3a-742525eb94c4-registry-tls\") pod \"image-registry-7955c67bcc-pkntv\" (UID: \"28b59d93-b9ce-4ceb-9f3a-742525eb94c4\") " pod="openshift-image-registry/image-registry-7955c67bcc-pkntv"
Oct 31 07:29:08.865184 ip-10-0-139-83 kubenswrapper[4720]: E1031 07:29:08.864946    4720 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Oct 31 07:29:08.865184 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:08.865114    4720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/ca31519e-aa90-4104-a175-f9af2b3bbeb4-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-6vkbv\" (UID: \"ca31519e-aa90-4104-a175-f9af2b3bbeb4\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-6vkbv"
Oct 31 07:29:08.865184 ip-10-0-139-83 kubenswrapper[4720]: E1031 07:29:08.865164    4720 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b44ed0e0-a72e-4746-a7de-ed8610da845d-metrics-certs podName:b44ed0e0-a72e-4746-a7de-ed8610da845d nodeName:}" failed. No retries permitted until 2026-10-31 07:29:10.865140869 +0000 UTC m=+4.996208932 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b44ed0e0-a72e-4746-a7de-ed8610da845d-metrics-certs") pod "router-default-6c88c45d57-c2bgn" (UID: "b44ed0e0-a72e-4746-a7de-ed8610da845d") : secret "router-metrics-certs-default" not found
Oct 31 07:29:08.865418 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:08.865203    4720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b1066ece-e6f3-4491-aa1f-a7b9a3aa248d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-8m768\" (UID: \"b1066ece-e6f3-4491-aa1f-a7b9a3aa248d\") " pod="openshift-insights/insights-runtime-extractor-8m768"
Oct 31 07:29:08.865418 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:08.865237    4720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/314c29aa-db6b-4f7f-86b4-c8952ce09c84-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-j94dz\" (UID: \"314c29aa-db6b-4f7f-86b4-c8952ce09c84\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-j94dz"
Oct 31 07:29:08.865418 ip-10-0-139-83 kubenswrapper[4720]: E1031 07:29:08.865209    4720 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 31 07:29:08.865418 ip-10-0-139-83 kubenswrapper[4720]: E1031 07:29:08.865285    4720 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7955c67bcc-pkntv: secret "image-registry-tls" not found
Oct 31 07:29:08.865418 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:08.865266    4720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/993cfed6-806d-44c5-bba8-c02203638fe7-registry-tls\") pod \"image-registry-7886c7d6c6-qdc9f\" (UID: \"993cfed6-806d-44c5-bba8-c02203638fe7\") " pod="openshift-image-registry/image-registry-7886c7d6c6-qdc9f"
Oct 31 07:29:08.865418 ip-10-0-139-83 kubenswrapper[4720]: E1031 07:29:08.865257    4720 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Oct 31 07:29:08.865418 ip-10-0-139-83 kubenswrapper[4720]: E1031 07:29:08.865351    4720 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 31 07:29:08.865418 ip-10-0-139-83 kubenswrapper[4720]: E1031 07:29:08.865359    4720 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Oct 31 07:29:08.865418 ip-10-0-139-83 kubenswrapper[4720]: E1031 07:29:08.865372    4720 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 31 07:29:08.865418 ip-10-0-139-83 kubenswrapper[4720]: E1031 07:29:08.865386    4720 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7886c7d6c6-qdc9f: secret "image-registry-tls" not found
Oct 31 07:29:08.865418 ip-10-0-139-83 kubenswrapper[4720]: E1031 07:29:08.865395    4720 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/28b59d93-b9ce-4ceb-9f3a-742525eb94c4-registry-tls podName:28b59d93-b9ce-4ceb-9f3a-742525eb94c4 nodeName:}" failed. No retries permitted until 2026-10-31 07:29:10.865376128 +0000 UTC m=+4.996444185 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/28b59d93-b9ce-4ceb-9f3a-742525eb94c4-registry-tls") pod "image-registry-7955c67bcc-pkntv" (UID: "28b59d93-b9ce-4ceb-9f3a-742525eb94c4") : secret "image-registry-tls" not found
Oct 31 07:29:08.865747 ip-10-0-139-83 kubenswrapper[4720]: E1031 07:29:08.865433    4720 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ca31519e-aa90-4104-a175-f9af2b3bbeb4-samples-operator-tls podName:ca31519e-aa90-4104-a175-f9af2b3bbeb4 nodeName:}" failed. No retries permitted until 2026-10-31 07:29:10.865417589 +0000 UTC m=+4.996485641 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/ca31519e-aa90-4104-a175-f9af2b3bbeb4-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-6vkbv" (UID: "ca31519e-aa90-4104-a175-f9af2b3bbeb4") : secret "samples-operator-tls" not found
Oct 31 07:29:08.865747 ip-10-0-139-83 kubenswrapper[4720]: E1031 07:29:08.865458    4720 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b1066ece-e6f3-4491-aa1f-a7b9a3aa248d-insights-runtime-extractor-tls podName:b1066ece-e6f3-4491-aa1f-a7b9a3aa248d nodeName:}" failed. No retries permitted until 2026-10-31 07:29:10.865447445 +0000 UTC m=+4.996515494 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/b1066ece-e6f3-4491-aa1f-a7b9a3aa248d-insights-runtime-extractor-tls") pod "insights-runtime-extractor-8m768" (UID: "b1066ece-e6f3-4491-aa1f-a7b9a3aa248d") : secret "insights-runtime-extractor-tls" not found
Oct 31 07:29:08.865747 ip-10-0-139-83 kubenswrapper[4720]: E1031 07:29:08.865474    4720 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/314c29aa-db6b-4f7f-86b4-c8952ce09c84-networking-console-plugin-cert podName:314c29aa-db6b-4f7f-86b4-c8952ce09c84 nodeName:}" failed. No retries permitted until 2026-10-31 07:29:10.865468088 +0000 UTC m=+4.996536136 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/314c29aa-db6b-4f7f-86b4-c8952ce09c84-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-j94dz" (UID: "314c29aa-db6b-4f7f-86b4-c8952ce09c84") : secret "networking-console-plugin-cert" not found
Oct 31 07:29:08.865747 ip-10-0-139-83 kubenswrapper[4720]: E1031 07:29:08.865490    4720 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/993cfed6-806d-44c5-bba8-c02203638fe7-registry-tls podName:993cfed6-806d-44c5-bba8-c02203638fe7 nodeName:}" failed. No retries permitted until 2026-10-31 07:29:10.865484736 +0000 UTC m=+4.996552786 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/993cfed6-806d-44c5-bba8-c02203638fe7-registry-tls") pod "image-registry-7886c7d6c6-qdc9f" (UID: "993cfed6-806d-44c5-bba8-c02203638fe7") : secret "image-registry-tls" not found
Oct 31 07:29:08.865747 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:08.865522    4720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c3ad95fd-e997-459c-93fd-27c4c030ea0d-metrics-certs\") pod \"network-metrics-daemon-r2zbv\" (UID: \"c3ad95fd-e997-459c-93fd-27c4c030ea0d\") " pod="openshift-multus/network-metrics-daemon-r2zbv"
Oct 31 07:29:08.865747 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:08.865555    4720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a73024cb-f581-450d-a09c-0565838c5baa-metrics-tls\") pod \"dns-default-9cmfx\" (UID: \"a73024cb-f581-450d-a09c-0565838c5baa\") " pod="openshift-dns/dns-default-9cmfx"
Oct 31 07:29:08.865747 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:08.865576    4720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/789f11d8-d951-4144-a589-a2bdbf6844e3-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-6k8mn\" (UID: \"789f11d8-d951-4144-a589-a2bdbf6844e3\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-6k8mn"
Oct 31 07:29:08.865747 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:08.865609    4720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b44ed0e0-a72e-4746-a7de-ed8610da845d-service-ca-bundle\") pod \"router-default-6c88c45d57-c2bgn\" (UID: \"b44ed0e0-a72e-4746-a7de-ed8610da845d\") " pod="openshift-ingress/router-default-6c88c45d57-c2bgn"
Oct 31 07:29:08.865747 ip-10-0-139-83 kubenswrapper[4720]: E1031 07:29:08.865623    4720 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Oct 31 07:29:08.865747 ip-10-0-139-83 kubenswrapper[4720]: E1031 07:29:08.865646    4720 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 31 07:29:08.865747 ip-10-0-139-83 kubenswrapper[4720]: E1031 07:29:08.865651    4720 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c3ad95fd-e997-459c-93fd-27c4c030ea0d-metrics-certs podName:c3ad95fd-e997-459c-93fd-27c4c030ea0d nodeName:}" failed. No retries permitted until 2026-10-31 07:29:10.865641959 +0000 UTC m=+4.996710008 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c3ad95fd-e997-459c-93fd-27c4c030ea0d-metrics-certs") pod "network-metrics-daemon-r2zbv" (UID: "c3ad95fd-e997-459c-93fd-27c4c030ea0d") : secret "metrics-daemon-secret" not found
Oct 31 07:29:08.865747 ip-10-0-139-83 kubenswrapper[4720]: E1031 07:29:08.865674    4720 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a73024cb-f581-450d-a09c-0565838c5baa-metrics-tls podName:a73024cb-f581-450d-a09c-0565838c5baa nodeName:}" failed. No retries permitted until 2026-10-31 07:29:10.865665579 +0000 UTC m=+4.996733662 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/a73024cb-f581-450d-a09c-0565838c5baa-metrics-tls") pod "dns-default-9cmfx" (UID: "a73024cb-f581-450d-a09c-0565838c5baa") : secret "dns-default-metrics-tls" not found
Oct 31 07:29:08.865747 ip-10-0-139-83 kubenswrapper[4720]: E1031 07:29:08.865699    4720 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Oct 31 07:29:08.865747 ip-10-0-139-83 kubenswrapper[4720]: E1031 07:29:08.865701    4720 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b44ed0e0-a72e-4746-a7de-ed8610da845d-service-ca-bundle podName:b44ed0e0-a72e-4746-a7de-ed8610da845d nodeName:}" failed. No retries permitted until 2026-10-31 07:29:10.865690134 +0000 UTC m=+4.996758183 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/b44ed0e0-a72e-4746-a7de-ed8610da845d-service-ca-bundle") pod "router-default-6c88c45d57-c2bgn" (UID: "b44ed0e0-a72e-4746-a7de-ed8610da845d") : configmap references non-existent config key: service-ca.crt
Oct 31 07:29:08.865747 ip-10-0-139-83 kubenswrapper[4720]: E1031 07:29:08.865738    4720 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/789f11d8-d951-4144-a589-a2bdbf6844e3-cluster-monitoring-operator-tls podName:789f11d8-d951-4144-a589-a2bdbf6844e3 nodeName:}" failed. No retries permitted until 2026-10-31 07:29:10.865729516 +0000 UTC m=+4.996797568 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/789f11d8-d951-4144-a589-a2bdbf6844e3-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-6k8mn" (UID: "789f11d8-d951-4144-a589-a2bdbf6844e3") : secret "cluster-monitoring-operator-tls" not found
Oct 31 07:29:09.095942 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:09.095909    4720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-5t4pm" event={"ID":"0b25f791-3520-481d-acbb-a3bef222fdac","Type":"ContainerStarted","Data":"0ee5bab6cd3e53e34e4775081ba33c98358eed1ff091937f440e3fecaf09d7e1"}
Oct 31 07:29:09.097637 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:09.097618    4720 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-4jszp_b1db0e3b-e85d-4769-8012-cf4b6f740d9d/console-operator/2.log"
Oct 31 07:29:09.097955 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:09.097939    4720 scope.go:122] "RemoveContainer" containerID="c65532fa5602117781458aaf71d70fac8484fefc6f0c87e31a745cca02403548"
Oct 31 07:29:09.098164 ip-10-0-139-83 kubenswrapper[4720]: E1031 07:29:09.098142    4720 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-4jszp_openshift-console-operator(b1db0e3b-e85d-4769-8012-cf4b6f740d9d)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-4jszp" podUID="b1db0e3b-e85d-4769-8012-cf4b6f740d9d"
Oct 31 07:29:10.386472 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:10.386431    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 07:29:10.731116 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:10.731039    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 07:29:10.881417 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:10.881390    4720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/314c29aa-db6b-4f7f-86b4-c8952ce09c84-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-j94dz\" (UID: \"314c29aa-db6b-4f7f-86b4-c8952ce09c84\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-j94dz"
Oct 31 07:29:10.881565 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:10.881422    4720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/993cfed6-806d-44c5-bba8-c02203638fe7-registry-tls\") pod \"image-registry-7886c7d6c6-qdc9f\" (UID: \"993cfed6-806d-44c5-bba8-c02203638fe7\") " pod="openshift-image-registry/image-registry-7886c7d6c6-qdc9f"
Oct 31 07:29:10.881565 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:10.881447    4720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c3ad95fd-e997-459c-93fd-27c4c030ea0d-metrics-certs\") pod \"network-metrics-daemon-r2zbv\" (UID: \"c3ad95fd-e997-459c-93fd-27c4c030ea0d\") " pod="openshift-multus/network-metrics-daemon-r2zbv"
Oct 31 07:29:10.881565 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:10.881467    4720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a73024cb-f581-450d-a09c-0565838c5baa-metrics-tls\") pod \"dns-default-9cmfx\" (UID: \"a73024cb-f581-450d-a09c-0565838c5baa\") " pod="openshift-dns/dns-default-9cmfx"
Oct 31 07:29:10.881565 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:10.881484    4720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/789f11d8-d951-4144-a589-a2bdbf6844e3-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-6k8mn\" (UID: \"789f11d8-d951-4144-a589-a2bdbf6844e3\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-6k8mn"
Oct 31 07:29:10.881565 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:10.881529    4720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b44ed0e0-a72e-4746-a7de-ed8610da845d-service-ca-bundle\") pod \"router-default-6c88c45d57-c2bgn\" (UID: \"b44ed0e0-a72e-4746-a7de-ed8610da845d\") " pod="openshift-ingress/router-default-6c88c45d57-c2bgn"
Oct 31 07:29:10.881565 ip-10-0-139-83 kubenswrapper[4720]: E1031 07:29:10.881554    4720 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Oct 31 07:29:10.881818 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:10.881567    4720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b44ed0e0-a72e-4746-a7de-ed8610da845d-metrics-certs\") pod \"router-default-6c88c45d57-c2bgn\" (UID: \"b44ed0e0-a72e-4746-a7de-ed8610da845d\") " pod="openshift-ingress/router-default-6c88c45d57-c2bgn"
Oct 31 07:29:10.881818 ip-10-0-139-83 kubenswrapper[4720]: E1031 07:29:10.881605    4720 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 31 07:29:10.881818 ip-10-0-139-83 kubenswrapper[4720]: E1031 07:29:10.881612    4720 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Oct 31 07:29:10.881818 ip-10-0-139-83 kubenswrapper[4720]: E1031 07:29:10.881623    4720 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c3ad95fd-e997-459c-93fd-27c4c030ea0d-metrics-certs podName:c3ad95fd-e997-459c-93fd-27c4c030ea0d nodeName:}" failed. No retries permitted until 2026-10-31 07:29:14.881604443 +0000 UTC m=+9.012672492 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c3ad95fd-e997-459c-93fd-27c4c030ea0d-metrics-certs") pod "network-metrics-daemon-r2zbv" (UID: "c3ad95fd-e997-459c-93fd-27c4c030ea0d") : secret "metrics-daemon-secret" not found
Oct 31 07:29:10.881818 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:10.881605    4720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/28b59d93-b9ce-4ceb-9f3a-742525eb94c4-registry-tls\") pod \"image-registry-7955c67bcc-pkntv\" (UID: \"28b59d93-b9ce-4ceb-9f3a-742525eb94c4\") " pod="openshift-image-registry/image-registry-7955c67bcc-pkntv"
Oct 31 07:29:10.881818 ip-10-0-139-83 kubenswrapper[4720]: E1031 07:29:10.881557    4720 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 31 07:29:10.881818 ip-10-0-139-83 kubenswrapper[4720]: E1031 07:29:10.881664    4720 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7886c7d6c6-qdc9f: secret "image-registry-tls" not found
Oct 31 07:29:10.881818 ip-10-0-139-83 kubenswrapper[4720]: E1031 07:29:10.881669    4720 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 31 07:29:10.881818 ip-10-0-139-83 kubenswrapper[4720]: E1031 07:29:10.881554    4720 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Oct 31 07:29:10.881818 ip-10-0-139-83 kubenswrapper[4720]: E1031 07:29:10.881680    4720 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7955c67bcc-pkntv: secret "image-registry-tls" not found
Oct 31 07:29:10.881818 ip-10-0-139-83 kubenswrapper[4720]: E1031 07:29:10.881665    4720 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b44ed0e0-a72e-4746-a7de-ed8610da845d-service-ca-bundle podName:b44ed0e0-a72e-4746-a7de-ed8610da845d nodeName:}" failed. No retries permitted until 2026-10-31 07:29:14.881648007 +0000 UTC m=+9.012716076 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/b44ed0e0-a72e-4746-a7de-ed8610da845d-service-ca-bundle") pod "router-default-6c88c45d57-c2bgn" (UID: "b44ed0e0-a72e-4746-a7de-ed8610da845d") : configmap references non-existent config key: service-ca.crt
Oct 31 07:29:10.881818 ip-10-0-139-83 kubenswrapper[4720]: E1031 07:29:10.881719    4720 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a73024cb-f581-450d-a09c-0565838c5baa-metrics-tls podName:a73024cb-f581-450d-a09c-0565838c5baa nodeName:}" failed. No retries permitted until 2026-10-31 07:29:14.881703396 +0000 UTC m=+9.012771448 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/a73024cb-f581-450d-a09c-0565838c5baa-metrics-tls") pod "dns-default-9cmfx" (UID: "a73024cb-f581-450d-a09c-0565838c5baa") : secret "dns-default-metrics-tls" not found
Oct 31 07:29:10.881818 ip-10-0-139-83 kubenswrapper[4720]: E1031 07:29:10.881737    4720 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/789f11d8-d951-4144-a589-a2bdbf6844e3-cluster-monitoring-operator-tls podName:789f11d8-d951-4144-a589-a2bdbf6844e3 nodeName:}" failed. No retries permitted until 2026-10-31 07:29:14.881727571 +0000 UTC m=+9.012795620 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/789f11d8-d951-4144-a589-a2bdbf6844e3-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-6k8mn" (UID: "789f11d8-d951-4144-a589-a2bdbf6844e3") : secret "cluster-monitoring-operator-tls" not found
Oct 31 07:29:10.881818 ip-10-0-139-83 kubenswrapper[4720]: E1031 07:29:10.881750    4720 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/993cfed6-806d-44c5-bba8-c02203638fe7-registry-tls podName:993cfed6-806d-44c5-bba8-c02203638fe7 nodeName:}" failed. No retries permitted until 2026-10-31 07:29:14.88174511 +0000 UTC m=+9.012813159 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/993cfed6-806d-44c5-bba8-c02203638fe7-registry-tls") pod "image-registry-7886c7d6c6-qdc9f" (UID: "993cfed6-806d-44c5-bba8-c02203638fe7") : secret "image-registry-tls" not found
Oct 31 07:29:10.881818 ip-10-0-139-83 kubenswrapper[4720]: E1031 07:29:10.881769    4720 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/314c29aa-db6b-4f7f-86b4-c8952ce09c84-networking-console-plugin-cert podName:314c29aa-db6b-4f7f-86b4-c8952ce09c84 nodeName:}" failed. No retries permitted until 2026-10-31 07:29:14.881763846 +0000 UTC m=+9.012831895 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/314c29aa-db6b-4f7f-86b4-c8952ce09c84-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-j94dz" (UID: "314c29aa-db6b-4f7f-86b4-c8952ce09c84") : secret "networking-console-plugin-cert" not found
Oct 31 07:29:10.881818 ip-10-0-139-83 kubenswrapper[4720]: E1031 07:29:10.881777    4720 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/28b59d93-b9ce-4ceb-9f3a-742525eb94c4-registry-tls podName:28b59d93-b9ce-4ceb-9f3a-742525eb94c4 nodeName:}" failed. No retries permitted until 2026-10-31 07:29:14.881773185 +0000 UTC m=+9.012841245 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/28b59d93-b9ce-4ceb-9f3a-742525eb94c4-registry-tls") pod "image-registry-7955c67bcc-pkntv" (UID: "28b59d93-b9ce-4ceb-9f3a-742525eb94c4") : secret "image-registry-tls" not found
Oct 31 07:29:10.881818 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:10.881788    4720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/ca31519e-aa90-4104-a175-f9af2b3bbeb4-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-6vkbv\" (UID: \"ca31519e-aa90-4104-a175-f9af2b3bbeb4\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-6vkbv"
Oct 31 07:29:10.881818 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:10.881816    4720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b1066ece-e6f3-4491-aa1f-a7b9a3aa248d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-8m768\" (UID: \"b1066ece-e6f3-4491-aa1f-a7b9a3aa248d\") " pod="openshift-insights/insights-runtime-extractor-8m768"
Oct 31 07:29:10.882416 ip-10-0-139-83 kubenswrapper[4720]: E1031 07:29:10.881873    4720 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 31 07:29:10.882416 ip-10-0-139-83 kubenswrapper[4720]: E1031 07:29:10.881893    4720 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b1066ece-e6f3-4491-aa1f-a7b9a3aa248d-insights-runtime-extractor-tls podName:b1066ece-e6f3-4491-aa1f-a7b9a3aa248d nodeName:}" failed. No retries permitted until 2026-10-31 07:29:14.881887557 +0000 UTC m=+9.012955605 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/b1066ece-e6f3-4491-aa1f-a7b9a3aa248d-insights-runtime-extractor-tls") pod "insights-runtime-extractor-8m768" (UID: "b1066ece-e6f3-4491-aa1f-a7b9a3aa248d") : secret "insights-runtime-extractor-tls" not found
Oct 31 07:29:10.882416 ip-10-0-139-83 kubenswrapper[4720]: E1031 07:29:10.881926    4720 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Oct 31 07:29:10.882416 ip-10-0-139-83 kubenswrapper[4720]: E1031 07:29:10.881966    4720 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ca31519e-aa90-4104-a175-f9af2b3bbeb4-samples-operator-tls podName:ca31519e-aa90-4104-a175-f9af2b3bbeb4 nodeName:}" failed. No retries permitted until 2026-10-31 07:29:14.881951486 +0000 UTC m=+9.013019549 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/ca31519e-aa90-4104-a175-f9af2b3bbeb4-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-6vkbv" (UID: "ca31519e-aa90-4104-a175-f9af2b3bbeb4") : secret "samples-operator-tls" not found
Oct 31 07:29:10.882416 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:10.882217    4720 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Oct 31 07:29:10.885786 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:10.885763    4720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b44ed0e0-a72e-4746-a7de-ed8610da845d-metrics-certs\") pod \"router-default-6c88c45d57-c2bgn\" (UID: \"b44ed0e0-a72e-4746-a7de-ed8610da845d\") " pod="openshift-ingress/router-default-6c88c45d57-c2bgn"
Oct 31 07:29:10.902056 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:10.902034    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 07:29:11.069581 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:11.069487    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 07:29:11.227417 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:11.227389    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 07:29:11.389180 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:11.389152    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 07:29:11.406744 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:11.406723    4720 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-tnw4j_c4233c69-4213-43be-8809-76612b6b6f3e/dns-node-resolver/0.log"
Oct 31 07:29:11.552392 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:11.552365    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 07:29:12.014162 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:12.014135    4720 kubelet.go:2727] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-7cd56b8dbc-4jszp"
Oct 31 07:29:12.014573 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:12.014556    4720 scope.go:122] "RemoveContainer" containerID="c65532fa5602117781458aaf71d70fac8484fefc6f0c87e31a745cca02403548"
Oct 31 07:29:12.014773 ip-10-0-139-83 kubenswrapper[4720]: E1031 07:29:12.014754    4720 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-4jszp_openshift-console-operator(b1db0e3b-e85d-4769-8012-cf4b6f740d9d)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-4jszp" podUID="b1db0e3b-e85d-4769-8012-cf4b6f740d9d"
Oct 31 07:29:12.215235 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:12.215196    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 07:29:12.407604 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:12.407585    4720 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-th7x7_fc64d31a-f48a-4293-a4de-262045d0b7e8/node-ca/0.log"
Oct 31 07:29:12.415110 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:12.415088    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 07:29:12.615322 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:12.615293    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 07:29:12.814985 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:12.814919    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 07:29:13.011006 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:13.010978    4720 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-kmcg5_2536c1c8-8a46-4af3-bc6d-81454a908d3f/serve-healthcheck-canary/0.log"
Oct 31 07:29:13.014883 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:13.014861    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 07:29:13.407841 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:13.407809    4720 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-66df5d496b-5t4pm_0b25f791-3520-481d-acbb-a3bef222fdac/migrator/0.log"
Oct 31 07:29:13.810605 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:13.810546    4720 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-7d4bc45dd6-p9z4r_cb9e4575-9265-43b5-bf02-7843611d6563/kube-storage-version-migrator-operator/0.log"
Oct 31 07:29:14.207862 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:14.207815    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 07:29:14.388954 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:14.388924    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 07:29:14.529784 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:14.529719    4720 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-5p7xw"
Oct 31 07:29:14.530128 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:14.529898    4720 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Oct 31 07:29:14.534907 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:14.534876    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 07:29:14.679061 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:14.679039    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 07:29:14.822206 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:14.822121    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 07:29:14.919132 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:14.919107    4720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b44ed0e0-a72e-4746-a7de-ed8610da845d-service-ca-bundle\") pod \"router-default-6c88c45d57-c2bgn\" (UID: \"b44ed0e0-a72e-4746-a7de-ed8610da845d\") " pod="openshift-ingress/router-default-6c88c45d57-c2bgn"
Oct 31 07:29:14.919248 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:14.919157    4720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/28b59d93-b9ce-4ceb-9f3a-742525eb94c4-registry-tls\") pod \"image-registry-7955c67bcc-pkntv\" (UID: \"28b59d93-b9ce-4ceb-9f3a-742525eb94c4\") " pod="openshift-image-registry/image-registry-7955c67bcc-pkntv"
Oct 31 07:29:14.919248 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:14.919187    4720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/ca31519e-aa90-4104-a175-f9af2b3bbeb4-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-6vkbv\" (UID: \"ca31519e-aa90-4104-a175-f9af2b3bbeb4\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-6vkbv"
Oct 31 07:29:14.919248 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:14.919229    4720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b1066ece-e6f3-4491-aa1f-a7b9a3aa248d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-8m768\" (UID: \"b1066ece-e6f3-4491-aa1f-a7b9a3aa248d\") " pod="openshift-insights/insights-runtime-extractor-8m768"
Oct 31 07:29:14.919431 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:14.919257    4720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/314c29aa-db6b-4f7f-86b4-c8952ce09c84-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-j94dz\" (UID: \"314c29aa-db6b-4f7f-86b4-c8952ce09c84\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-j94dz"
Oct 31 07:29:14.919431 ip-10-0-139-83 kubenswrapper[4720]: E1031 07:29:14.919291    4720 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b44ed0e0-a72e-4746-a7de-ed8610da845d-service-ca-bundle podName:b44ed0e0-a72e-4746-a7de-ed8610da845d nodeName:}" failed. No retries permitted until 2026-10-31 07:29:22.919275353 +0000 UTC m=+17.050343401 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/b44ed0e0-a72e-4746-a7de-ed8610da845d-service-ca-bundle") pod "router-default-6c88c45d57-c2bgn" (UID: "b44ed0e0-a72e-4746-a7de-ed8610da845d") : configmap references non-existent config key: service-ca.crt
Oct 31 07:29:14.919431 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:14.919310    4720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/993cfed6-806d-44c5-bba8-c02203638fe7-registry-tls\") pod \"image-registry-7886c7d6c6-qdc9f\" (UID: \"993cfed6-806d-44c5-bba8-c02203638fe7\") " pod="openshift-image-registry/image-registry-7886c7d6c6-qdc9f"
Oct 31 07:29:14.919431 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:14.919367    4720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c3ad95fd-e997-459c-93fd-27c4c030ea0d-metrics-certs\") pod \"network-metrics-daemon-r2zbv\" (UID: \"c3ad95fd-e997-459c-93fd-27c4c030ea0d\") " pod="openshift-multus/network-metrics-daemon-r2zbv"
Oct 31 07:29:14.919431 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:14.919397    4720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a73024cb-f581-450d-a09c-0565838c5baa-metrics-tls\") pod \"dns-default-9cmfx\" (UID: \"a73024cb-f581-450d-a09c-0565838c5baa\") " pod="openshift-dns/dns-default-9cmfx"
Oct 31 07:29:14.919695 ip-10-0-139-83 kubenswrapper[4720]: E1031 07:29:14.919448    4720 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 31 07:29:14.919695 ip-10-0-139-83 kubenswrapper[4720]: E1031 07:29:14.919469    4720 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 31 07:29:14.919695 ip-10-0-139-83 kubenswrapper[4720]: E1031 07:29:14.919523    4720 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b1066ece-e6f3-4491-aa1f-a7b9a3aa248d-insights-runtime-extractor-tls podName:b1066ece-e6f3-4491-aa1f-a7b9a3aa248d nodeName:}" failed. No retries permitted until 2026-10-31 07:29:22.919502486 +0000 UTC m=+17.050570550 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/b1066ece-e6f3-4491-aa1f-a7b9a3aa248d-insights-runtime-extractor-tls") pod "insights-runtime-extractor-8m768" (UID: "b1066ece-e6f3-4491-aa1f-a7b9a3aa248d") : secret "insights-runtime-extractor-tls" not found
Oct 31 07:29:14.919695 ip-10-0-139-83 kubenswrapper[4720]: E1031 07:29:14.919543    4720 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a73024cb-f581-450d-a09c-0565838c5baa-metrics-tls podName:a73024cb-f581-450d-a09c-0565838c5baa nodeName:}" failed. No retries permitted until 2026-10-31 07:29:22.919532062 +0000 UTC m=+17.050600111 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/a73024cb-f581-450d-a09c-0565838c5baa-metrics-tls") pod "dns-default-9cmfx" (UID: "a73024cb-f581-450d-a09c-0565838c5baa") : secret "dns-default-metrics-tls" not found
Oct 31 07:29:14.919695 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:14.919581    4720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/789f11d8-d951-4144-a589-a2bdbf6844e3-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-6k8mn\" (UID: \"789f11d8-d951-4144-a589-a2bdbf6844e3\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-6k8mn"
Oct 31 07:29:14.919695 ip-10-0-139-83 kubenswrapper[4720]: E1031 07:29:14.919679    4720 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Oct 31 07:29:14.920000 ip-10-0-139-83 kubenswrapper[4720]: E1031 07:29:14.919713    4720 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/789f11d8-d951-4144-a589-a2bdbf6844e3-cluster-monitoring-operator-tls podName:789f11d8-d951-4144-a589-a2bdbf6844e3 nodeName:}" failed. No retries permitted until 2026-10-31 07:29:22.91970459 +0000 UTC m=+17.050772652 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/789f11d8-d951-4144-a589-a2bdbf6844e3-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-6k8mn" (UID: "789f11d8-d951-4144-a589-a2bdbf6844e3") : secret "cluster-monitoring-operator-tls" not found
Oct 31 07:29:14.921807 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:14.921781    4720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c3ad95fd-e997-459c-93fd-27c4c030ea0d-metrics-certs\") pod \"network-metrics-daemon-r2zbv\" (UID: \"c3ad95fd-e997-459c-93fd-27c4c030ea0d\") " pod="openshift-multus/network-metrics-daemon-r2zbv"
Oct 31 07:29:14.921976 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:14.921948    4720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/993cfed6-806d-44c5-bba8-c02203638fe7-registry-tls\") pod \"image-registry-7886c7d6c6-qdc9f\" (UID: \"993cfed6-806d-44c5-bba8-c02203638fe7\") " pod="openshift-image-registry/image-registry-7886c7d6c6-qdc9f"
Oct 31 07:29:14.922250 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:14.922227    4720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/ca31519e-aa90-4104-a175-f9af2b3bbeb4-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-6vkbv\" (UID: \"ca31519e-aa90-4104-a175-f9af2b3bbeb4\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-6vkbv"
Oct 31 07:29:14.922294 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:14.922234    4720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/28b59d93-b9ce-4ceb-9f3a-742525eb94c4-registry-tls\") pod \"image-registry-7955c67bcc-pkntv\" (UID: \"28b59d93-b9ce-4ceb-9f3a-742525eb94c4\") " pod="openshift-image-registry/image-registry-7955c67bcc-pkntv"
Oct 31 07:29:14.922419 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:14.922406    4720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/314c29aa-db6b-4f7f-86b4-c8952ce09c84-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-j94dz\" (UID: \"314c29aa-db6b-4f7f-86b4-c8952ce09c84\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-j94dz"
Oct 31 07:29:15.007642 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:15.007616    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 07:29:15.173462 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:15.173431    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 07:29:15.210330 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:15.210298    4720 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7886c7d6c6-qdc9f"
Oct 31 07:29:15.211385 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:15.211368    4720 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-6vkbv"
Oct 31 07:29:15.213793 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:15.213777    4720 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-86c478bb88-j94dz"
Oct 31 07:29:15.219262 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:15.219244    4720 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r2zbv"
Oct 31 07:29:15.222229 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:15.222210    4720 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7955c67bcc-pkntv"
Oct 31 07:29:15.595764 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:15.595735    4720 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-6vkbv"]
Oct 31 07:29:15.697608 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:15.697572    4720 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-j94dz"]
Oct 31 07:29:15.701010 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:15.700990    4720 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-7955c67bcc-pkntv"]
Oct 31 07:29:15.701679 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:15.701643    4720 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod314c29aa_db6b_4f7f_86b4_c8952ce09c84.slice/crio-5cbf61951eb0c82ca6b80abab51f90e3d84ba770716a5f448002b2cb3974b2c9 WatchSource:0}: Error finding container 5cbf61951eb0c82ca6b80abab51f90e3d84ba770716a5f448002b2cb3974b2c9: Status 404 returned error can't find the container with id 5cbf61951eb0c82ca6b80abab51f90e3d84ba770716a5f448002b2cb3974b2c9
Oct 31 07:29:15.704558 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:15.704539    4720 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod28b59d93_b9ce_4ceb_9f3a_742525eb94c4.slice/crio-629d17fd8d940802e6ed7c5e94e98da44d1d2b16184f84afd22fdf51628c21b4 WatchSource:0}: Error finding container 629d17fd8d940802e6ed7c5e94e98da44d1d2b16184f84afd22fdf51628c21b4: Status 404 returned error can't find the container with id 629d17fd8d940802e6ed7c5e94e98da44d1d2b16184f84afd22fdf51628c21b4
Oct 31 07:29:15.748112 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:15.748087    4720 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-r2zbv"]
Oct 31 07:29:15.751277 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:15.751256    4720 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-7886c7d6c6-qdc9f"]
Oct 31 07:29:15.751923 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:15.751892    4720 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc3ad95fd_e997_459c_93fd_27c4c030ea0d.slice/crio-2b865880f9bad8e1a8ddccba7b2265ad00b441fb021ae004777bee706a147aec WatchSource:0}: Error finding container 2b865880f9bad8e1a8ddccba7b2265ad00b441fb021ae004777bee706a147aec: Status 404 returned error can't find the container with id 2b865880f9bad8e1a8ddccba7b2265ad00b441fb021ae004777bee706a147aec
Oct 31 07:29:15.756069 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:15.756047    4720 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod993cfed6_806d_44c5_bba8_c02203638fe7.slice/crio-40db303d05df9e64756d1fe275ba37eb942abd352ec78fce0a9b59037bfae450 WatchSource:0}: Error finding container 40db303d05df9e64756d1fe275ba37eb942abd352ec78fce0a9b59037bfae450: Status 404 returned error can't find the container with id 40db303d05df9e64756d1fe275ba37eb942abd352ec78fce0a9b59037bfae450
Oct 31 07:29:16.121436 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:16.121402    4720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-r2zbv" event={"ID":"c3ad95fd-e997-459c-93fd-27c4c030ea0d","Type":"ContainerStarted","Data":"2b865880f9bad8e1a8ddccba7b2265ad00b441fb021ae004777bee706a147aec"}
Oct 31 07:29:16.122575 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:16.122547    4720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-6vkbv" event={"ID":"ca31519e-aa90-4104-a175-f9af2b3bbeb4","Type":"ContainerStarted","Data":"5a2c3a24c7b31579cbd4499342c19a21a1dec9a5252d31a55805d69a93f561ef"}
Oct 31 07:29:16.123712 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:16.123688    4720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-86c478bb88-j94dz" event={"ID":"314c29aa-db6b-4f7f-86b4-c8952ce09c84","Type":"ContainerStarted","Data":"5cbf61951eb0c82ca6b80abab51f90e3d84ba770716a5f448002b2cb3974b2c9"}
Oct 31 07:29:16.125815 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:16.125686    4720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7886c7d6c6-qdc9f" event={"ID":"993cfed6-806d-44c5-bba8-c02203638fe7","Type":"ContainerStarted","Data":"c1afbd4a198d8efa93789314bcb177363aaf598ad88d64e78122fedef482f75c"}
Oct 31 07:29:16.125815 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:16.125771    4720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7886c7d6c6-qdc9f" event={"ID":"993cfed6-806d-44c5-bba8-c02203638fe7","Type":"ContainerStarted","Data":"40db303d05df9e64756d1fe275ba37eb942abd352ec78fce0a9b59037bfae450"}
Oct 31 07:29:16.130136 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:16.130117    4720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7955c67bcc-pkntv" event={"ID":"28b59d93-b9ce-4ceb-9f3a-742525eb94c4","Type":"ContainerStarted","Data":"9dfc89ae8f837cd9cd34ff414c2ea73f9645696fad30d9e8618c22942bbb61db"}
Oct 31 07:29:16.130215 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:16.130146    4720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7955c67bcc-pkntv" event={"ID":"28b59d93-b9ce-4ceb-9f3a-742525eb94c4","Type":"ContainerStarted","Data":"629d17fd8d940802e6ed7c5e94e98da44d1d2b16184f84afd22fdf51628c21b4"}
Oct 31 07:29:16.207739 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:16.207716    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 07:29:16.413621 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:16.413559    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 07:29:16.609465 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:16.609428    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 07:29:16.807255 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:16.807177    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 07:29:17.006496 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:17.006426    4720 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-7955c67bcc-pkntv"
Oct 31 07:29:17.013319 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:17.013285    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 07:29:17.134839 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:17.134804    4720 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Oct 31 07:29:18.139561 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:18.139481    4720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-r2zbv" event={"ID":"c3ad95fd-e997-459c-93fd-27c4c030ea0d","Type":"ContainerStarted","Data":"93f9540917bd1bd7f82766c59a5c35f9ffd6f6c89b3df7a58f7a6290ca3528e8"}
Oct 31 07:29:18.139561 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:18.139528    4720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-r2zbv" event={"ID":"c3ad95fd-e997-459c-93fd-27c4c030ea0d","Type":"ContainerStarted","Data":"26f1ea20bae10a6ab2a20318a9124f249e024286ef86b514aa561d196e9fc32f"}
Oct 31 07:29:18.141394 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:18.141357    4720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-6vkbv" event={"ID":"ca31519e-aa90-4104-a175-f9af2b3bbeb4","Type":"ContainerStarted","Data":"709f3704c1ddbd70fd94aac4d02450397d74846044e277bc08155d0e6db3b05c"}
Oct 31 07:29:18.141527 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:18.141406    4720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-6vkbv" event={"ID":"ca31519e-aa90-4104-a175-f9af2b3bbeb4","Type":"ContainerStarted","Data":"7392281f69e665ee346ff3082fcbdc08d68fa797b7c40a0ce4fccc25b036b8d1"}
Oct 31 07:29:18.142820 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:18.142799    4720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-86c478bb88-j94dz" event={"ID":"314c29aa-db6b-4f7f-86b4-c8952ce09c84","Type":"ContainerStarted","Data":"04ebf521e4c50929231aec2f5ad537618ae795389c7dcf478dc813b92e0f449f"}
Oct 31 07:29:18.262040 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:18.261999    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 07:29:18.405374 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:18.405272    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 07:29:18.617215 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:18.617180    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 07:29:18.826531 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:18.826451    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 07:29:19.008882 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:19.008838    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 07:29:20.020303 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:20.020274    4720 scope.go:122] "RemoveContainer" containerID="c65532fa5602117781458aaf71d70fac8484fefc6f0c87e31a745cca02403548"
Oct 31 07:29:20.151345 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:20.151312    4720 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-4jszp_b1db0e3b-e85d-4769-8012-cf4b6f740d9d/console-operator/2.log"
Oct 31 07:29:20.151510 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:20.151426    4720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-4jszp" event={"ID":"b1db0e3b-e85d-4769-8012-cf4b6f740d9d","Type":"ContainerStarted","Data":"faf74e47850279aed0868e046eaac226186478a621ae3160e9ac1f241977f830"}
Oct 31 07:29:20.151702 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:20.151675    4720 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-4jszp"
Oct 31 07:29:20.152590 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:20.152563    4720 patch_prober.go:29] interesting pod/console-operator-7cd56b8dbc-4jszp container/console-operator namespace/openshift-console-operator: Readiness probe status=failure output="Get \"https://10.132.0.20:8443/readyz\": dial tcp 10.132.0.20:8443: connect: connection refused" start-of-body=
Oct 31 07:29:20.152679 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:20.152611    4720 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-console-operator/console-operator-7cd56b8dbc-4jszp" podUID="b1db0e3b-e85d-4769-8012-cf4b6f740d9d" containerName="console-operator" probeResult="failure" output="Get \"https://10.132.0.20:8443/readyz\": dial tcp 10.132.0.20:8443: connect: connection refused"
Oct 31 07:29:20.204924 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:20.204903    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 07:29:20.396822 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:20.396795    4720 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-7886c7d6c6-qdc9f"
Oct 31 07:29:20.412260 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:20.412233    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 07:29:20.600242 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:20.600196    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 07:29:20.800367 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:20.800269    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 07:29:21.008930 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:21.008901    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 07:29:21.159050 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:21.159027    4720 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-4jszp"
Oct 31 07:29:21.624590 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:21.624564    4720 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-h4k2m"
Oct 31 07:29:21.625570 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:21.625545    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 07:29:21.790593 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:21.790562    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 07:29:22.006646 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:22.006572    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 07:29:22.055253 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:22.055221    4720 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-7886c7d6c6-qdc9f"]
Oct 31 07:29:22.276929 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:22.276850    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 07:29:22.433462 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:22.433430    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 07:29:22.757704 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:22.757671    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 07:29:22.912175 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:22.912145    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 07:29:22.989394 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:22.989371    4720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b1066ece-e6f3-4491-aa1f-a7b9a3aa248d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-8m768\" (UID: \"b1066ece-e6f3-4491-aa1f-a7b9a3aa248d\") " pod="openshift-insights/insights-runtime-extractor-8m768"
Oct 31 07:29:22.989525 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:22.989408    4720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a73024cb-f581-450d-a09c-0565838c5baa-metrics-tls\") pod \"dns-default-9cmfx\" (UID: \"a73024cb-f581-450d-a09c-0565838c5baa\") " pod="openshift-dns/dns-default-9cmfx"
Oct 31 07:29:22.989525 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:22.989429    4720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/789f11d8-d951-4144-a589-a2bdbf6844e3-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-6k8mn\" (UID: \"789f11d8-d951-4144-a589-a2bdbf6844e3\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-6k8mn"
Oct 31 07:29:22.989525 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:22.989466    4720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b44ed0e0-a72e-4746-a7de-ed8610da845d-service-ca-bundle\") pod \"router-default-6c88c45d57-c2bgn\" (UID: \"b44ed0e0-a72e-4746-a7de-ed8610da845d\") " pod="openshift-ingress/router-default-6c88c45d57-c2bgn"
Oct 31 07:29:22.990123 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:22.990101    4720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b44ed0e0-a72e-4746-a7de-ed8610da845d-service-ca-bundle\") pod \"router-default-6c88c45d57-c2bgn\" (UID: \"b44ed0e0-a72e-4746-a7de-ed8610da845d\") " pod="openshift-ingress/router-default-6c88c45d57-c2bgn"
Oct 31 07:29:22.991931 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:22.991907    4720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/789f11d8-d951-4144-a589-a2bdbf6844e3-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-6k8mn\" (UID: \"789f11d8-d951-4144-a589-a2bdbf6844e3\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-6k8mn"
Oct 31 07:29:22.992044 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:22.991946    4720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a73024cb-f581-450d-a09c-0565838c5baa-metrics-tls\") pod \"dns-default-9cmfx\" (UID: \"a73024cb-f581-450d-a09c-0565838c5baa\") " pod="openshift-dns/dns-default-9cmfx"
Oct 31 07:29:22.992044 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:22.991985    4720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b1066ece-e6f3-4491-aa1f-a7b9a3aa248d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-8m768\" (UID: \"b1066ece-e6f3-4491-aa1f-a7b9a3aa248d\") " pod="openshift-insights/insights-runtime-extractor-8m768"
Oct 31 07:29:23.010513 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:23.010462    4720 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-6c88c45d57-c2bgn"
Oct 31 07:29:23.011528 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:23.011514    4720 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-6k8mn"
Oct 31 07:29:23.022710 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:23.022689    4720 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-9cmfx"
Oct 31 07:29:23.023813 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:23.023790    4720 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-8m768"
Oct 31 07:29:23.378915 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:23.378872    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 07:29:23.492821 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:23.492789    4720 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-6k8mn"]
Oct 31 07:29:23.528346 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:23.528305    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 07:29:23.536079 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:23.536054    4720 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda73024cb_f581_450d_a09c_0565838c5baa.slice/crio-85152796bddf118015a265f70d274a2fee7a3ac886793f4ac1cc322936a72539 WatchSource:0}: Error finding container 85152796bddf118015a265f70d274a2fee7a3ac886793f4ac1cc322936a72539: Status 404 returned error can't find the container with id 85152796bddf118015a265f70d274a2fee7a3ac886793f4ac1cc322936a72539
Oct 31 07:29:23.536764 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:23.536741    4720 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod789f11d8_d951_4144_a589_a2bdbf6844e3.slice/crio-64ff346eaef10adb0a22129609f90302da602f178fadbfddec8a2b3f39ab3b5c WatchSource:0}: Error finding container 64ff346eaef10adb0a22129609f90302da602f178fadbfddec8a2b3f39ab3b5c: Status 404 returned error can't find the container with id 64ff346eaef10adb0a22129609f90302da602f178fadbfddec8a2b3f39ab3b5c
Oct 31 07:29:23.542087 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:23.542060    4720 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-8m768"]
Oct 31 07:29:23.542708 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:23.542688    4720 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-6c88c45d57-c2bgn"]
Oct 31 07:29:23.543369 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:23.543350    4720 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb1066ece_e6f3_4491_aa1f_a7b9a3aa248d.slice/crio-aea4af7b4604dd056ba32a95050d5332b64eaa9cc887805a7a4865f5f9526c9c WatchSource:0}: Error finding container aea4af7b4604dd056ba32a95050d5332b64eaa9cc887805a7a4865f5f9526c9c: Status 404 returned error can't find the container with id aea4af7b4604dd056ba32a95050d5332b64eaa9cc887805a7a4865f5f9526c9c
Oct 31 07:29:23.543790 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:23.543747    4720 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-9cmfx"]
Oct 31 07:29:23.546852 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:23.546833    4720 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb44ed0e0_a72e_4746_a7de_ed8610da845d.slice/crio-9741f79adb6633b6b535f2ceb48ba38cc11897546c8893b2d554334b5d35b7f5 WatchSource:0}: Error finding container 9741f79adb6633b6b535f2ceb48ba38cc11897546c8893b2d554334b5d35b7f5: Status 404 returned error can't find the container with id 9741f79adb6633b6b535f2ceb48ba38cc11897546c8893b2d554334b5d35b7f5
Oct 31 07:29:23.820492 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:23.820412    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 07:29:23.987157 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:23.987112    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 07:29:24.170859 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:24.170831    4720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-8m768" event={"ID":"b1066ece-e6f3-4491-aa1f-a7b9a3aa248d","Type":"ContainerStarted","Data":"ff6be6dabdd8336af25d76fea9ead1b0deba6dd3c7fa01725f156c9a64aa158c"}
Oct 31 07:29:24.170998 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:24.170896    4720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-8m768" event={"ID":"b1066ece-e6f3-4491-aa1f-a7b9a3aa248d","Type":"ContainerStarted","Data":"aea4af7b4604dd056ba32a95050d5332b64eaa9cc887805a7a4865f5f9526c9c"}
Oct 31 07:29:24.172079 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:24.172051    4720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-6k8mn" event={"ID":"789f11d8-d951-4144-a589-a2bdbf6844e3","Type":"ContainerStarted","Data":"64ff346eaef10adb0a22129609f90302da602f178fadbfddec8a2b3f39ab3b5c"}
Oct 31 07:29:24.173502 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:24.173482    4720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-6c88c45d57-c2bgn" event={"ID":"b44ed0e0-a72e-4746-a7de-ed8610da845d","Type":"ContainerStarted","Data":"e41324e4c8f7b03b18d7e7d832c111de187efb77b0a8666458d0f4827c828b2a"}
Oct 31 07:29:24.173586 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:24.173523    4720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-6c88c45d57-c2bgn" event={"ID":"b44ed0e0-a72e-4746-a7de-ed8610da845d","Type":"ContainerStarted","Data":"9741f79adb6633b6b535f2ceb48ba38cc11897546c8893b2d554334b5d35b7f5"}
Oct 31 07:29:24.174545 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:24.174526    4720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-9cmfx" event={"ID":"a73024cb-f581-450d-a09c-0565838c5baa","Type":"ContainerStarted","Data":"85152796bddf118015a265f70d274a2fee7a3ac886793f4ac1cc322936a72539"}
Oct 31 07:29:24.221483 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:24.221450    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 07:29:24.392187 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:24.392160    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 07:29:25.011351 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:25.011248    4720 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-6c88c45d57-c2bgn"
Oct 31 07:29:25.014497 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:25.014472    4720 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-6c88c45d57-c2bgn"
Oct 31 07:29:25.182247 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:25.182200    4720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-8m768" event={"ID":"b1066ece-e6f3-4491-aa1f-a7b9a3aa248d","Type":"ContainerStarted","Data":"969c3e4776fbfb7dd37855715181f2a4ba3ca9c4aa171c8338f6b822aa659218"}
Oct 31 07:29:25.182476 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:25.182457    4720 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-6c88c45d57-c2bgn"
Oct 31 07:29:25.184027 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:25.184007    4720 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-6c88c45d57-c2bgn"
Oct 31 07:29:26.187529 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:26.187443    4720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-9cmfx" event={"ID":"a73024cb-f581-450d-a09c-0565838c5baa","Type":"ContainerStarted","Data":"bb8af1068f8a78ca5681ab348005942c2cf65d4896af3907982754b3a614d394"}
Oct 31 07:29:26.187529 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:26.187504    4720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-9cmfx" event={"ID":"a73024cb-f581-450d-a09c-0565838c5baa","Type":"ContainerStarted","Data":"a11c040031e4555a90443fadbab5cb7ca484fd3c3b9c68d30b2841abc39c3b28"}
Oct 31 07:29:26.188061 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:26.187532    4720 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-9cmfx"
Oct 31 07:29:26.189634 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:26.189609    4720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-6k8mn" event={"ID":"789f11d8-d951-4144-a589-a2bdbf6844e3","Type":"ContainerStarted","Data":"30401fd5a3bcd2f61085147644bdba7fc72b12c5862536f0a1a77983612aceb1"}
Oct 31 07:29:26.901261 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:26.901232    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 07:29:27.102164 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:27.102136    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 07:29:27.195687 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:27.195602    4720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-8m768" event={"ID":"b1066ece-e6f3-4491-aa1f-a7b9a3aa248d","Type":"ContainerStarted","Data":"01257189a6db5dfb0bc337e5bcde3f51f7843f32fd8f08ba6b7ff798e41a1623"}
Oct 31 07:29:27.291402 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:27.291373    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 07:29:27.918679 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:27.918652    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 07:29:28.078128 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:28.078089    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 07:29:32.060661 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:32.060630    4720 patch_prober.go:29] interesting pod/image-registry-7886c7d6c6-qdc9f container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Oct 31 07:29:32.061105 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:32.060677    4720 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-7886c7d6c6-qdc9f" podUID="993cfed6-806d-44c5-bba8-c02203638fe7" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 07:29:32.188220 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:32.188189    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 07:29:32.334245 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:32.334212    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 07:29:32.488091 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:32.488063    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 07:29:32.646712 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:32.646641    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 07:29:33.250241 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:33.250212    4720 patch_prober.go:29] interesting pod/image-registry-7955c67bcc-pkntv container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Oct 31 07:29:33.250654 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:33.250262    4720 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-7955c67bcc-pkntv" podUID="28b59d93-b9ce-4ceb-9f3a-742525eb94c4" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 07:29:36.197762 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:36.197734    4720 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-9cmfx"
Oct 31 07:29:36.297920 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:36.297893    4720 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-5p7xw"
Oct 31 07:29:37.488592 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:37.488560    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 07:29:38.093215 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:38.093179    4720 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-9zp4l"
Oct 31 07:29:38.147457 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:38.147434    4720 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-7955c67bcc-pkntv"
Oct 31 07:29:38.386225 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:38.386141    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 07:29:38.549432 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:38.549395    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 07:29:38.730268 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:38.730204    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 07:29:38.937131 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:38.937099    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 07:29:39.104200 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:39.104166    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 07:29:39.209267 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:39.209236    4720 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-p56z2"]
Oct 31 07:29:39.244929 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:39.244905    4720 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-p56z2"
Oct 31 07:29:39.272396 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:39.272366    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 07:29:39.306016 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:39.305981    4720 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Oct 31 07:29:39.309251 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:39.309228    4720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/bc1939c5-394d-4174-a903-7c23b89e113b-metrics-client-ca\") pod \"node-exporter-p56z2\" (UID: \"bc1939c5-394d-4174-a903-7c23b89e113b\") " pod="openshift-monitoring/node-exporter-p56z2"
Oct 31 07:29:39.309400 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:39.309263    4720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/bc1939c5-394d-4174-a903-7c23b89e113b-root\") pod \"node-exporter-p56z2\" (UID: \"bc1939c5-394d-4174-a903-7c23b89e113b\") " pod="openshift-monitoring/node-exporter-p56z2"
Oct 31 07:29:39.309400 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:39.309290    4720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/bc1939c5-394d-4174-a903-7c23b89e113b-node-exporter-wtmp\") pod \"node-exporter-p56z2\" (UID: \"bc1939c5-394d-4174-a903-7c23b89e113b\") " pod="openshift-monitoring/node-exporter-p56z2"
Oct 31 07:29:39.309400 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:39.309388    4720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/bc1939c5-394d-4174-a903-7c23b89e113b-node-exporter-textfile\") pod \"node-exporter-p56z2\" (UID: \"bc1939c5-394d-4174-a903-7c23b89e113b\") " pod="openshift-monitoring/node-exporter-p56z2"
Oct 31 07:29:39.309512 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:39.309429    4720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/bc1939c5-394d-4174-a903-7c23b89e113b-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-p56z2\" (UID: \"bc1939c5-394d-4174-a903-7c23b89e113b\") " pod="openshift-monitoring/node-exporter-p56z2"
Oct 31 07:29:39.309512 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:39.309452    4720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/bc1939c5-394d-4174-a903-7c23b89e113b-node-exporter-tls\") pod \"node-exporter-p56z2\" (UID: \"bc1939c5-394d-4174-a903-7c23b89e113b\") " pod="openshift-monitoring/node-exporter-p56z2"
Oct 31 07:29:39.309512 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:39.309480    4720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/bc1939c5-394d-4174-a903-7c23b89e113b-sys\") pod \"node-exporter-p56z2\" (UID: \"bc1939c5-394d-4174-a903-7c23b89e113b\") " pod="openshift-monitoring/node-exporter-p56z2"
Oct 31 07:29:39.309600 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:39.309538    4720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/bc1939c5-394d-4174-a903-7c23b89e113b-node-exporter-accelerators-collector-config\") pod \"node-exporter-p56z2\" (UID: \"bc1939c5-394d-4174-a903-7c23b89e113b\") " pod="openshift-monitoring/node-exporter-p56z2"
Oct 31 07:29:39.309641 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:39.309628    4720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-njw4l\" (UniqueName: \"kubernetes.io/projected/bc1939c5-394d-4174-a903-7c23b89e113b-kube-api-access-njw4l\") pod \"node-exporter-p56z2\" (UID: \"bc1939c5-394d-4174-a903-7c23b89e113b\") " pod="openshift-monitoring/node-exporter-p56z2"
Oct 31 07:29:39.346052 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:39.346023    4720 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Oct 31 07:29:39.346052 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:39.346049    4720 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-c7ffn\""
Oct 31 07:29:39.350626 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:39.350608    4720 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Oct 31 07:29:39.350732 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:39.350691    4720 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Oct 31 07:29:39.410892 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:39.410830    4720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/bc1939c5-394d-4174-a903-7c23b89e113b-node-exporter-wtmp\") pod \"node-exporter-p56z2\" (UID: \"bc1939c5-394d-4174-a903-7c23b89e113b\") " pod="openshift-monitoring/node-exporter-p56z2"
Oct 31 07:29:39.410892 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:39.410858    4720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/bc1939c5-394d-4174-a903-7c23b89e113b-node-exporter-textfile\") pod \"node-exporter-p56z2\" (UID: \"bc1939c5-394d-4174-a903-7c23b89e113b\") " pod="openshift-monitoring/node-exporter-p56z2"
Oct 31 07:29:39.410892 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:39.410885    4720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/bc1939c5-394d-4174-a903-7c23b89e113b-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-p56z2\" (UID: \"bc1939c5-394d-4174-a903-7c23b89e113b\") " pod="openshift-monitoring/node-exporter-p56z2"
Oct 31 07:29:39.411078 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:39.410998    4720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/bc1939c5-394d-4174-a903-7c23b89e113b-node-exporter-tls\") pod \"node-exporter-p56z2\" (UID: \"bc1939c5-394d-4174-a903-7c23b89e113b\") " pod="openshift-monitoring/node-exporter-p56z2"
Oct 31 07:29:39.411078 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:39.411010    4720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/bc1939c5-394d-4174-a903-7c23b89e113b-node-exporter-wtmp\") pod \"node-exporter-p56z2\" (UID: \"bc1939c5-394d-4174-a903-7c23b89e113b\") " pod="openshift-monitoring/node-exporter-p56z2"
Oct 31 07:29:39.411078 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:39.411052    4720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/bc1939c5-394d-4174-a903-7c23b89e113b-sys\") pod \"node-exporter-p56z2\" (UID: \"bc1939c5-394d-4174-a903-7c23b89e113b\") " pod="openshift-monitoring/node-exporter-p56z2"
Oct 31 07:29:39.411207 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:39.411084    4720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/bc1939c5-394d-4174-a903-7c23b89e113b-node-exporter-accelerators-collector-config\") pod \"node-exporter-p56z2\" (UID: \"bc1939c5-394d-4174-a903-7c23b89e113b\") " pod="openshift-monitoring/node-exporter-p56z2"
Oct 31 07:29:39.411207 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:39.411113    4720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-njw4l\" (UniqueName: \"kubernetes.io/projected/bc1939c5-394d-4174-a903-7c23b89e113b-kube-api-access-njw4l\") pod \"node-exporter-p56z2\" (UID: \"bc1939c5-394d-4174-a903-7c23b89e113b\") " pod="openshift-monitoring/node-exporter-p56z2"
Oct 31 07:29:39.411207 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:39.411128    4720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/bc1939c5-394d-4174-a903-7c23b89e113b-node-exporter-textfile\") pod \"node-exporter-p56z2\" (UID: \"bc1939c5-394d-4174-a903-7c23b89e113b\") " pod="openshift-monitoring/node-exporter-p56z2"
Oct 31 07:29:39.411207 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:39.411146    4720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/bc1939c5-394d-4174-a903-7c23b89e113b-metrics-client-ca\") pod \"node-exporter-p56z2\" (UID: \"bc1939c5-394d-4174-a903-7c23b89e113b\") " pod="openshift-monitoring/node-exporter-p56z2"
Oct 31 07:29:39.411207 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:39.411177    4720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/bc1939c5-394d-4174-a903-7c23b89e113b-root\") pod \"node-exporter-p56z2\" (UID: \"bc1939c5-394d-4174-a903-7c23b89e113b\") " pod="openshift-monitoring/node-exporter-p56z2"
Oct 31 07:29:39.411207 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:39.411189    4720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/bc1939c5-394d-4174-a903-7c23b89e113b-sys\") pod \"node-exporter-p56z2\" (UID: \"bc1939c5-394d-4174-a903-7c23b89e113b\") " pod="openshift-monitoring/node-exporter-p56z2"
Oct 31 07:29:39.411545 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:39.411250    4720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/bc1939c5-394d-4174-a903-7c23b89e113b-root\") pod \"node-exporter-p56z2\" (UID: \"bc1939c5-394d-4174-a903-7c23b89e113b\") " pod="openshift-monitoring/node-exporter-p56z2"
Oct 31 07:29:39.411655 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:39.411636    4720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/bc1939c5-394d-4174-a903-7c23b89e113b-node-exporter-accelerators-collector-config\") pod \"node-exporter-p56z2\" (UID: \"bc1939c5-394d-4174-a903-7c23b89e113b\") " pod="openshift-monitoring/node-exporter-p56z2"
Oct 31 07:29:39.411876 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:39.411637    4720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/bc1939c5-394d-4174-a903-7c23b89e113b-metrics-client-ca\") pod \"node-exporter-p56z2\" (UID: \"bc1939c5-394d-4174-a903-7c23b89e113b\") " pod="openshift-monitoring/node-exporter-p56z2"
Oct 31 07:29:39.413381 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:39.413361    4720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/bc1939c5-394d-4174-a903-7c23b89e113b-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-p56z2\" (UID: \"bc1939c5-394d-4174-a903-7c23b89e113b\") " pod="openshift-monitoring/node-exporter-p56z2"
Oct 31 07:29:39.413465 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:39.413449    4720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/bc1939c5-394d-4174-a903-7c23b89e113b-node-exporter-tls\") pod \"node-exporter-p56z2\" (UID: \"bc1939c5-394d-4174-a903-7c23b89e113b\") " pod="openshift-monitoring/node-exporter-p56z2"
Oct 31 07:29:39.439679 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:39.439652    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 07:29:39.467163 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:39.467137    4720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-njw4l\" (UniqueName: \"kubernetes.io/projected/bc1939c5-394d-4174-a903-7c23b89e113b-kube-api-access-njw4l\") pod \"node-exporter-p56z2\" (UID: \"bc1939c5-394d-4174-a903-7c23b89e113b\") " pod="openshift-monitoring/node-exporter-p56z2"
Oct 31 07:29:39.590907 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:39.590877    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 07:29:39.591864 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:39.591845    4720 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-p56z2"
Oct 31 07:29:39.621528 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:29:39.621499    4720 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbc1939c5_394d_4174_a903_7c23b89e113b.slice/crio-c7f4e05dd4bdf7d83d6da80d9082980e5515a8c83029fddbe897e495bbfa1091 WatchSource:0}: Error finding container c7f4e05dd4bdf7d83d6da80d9082980e5515a8c83029fddbe897e495bbfa1091: Status 404 returned error can't find the container with id c7f4e05dd4bdf7d83d6da80d9082980e5515a8c83029fddbe897e495bbfa1091
Oct 31 07:29:39.736621 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:39.736556    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 07:29:40.247781 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:40.247749    4720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-p56z2" event={"ID":"bc1939c5-394d-4174-a903-7c23b89e113b","Type":"ContainerStarted","Data":"c7f4e05dd4bdf7d83d6da80d9082980e5515a8c83029fddbe897e495bbfa1091"}
Oct 31 07:29:40.498269 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:40.498206    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 07:29:40.765153 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:40.765059    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 07:29:41.364187 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:41.364153    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 07:29:42.062674 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:42.062598    4720 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-7886c7d6c6-qdc9f"
Oct 31 07:29:42.063479 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:42.063444    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 07:29:42.225670 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:42.225627    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 07:29:42.257923 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:42.257889    4720 generic.go:356] "Generic (PLEG): container finished" podID="bc1939c5-394d-4174-a903-7c23b89e113b" containerID="523d0fe6b094e31c7276798df4554664d0a028b4262bfff6a47cf983f7708ba3" exitCode=0
Oct 31 07:29:42.258082 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:42.257970    4720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-p56z2" event={"ID":"bc1939c5-394d-4174-a903-7c23b89e113b","Type":"ContainerDied","Data":"523d0fe6b094e31c7276798df4554664d0a028b4262bfff6a47cf983f7708ba3"}
Oct 31 07:29:42.378282 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:42.378253    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 07:29:42.642410 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:42.642338    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 07:29:42.872077 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:42.872047    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 07:29:43.205390 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:43.205354    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 07:29:43.264319 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:43.264286    4720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-p56z2" event={"ID":"bc1939c5-394d-4174-a903-7c23b89e113b","Type":"ContainerStarted","Data":"ce98825cf878e14b8fc43d75132faf4a42aa3e5737134d14f72d8dca21a2c19a"}
Oct 31 07:29:43.264488 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:43.264355    4720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-p56z2" event={"ID":"bc1939c5-394d-4174-a903-7c23b89e113b","Type":"ContainerStarted","Data":"9e4229f5654932f52decb10e63c70404361fb12b59a6bb631ec229128b132f93"}
Oct 31 07:29:43.361294 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:43.361257    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 07:29:43.415292 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:43.415242    4720 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-p56z2" podStartSLOduration=2.514434188 podStartE2EDuration="4.415228571s" totalImagesPullingTime="1.80966039s" totalInitContainerRuntime="91.133993ms" isStatefulPod=true podCreationTimestamp="2026-10-31 07:29:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 07:29:43.379404424 +0000 UTC m=+37.510472496" watchObservedRunningTime="2026-10-31 07:29:43.415228571 +0000 UTC m=+37.546296641"
Oct 31 07:29:43.460193 ip-10-0-139-83 kubenswrapper[4720]: E1031 07:29:43.460118    4720 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8m768" containerName="extractor"
Oct 31 07:29:43.560638 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:43.560613    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 07:29:43.704497 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:43.704469    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 07:29:43.860906 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:43.860879    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 07:29:44.022751 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:44.022719    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 07:29:44.925753 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:44.925722    4720 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-7955c67bcc-pkntv"]
Oct 31 07:29:45.053544 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:45.053510    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 07:29:45.204346 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:45.204210    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 07:29:45.347703 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:45.347677    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 07:29:47.075513 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:47.075473    4720 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-image-registry/image-registry-7886c7d6c6-qdc9f" podUID="993cfed6-806d-44c5-bba8-c02203638fe7" containerName="registry" containerID="cri-o://c1afbd4a198d8efa93789314bcb177363aaf598ad88d64e78122fedef482f75c" gracePeriod=30
Oct 31 07:29:47.280622 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:47.280586    4720 generic.go:356] "Generic (PLEG): container finished" podID="993cfed6-806d-44c5-bba8-c02203638fe7" containerID="c1afbd4a198d8efa93789314bcb177363aaf598ad88d64e78122fedef482f75c" exitCode=0
Oct 31 07:29:47.280820 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:47.280667    4720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7886c7d6c6-qdc9f" event={"ID":"993cfed6-806d-44c5-bba8-c02203638fe7","Type":"ContainerDied","Data":"c1afbd4a198d8efa93789314bcb177363aaf598ad88d64e78122fedef482f75c"}
Oct 31 07:29:47.320378 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:47.320354    4720 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7886c7d6c6-qdc9f"
Oct 31 07:29:47.374009 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:47.373980    4720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/993cfed6-806d-44c5-bba8-c02203638fe7-registry-tls\" (UniqueName: \"kubernetes.io/projected/993cfed6-806d-44c5-bba8-c02203638fe7-registry-tls\") pod \"993cfed6-806d-44c5-bba8-c02203638fe7\" (UID: \"993cfed6-806d-44c5-bba8-c02203638fe7\") "
Oct 31 07:29:47.374224 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:47.374022    4720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/993cfed6-806d-44c5-bba8-c02203638fe7-registry-certificates\" (UniqueName: \"kubernetes.io/configmap/993cfed6-806d-44c5-bba8-c02203638fe7-registry-certificates\") pod \"993cfed6-806d-44c5-bba8-c02203638fe7\" (UID: \"993cfed6-806d-44c5-bba8-c02203638fe7\") "
Oct 31 07:29:47.374224 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:47.374097    4720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/993cfed6-806d-44c5-bba8-c02203638fe7-bound-sa-token\" (UniqueName: \"kubernetes.io/projected/993cfed6-806d-44c5-bba8-c02203638fe7-bound-sa-token\") pod \"993cfed6-806d-44c5-bba8-c02203638fe7\" (UID: \"993cfed6-806d-44c5-bba8-c02203638fe7\") "
Oct 31 07:29:47.374224 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:47.374131    4720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/993cfed6-806d-44c5-bba8-c02203638fe7-ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/993cfed6-806d-44c5-bba8-c02203638fe7-ca-trust-extracted\") pod \"993cfed6-806d-44c5-bba8-c02203638fe7\" (UID: \"993cfed6-806d-44c5-bba8-c02203638fe7\") "
Oct 31 07:29:47.374224 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:47.374174    4720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/993cfed6-806d-44c5-bba8-c02203638fe7-image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/993cfed6-806d-44c5-bba8-c02203638fe7-image-registry-private-configuration\") pod \"993cfed6-806d-44c5-bba8-c02203638fe7\" (UID: \"993cfed6-806d-44c5-bba8-c02203638fe7\") "
Oct 31 07:29:47.374224 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:47.374215    4720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/993cfed6-806d-44c5-bba8-c02203638fe7-trusted-ca\" (UniqueName: \"kubernetes.io/configmap/993cfed6-806d-44c5-bba8-c02203638fe7-trusted-ca\") pod \"993cfed6-806d-44c5-bba8-c02203638fe7\" (UID: \"993cfed6-806d-44c5-bba8-c02203638fe7\") "
Oct 31 07:29:47.374499 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:47.374257    4720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/993cfed6-806d-44c5-bba8-c02203638fe7-installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/993cfed6-806d-44c5-bba8-c02203638fe7-installation-pull-secrets\") pod \"993cfed6-806d-44c5-bba8-c02203638fe7\" (UID: \"993cfed6-806d-44c5-bba8-c02203638fe7\") "
Oct 31 07:29:47.374499 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:47.374289    4720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/993cfed6-806d-44c5-bba8-c02203638fe7-kube-api-access-v2zhk\" (UniqueName: \"kubernetes.io/projected/993cfed6-806d-44c5-bba8-c02203638fe7-kube-api-access-v2zhk\") pod \"993cfed6-806d-44c5-bba8-c02203638fe7\" (UID: \"993cfed6-806d-44c5-bba8-c02203638fe7\") "
Oct 31 07:29:47.374603 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:47.374583    4720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/993cfed6-806d-44c5-bba8-c02203638fe7-registry-certificates" pod "993cfed6-806d-44c5-bba8-c02203638fe7" (UID: "993cfed6-806d-44c5-bba8-c02203638fe7"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 07:29:47.375032 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:47.374999    4720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/993cfed6-806d-44c5-bba8-c02203638fe7-trusted-ca" pod "993cfed6-806d-44c5-bba8-c02203638fe7" (UID: "993cfed6-806d-44c5-bba8-c02203638fe7"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 07:29:47.376788 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:47.376760    4720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/993cfed6-806d-44c5-bba8-c02203638fe7-bound-sa-token" pod "993cfed6-806d-44c5-bba8-c02203638fe7" (UID: "993cfed6-806d-44c5-bba8-c02203638fe7"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 07:29:47.376989 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:47.376954    4720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/993cfed6-806d-44c5-bba8-c02203638fe7-registry-tls" pod "993cfed6-806d-44c5-bba8-c02203638fe7" (UID: "993cfed6-806d-44c5-bba8-c02203638fe7"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 07:29:47.376989 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:47.376966    4720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/993cfed6-806d-44c5-bba8-c02203638fe7-kube-api-access-v2zhk" pod "993cfed6-806d-44c5-bba8-c02203638fe7" (UID: "993cfed6-806d-44c5-bba8-c02203638fe7"). InnerVolumeSpecName "kube-api-access-v2zhk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 07:29:47.377156 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:47.377042    4720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/993cfed6-806d-44c5-bba8-c02203638fe7-installation-pull-secrets" pod "993cfed6-806d-44c5-bba8-c02203638fe7" (UID: "993cfed6-806d-44c5-bba8-c02203638fe7"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 07:29:47.377292 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:47.377271    4720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/993cfed6-806d-44c5-bba8-c02203638fe7-image-registry-private-configuration" pod "993cfed6-806d-44c5-bba8-c02203638fe7" (UID: "993cfed6-806d-44c5-bba8-c02203638fe7"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 07:29:47.381607 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:47.381578    4720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/993cfed6-806d-44c5-bba8-c02203638fe7-ca-trust-extracted" pod "993cfed6-806d-44c5-bba8-c02203638fe7" (UID: "993cfed6-806d-44c5-bba8-c02203638fe7"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 07:29:47.475715 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:47.475685    4720 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/993cfed6-806d-44c5-bba8-c02203638fe7-bound-sa-token\") on node \"ip-10-0-139-83.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 07:29:47.475715 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:47.475710    4720 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/993cfed6-806d-44c5-bba8-c02203638fe7-ca-trust-extracted\") on node \"ip-10-0-139-83.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 07:29:47.475909 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:47.475733    4720 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/993cfed6-806d-44c5-bba8-c02203638fe7-image-registry-private-configuration\") on node \"ip-10-0-139-83.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 07:29:47.475909 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:47.475742    4720 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/993cfed6-806d-44c5-bba8-c02203638fe7-trusted-ca\") on node \"ip-10-0-139-83.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 07:29:47.475909 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:47.475751    4720 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/993cfed6-806d-44c5-bba8-c02203638fe7-installation-pull-secrets\") on node \"ip-10-0-139-83.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 07:29:47.475909 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:47.475762    4720 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v2zhk\" (UniqueName: \"kubernetes.io/projected/993cfed6-806d-44c5-bba8-c02203638fe7-kube-api-access-v2zhk\") on node \"ip-10-0-139-83.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 07:29:47.475909 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:47.475771    4720 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/993cfed6-806d-44c5-bba8-c02203638fe7-registry-tls\") on node \"ip-10-0-139-83.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 07:29:47.475909 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:47.475781    4720 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/993cfed6-806d-44c5-bba8-c02203638fe7-registry-certificates\") on node \"ip-10-0-139-83.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 07:29:48.163766 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:48.163732    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 07:29:48.285991 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:48.285920    4720 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7886c7d6c6-qdc9f"
Oct 31 07:29:48.285991 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:48.285935    4720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7886c7d6c6-qdc9f" event={"ID":"993cfed6-806d-44c5-bba8-c02203638fe7","Type":"ContainerDied","Data":"40db303d05df9e64756d1fe275ba37eb942abd352ec78fce0a9b59037bfae450"}
Oct 31 07:29:48.285991 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:48.285985    4720 scope.go:122] "RemoveContainer" containerID="c1afbd4a198d8efa93789314bcb177363aaf598ad88d64e78122fedef482f75c"
Oct 31 07:29:48.495987 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:48.495959    4720 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-7886c7d6c6-qdc9f"]
Oct 31 07:29:48.545101 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:48.545045    4720 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-7886c7d6c6-qdc9f"]
Oct 31 07:29:50.024614 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:50.024581    4720 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="993cfed6-806d-44c5-bba8-c02203638fe7" path="/var/lib/kubelet/pods/993cfed6-806d-44c5-bba8-c02203638fe7/volumes"
Oct 31 07:29:51.028867 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:51.028833    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 07:29:51.194066 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:51.194036    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 07:29:53.387678 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:53.387648    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 07:29:53.554182 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:53.554146    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 07:29:55.242152 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:55.242121    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 07:29:55.401716 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:55.401690    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 07:29:56.313664 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:56.313623    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 07:29:56.500608 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:56.500571    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 07:29:58.028884 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:58.028849    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 07:29:58.194878 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:29:58.194850    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 07:30:00.320083 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:30:00.320047    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 07:30:00.469125 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:30:00.469092    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 07:30:02.023021 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:30:02.022958    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 07:30:02.167537 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:30:02.167492    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 07:30:04.623070 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:30:04.623037    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 07:30:04.775049 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:30:04.775023    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 07:30:05.433837 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:30:05.433803    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 07:30:05.612993 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:30:05.612960    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 07:30:05.827589 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:30:05.827504    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 07:30:05.974593 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:30:05.974559    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 07:30:08.541122 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:30:08.541075    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 07:30:08.692872 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:30:08.692840    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 07:30:09.946198 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:30:09.946145    4720 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-image-registry/image-registry-7955c67bcc-pkntv" podUID="28b59d93-b9ce-4ceb-9f3a-742525eb94c4" containerName="registry" containerID="cri-o://9dfc89ae8f837cd9cd34ff414c2ea73f9645696fad30d9e8618c22942bbb61db" gracePeriod=30
Oct 31 07:30:10.179156 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:30:10.179133    4720 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7955c67bcc-pkntv"
Oct 31 07:30:10.347773 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:30:10.347748    4720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/28b59d93-b9ce-4ceb-9f3a-742525eb94c4-installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/28b59d93-b9ce-4ceb-9f3a-742525eb94c4-installation-pull-secrets\") pod \"28b59d93-b9ce-4ceb-9f3a-742525eb94c4\" (UID: \"28b59d93-b9ce-4ceb-9f3a-742525eb94c4\") "
Oct 31 07:30:10.347915 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:30:10.347790    4720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/28b59d93-b9ce-4ceb-9f3a-742525eb94c4-registry-tls\" (UniqueName: \"kubernetes.io/projected/28b59d93-b9ce-4ceb-9f3a-742525eb94c4-registry-tls\") pod \"28b59d93-b9ce-4ceb-9f3a-742525eb94c4\" (UID: \"28b59d93-b9ce-4ceb-9f3a-742525eb94c4\") "
Oct 31 07:30:10.347915 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:30:10.347811    4720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/28b59d93-b9ce-4ceb-9f3a-742525eb94c4-bound-sa-token\" (UniqueName: \"kubernetes.io/projected/28b59d93-b9ce-4ceb-9f3a-742525eb94c4-bound-sa-token\") pod \"28b59d93-b9ce-4ceb-9f3a-742525eb94c4\" (UID: \"28b59d93-b9ce-4ceb-9f3a-742525eb94c4\") "
Oct 31 07:30:10.347915 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:30:10.347831    4720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/28b59d93-b9ce-4ceb-9f3a-742525eb94c4-image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/28b59d93-b9ce-4ceb-9f3a-742525eb94c4-image-registry-private-configuration\") pod \"28b59d93-b9ce-4ceb-9f3a-742525eb94c4\" (UID: \"28b59d93-b9ce-4ceb-9f3a-742525eb94c4\") "
Oct 31 07:30:10.347915 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:30:10.347870    4720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/28b59d93-b9ce-4ceb-9f3a-742525eb94c4-ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/28b59d93-b9ce-4ceb-9f3a-742525eb94c4-ca-trust-extracted\") pod \"28b59d93-b9ce-4ceb-9f3a-742525eb94c4\" (UID: \"28b59d93-b9ce-4ceb-9f3a-742525eb94c4\") "
Oct 31 07:30:10.347915 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:30:10.347894    4720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/28b59d93-b9ce-4ceb-9f3a-742525eb94c4-kube-api-access-xvv26\" (UniqueName: \"kubernetes.io/projected/28b59d93-b9ce-4ceb-9f3a-742525eb94c4-kube-api-access-xvv26\") pod \"28b59d93-b9ce-4ceb-9f3a-742525eb94c4\" (UID: \"28b59d93-b9ce-4ceb-9f3a-742525eb94c4\") "
Oct 31 07:30:10.347915 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:30:10.347914    4720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/28b59d93-b9ce-4ceb-9f3a-742525eb94c4-registry-certificates\" (UniqueName: \"kubernetes.io/configmap/28b59d93-b9ce-4ceb-9f3a-742525eb94c4-registry-certificates\") pod \"28b59d93-b9ce-4ceb-9f3a-742525eb94c4\" (UID: \"28b59d93-b9ce-4ceb-9f3a-742525eb94c4\") "
Oct 31 07:30:10.348224 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:30:10.347980    4720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/28b59d93-b9ce-4ceb-9f3a-742525eb94c4-trusted-ca\" (UniqueName: \"kubernetes.io/configmap/28b59d93-b9ce-4ceb-9f3a-742525eb94c4-trusted-ca\") pod \"28b59d93-b9ce-4ceb-9f3a-742525eb94c4\" (UID: \"28b59d93-b9ce-4ceb-9f3a-742525eb94c4\") "
Oct 31 07:30:10.348438 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:30:10.348405    4720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/28b59d93-b9ce-4ceb-9f3a-742525eb94c4-registry-certificates" pod "28b59d93-b9ce-4ceb-9f3a-742525eb94c4" (UID: "28b59d93-b9ce-4ceb-9f3a-742525eb94c4"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 07:30:10.348872 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:30:10.348843    4720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/28b59d93-b9ce-4ceb-9f3a-742525eb94c4-trusted-ca" pod "28b59d93-b9ce-4ceb-9f3a-742525eb94c4" (UID: "28b59d93-b9ce-4ceb-9f3a-742525eb94c4"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 07:30:10.350748 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:30:10.350720    4720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/28b59d93-b9ce-4ceb-9f3a-742525eb94c4-registry-tls" pod "28b59d93-b9ce-4ceb-9f3a-742525eb94c4" (UID: "28b59d93-b9ce-4ceb-9f3a-742525eb94c4"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 07:30:10.350910 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:30:10.350854    4720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/28b59d93-b9ce-4ceb-9f3a-742525eb94c4-image-registry-private-configuration" pod "28b59d93-b9ce-4ceb-9f3a-742525eb94c4" (UID: "28b59d93-b9ce-4ceb-9f3a-742525eb94c4"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 07:30:10.350910 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:30:10.350897    4720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/28b59d93-b9ce-4ceb-9f3a-742525eb94c4-bound-sa-token" pod "28b59d93-b9ce-4ceb-9f3a-742525eb94c4" (UID: "28b59d93-b9ce-4ceb-9f3a-742525eb94c4"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 07:30:10.351035 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:30:10.350950    4720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/28b59d93-b9ce-4ceb-9f3a-742525eb94c4-kube-api-access-xvv26" pod "28b59d93-b9ce-4ceb-9f3a-742525eb94c4" (UID: "28b59d93-b9ce-4ceb-9f3a-742525eb94c4"). InnerVolumeSpecName "kube-api-access-xvv26". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 07:30:10.351080 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:30:10.351022    4720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/28b59d93-b9ce-4ceb-9f3a-742525eb94c4-installation-pull-secrets" pod "28b59d93-b9ce-4ceb-9f3a-742525eb94c4" (UID: "28b59d93-b9ce-4ceb-9f3a-742525eb94c4"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 07:30:10.355462 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:30:10.355442    4720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/28b59d93-b9ce-4ceb-9f3a-742525eb94c4-ca-trust-extracted" pod "28b59d93-b9ce-4ceb-9f3a-742525eb94c4" (UID: "28b59d93-b9ce-4ceb-9f3a-742525eb94c4"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 07:30:10.371664 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:30:10.371644    4720 generic.go:356] "Generic (PLEG): container finished" podID="28b59d93-b9ce-4ceb-9f3a-742525eb94c4" containerID="9dfc89ae8f837cd9cd34ff414c2ea73f9645696fad30d9e8618c22942bbb61db" exitCode=0
Oct 31 07:30:10.371741 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:30:10.371676    4720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7955c67bcc-pkntv" event={"ID":"28b59d93-b9ce-4ceb-9f3a-742525eb94c4","Type":"ContainerDied","Data":"9dfc89ae8f837cd9cd34ff414c2ea73f9645696fad30d9e8618c22942bbb61db"}
Oct 31 07:30:10.371741 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:30:10.371688    4720 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7955c67bcc-pkntv"
Oct 31 07:30:10.371741 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:30:10.371709    4720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7955c67bcc-pkntv" event={"ID":"28b59d93-b9ce-4ceb-9f3a-742525eb94c4","Type":"ContainerDied","Data":"629d17fd8d940802e6ed7c5e94e98da44d1d2b16184f84afd22fdf51628c21b4"}
Oct 31 07:30:10.371741 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:30:10.371727    4720 scope.go:122] "RemoveContainer" containerID="9dfc89ae8f837cd9cd34ff414c2ea73f9645696fad30d9e8618c22942bbb61db"
Oct 31 07:30:10.379686 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:30:10.379671    4720 scope.go:122] "RemoveContainer" containerID="9dfc89ae8f837cd9cd34ff414c2ea73f9645696fad30d9e8618c22942bbb61db"
Oct 31 07:30:10.379937 ip-10-0-139-83 kubenswrapper[4720]: E1031 07:30:10.379918    4720 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9dfc89ae8f837cd9cd34ff414c2ea73f9645696fad30d9e8618c22942bbb61db\": container with ID starting with 9dfc89ae8f837cd9cd34ff414c2ea73f9645696fad30d9e8618c22942bbb61db not found: ID does not exist" containerID="9dfc89ae8f837cd9cd34ff414c2ea73f9645696fad30d9e8618c22942bbb61db"
Oct 31 07:30:10.379986 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:30:10.379944    4720 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9dfc89ae8f837cd9cd34ff414c2ea73f9645696fad30d9e8618c22942bbb61db"} err="failed to get container status \"9dfc89ae8f837cd9cd34ff414c2ea73f9645696fad30d9e8618c22942bbb61db\": rpc error: code = NotFound desc = could not find container \"9dfc89ae8f837cd9cd34ff414c2ea73f9645696fad30d9e8618c22942bbb61db\": container with ID starting with 9dfc89ae8f837cd9cd34ff414c2ea73f9645696fad30d9e8618c22942bbb61db not found: ID does not exist"
Oct 31 07:30:10.449470 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:30:10.449447    4720 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/28b59d93-b9ce-4ceb-9f3a-742525eb94c4-ca-trust-extracted\") on node \"ip-10-0-139-83.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 07:30:10.449470 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:30:10.449466    4720 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xvv26\" (UniqueName: \"kubernetes.io/projected/28b59d93-b9ce-4ceb-9f3a-742525eb94c4-kube-api-access-xvv26\") on node \"ip-10-0-139-83.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 07:30:10.449617 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:30:10.449478    4720 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/28b59d93-b9ce-4ceb-9f3a-742525eb94c4-registry-certificates\") on node \"ip-10-0-139-83.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 07:30:10.449617 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:30:10.449487    4720 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/28b59d93-b9ce-4ceb-9f3a-742525eb94c4-trusted-ca\") on node \"ip-10-0-139-83.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 07:30:10.449617 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:30:10.449496    4720 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/28b59d93-b9ce-4ceb-9f3a-742525eb94c4-installation-pull-secrets\") on node \"ip-10-0-139-83.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 07:30:10.449617 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:30:10.449504    4720 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/28b59d93-b9ce-4ceb-9f3a-742525eb94c4-registry-tls\") on node \"ip-10-0-139-83.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 07:30:10.449617 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:30:10.449512    4720 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/28b59d93-b9ce-4ceb-9f3a-742525eb94c4-bound-sa-token\") on node \"ip-10-0-139-83.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 07:30:10.449617 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:30:10.449521    4720 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/28b59d93-b9ce-4ceb-9f3a-742525eb94c4-image-registry-private-configuration\") on node \"ip-10-0-139-83.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 07:30:10.532137 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:30:10.532085    4720 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-7955c67bcc-pkntv"]
Oct 31 07:30:10.535658 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:30:10.535634    4720 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-7955c67bcc-pkntv"]
Oct 31 07:30:12.023987 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:30:12.023955    4720 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="28b59d93-b9ce-4ceb-9f3a-742525eb94c4" path="/var/lib/kubelet/pods/28b59d93-b9ce-4ceb-9f3a-742525eb94c4/volumes"
Oct 31 07:30:13.387717 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:30:13.387684    4720 generic.go:356] "Generic (PLEG): container finished" podID="cb9e4575-9265-43b5-bf02-7843611d6563" containerID="e25341347c884d204f4de3bdc1eb8e6f14c7fb76f88609e30c77e2007599130e" exitCode=0
Oct 31 07:30:13.388081 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:30:13.387759    4720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-p9z4r" event={"ID":"cb9e4575-9265-43b5-bf02-7843611d6563","Type":"ContainerDied","Data":"e25341347c884d204f4de3bdc1eb8e6f14c7fb76f88609e30c77e2007599130e"}
Oct 31 07:30:13.388081 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:30:13.388027    4720 scope.go:122] "RemoveContainer" containerID="e25341347c884d204f4de3bdc1eb8e6f14c7fb76f88609e30c77e2007599130e"
Oct 31 07:30:13.463949 ip-10-0-139-83 kubenswrapper[4720]: E1031 07:30:13.463910    4720 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8m768" containerName="extractor"
Oct 31 07:30:14.392952 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:30:14.392924    4720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-p9z4r" event={"ID":"cb9e4575-9265-43b5-bf02-7843611d6563","Type":"ContainerStarted","Data":"579e60277babf7d8bab4702a228ec2f1447d8b26f46fe959c9f506b6a462f99d"}
Oct 31 07:30:17.143030 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:30:17.142994    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 07:30:17.287496 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:30:17.287462    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 07:30:17.489020 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:30:17.488993    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 07:30:17.642652 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:30:17.642607    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 07:30:18.408090 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:30:18.408056    4720 generic.go:356] "Generic (PLEG): container finished" podID="4db4c33f-0c58-40f8-bf74-51b3eae3c9db" containerID="acc53d1d3f53d66239c0908e4ff0412ae73dbf42bd87be51c0d44b406a269631" exitCode=0
Oct 31 07:30:18.408481 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:30:18.408130    4720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-hb49h" event={"ID":"4db4c33f-0c58-40f8-bf74-51b3eae3c9db","Type":"ContainerDied","Data":"acc53d1d3f53d66239c0908e4ff0412ae73dbf42bd87be51c0d44b406a269631"}
Oct 31 07:30:18.408481 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:30:18.408433    4720 scope.go:122] "RemoveContainer" containerID="acc53d1d3f53d66239c0908e4ff0412ae73dbf42bd87be51c0d44b406a269631"
Oct 31 07:30:19.230342 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:30:19.230290    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 07:30:19.388057 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:30:19.388024    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 07:30:19.413533 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:30:19.413508    4720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-hb49h" event={"ID":"4db4c33f-0c58-40f8-bf74-51b3eae3c9db","Type":"ContainerStarted","Data":"39f39897f8f0af6125d1f9cf27d37185d3c0f74ab12e858e911c124af28ec986"}
Oct 31 07:30:27.441163 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:30:27.441131    4720 generic.go:356] "Generic (PLEG): container finished" podID="81829763-27b1-4a48-a568-cd5f6a39a953" containerID="44664141967c1ff5700a167e429d15365939c9433c32292fcd3583ff3e207b08" exitCode=0
Oct 31 07:30:27.441545 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:30:27.441167    4720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-dwc4p" event={"ID":"81829763-27b1-4a48-a568-cd5f6a39a953","Type":"ContainerDied","Data":"44664141967c1ff5700a167e429d15365939c9433c32292fcd3583ff3e207b08"}
Oct 31 07:30:27.441545 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:30:27.441463    4720 scope.go:122] "RemoveContainer" containerID="44664141967c1ff5700a167e429d15365939c9433c32292fcd3583ff3e207b08"
Oct 31 07:30:28.446503 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:30:28.446473    4720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-dwc4p" event={"ID":"81829763-27b1-4a48-a568-cd5f6a39a953","Type":"ContainerStarted","Data":"3d02d80b063b053a20b3b519c8467c4e2199ef9484b363753aae5a81ea24aba0"}
Oct 31 07:30:43.460374 ip-10-0-139-83 kubenswrapper[4720]: E1031 07:30:43.460321    4720 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8m768" containerName="extractor"
Oct 31 07:30:56.649640 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:30:56.649592    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 07:30:57.846024 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:30:57.845991    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 07:30:57.989687 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:30:57.989655    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 07:31:01.313579 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:31:01.313547    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 07:31:10.376625 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:31:10.376592    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 07:31:10.520897 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:31:10.520847    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 07:31:13.461010 ip-10-0-139-83 kubenswrapper[4720]: E1031 07:31:13.460976    4720 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8m768" containerName="extractor"
Oct 31 07:31:20.243243 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:31:20.243162    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 07:31:20.386774 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:31:20.386745    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 07:31:20.623361 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:31:20.623304    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 07:31:23.383174 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:31:23.383135    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 07:31:23.535105 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:31:23.535065    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 07:31:43.460997 ip-10-0-139-83 kubenswrapper[4720]: E1031 07:31:43.460959    4720 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8m768" containerName="extractor"
Oct 31 07:31:57.011899 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:31:57.011862    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 07:32:03.944401 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:32:03.944364    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 07:32:11.654591 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:32:11.654546    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 07:32:13.460475 ip-10-0-139-83 kubenswrapper[4720]: E1031 07:32:13.460440    4720 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8m768" containerName="extractor"
Oct 31 07:32:43.460490 ip-10-0-139-83 kubenswrapper[4720]: E1031 07:32:43.460455    4720 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8m768" containerName="extractor"
Oct 31 07:32:57.342892 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:32:57.342807    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 07:33:01.988721 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:33:01.988690    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 07:33:11.987987 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:33:11.987941    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 07:33:13.460249 ip-10-0-139-83 kubenswrapper[4720]: E1031 07:33:13.460214    4720 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8m768" containerName="extractor"
Oct 31 07:33:30.769526 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:33:30.769474    4720 prober.go:122] "Probe failed" probeType="Liveness" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-83.us-west-1.compute.internal" podUID="2183ffc19200ec816330ede9883f3bf6" containerName="haproxy" probeResult="failure" output="Get \"https://172.20.0.1:6443/version\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 07:33:41.258394 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:33:41.258313    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 07:33:42.711570 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:33:42.711539    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 07:33:43.460979 ip-10-0-139-83 kubenswrapper[4720]: E1031 07:33:43.460946    4720 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8m768" containerName="extractor"
Oct 31 07:33:51.432322 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:33:51.432291    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 07:33:57.655249 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:33:57.655216    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 07:34:02.334076 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:34:02.334042    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 07:34:06.031905 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:34:06.031878    4720 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-4jszp_b1db0e3b-e85d-4769-8012-cf4b6f740d9d/console-operator/2.log"
Oct 31 07:34:06.032354 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:34:06.032319    4720 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-4jszp_b1db0e3b-e85d-4769-8012-cf4b6f740d9d/console-operator/2.log"
Oct 31 07:34:12.372343 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:34:12.372300    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 07:34:13.459560 ip-10-0-139-83 kubenswrapper[4720]: E1031 07:34:13.459527    4720 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8m768" containerName="extractor"
Oct 31 07:34:18.664649 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:34:18.664573    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 07:34:18.933806 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:34:18.933716    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 07:34:25.936079 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:34:25.936047    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 07:34:27.683622 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:34:27.683588    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 07:34:27.951872 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:34:27.951788    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 07:34:29.291062 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:34:29.291031    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 07:34:29.670320 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:34:29.670282    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 07:34:29.816887 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:34:29.816843    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 07:34:30.011754 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:34:30.011665    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 07:34:30.183602 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:34:30.183567    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 07:34:32.277978 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:34:32.277919    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 07:34:33.274775 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:34:33.274739    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 07:34:33.774696 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:34:33.774657    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 07:34:34.036389 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:34:34.036282    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 07:34:43.460664 ip-10-0-139-83 kubenswrapper[4720]: E1031 07:34:43.460622    4720 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8m768" containerName="extractor"
Oct 31 07:34:55.320561 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:34:55.320521    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 07:35:02.668497 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:35:02.668452    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 07:35:07.997263 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:35:07.997229    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 07:35:12.658134 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:35:12.658100    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 07:35:13.460634 ip-10-0-139-83 kubenswrapper[4720]: E1031 07:35:13.460600    4720 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8m768" containerName="extractor"
Oct 31 07:35:18.637261 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:35:18.637217    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 07:35:18.794833 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:35:18.794803    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 07:35:20.960421 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:35:20.960387    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 07:35:43.459615 ip-10-0-139-83 kubenswrapper[4720]: E1031 07:35:43.459577    4720 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8m768" containerName="extractor"
Oct 31 07:35:54.093187 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:35:54.093079    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 07:35:54.329261 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:35:54.329192    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 07:35:58.271306 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:35:58.271256    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 07:35:59.610169 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:35:59.610113    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 07:35:59.766183 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:35:59.766152    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 07:36:00.011095 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:36:00.011017    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 07:36:02.553439 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:36:02.553403    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 07:36:02.708766 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:36:02.708737    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 07:36:02.868993 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:36:02.868962    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 07:36:03.025007 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:36:03.024979    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 07:36:03.631738 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:36:03.631703    4720 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-9tnqq"]
Oct 31 07:36:03.632086 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:36:03.631998    4720 cpu_manager.go:414] "containerMap: removing container" podUID="993cfed6-806d-44c5-bba8-c02203638fe7" containerName="registry"
Oct 31 07:36:03.632086 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:36:03.632008    4720 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="993cfed6-806d-44c5-bba8-c02203638fe7" containerName="registry"
Oct 31 07:36:03.632086 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:36:03.632030    4720 cpu_manager.go:414] "containerMap: removing container" podUID="28b59d93-b9ce-4ceb-9f3a-742525eb94c4" containerName="registry"
Oct 31 07:36:03.632086 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:36:03.632034    4720 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="28b59d93-b9ce-4ceb-9f3a-742525eb94c4" containerName="registry"
Oct 31 07:36:03.632086 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:36:03.632079    4720 memory_manager.go:365] "RemoveStaleState removing state" podUID="993cfed6-806d-44c5-bba8-c02203638fe7" containerName="registry"
Oct 31 07:36:03.632086 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:36:03.632089    4720 memory_manager.go:365] "RemoveStaleState removing state" podUID="28b59d93-b9ce-4ceb-9f3a-742525eb94c4" containerName="registry"
Oct 31 07:36:03.634874 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:36:03.634859    4720 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-9tnqq"
Oct 31 07:36:03.735959 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:36:03.735930    4720 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\""
Oct 31 07:36:03.735959 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:36:03.735947    4720 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\""
Oct 31 07:36:03.736148 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:36:03.736005    4720 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-gcj9k\""
Oct 31 07:36:03.736977 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:36:03.736956    4720 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-9tnqq"]
Oct 31 07:36:03.767454 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:36:03.767420    4720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zmclz\" (UniqueName: \"kubernetes.io/projected/14b80f1a-6d77-43e5-8e5c-a16c806d543d-kube-api-access-zmclz\") pod \"cert-manager-webhook-5dbbbf75cc-9tnqq\" (UID: \"14b80f1a-6d77-43e5-8e5c-a16c806d543d\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-9tnqq"
Oct 31 07:36:03.767454 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:36:03.767457    4720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/14b80f1a-6d77-43e5-8e5c-a16c806d543d-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-9tnqq\" (UID: \"14b80f1a-6d77-43e5-8e5c-a16c806d543d\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-9tnqq"
Oct 31 07:36:03.868643 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:36:03.868613    4720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zmclz\" (UniqueName: \"kubernetes.io/projected/14b80f1a-6d77-43e5-8e5c-a16c806d543d-kube-api-access-zmclz\") pod \"cert-manager-webhook-5dbbbf75cc-9tnqq\" (UID: \"14b80f1a-6d77-43e5-8e5c-a16c806d543d\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-9tnqq"
Oct 31 07:36:03.868775 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:36:03.868648    4720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/14b80f1a-6d77-43e5-8e5c-a16c806d543d-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-9tnqq\" (UID: \"14b80f1a-6d77-43e5-8e5c-a16c806d543d\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-9tnqq"
Oct 31 07:36:03.924817 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:36:03.924755    4720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/14b80f1a-6d77-43e5-8e5c-a16c806d543d-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-9tnqq\" (UID: \"14b80f1a-6d77-43e5-8e5c-a16c806d543d\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-9tnqq"
Oct 31 07:36:03.925091 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:36:03.925066    4720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zmclz\" (UniqueName: \"kubernetes.io/projected/14b80f1a-6d77-43e5-8e5c-a16c806d543d-kube-api-access-zmclz\") pod \"cert-manager-webhook-5dbbbf75cc-9tnqq\" (UID: \"14b80f1a-6d77-43e5-8e5c-a16c806d543d\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-9tnqq"
Oct 31 07:36:03.953924 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:36:03.953900    4720 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-9tnqq"
Oct 31 07:36:04.342547 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:36:04.342514    4720 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-9tnqq"]
Oct 31 07:36:04.346359 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:36:04.346314    4720 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod14b80f1a_6d77_43e5_8e5c_a16c806d543d.slice/crio-9eba8b59a436f41d565e9b1a2d1e0c5c40fee7fd3d185acd5dbce218b524285c WatchSource:0}: Error finding container 9eba8b59a436f41d565e9b1a2d1e0c5c40fee7fd3d185acd5dbce218b524285c: Status 404 returned error can't find the container with id 9eba8b59a436f41d565e9b1a2d1e0c5c40fee7fd3d185acd5dbce218b524285c
Oct 31 07:36:04.348345 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:36:04.348312    4720 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 31 07:36:04.591145 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:36:04.591114    4720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-9tnqq" event={"ID":"14b80f1a-6d77-43e5-8e5c-a16c806d543d","Type":"ContainerStarted","Data":"9eba8b59a436f41d565e9b1a2d1e0c5c40fee7fd3d185acd5dbce218b524285c"}
Oct 31 07:36:04.940130 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:36:04.940064    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 07:36:05.130757 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:36:05.130707    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 07:36:05.620394 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:36:05.620363    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 07:36:06.697640 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:36:06.697613    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 07:36:06.858684 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:36:06.858652    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 07:36:07.014383 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:36:07.014287    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 07:36:09.370964 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:36:09.370935    4720 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-5c57bbd4fc-hp6dv"]
Oct 31 07:36:09.374211 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:36:09.374195    4720 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-hp6dv"
Oct 31 07:36:09.429462 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:36:09.429440    4720 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-5c57bbd4fc-hp6dv"]
Oct 31 07:36:09.477612 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:36:09.477589    4720 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-fzkxj\""
Oct 31 07:36:09.518168 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:36:09.518144    4720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jnzln\" (UniqueName: \"kubernetes.io/projected/efb67732-6546-4290-a7b9-cbd9c0d28d9c-kube-api-access-jnzln\") pod \"cert-manager-cainjector-5c57bbd4fc-hp6dv\" (UID: \"efb67732-6546-4290-a7b9-cbd9c0d28d9c\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-hp6dv"
Oct 31 07:36:09.518277 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:36:09.518177    4720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/efb67732-6546-4290-a7b9-cbd9c0d28d9c-bound-sa-token\") pod \"cert-manager-cainjector-5c57bbd4fc-hp6dv\" (UID: \"efb67732-6546-4290-a7b9-cbd9c0d28d9c\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-hp6dv"
Oct 31 07:36:09.612110 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:36:09.612073    4720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-9tnqq" event={"ID":"14b80f1a-6d77-43e5-8e5c-a16c806d543d","Type":"ContainerStarted","Data":"2e986da174e4166fbc682a8dfa309a4d004be143513990cce421111a5e0ce5fc"}
Oct 31 07:36:09.612247 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:36:09.612127    4720 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-9tnqq"
Oct 31 07:36:09.618515 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:36:09.618493    4720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jnzln\" (UniqueName: \"kubernetes.io/projected/efb67732-6546-4290-a7b9-cbd9c0d28d9c-kube-api-access-jnzln\") pod \"cert-manager-cainjector-5c57bbd4fc-hp6dv\" (UID: \"efb67732-6546-4290-a7b9-cbd9c0d28d9c\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-hp6dv"
Oct 31 07:36:09.618602 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:36:09.618526    4720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/efb67732-6546-4290-a7b9-cbd9c0d28d9c-bound-sa-token\") pod \"cert-manager-cainjector-5c57bbd4fc-hp6dv\" (UID: \"efb67732-6546-4290-a7b9-cbd9c0d28d9c\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-hp6dv"
Oct 31 07:36:09.675157 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:36:09.675107    4720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/efb67732-6546-4290-a7b9-cbd9c0d28d9c-bound-sa-token\") pod \"cert-manager-cainjector-5c57bbd4fc-hp6dv\" (UID: \"efb67732-6546-4290-a7b9-cbd9c0d28d9c\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-hp6dv"
Oct 31 07:36:09.675241 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:36:09.675170    4720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jnzln\" (UniqueName: \"kubernetes.io/projected/efb67732-6546-4290-a7b9-cbd9c0d28d9c-kube-api-access-jnzln\") pod \"cert-manager-cainjector-5c57bbd4fc-hp6dv\" (UID: \"efb67732-6546-4290-a7b9-cbd9c0d28d9c\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-hp6dv"
Oct 31 07:36:09.684926 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:36:09.684899    4720 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-hp6dv"
Oct 31 07:36:09.724830 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:36:09.724773    4720 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-9tnqq" podStartSLOduration=2.304992702 podStartE2EDuration="6.724754905s" totalImagesPullingTime="4.419762203s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 07:36:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 07:36:09.72409625 +0000 UTC m=+423.855164322" watchObservedRunningTime="2026-10-31 07:36:09.724754905 +0000 UTC m=+423.855822976"
Oct 31 07:36:10.052567 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:36:10.052500    4720 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-5c57bbd4fc-hp6dv"]
Oct 31 07:36:10.056387 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:36:10.056358    4720 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podefb67732_6546_4290_a7b9_cbd9c0d28d9c.slice/crio-75833ca2234ec17f9b6f5ab64afad08cef7750d9d843830afce55c4b2542d9d1 WatchSource:0}: Error finding container 75833ca2234ec17f9b6f5ab64afad08cef7750d9d843830afce55c4b2542d9d1: Status 404 returned error can't find the container with id 75833ca2234ec17f9b6f5ab64afad08cef7750d9d843830afce55c4b2542d9d1
Oct 31 07:36:10.619787 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:36:10.619747    4720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-hp6dv" event={"ID":"efb67732-6546-4290-a7b9-cbd9c0d28d9c","Type":"ContainerStarted","Data":"c624861e0f008483b390c322b3cfd17bd3af8157f96465f84c371b46393fbd0b"}
Oct 31 07:36:10.620275 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:36:10.619795    4720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-hp6dv" event={"ID":"efb67732-6546-4290-a7b9-cbd9c0d28d9c","Type":"ContainerStarted","Data":"75833ca2234ec17f9b6f5ab64afad08cef7750d9d843830afce55c4b2542d9d1"}
Oct 31 07:36:10.732090 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:36:10.732044    4720 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-hp6dv" podStartSLOduration=1.732030145 podStartE2EDuration="1.732030145s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 07:36:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 07:36:10.730754679 +0000 UTC m=+424.861822771" watchObservedRunningTime="2026-10-31 07:36:10.732030145 +0000 UTC m=+424.863098212"
Oct 31 07:36:11.612884 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:36:11.612852    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 07:36:11.773170 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:36:11.773139    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 07:36:12.983015 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:36:12.982984    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 07:36:13.329667 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:36:13.329595    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 07:36:13.460000 ip-10-0-139-83 kubenswrapper[4720]: E1031 07:36:13.459960    4720 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8m768" containerName="extractor"
Oct 31 07:36:15.621685 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:36:15.621654    4720 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-9tnqq"
Oct 31 07:36:20.430491 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:36:20.430447    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 07:36:20.583114 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:36:20.583089    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 07:36:31.105954 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:36:31.105918    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 07:36:35.722453 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:36:35.722420    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 07:36:35.885537 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:36:35.885506    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 07:36:36.209298 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:36:36.209269    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 07:36:36.364798 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:36:36.364769    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 07:36:36.515068 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:36:36.515000    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 07:36:36.621074 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:36:36.621054    4720 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-64dc655d5d-h8wpv"]
Oct 31 07:36:36.624844 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:36:36.624829    4720 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-64dc655d5d-h8wpv"
Oct 31 07:36:36.675892 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:36:36.675867    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 07:36:36.680546 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:36:36.680527    4720 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-64dc655d5d-h8wpv"]
Oct 31 07:36:36.770014 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:36:36.769964    4720 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\""
Oct 31 07:36:36.785701 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:36:36.785682    4720 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-6jm8r\""
Oct 31 07:36:36.786062 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:36:36.786046    4720 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\""
Oct 31 07:36:36.786137 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:36:36.786085    4720 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\""
Oct 31 07:36:36.786137 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:36:36.786100    4720 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\""
Oct 31 07:36:36.786137 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:36:36.786122    4720 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\""
Oct 31 07:36:36.793449 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:36:36.793421    4720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a30be790-954e-4d1f-9b10-1da5233852a2-cert\") pod \"lws-controller-manager-64dc655d5d-h8wpv\" (UID: \"a30be790-954e-4d1f-9b10-1da5233852a2\") " pod="openshift-lws-operator/lws-controller-manager-64dc655d5d-h8wpv"
Oct 31 07:36:36.793513 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:36:36.793461    4720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/a30be790-954e-4d1f-9b10-1da5233852a2-manager-config\") pod \"lws-controller-manager-64dc655d5d-h8wpv\" (UID: \"a30be790-954e-4d1f-9b10-1da5233852a2\") " pod="openshift-lws-operator/lws-controller-manager-64dc655d5d-h8wpv"
Oct 31 07:36:36.793513 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:36:36.793481    4720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lsr2r\" (UniqueName: \"kubernetes.io/projected/a30be790-954e-4d1f-9b10-1da5233852a2-kube-api-access-lsr2r\") pod \"lws-controller-manager-64dc655d5d-h8wpv\" (UID: \"a30be790-954e-4d1f-9b10-1da5233852a2\") " pod="openshift-lws-operator/lws-controller-manager-64dc655d5d-h8wpv"
Oct 31 07:36:36.793636 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:36:36.793576    4720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/a30be790-954e-4d1f-9b10-1da5233852a2-metrics-cert\") pod \"lws-controller-manager-64dc655d5d-h8wpv\" (UID: \"a30be790-954e-4d1f-9b10-1da5233852a2\") " pod="openshift-lws-operator/lws-controller-manager-64dc655d5d-h8wpv"
Oct 31 07:36:36.832156 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:36:36.832130    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 07:36:36.893931 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:36:36.893904    4720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a30be790-954e-4d1f-9b10-1da5233852a2-cert\") pod \"lws-controller-manager-64dc655d5d-h8wpv\" (UID: \"a30be790-954e-4d1f-9b10-1da5233852a2\") " pod="openshift-lws-operator/lws-controller-manager-64dc655d5d-h8wpv"
Oct 31 07:36:36.894056 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:36:36.893941    4720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/a30be790-954e-4d1f-9b10-1da5233852a2-manager-config\") pod \"lws-controller-manager-64dc655d5d-h8wpv\" (UID: \"a30be790-954e-4d1f-9b10-1da5233852a2\") " pod="openshift-lws-operator/lws-controller-manager-64dc655d5d-h8wpv"
Oct 31 07:36:36.894056 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:36:36.893962    4720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lsr2r\" (UniqueName: \"kubernetes.io/projected/a30be790-954e-4d1f-9b10-1da5233852a2-kube-api-access-lsr2r\") pod \"lws-controller-manager-64dc655d5d-h8wpv\" (UID: \"a30be790-954e-4d1f-9b10-1da5233852a2\") " pod="openshift-lws-operator/lws-controller-manager-64dc655d5d-h8wpv"
Oct 31 07:36:36.894172 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:36:36.894079    4720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/a30be790-954e-4d1f-9b10-1da5233852a2-metrics-cert\") pod \"lws-controller-manager-64dc655d5d-h8wpv\" (UID: \"a30be790-954e-4d1f-9b10-1da5233852a2\") " pod="openshift-lws-operator/lws-controller-manager-64dc655d5d-h8wpv"
Oct 31 07:36:36.894697 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:36:36.894675    4720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/a30be790-954e-4d1f-9b10-1da5233852a2-manager-config\") pod \"lws-controller-manager-64dc655d5d-h8wpv\" (UID: \"a30be790-954e-4d1f-9b10-1da5233852a2\") " pod="openshift-lws-operator/lws-controller-manager-64dc655d5d-h8wpv"
Oct 31 07:36:36.896659 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:36:36.896636    4720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a30be790-954e-4d1f-9b10-1da5233852a2-cert\") pod \"lws-controller-manager-64dc655d5d-h8wpv\" (UID: \"a30be790-954e-4d1f-9b10-1da5233852a2\") " pod="openshift-lws-operator/lws-controller-manager-64dc655d5d-h8wpv"
Oct 31 07:36:36.896737 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:36:36.896638    4720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/a30be790-954e-4d1f-9b10-1da5233852a2-metrics-cert\") pod \"lws-controller-manager-64dc655d5d-h8wpv\" (UID: \"a30be790-954e-4d1f-9b10-1da5233852a2\") " pod="openshift-lws-operator/lws-controller-manager-64dc655d5d-h8wpv"
Oct 31 07:36:36.949973 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:36:36.949948    4720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lsr2r\" (UniqueName: \"kubernetes.io/projected/a30be790-954e-4d1f-9b10-1da5233852a2-kube-api-access-lsr2r\") pod \"lws-controller-manager-64dc655d5d-h8wpv\" (UID: \"a30be790-954e-4d1f-9b10-1da5233852a2\") " pod="openshift-lws-operator/lws-controller-manager-64dc655d5d-h8wpv"
Oct 31 07:36:37.234458 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:36:37.234434    4720 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-64dc655d5d-h8wpv"
Oct 31 07:36:37.603528 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:36:37.603499    4720 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-64dc655d5d-h8wpv"]
Oct 31 07:36:37.606802 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:36:37.606773    4720 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda30be790_954e_4d1f_9b10_1da5233852a2.slice/crio-e9d6d169d0ce1bb169da1d2734ca276224861e9da70af98e7caea22562e173a1 WatchSource:0}: Error finding container e9d6d169d0ce1bb169da1d2734ca276224861e9da70af98e7caea22562e173a1: Status 404 returned error can't find the container with id e9d6d169d0ce1bb169da1d2734ca276224861e9da70af98e7caea22562e173a1
Oct 31 07:36:37.723620 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:36:37.723589    4720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-64dc655d5d-h8wpv" event={"ID":"a30be790-954e-4d1f-9b10-1da5233852a2","Type":"ContainerStarted","Data":"e9d6d169d0ce1bb169da1d2734ca276224861e9da70af98e7caea22562e173a1"}
Oct 31 07:36:41.183847 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:36:41.183816    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 07:36:41.340574 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:36:41.340494    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 07:36:41.740812 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:36:41.740778    4720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-64dc655d5d-h8wpv" event={"ID":"a30be790-954e-4d1f-9b10-1da5233852a2","Type":"ContainerStarted","Data":"287c2fa3a2e2f8d35952b3f67cb533cc63cb47951d0240df7f292493b8f8d0e7"}
Oct 31 07:36:41.740952 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:36:41.740904    4720 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-64dc655d5d-h8wpv"
Oct 31 07:36:41.853547 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:36:41.853509    4720 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-64dc655d5d-h8wpv" podStartSLOduration=2.2942546520000002 podStartE2EDuration="5.853497628s" totalImagesPullingTime="3.559242976s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 07:36:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 07:36:41.852520736 +0000 UTC m=+455.983588806" watchObservedRunningTime="2026-10-31 07:36:41.853497628 +0000 UTC m=+455.984565698"
Oct 31 07:36:43.460286 ip-10-0-139-83 kubenswrapper[4720]: E1031 07:36:43.460253    4720 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8m768" containerName="extractor"
Oct 31 07:36:45.348087 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:36:45.348054    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 07:36:45.490375 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:36:45.490317    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 07:36:49.095732 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:36:49.095705    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 07:36:52.746543 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:36:52.746515    4720 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-64dc655d5d-h8wpv"
Oct 31 07:36:53.033992 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:36:53.033909    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 07:36:53.197667 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:36:53.197625    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 07:36:55.100411 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:36:55.100384    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 07:37:03.344691 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:37:03.344645    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 07:37:08.279270 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:37:08.279235    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 07:37:08.847237 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:37:08.847181    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 07:37:11.913977 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:37:11.913938    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 07:37:12.708193 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:37:12.708159    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 07:37:13.321456 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:37:13.321422    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 07:37:13.461834 ip-10-0-139-83 kubenswrapper[4720]: E1031 07:37:13.461800    4720 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8m768" containerName="extractor"
Oct 31 07:37:13.755545 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:37:13.755515    4720 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-68df894cb4-xjstz"]
Oct 31 07:37:13.763044 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:37:13.763021    4720 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-68df894cb4-xjstz"
Oct 31 07:37:13.818125 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:37:13.818102    4720 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\""
Oct 31 07:37:13.821447 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:37:13.821434    4720 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Oct 31 07:37:13.852015 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:37:13.851993    4720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xlcl2\" (UniqueName: \"kubernetes.io/projected/33530b7d-e4c2-490b-b7ea-7a7915cbe769-kube-api-access-xlcl2\") pod \"dns-operator-controller-manager-68df894cb4-xjstz\" (UID: \"33530b7d-e4c2-490b-b7ea-7a7915cbe769\") " pod="kuadrant-system/dns-operator-controller-manager-68df894cb4-xjstz"
Oct 31 07:37:13.867762 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:37:13.867741    4720 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Oct 31 07:37:13.870934 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:37:13.870916    4720 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-ffr4f\""
Oct 31 07:37:13.872055 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:37:13.872037    4720 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-68df894cb4-xjstz"]
Oct 31 07:37:13.952759 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:37:13.952737    4720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xlcl2\" (UniqueName: \"kubernetes.io/projected/33530b7d-e4c2-490b-b7ea-7a7915cbe769-kube-api-access-xlcl2\") pod \"dns-operator-controller-manager-68df894cb4-xjstz\" (UID: \"33530b7d-e4c2-490b-b7ea-7a7915cbe769\") " pod="kuadrant-system/dns-operator-controller-manager-68df894cb4-xjstz"
Oct 31 07:37:14.013909 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:37:14.013851    4720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xlcl2\" (UniqueName: \"kubernetes.io/projected/33530b7d-e4c2-490b-b7ea-7a7915cbe769-kube-api-access-xlcl2\") pod \"dns-operator-controller-manager-68df894cb4-xjstz\" (UID: \"33530b7d-e4c2-490b-b7ea-7a7915cbe769\") " pod="kuadrant-system/dns-operator-controller-manager-68df894cb4-xjstz"
Oct 31 07:37:14.074277 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:37:14.074254    4720 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-68df894cb4-xjstz"
Oct 31 07:37:14.085308 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:37:14.085283    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 07:37:14.444766 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:37:14.444740    4720 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-68df894cb4-xjstz"]
Oct 31 07:37:14.448939 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:37:14.448906    4720 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod33530b7d_e4c2_490b_b7ea_7a7915cbe769.slice/crio-ee817d77008b52bca5f9ec64b7b3bc1bfbbf58f1da2e16e9edadaa8af3e900ae WatchSource:0}: Error finding container ee817d77008b52bca5f9ec64b7b3bc1bfbbf58f1da2e16e9edadaa8af3e900ae: Status 404 returned error can't find the container with id ee817d77008b52bca5f9ec64b7b3bc1bfbbf58f1da2e16e9edadaa8af3e900ae
Oct 31 07:37:14.499306 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:37:14.499283    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 07:37:14.863450 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:37:14.863421    4720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-68df894cb4-xjstz" event={"ID":"33530b7d-e4c2-490b-b7ea-7a7915cbe769","Type":"ContainerStarted","Data":"ee817d77008b52bca5f9ec64b7b3bc1bfbbf58f1da2e16e9edadaa8af3e900ae"}
Oct 31 07:37:15.908692 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:37:15.908656    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 07:37:16.154021 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:37:16.153955    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 07:37:16.880603 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:37:16.880573    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 07:37:17.345583 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:37:17.345554    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 07:37:17.536482 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:37:17.536455    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 07:37:17.709868 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:37:17.709783    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 07:37:18.542226 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:37:18.542194    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 07:37:18.881077 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:37:18.881047    4720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-68df894cb4-xjstz" event={"ID":"33530b7d-e4c2-490b-b7ea-7a7915cbe769","Type":"ContainerStarted","Data":"3e5ea20ef89f94a323d2f18e50ebd40efd802205d50afc5767aed910951d812e"}
Oct 31 07:37:18.881304 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:37:18.881286    4720 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-68df894cb4-xjstz"
Oct 31 07:37:18.996248 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:37:18.996188    4720 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-68df894cb4-xjstz" podStartSLOduration=2.5101905799999997 podStartE2EDuration="5.996171358s" totalImagesPullingTime="3.485980778s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 07:37:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 07:37:18.995779912 +0000 UTC m=+493.126847992" watchObservedRunningTime="2026-10-31 07:37:18.996171358 +0000 UTC m=+493.127239424"
Oct 31 07:37:20.604754 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:37:20.604727    4720 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-658c459c99-nnrhp"]
Oct 31 07:37:20.607778 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:37:20.607761    4720 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-658c459c99-nnrhp"
Oct 31 07:37:20.653602 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:37:20.653582    4720 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-658c459c99-nnrhp"]
Oct 31 07:37:20.700593 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:37:20.700566    4720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/c4d53665-7863-48a0-a0a2-7e911345bd4f-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-658c459c99-nnrhp\" (UID: \"c4d53665-7863-48a0-a0a2-7e911345bd4f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-658c459c99-nnrhp"
Oct 31 07:37:20.700702 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:37:20.700599    4720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w8bvl\" (UniqueName: \"kubernetes.io/projected/c4d53665-7863-48a0-a0a2-7e911345bd4f-kube-api-access-w8bvl\") pod \"kuadrant-operator-controller-manager-658c459c99-nnrhp\" (UID: \"c4d53665-7863-48a0-a0a2-7e911345bd4f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-658c459c99-nnrhp"
Oct 31 07:37:20.701985 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:37:20.701970    4720 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-xhv4l\""
Oct 31 07:37:20.801032 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:37:20.801002    4720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/c4d53665-7863-48a0-a0a2-7e911345bd4f-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-658c459c99-nnrhp\" (UID: \"c4d53665-7863-48a0-a0a2-7e911345bd4f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-658c459c99-nnrhp"
Oct 31 07:37:20.801134 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:37:20.801037    4720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w8bvl\" (UniqueName: \"kubernetes.io/projected/c4d53665-7863-48a0-a0a2-7e911345bd4f-kube-api-access-w8bvl\") pod \"kuadrant-operator-controller-manager-658c459c99-nnrhp\" (UID: \"c4d53665-7863-48a0-a0a2-7e911345bd4f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-658c459c99-nnrhp"
Oct 31 07:37:20.801375 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:37:20.801360    4720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/c4d53665-7863-48a0-a0a2-7e911345bd4f-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-658c459c99-nnrhp\" (UID: \"c4d53665-7863-48a0-a0a2-7e911345bd4f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-658c459c99-nnrhp"
Oct 31 07:37:20.857787 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:37:20.857739    4720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w8bvl\" (UniqueName: \"kubernetes.io/projected/c4d53665-7863-48a0-a0a2-7e911345bd4f-kube-api-access-w8bvl\") pod \"kuadrant-operator-controller-manager-658c459c99-nnrhp\" (UID: \"c4d53665-7863-48a0-a0a2-7e911345bd4f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-658c459c99-nnrhp"
Oct 31 07:37:20.919511 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:37:20.919493    4720 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-658c459c99-nnrhp"
Oct 31 07:37:21.343550 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:37:21.343514    4720 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-658c459c99-nnrhp"]
Oct 31 07:37:21.347581 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:37:21.347555    4720 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc4d53665_7863_48a0_a0a2_7e911345bd4f.slice/crio-c021e209a0aab7409785d6d88ff85ecf9725611ce79082732ce27d6e57b4097f WatchSource:0}: Error finding container c021e209a0aab7409785d6d88ff85ecf9725611ce79082732ce27d6e57b4097f: Status 404 returned error can't find the container with id c021e209a0aab7409785d6d88ff85ecf9725611ce79082732ce27d6e57b4097f
Oct 31 07:37:21.894403 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:37:21.894373    4720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-658c459c99-nnrhp" event={"ID":"c4d53665-7863-48a0-a0a2-7e911345bd4f","Type":"ContainerStarted","Data":"c021e209a0aab7409785d6d88ff85ecf9725611ce79082732ce27d6e57b4097f"}
Oct 31 07:37:22.301498 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:37:22.301433    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 07:37:22.452575 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:37:22.452537    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 07:37:24.885072 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:37:24.885041    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 07:37:25.030600 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:37:25.030565    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 07:37:26.916939 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:37:26.916902    4720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-658c459c99-nnrhp" event={"ID":"c4d53665-7863-48a0-a0a2-7e911345bd4f","Type":"ContainerStarted","Data":"9a4d5a74573d71eee6300f6a1c4458deeee8679ac1ec1835a6af829703726077"}
Oct 31 07:37:26.917294 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:37:26.916977    4720 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-658c459c99-nnrhp"
Oct 31 07:37:27.159819 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:37:27.159450    4720 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-658c459c99-nnrhp" podStartSLOduration=2.073651403 podStartE2EDuration="7.159433063s" totalImagesPullingTime="5.08578166s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 07:37:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 07:37:27.108895724 +0000 UTC m=+501.239963807" watchObservedRunningTime="2026-10-31 07:37:27.159433063 +0000 UTC m=+501.290501135"
Oct 31 07:37:29.887431 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:37:29.887398    4720 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-68df894cb4-xjstz"
Oct 31 07:37:30.714244 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:37:30.714214    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 07:37:30.931547 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:37:30.931514    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 07:37:31.818683 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:37:31.818654    4720 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-wmfgm"]
Oct 31 07:37:31.821558 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:37:31.821534    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 07:37:31.821864 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:37:31.821848    4720 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-wmfgm"
Oct 31 07:37:31.873917 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:37:31.873894    4720 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\""
Oct 31 07:37:31.882722 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:37:31.882697    4720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6zbj8\" (UniqueName: \"kubernetes.io/projected/601ccf10-fbcd-41c1-ae75-6d7c01bc1b8a-kube-api-access-6zbj8\") pod \"kuadrant-console-plugin-66c4fb575d-wmfgm\" (UID: \"601ccf10-fbcd-41c1-ae75-6d7c01bc1b8a\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-wmfgm"
Oct 31 07:37:31.882853 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:37:31.882728    4720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/601ccf10-fbcd-41c1-ae75-6d7c01bc1b8a-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-wmfgm\" (UID: \"601ccf10-fbcd-41c1-ae75-6d7c01bc1b8a\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-wmfgm"
Oct 31 07:37:31.882853 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:37:31.882763    4720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/601ccf10-fbcd-41c1-ae75-6d7c01bc1b8a-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-wmfgm\" (UID: \"601ccf10-fbcd-41c1-ae75-6d7c01bc1b8a\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-wmfgm"
Oct 31 07:37:31.923863 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:37:31.923839    4720 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-2mh85\""
Oct 31 07:37:31.923863 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:37:31.923845    4720 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\""
Oct 31 07:37:31.925965 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:37:31.925941    4720 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-wmfgm"]
Oct 31 07:37:31.969118 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:37:31.969091    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 07:37:31.983572 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:37:31.983548    4720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6zbj8\" (UniqueName: \"kubernetes.io/projected/601ccf10-fbcd-41c1-ae75-6d7c01bc1b8a-kube-api-access-6zbj8\") pod \"kuadrant-console-plugin-66c4fb575d-wmfgm\" (UID: \"601ccf10-fbcd-41c1-ae75-6d7c01bc1b8a\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-wmfgm"
Oct 31 07:37:31.983675 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:37:31.983585    4720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/601ccf10-fbcd-41c1-ae75-6d7c01bc1b8a-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-wmfgm\" (UID: \"601ccf10-fbcd-41c1-ae75-6d7c01bc1b8a\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-wmfgm"
Oct 31 07:37:31.983675 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:37:31.983641    4720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/601ccf10-fbcd-41c1-ae75-6d7c01bc1b8a-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-wmfgm\" (UID: \"601ccf10-fbcd-41c1-ae75-6d7c01bc1b8a\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-wmfgm"
Oct 31 07:37:31.984233 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:37:31.984213    4720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/601ccf10-fbcd-41c1-ae75-6d7c01bc1b8a-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-wmfgm\" (UID: \"601ccf10-fbcd-41c1-ae75-6d7c01bc1b8a\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-wmfgm"
Oct 31 07:37:31.986221 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:37:31.986190    4720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/601ccf10-fbcd-41c1-ae75-6d7c01bc1b8a-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-wmfgm\" (UID: \"601ccf10-fbcd-41c1-ae75-6d7c01bc1b8a\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-wmfgm"
Oct 31 07:37:32.039945 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:37:32.039921    4720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6zbj8\" (UniqueName: \"kubernetes.io/projected/601ccf10-fbcd-41c1-ae75-6d7c01bc1b8a-kube-api-access-6zbj8\") pod \"kuadrant-console-plugin-66c4fb575d-wmfgm\" (UID: \"601ccf10-fbcd-41c1-ae75-6d7c01bc1b8a\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-wmfgm"
Oct 31 07:37:32.132827 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:37:32.132798    4720 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-wmfgm"
Oct 31 07:37:32.501034 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:37:32.500944    4720 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-wmfgm"]
Oct 31 07:37:32.504821 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:37:32.504789    4720 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod601ccf10_fbcd_41c1_ae75_6d7c01bc1b8a.slice/crio-a334a1360769af6b040ddae3855730cc24b75378737159675a1725950dd8ffc8 WatchSource:0}: Error finding container a334a1360769af6b040ddae3855730cc24b75378737159675a1725950dd8ffc8: Status 404 returned error can't find the container with id a334a1360769af6b040ddae3855730cc24b75378737159675a1725950dd8ffc8
Oct 31 07:37:32.941778 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:37:32.941740    4720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-wmfgm" event={"ID":"601ccf10-fbcd-41c1-ae75-6d7c01bc1b8a","Type":"ContainerStarted","Data":"a334a1360769af6b040ddae3855730cc24b75378737159675a1725950dd8ffc8"}
Oct 31 07:37:35.465465 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:37:35.465429    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 07:37:35.612154 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:37:35.612111    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 07:37:37.924314 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:37:37.924274    4720 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-658c459c99-nnrhp"
Oct 31 07:37:37.965598 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:37:37.965572    4720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-wmfgm" event={"ID":"601ccf10-fbcd-41c1-ae75-6d7c01bc1b8a","Type":"ContainerStarted","Data":"c34aac67c1a2b23ee48aeee21e2e673ec77e7d52269eb6d77d0d49bdc3c66ff9"}
Oct 31 07:37:38.149380 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:37:38.149260    4720 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-wmfgm" podStartSLOduration=1.777617792 podStartE2EDuration="7.14924666s" totalImagesPullingTime="5.371628868s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 07:37:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 07:37:38.148882454 +0000 UTC m=+512.279950522" watchObservedRunningTime="2026-10-31 07:37:38.14924666 +0000 UTC m=+512.280314725"
Oct 31 07:37:42.103051 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:37:42.103014    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 07:37:42.254756 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:37:42.254723    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 07:37:43.459502 ip-10-0-139-83 kubenswrapper[4720]: E1031 07:37:43.459469    4720 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8m768" containerName="extractor"
Oct 31 07:37:45.102735 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:37:45.102702    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 07:37:45.254876 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:37:45.254843    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 07:37:50.892617 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:37:50.892562    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 07:37:51.041547 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:37:51.041512    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 07:37:51.497736 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:37:51.497702    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 07:37:51.656606 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:37:51.656578    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 07:37:54.955128 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:37:54.955093    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 07:37:55.111044 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:37:55.111015    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 07:38:03.669603 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:38:03.669566    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 07:38:13.177472 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:38:13.177437    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 07:38:13.461766 ip-10-0-139-83 kubenswrapper[4720]: E1031 07:38:13.461680    4720 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8m768" containerName="extractor"
Oct 31 07:38:13.542795 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:38:13.542766    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 07:38:13.736849 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:38:13.736755    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 07:38:14.496468 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:38:14.496437    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 07:38:14.641238 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:38:14.641203    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 07:38:14.862818 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:38:14.862775    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 07:38:15.012621 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:38:15.012592    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 07:38:17.290856 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:38:17.290826    4720 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-hpzgv"]
Oct 31 07:38:17.293914 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:38:17.293896    4720 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-hpzgv"
Oct 31 07:38:17.347091 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:38:17.347073    4720 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-tmh6h\""
Oct 31 07:38:17.397224 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:38:17.397198    4720 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-hpzgv"]
Oct 31 07:38:17.405871 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:38:17.405852    4720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wj2pw\" (UniqueName: \"kubernetes.io/projected/e6429727-3f72-4290-8fa4-7ae95e699a98-kube-api-access-wj2pw\") pod \"authorino-56c6c9ddd5-hpzgv\" (UID: \"e6429727-3f72-4290-8fa4-7ae95e699a98\") " pod="kuadrant-system/authorino-56c6c9ddd5-hpzgv"
Oct 31 07:38:17.506417 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:38:17.506392    4720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wj2pw\" (UniqueName: \"kubernetes.io/projected/e6429727-3f72-4290-8fa4-7ae95e699a98-kube-api-access-wj2pw\") pod \"authorino-56c6c9ddd5-hpzgv\" (UID: \"e6429727-3f72-4290-8fa4-7ae95e699a98\") " pod="kuadrant-system/authorino-56c6c9ddd5-hpzgv"
Oct 31 07:38:17.562408 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:38:17.562352    4720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wj2pw\" (UniqueName: \"kubernetes.io/projected/e6429727-3f72-4290-8fa4-7ae95e699a98-kube-api-access-wj2pw\") pod \"authorino-56c6c9ddd5-hpzgv\" (UID: \"e6429727-3f72-4290-8fa4-7ae95e699a98\") " pod="kuadrant-system/authorino-56c6c9ddd5-hpzgv"
Oct 31 07:38:17.604510 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:38:17.604491    4720 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-hpzgv"
Oct 31 07:38:17.968410 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:38:17.968343    4720 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-hpzgv"]
Oct 31 07:38:17.972050 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:38:17.972025    4720 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode6429727_3f72_4290_8fa4_7ae95e699a98.slice/crio-c89647a06e6929bed48c47b09c3af5d3c79410aa652abfedf0657fd665b13f0b WatchSource:0}: Error finding container c89647a06e6929bed48c47b09c3af5d3c79410aa652abfedf0657fd665b13f0b: Status 404 returned error can't find the container with id c89647a06e6929bed48c47b09c3af5d3c79410aa652abfedf0657fd665b13f0b
Oct 31 07:38:18.114733 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:38:18.114702    4720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-hpzgv" event={"ID":"e6429727-3f72-4290-8fa4-7ae95e699a98","Type":"ContainerStarted","Data":"c89647a06e6929bed48c47b09c3af5d3c79410aa652abfedf0657fd665b13f0b"}
Oct 31 07:38:21.127476 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:38:21.127435    4720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-hpzgv" event={"ID":"e6429727-3f72-4290-8fa4-7ae95e699a98","Type":"ContainerStarted","Data":"4e19e50567ef64c66765febf756f85888ef454da69cc501bc4454c8a24db205e"}
Oct 31 07:38:21.239163 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:38:21.239112    4720 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-56c6c9ddd5-hpzgv" podStartSLOduration=1.601193675 podStartE2EDuration="4.239100168s" totalImagesPullingTime="2.637906493s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 07:38:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 07:38:21.238621098 +0000 UTC m=+555.369689160" watchObservedRunningTime="2026-10-31 07:38:21.239100168 +0000 UTC m=+555.370168263"
Oct 31 07:38:21.707006 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:38:21.706974    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 07:38:43.460077 ip-10-0-139-83 kubenswrapper[4720]: E1031 07:38:43.460039    4720 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8m768" containerName="extractor"
Oct 31 07:38:58.727320 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:38:58.727241    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 07:39:00.186242 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:39:00.186211    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 07:39:04.005306 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:39:04.005266    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 07:39:06.057765 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:39:06.057741    4720 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-4jszp_b1db0e3b-e85d-4769-8012-cf4b6f740d9d/console-operator/2.log"
Oct 31 07:39:06.058159 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:39:06.057807    4720 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-4jszp_b1db0e3b-e85d-4769-8012-cf4b6f740d9d/console-operator/2.log"
Oct 31 07:39:13.459548 ip-10-0-139-83 kubenswrapper[4720]: E1031 07:39:13.459513    4720 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8m768" containerName="extractor"
Oct 31 07:39:14.057606 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:39:14.057576    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 07:39:14.821247 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:39:14.820866    4720 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-hpzgv"]
Oct 31 07:39:14.821247 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:39:14.821062    4720 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-56c6c9ddd5-hpzgv" podUID="e6429727-3f72-4290-8fa4-7ae95e699a98" containerName="authorino" containerID="cri-o://4e19e50567ef64c66765febf756f85888ef454da69cc501bc4454c8a24db205e" gracePeriod=30
Oct 31 07:39:15.066128 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:39:15.066102    4720 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-hpzgv"
Oct 31 07:39:15.192721 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:39:15.192700    4720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e6429727-3f72-4290-8fa4-7ae95e699a98-kube-api-access-wj2pw\" (UniqueName: \"kubernetes.io/projected/e6429727-3f72-4290-8fa4-7ae95e699a98-kube-api-access-wj2pw\") pod \"e6429727-3f72-4290-8fa4-7ae95e699a98\" (UID: \"e6429727-3f72-4290-8fa4-7ae95e699a98\") "
Oct 31 07:39:15.194877 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:39:15.194854    4720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e6429727-3f72-4290-8fa4-7ae95e699a98-kube-api-access-wj2pw" pod "e6429727-3f72-4290-8fa4-7ae95e699a98" (UID: "e6429727-3f72-4290-8fa4-7ae95e699a98"). InnerVolumeSpecName "kube-api-access-wj2pw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 07:39:15.293714 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:39:15.293679    4720 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wj2pw\" (UniqueName: \"kubernetes.io/projected/e6429727-3f72-4290-8fa4-7ae95e699a98-kube-api-access-wj2pw\") on node \"ip-10-0-139-83.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 07:39:15.332173 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:39:15.332147    4720 generic.go:356] "Generic (PLEG): container finished" podID="e6429727-3f72-4290-8fa4-7ae95e699a98" containerID="4e19e50567ef64c66765febf756f85888ef454da69cc501bc4454c8a24db205e" exitCode=0
Oct 31 07:39:15.332285 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:39:15.332190    4720 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-hpzgv"
Oct 31 07:39:15.332285 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:39:15.332198    4720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-hpzgv" event={"ID":"e6429727-3f72-4290-8fa4-7ae95e699a98","Type":"ContainerDied","Data":"4e19e50567ef64c66765febf756f85888ef454da69cc501bc4454c8a24db205e"}
Oct 31 07:39:15.332285 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:39:15.332227    4720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-hpzgv" event={"ID":"e6429727-3f72-4290-8fa4-7ae95e699a98","Type":"ContainerDied","Data":"c89647a06e6929bed48c47b09c3af5d3c79410aa652abfedf0657fd665b13f0b"}
Oct 31 07:39:15.332285 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:39:15.332241    4720 scope.go:122] "RemoveContainer" containerID="4e19e50567ef64c66765febf756f85888ef454da69cc501bc4454c8a24db205e"
Oct 31 07:39:15.340807 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:39:15.340790    4720 scope.go:122] "RemoveContainer" containerID="4e19e50567ef64c66765febf756f85888ef454da69cc501bc4454c8a24db205e"
Oct 31 07:39:15.341051 ip-10-0-139-83 kubenswrapper[4720]: E1031 07:39:15.341033    4720 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4e19e50567ef64c66765febf756f85888ef454da69cc501bc4454c8a24db205e\": container with ID starting with 4e19e50567ef64c66765febf756f85888ef454da69cc501bc4454c8a24db205e not found: ID does not exist" containerID="4e19e50567ef64c66765febf756f85888ef454da69cc501bc4454c8a24db205e"
Oct 31 07:39:15.341095 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:39:15.341059    4720 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4e19e50567ef64c66765febf756f85888ef454da69cc501bc4454c8a24db205e"} err="failed to get container status \"4e19e50567ef64c66765febf756f85888ef454da69cc501bc4454c8a24db205e\": rpc error: code = NotFound desc = could not find container \"4e19e50567ef64c66765febf756f85888ef454da69cc501bc4454c8a24db205e\": container with ID starting with 4e19e50567ef64c66765febf756f85888ef454da69cc501bc4454c8a24db205e not found: ID does not exist"
Oct 31 07:39:15.491886 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:39:15.491839    4720 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-hpzgv"]
Oct 31 07:39:15.495465 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:39:15.495440    4720 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-hpzgv"]
Oct 31 07:39:16.025300 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:39:16.025272    4720 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e6429727-3f72-4290-8fa4-7ae95e699a98" path="/var/lib/kubelet/pods/e6429727-3f72-4290-8fa4-7ae95e699a98/volumes"
Oct 31 07:39:17.832442 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:39:17.832411    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 07:39:18.066444 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:39:18.066417    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 07:39:24.012213 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:39:24.012185    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 07:39:43.459590 ip-10-0-139-83 kubenswrapper[4720]: E1031 07:39:43.459554    4720 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8m768" containerName="extractor"
Oct 31 07:39:44.761126 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:39:44.761067    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 07:39:44.906122 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:39:44.906090    4720 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kserve/seaweedfs-78bb9c6ff8-bbmmt"]
Oct 31 07:39:44.906505 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:39:44.906490    4720 cpu_manager.go:414] "containerMap: removing container" podUID="e6429727-3f72-4290-8fa4-7ae95e699a98" containerName="authorino"
Oct 31 07:39:44.906565 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:39:44.906507    4720 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e6429727-3f72-4290-8fa4-7ae95e699a98" containerName="authorino"
Oct 31 07:39:44.906599 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:39:44.906584    4720 memory_manager.go:365] "RemoveStaleState removing state" podUID="e6429727-3f72-4290-8fa4-7ae95e699a98" containerName="authorino"
Oct 31 07:39:44.909502 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:39:44.909486    4720 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-78bb9c6ff8-bbmmt"
Oct 31 07:39:44.923666 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:39:44.923644    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 07:39:44.967077 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:39:44.967058    4720 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-78bb9c6ff8-bbmmt"]
Oct 31 07:39:44.997194 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:39:44.997174    4720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fb8wb\" (UniqueName: \"kubernetes.io/projected/920cc099-a4e4-4863-8968-76ca707b293a-kube-api-access-fb8wb\") pod \"seaweedfs-78bb9c6ff8-bbmmt\" (UID: \"920cc099-a4e4-4863-8968-76ca707b293a\") " pod="kserve/seaweedfs-78bb9c6ff8-bbmmt"
Oct 31 07:39:44.997310 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:39:44.997286    4720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/920cc099-a4e4-4863-8968-76ca707b293a-data\") pod \"seaweedfs-78bb9c6ff8-bbmmt\" (UID: \"920cc099-a4e4-4863-8968-76ca707b293a\") " pod="kserve/seaweedfs-78bb9c6ff8-bbmmt"
Oct 31 07:39:45.012308 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:39:45.012261    4720 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
Oct 31 07:39:45.012308 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:39:45.012275    4720 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
Oct 31 07:39:45.012308 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:39:45.012267    4720 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"mlpipeline-s3-artifact\""
Oct 31 07:39:45.012453 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:39:45.012311    4720 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-zv87w\""
Oct 31 07:39:45.074583 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:39:45.074546    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 07:39:45.098629 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:39:45.098608    4720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/920cc099-a4e4-4863-8968-76ca707b293a-data\") pod \"seaweedfs-78bb9c6ff8-bbmmt\" (UID: \"920cc099-a4e4-4863-8968-76ca707b293a\") " pod="kserve/seaweedfs-78bb9c6ff8-bbmmt"
Oct 31 07:39:45.098703 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:39:45.098659    4720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fb8wb\" (UniqueName: \"kubernetes.io/projected/920cc099-a4e4-4863-8968-76ca707b293a-kube-api-access-fb8wb\") pod \"seaweedfs-78bb9c6ff8-bbmmt\" (UID: \"920cc099-a4e4-4863-8968-76ca707b293a\") " pod="kserve/seaweedfs-78bb9c6ff8-bbmmt"
Oct 31 07:39:45.098995 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:39:45.098980    4720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/920cc099-a4e4-4863-8968-76ca707b293a-data\") pod \"seaweedfs-78bb9c6ff8-bbmmt\" (UID: \"920cc099-a4e4-4863-8968-76ca707b293a\") " pod="kserve/seaweedfs-78bb9c6ff8-bbmmt"
Oct 31 07:39:45.155202 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:39:45.155183    4720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fb8wb\" (UniqueName: \"kubernetes.io/projected/920cc099-a4e4-4863-8968-76ca707b293a-kube-api-access-fb8wb\") pod \"seaweedfs-78bb9c6ff8-bbmmt\" (UID: \"920cc099-a4e4-4863-8968-76ca707b293a\") " pod="kserve/seaweedfs-78bb9c6ff8-bbmmt"
Oct 31 07:39:45.220110 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:39:45.220085    4720 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-78bb9c6ff8-bbmmt"
Oct 31 07:39:45.225915 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:39:45.225873    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 07:39:45.591509 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:39:45.591479    4720 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-78bb9c6ff8-bbmmt"]
Oct 31 07:39:45.594405 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:39:45.594380    4720 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod920cc099_a4e4_4863_8968_76ca707b293a.slice/crio-b1bde49f03a998f916e81d62f0a96a0b265fb6cec4d4006819eae61f9d4ebf82 WatchSource:0}: Error finding container b1bde49f03a998f916e81d62f0a96a0b265fb6cec4d4006819eae61f9d4ebf82: Status 404 returned error can't find the container with id b1bde49f03a998f916e81d62f0a96a0b265fb6cec4d4006819eae61f9d4ebf82
Oct 31 07:39:46.248580 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:39:46.248553    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 07:39:46.388511 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:39:46.388479    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 07:39:46.454989 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:39:46.454959    4720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-78bb9c6ff8-bbmmt" event={"ID":"920cc099-a4e4-4863-8968-76ca707b293a","Type":"ContainerStarted","Data":"b1bde49f03a998f916e81d62f0a96a0b265fb6cec4d4006819eae61f9d4ebf82"}
Oct 31 07:39:46.539219 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:39:46.539160    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 07:39:49.469496 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:39:49.469464    4720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-78bb9c6ff8-bbmmt" event={"ID":"920cc099-a4e4-4863-8968-76ca707b293a","Type":"ContainerStarted","Data":"db3cc239a3ef0c130432b849fbf070b27ad7f2b640e2c44f5aade63c0528569b"}
Oct 31 07:39:49.470070 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:39:49.469605    4720 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/seaweedfs-78bb9c6ff8-bbmmt"
Oct 31 07:39:49.581531 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:39:49.581488    4720 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kserve/seaweedfs-78bb9c6ff8-bbmmt" podStartSLOduration=2.6245800299999997 podStartE2EDuration="5.58147473s" totalImagesPullingTime="2.9568947s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 07:39:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 07:39:49.580625516 +0000 UTC m=+643.711693655" watchObservedRunningTime="2026-10-31 07:39:49.58147473 +0000 UTC m=+643.712542800"
Oct 31 07:39:49.637553 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:39:49.637532    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 07:39:49.786099 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:39:49.786044    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 07:39:55.475776 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:39:55.475745    4720 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/seaweedfs-78bb9c6ff8-bbmmt"
Oct 31 07:40:13.460046 ip-10-0-139-83 kubenswrapper[4720]: E1031 07:40:13.460009    4720 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8m768" containerName="extractor"
Oct 31 07:40:14.338625 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:40:14.338591    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 07:40:24.335670 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:40:24.335599    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 07:40:24.482627 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:40:24.482599    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 07:40:43.460472 ip-10-0-139-83 kubenswrapper[4720]: E1031 07:40:43.460437    4720 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8m768" containerName="extractor"
Oct 31 07:41:13.460552 ip-10-0-139-83 kubenswrapper[4720]: E1031 07:41:13.460513    4720 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8m768" containerName="extractor"
Oct 31 07:41:14.682397 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:41:14.682364    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 07:41:21.551994 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:41:21.551963    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 07:41:21.708634 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:41:21.708596    4720 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kserve/odh-model-controller-7f9f8cdc65-ljxhx"]
Oct 31 07:41:21.711902 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:41:21.711885    4720 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/odh-model-controller-7f9f8cdc65-ljxhx"
Oct 31 07:41:21.717305 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:41:21.717281    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 07:41:21.768429 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:41:21.768400    4720 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kserve/odh-model-controller-7f9f8cdc65-ljxhx"]
Oct 31 07:41:21.792126 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:41:21.792101    4720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/dbc16c22-0395-415b-95df-da2db7df78a0-cert\") pod \"odh-model-controller-7f9f8cdc65-ljxhx\" (UID: \"dbc16c22-0395-415b-95df-da2db7df78a0\") " pod="kserve/odh-model-controller-7f9f8cdc65-ljxhx"
Oct 31 07:41:21.792237 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:41:21.792128    4720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lsrs6\" (UniqueName: \"kubernetes.io/projected/dbc16c22-0395-415b-95df-da2db7df78a0-kube-api-access-lsrs6\") pod \"odh-model-controller-7f9f8cdc65-ljxhx\" (UID: \"dbc16c22-0395-415b-95df-da2db7df78a0\") " pod="kserve/odh-model-controller-7f9f8cdc65-ljxhx"
Oct 31 07:41:21.792237 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:41:21.792149    4720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dbc16c22-0395-415b-95df-da2db7df78a0-metrics-certs\") pod \"odh-model-controller-7f9f8cdc65-ljxhx\" (UID: \"dbc16c22-0395-415b-95df-da2db7df78a0\") " pod="kserve/odh-model-controller-7f9f8cdc65-ljxhx"
Oct 31 07:41:21.816301 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:41:21.816238    4720 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-dockercfg-s8mnc\""
Oct 31 07:41:21.816424 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:41:21.816304    4720 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-metrics-tls\""
Oct 31 07:41:21.816424 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:41:21.816350    4720 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-webhook-cert\""
Oct 31 07:41:21.865480 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:41:21.865456    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 07:41:21.893408 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:41:21.893385    4720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/dbc16c22-0395-415b-95df-da2db7df78a0-cert\") pod \"odh-model-controller-7f9f8cdc65-ljxhx\" (UID: \"dbc16c22-0395-415b-95df-da2db7df78a0\") " pod="kserve/odh-model-controller-7f9f8cdc65-ljxhx"
Oct 31 07:41:21.893490 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:41:21.893412    4720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lsrs6\" (UniqueName: \"kubernetes.io/projected/dbc16c22-0395-415b-95df-da2db7df78a0-kube-api-access-lsrs6\") pod \"odh-model-controller-7f9f8cdc65-ljxhx\" (UID: \"dbc16c22-0395-415b-95df-da2db7df78a0\") " pod="kserve/odh-model-controller-7f9f8cdc65-ljxhx"
Oct 31 07:41:21.893490 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:41:21.893432    4720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dbc16c22-0395-415b-95df-da2db7df78a0-metrics-certs\") pod \"odh-model-controller-7f9f8cdc65-ljxhx\" (UID: \"dbc16c22-0395-415b-95df-da2db7df78a0\") " pod="kserve/odh-model-controller-7f9f8cdc65-ljxhx"
Oct 31 07:41:21.895818 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:41:21.895785    4720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/dbc16c22-0395-415b-95df-da2db7df78a0-cert\") pod \"odh-model-controller-7f9f8cdc65-ljxhx\" (UID: \"dbc16c22-0395-415b-95df-da2db7df78a0\") " pod="kserve/odh-model-controller-7f9f8cdc65-ljxhx"
Oct 31 07:41:21.895907 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:41:21.895819    4720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dbc16c22-0395-415b-95df-da2db7df78a0-metrics-certs\") pod \"odh-model-controller-7f9f8cdc65-ljxhx\" (UID: \"dbc16c22-0395-415b-95df-da2db7df78a0\") " pod="kserve/odh-model-controller-7f9f8cdc65-ljxhx"
Oct 31 07:41:21.949049 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:41:21.949018    4720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lsrs6\" (UniqueName: \"kubernetes.io/projected/dbc16c22-0395-415b-95df-da2db7df78a0-kube-api-access-lsrs6\") pod \"odh-model-controller-7f9f8cdc65-ljxhx\" (UID: \"dbc16c22-0395-415b-95df-da2db7df78a0\") " pod="kserve/odh-model-controller-7f9f8cdc65-ljxhx"
Oct 31 07:41:22.011047 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:41:22.011025    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 07:41:22.024405 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:41:22.024388    4720 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/odh-model-controller-7f9f8cdc65-ljxhx"
Oct 31 07:41:22.395582 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:41:22.395556    4720 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kserve/odh-model-controller-7f9f8cdc65-ljxhx"]
Oct 31 07:41:22.401446 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:41:22.401428    4720 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 31 07:41:22.848296 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:41:22.848266    4720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kserve/odh-model-controller-7f9f8cdc65-ljxhx" event={"ID":"dbc16c22-0395-415b-95df-da2db7df78a0","Type":"ContainerStarted","Data":"764c92f9d8359bac0fe9d14f163fcbaea9459fdbee7e6a6cbe48fc905916d702"}
Oct 31 07:41:24.678603 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:41:24.678575    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 07:41:25.864450 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:41:25.864418    4720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kserve/odh-model-controller-7f9f8cdc65-ljxhx" event={"ID":"dbc16c22-0395-415b-95df-da2db7df78a0","Type":"ContainerStarted","Data":"5662220f1caa419c9762efa5c36b8a74089ffcabc4d57937f2e421f552347544"}
Oct 31 07:41:25.864828 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:41:25.864534    4720 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/odh-model-controller-7f9f8cdc65-ljxhx"
Oct 31 07:41:25.978599 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:41:25.978558    4720 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kserve/odh-model-controller-7f9f8cdc65-ljxhx" podStartSLOduration=2.191264959 podStartE2EDuration="4.978546161s" totalImagesPullingTime="2.787281202s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 07:41:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 07:41:25.977540772 +0000 UTC m=+740.108608842" watchObservedRunningTime="2026-10-31 07:41:25.978546161 +0000 UTC m=+740.109614267"
Oct 31 07:41:26.610456 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:41:26.610425    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 07:41:26.753697 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:41:26.753660    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 07:41:36.870931 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:41:36.870901    4720 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/odh-model-controller-7f9f8cdc65-ljxhx"
Oct 31 07:41:43.462491 ip-10-0-139-83 kubenswrapper[4720]: E1031 07:41:43.462445    4720 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8m768" containerName="extractor"
Oct 31 07:41:48.267114 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:41:48.267038    4720 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kserve/s3-init-nx5tn"]
Oct 31 07:41:48.270507 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:41:48.270491    4720 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-nx5tn"
Oct 31 07:41:48.322208 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:41:48.322186    4720 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"hf-token\""
Oct 31 07:41:48.372736 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:41:48.372708    4720 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kserve/s3-init-nx5tn"]
Oct 31 07:41:48.381062 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:41:48.381032    4720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dnjn8\" (UniqueName: \"kubernetes.io/projected/a54b3c28-6f13-4bee-8efc-5619d28a3b8c-kube-api-access-dnjn8\") pod \"s3-init-nx5tn\" (UID: \"a54b3c28-6f13-4bee-8efc-5619d28a3b8c\") " pod="kserve/s3-init-nx5tn"
Oct 31 07:41:48.381153 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:41:48.381103    4720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a54b3c28-6f13-4bee-8efc-5619d28a3b8c-model-cache\") pod \"s3-init-nx5tn\" (UID: \"a54b3c28-6f13-4bee-8efc-5619d28a3b8c\") " pod="kserve/s3-init-nx5tn"
Oct 31 07:41:48.482418 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:41:48.482389    4720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a54b3c28-6f13-4bee-8efc-5619d28a3b8c-model-cache\") pod \"s3-init-nx5tn\" (UID: \"a54b3c28-6f13-4bee-8efc-5619d28a3b8c\") " pod="kserve/s3-init-nx5tn"
Oct 31 07:41:48.482538 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:41:48.482439    4720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dnjn8\" (UniqueName: \"kubernetes.io/projected/a54b3c28-6f13-4bee-8efc-5619d28a3b8c-kube-api-access-dnjn8\") pod \"s3-init-nx5tn\" (UID: \"a54b3c28-6f13-4bee-8efc-5619d28a3b8c\") " pod="kserve/s3-init-nx5tn"
Oct 31 07:41:48.482765 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:41:48.482746    4720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a54b3c28-6f13-4bee-8efc-5619d28a3b8c-model-cache\") pod \"s3-init-nx5tn\" (UID: \"a54b3c28-6f13-4bee-8efc-5619d28a3b8c\") " pod="kserve/s3-init-nx5tn"
Oct 31 07:41:48.538586 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:41:48.538529    4720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dnjn8\" (UniqueName: \"kubernetes.io/projected/a54b3c28-6f13-4bee-8efc-5619d28a3b8c-kube-api-access-dnjn8\") pod \"s3-init-nx5tn\" (UID: \"a54b3c28-6f13-4bee-8efc-5619d28a3b8c\") " pod="kserve/s3-init-nx5tn"
Oct 31 07:41:48.583647 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:41:48.583622    4720 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-nx5tn"
Oct 31 07:41:48.951237 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:41:48.951209    4720 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kserve/s3-init-nx5tn"]
Oct 31 07:41:48.954745 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:41:48.954704    4720 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda54b3c28_6f13_4bee_8efc_5619d28a3b8c.slice/crio-700f1ea7a34f70672d873d7e783ac42225e47880c31c51365c12450bf4052dc7 WatchSource:0}: Error finding container 700f1ea7a34f70672d873d7e783ac42225e47880c31c51365c12450bf4052dc7: Status 404 returned error can't find the container with id 700f1ea7a34f70672d873d7e783ac42225e47880c31c51365c12450bf4052dc7
Oct 31 07:41:49.963977 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:41:49.963943    4720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-nx5tn" event={"ID":"a54b3c28-6f13-4bee-8efc-5619d28a3b8c","Type":"ContainerStarted","Data":"700f1ea7a34f70672d873d7e783ac42225e47880c31c51365c12450bf4052dc7"}
Oct 31 07:41:55.993151 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:41:55.993067    4720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-nx5tn" event={"ID":"a54b3c28-6f13-4bee-8efc-5619d28a3b8c","Type":"ContainerStarted","Data":"9abe5d6a0c2c930b490b74fbecb18667c34664a17419e79bef6acf1e8d952440"}
Oct 31 07:41:57.194166 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:41:57.194134    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 07:41:57.335903 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:41:57.335872    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 07:41:58.002199 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:41:58.002163    4720 generic.go:356] "Generic (PLEG): container finished" podID="a54b3c28-6f13-4bee-8efc-5619d28a3b8c" containerID="9abe5d6a0c2c930b490b74fbecb18667c34664a17419e79bef6acf1e8d952440" exitCode=0
Oct 31 07:41:58.002363 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:41:58.002237    4720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-nx5tn" event={"ID":"a54b3c28-6f13-4bee-8efc-5619d28a3b8c","Type":"ContainerDied","Data":"9abe5d6a0c2c930b490b74fbecb18667c34664a17419e79bef6acf1e8d952440"}
Oct 31 07:42:04.034084 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:42:04.034042    4720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-nx5tn" event={"ID":"a54b3c28-6f13-4bee-8efc-5619d28a3b8c","Type":"ContainerStarted","Data":"576cc19b4a4d322fb32cc797b06faaef4ce6c05e0f3416b0ba28de3944e56401"}
Oct 31 07:42:04.146897 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:42:04.146835    4720 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kserve/s3-init-nx5tn" podStartSLOduration=2.230925749 podStartE2EDuration="16.146817154s" totalImagesPullingTime="11.737466399s" totalInitContainerRuntime="2.178425006s" isStatefulPod=false podCreationTimestamp="2026-10-31 07:41:48 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 07:42:04.146281193 +0000 UTC m=+778.277349259" watchObservedRunningTime="2026-10-31 07:42:04.146817154 +0000 UTC m=+778.277885228"
Oct 31 07:42:13.461672 ip-10-0-139-83 kubenswrapper[4720]: E1031 07:42:13.461627    4720 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8m768" containerName="extractor"
Oct 31 07:42:25.022863 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:42:25.022827    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 07:42:43.460956 ip-10-0-139-83 kubenswrapper[4720]: E1031 07:42:43.460911    4720 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8m768" containerName="extractor"
Oct 31 07:42:55.021304 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:42:55.021269    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 07:42:56.938880 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:42:56.938845    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 07:42:57.556080 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:42:57.556048    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 07:43:05.839955 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:43:05.839920    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 07:43:08.166020 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:43:08.165993    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 07:43:08.310216 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:43:08.310183    4720 generic.go:356] "Generic (PLEG): container finished" podID="a54b3c28-6f13-4bee-8efc-5619d28a3b8c" containerID="576cc19b4a4d322fb32cc797b06faaef4ce6c05e0f3416b0ba28de3944e56401" exitCode=0
Oct 31 07:43:08.310388 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:43:08.310245    4720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-nx5tn" event={"ID":"a54b3c28-6f13-4bee-8efc-5619d28a3b8c","Type":"ContainerDied","Data":"576cc19b4a4d322fb32cc797b06faaef4ce6c05e0f3416b0ba28de3944e56401"}
Oct 31 07:43:08.820532 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:43:08.820497    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 07:43:09.444626 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:43:09.444600    4720 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-nx5tn"
Oct 31 07:43:09.460906 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:43:09.460880    4720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a54b3c28-6f13-4bee-8efc-5619d28a3b8c-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a54b3c28-6f13-4bee-8efc-5619d28a3b8c-model-cache\") pod \"a54b3c28-6f13-4bee-8efc-5619d28a3b8c\" (UID: \"a54b3c28-6f13-4bee-8efc-5619d28a3b8c\") "
Oct 31 07:43:09.461027 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:43:09.460955    4720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a54b3c28-6f13-4bee-8efc-5619d28a3b8c-kube-api-access-dnjn8\" (UniqueName: \"kubernetes.io/projected/a54b3c28-6f13-4bee-8efc-5619d28a3b8c-kube-api-access-dnjn8\") pod \"a54b3c28-6f13-4bee-8efc-5619d28a3b8c\" (UID: \"a54b3c28-6f13-4bee-8efc-5619d28a3b8c\") "
Oct 31 07:43:09.463108 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:43:09.463082    4720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a54b3c28-6f13-4bee-8efc-5619d28a3b8c-kube-api-access-dnjn8" pod "a54b3c28-6f13-4bee-8efc-5619d28a3b8c" (UID: "a54b3c28-6f13-4bee-8efc-5619d28a3b8c"). InnerVolumeSpecName "kube-api-access-dnjn8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 07:43:09.479590 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:43:09.479558    4720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a54b3c28-6f13-4bee-8efc-5619d28a3b8c-model-cache" pod "a54b3c28-6f13-4bee-8efc-5619d28a3b8c" (UID: "a54b3c28-6f13-4bee-8efc-5619d28a3b8c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 07:43:09.561459 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:43:09.561429    4720 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a54b3c28-6f13-4bee-8efc-5619d28a3b8c-model-cache\") on node \"ip-10-0-139-83.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 07:43:09.561459 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:43:09.561456    4720 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dnjn8\" (UniqueName: \"kubernetes.io/projected/a54b3c28-6f13-4bee-8efc-5619d28a3b8c-kube-api-access-dnjn8\") on node \"ip-10-0-139-83.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 07:43:10.320971 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:43:10.320890    4720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-nx5tn" event={"ID":"a54b3c28-6f13-4bee-8efc-5619d28a3b8c","Type":"ContainerDied","Data":"700f1ea7a34f70672d873d7e783ac42225e47880c31c51365c12450bf4052dc7"}
Oct 31 07:43:10.320971 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:43:10.320918    4720 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-nx5tn"
Oct 31 07:43:10.320971 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:43:10.320927    4720 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="700f1ea7a34f70672d873d7e783ac42225e47880c31c51365c12450bf4052dc7"
Oct 31 07:43:10.993733 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:43:10.993698    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 07:43:11.173606 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:43:11.173572    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 07:43:13.461459 ip-10-0-139-83 kubenswrapper[4720]: E1031 07:43:13.461421    4720 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8m768" containerName="extractor"
Oct 31 07:43:20.045006 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:43:20.044936    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 07:43:25.369724 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:43:25.369694    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 07:43:43.460510 ip-10-0-139-83 kubenswrapper[4720]: E1031 07:43:43.460410    4720 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8m768" containerName="extractor"
Oct 31 07:43:55.358815 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:43:55.358783    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 07:44:06.082996 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:44:06.082964    4720 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-4jszp_b1db0e3b-e85d-4769-8012-cf4b6f740d9d/console-operator/2.log"
Oct 31 07:44:06.084104 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:44:06.084080    4720 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-4jszp_b1db0e3b-e85d-4769-8012-cf4b6f740d9d/console-operator/2.log"
Oct 31 07:44:13.280998 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:44:13.280966    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 07:44:13.459367 ip-10-0-139-83 kubenswrapper[4720]: E1031 07:44:13.459314    4720 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8m768" containerName="extractor"
Oct 31 07:44:14.988688 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:44:14.988653    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 07:44:20.157488 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:44:20.157456    4720 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-test-cpu-accelerator-preset-applies-to-workload-8e506af7/cpu-accel-preset-kserve-5bf588fdc9-hjg94"]
Oct 31 07:44:20.157848 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:44:20.157761    4720 cpu_manager.go:414] "containerMap: removing container" podUID="a54b3c28-6f13-4bee-8efc-5619d28a3b8c" containerName="s3-init"
Oct 31 07:44:20.157848 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:44:20.157771    4720 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a54b3c28-6f13-4bee-8efc-5619d28a3b8c" containerName="s3-init"
Oct 31 07:44:20.157848 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:44:20.157802    4720 cpu_manager.go:414] "containerMap: removing container" podUID="a54b3c28-6f13-4bee-8efc-5619d28a3b8c" containerName="download-hf-model"
Oct 31 07:44:20.157848 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:44:20.157807    4720 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a54b3c28-6f13-4bee-8efc-5619d28a3b8c" containerName="download-hf-model"
Oct 31 07:44:20.157982 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:44:20.157860    4720 memory_manager.go:365] "RemoveStaleState removing state" podUID="a54b3c28-6f13-4bee-8efc-5619d28a3b8c" containerName="s3-init"
Oct 31 07:44:20.160828 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:44:20.160812    4720 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-cpu-accelerator-preset-applies-to-workload-8e506af7/cpu-accel-preset-kserve-5bf588fdc9-hjg94"
Oct 31 07:44:20.204800 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:44:20.204776    4720 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-cpu-accelerator-preset-applies-to-workload-8e506af7/cpu-accel-preset-kserve-5bf588fdc9-hjg94"]
Oct 31 07:44:20.220813 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:44:20.220790    4720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/274101ad-39fb-4fe4-9227-555be0aa17ff-model-cache\") pod \"cpu-accel-preset-kserve-5bf588fdc9-hjg94\" (UID: \"274101ad-39fb-4fe4-9227-555be0aa17ff\") " pod="e2e-test-cpu-accelerator-preset-applies-to-workload-8e506af7/cpu-accel-preset-kserve-5bf588fdc9-hjg94"
Oct 31 07:44:20.220942 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:44:20.220821    4720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/274101ad-39fb-4fe4-9227-555be0aa17ff-dshm\") pod \"cpu-accel-preset-kserve-5bf588fdc9-hjg94\" (UID: \"274101ad-39fb-4fe4-9227-555be0aa17ff\") " pod="e2e-test-cpu-accelerator-preset-applies-to-workload-8e506af7/cpu-accel-preset-kserve-5bf588fdc9-hjg94"
Oct 31 07:44:20.220942 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:44:20.220840    4720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/274101ad-39fb-4fe4-9227-555be0aa17ff-tmp-dir\") pod \"cpu-accel-preset-kserve-5bf588fdc9-hjg94\" (UID: \"274101ad-39fb-4fe4-9227-555be0aa17ff\") " pod="e2e-test-cpu-accelerator-preset-applies-to-workload-8e506af7/cpu-accel-preset-kserve-5bf588fdc9-hjg94"
Oct 31 07:44:20.220942 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:44:20.220864    4720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/274101ad-39fb-4fe4-9227-555be0aa17ff-kserve-provision-location\") pod \"cpu-accel-preset-kserve-5bf588fdc9-hjg94\" (UID: \"274101ad-39fb-4fe4-9227-555be0aa17ff\") " pod="e2e-test-cpu-accelerator-preset-applies-to-workload-8e506af7/cpu-accel-preset-kserve-5bf588fdc9-hjg94"
Oct 31 07:44:20.221043 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:44:20.220985    4720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rvpr9\" (UniqueName: \"kubernetes.io/projected/274101ad-39fb-4fe4-9227-555be0aa17ff-kube-api-access-rvpr9\") pod \"cpu-accel-preset-kserve-5bf588fdc9-hjg94\" (UID: \"274101ad-39fb-4fe4-9227-555be0aa17ff\") " pod="e2e-test-cpu-accelerator-preset-applies-to-workload-8e506af7/cpu-accel-preset-kserve-5bf588fdc9-hjg94"
Oct 31 07:44:20.221043 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:44:20.221022    4720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/274101ad-39fb-4fe4-9227-555be0aa17ff-home\") pod \"cpu-accel-preset-kserve-5bf588fdc9-hjg94\" (UID: \"274101ad-39fb-4fe4-9227-555be0aa17ff\") " pod="e2e-test-cpu-accelerator-preset-applies-to-workload-8e506af7/cpu-accel-preset-kserve-5bf588fdc9-hjg94"
Oct 31 07:44:20.221102 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:44:20.221051    4720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/274101ad-39fb-4fe4-9227-555be0aa17ff-tls-certs\") pod \"cpu-accel-preset-kserve-5bf588fdc9-hjg94\" (UID: \"274101ad-39fb-4fe4-9227-555be0aa17ff\") " pod="e2e-test-cpu-accelerator-preset-applies-to-workload-8e506af7/cpu-accel-preset-kserve-5bf588fdc9-hjg94"
Oct 31 07:44:20.248244 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:44:20.248220    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 07:44:20.252713 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:44:20.252689    4720 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-cpu-accelerator-preset-applies-to-workload-8e506af7\"/\"openshift-service-ca.crt\""
Oct 31 07:44:20.252713 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:44:20.252689    4720 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-cpu-accelerator-preset-applies-to-workload-8e506af7\"/\"kube-root-ca.crt\""
Oct 31 07:44:20.252866 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:44:20.252721    4720 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-cpu-accelerator-preset-applies-to-workload-8e506af7\"/\"seaweedfs-s3-creds\""
Oct 31 07:44:20.252866 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:44:20.252760    4720 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-cpu-accelerator-preset-applies-to-workload-8e506af7\"/\"default-dockercfg-zgnn8\""
Oct 31 07:44:20.252866 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:44:20.252855    4720 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-cpu-accelerator-preset-applies-to-workload-8e506af7\"/\"cpu-accel-preset-kserve-self-signed-certs\""
Oct 31 07:44:20.321981 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:44:20.321953    4720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rvpr9\" (UniqueName: \"kubernetes.io/projected/274101ad-39fb-4fe4-9227-555be0aa17ff-kube-api-access-rvpr9\") pod \"cpu-accel-preset-kserve-5bf588fdc9-hjg94\" (UID: \"274101ad-39fb-4fe4-9227-555be0aa17ff\") " pod="e2e-test-cpu-accelerator-preset-applies-to-workload-8e506af7/cpu-accel-preset-kserve-5bf588fdc9-hjg94"
Oct 31 07:44:20.321981 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:44:20.321984    4720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/274101ad-39fb-4fe4-9227-555be0aa17ff-home\") pod \"cpu-accel-preset-kserve-5bf588fdc9-hjg94\" (UID: \"274101ad-39fb-4fe4-9227-555be0aa17ff\") " pod="e2e-test-cpu-accelerator-preset-applies-to-workload-8e506af7/cpu-accel-preset-kserve-5bf588fdc9-hjg94"
Oct 31 07:44:20.322170 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:44:20.322007    4720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/274101ad-39fb-4fe4-9227-555be0aa17ff-tls-certs\") pod \"cpu-accel-preset-kserve-5bf588fdc9-hjg94\" (UID: \"274101ad-39fb-4fe4-9227-555be0aa17ff\") " pod="e2e-test-cpu-accelerator-preset-applies-to-workload-8e506af7/cpu-accel-preset-kserve-5bf588fdc9-hjg94"
Oct 31 07:44:20.322170 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:44:20.322031    4720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/274101ad-39fb-4fe4-9227-555be0aa17ff-model-cache\") pod \"cpu-accel-preset-kserve-5bf588fdc9-hjg94\" (UID: \"274101ad-39fb-4fe4-9227-555be0aa17ff\") " pod="e2e-test-cpu-accelerator-preset-applies-to-workload-8e506af7/cpu-accel-preset-kserve-5bf588fdc9-hjg94"
Oct 31 07:44:20.322170 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:44:20.322049    4720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/274101ad-39fb-4fe4-9227-555be0aa17ff-dshm\") pod \"cpu-accel-preset-kserve-5bf588fdc9-hjg94\" (UID: \"274101ad-39fb-4fe4-9227-555be0aa17ff\") " pod="e2e-test-cpu-accelerator-preset-applies-to-workload-8e506af7/cpu-accel-preset-kserve-5bf588fdc9-hjg94"
Oct 31 07:44:20.322170 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:44:20.322069    4720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/274101ad-39fb-4fe4-9227-555be0aa17ff-tmp-dir\") pod \"cpu-accel-preset-kserve-5bf588fdc9-hjg94\" (UID: \"274101ad-39fb-4fe4-9227-555be0aa17ff\") " pod="e2e-test-cpu-accelerator-preset-applies-to-workload-8e506af7/cpu-accel-preset-kserve-5bf588fdc9-hjg94"
Oct 31 07:44:20.322170 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:44:20.322095    4720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/274101ad-39fb-4fe4-9227-555be0aa17ff-kserve-provision-location\") pod \"cpu-accel-preset-kserve-5bf588fdc9-hjg94\" (UID: \"274101ad-39fb-4fe4-9227-555be0aa17ff\") " pod="e2e-test-cpu-accelerator-preset-applies-to-workload-8e506af7/cpu-accel-preset-kserve-5bf588fdc9-hjg94"
Oct 31 07:44:20.322553 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:44:20.322417    4720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/274101ad-39fb-4fe4-9227-555be0aa17ff-home\") pod \"cpu-accel-preset-kserve-5bf588fdc9-hjg94\" (UID: \"274101ad-39fb-4fe4-9227-555be0aa17ff\") " pod="e2e-test-cpu-accelerator-preset-applies-to-workload-8e506af7/cpu-accel-preset-kserve-5bf588fdc9-hjg94"
Oct 31 07:44:20.322553 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:44:20.322488    4720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/274101ad-39fb-4fe4-9227-555be0aa17ff-model-cache\") pod \"cpu-accel-preset-kserve-5bf588fdc9-hjg94\" (UID: \"274101ad-39fb-4fe4-9227-555be0aa17ff\") " pod="e2e-test-cpu-accelerator-preset-applies-to-workload-8e506af7/cpu-accel-preset-kserve-5bf588fdc9-hjg94"
Oct 31 07:44:20.322553 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:44:20.322515    4720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/274101ad-39fb-4fe4-9227-555be0aa17ff-kserve-provision-location\") pod \"cpu-accel-preset-kserve-5bf588fdc9-hjg94\" (UID: \"274101ad-39fb-4fe4-9227-555be0aa17ff\") " pod="e2e-test-cpu-accelerator-preset-applies-to-workload-8e506af7/cpu-accel-preset-kserve-5bf588fdc9-hjg94"
Oct 31 07:44:20.322660 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:44:20.322560    4720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/274101ad-39fb-4fe4-9227-555be0aa17ff-tmp-dir\") pod \"cpu-accel-preset-kserve-5bf588fdc9-hjg94\" (UID: \"274101ad-39fb-4fe4-9227-555be0aa17ff\") " pod="e2e-test-cpu-accelerator-preset-applies-to-workload-8e506af7/cpu-accel-preset-kserve-5bf588fdc9-hjg94"
Oct 31 07:44:20.324561 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:44:20.324540    4720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/274101ad-39fb-4fe4-9227-555be0aa17ff-dshm\") pod \"cpu-accel-preset-kserve-5bf588fdc9-hjg94\" (UID: \"274101ad-39fb-4fe4-9227-555be0aa17ff\") " pod="e2e-test-cpu-accelerator-preset-applies-to-workload-8e506af7/cpu-accel-preset-kserve-5bf588fdc9-hjg94"
Oct 31 07:44:20.324784 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:44:20.324769    4720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/274101ad-39fb-4fe4-9227-555be0aa17ff-tls-certs\") pod \"cpu-accel-preset-kserve-5bf588fdc9-hjg94\" (UID: \"274101ad-39fb-4fe4-9227-555be0aa17ff\") " pod="e2e-test-cpu-accelerator-preset-applies-to-workload-8e506af7/cpu-accel-preset-kserve-5bf588fdc9-hjg94"
Oct 31 07:44:20.385196 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:44:20.385175    4720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rvpr9\" (UniqueName: \"kubernetes.io/projected/274101ad-39fb-4fe4-9227-555be0aa17ff-kube-api-access-rvpr9\") pod \"cpu-accel-preset-kserve-5bf588fdc9-hjg94\" (UID: \"274101ad-39fb-4fe4-9227-555be0aa17ff\") " pod="e2e-test-cpu-accelerator-preset-applies-to-workload-8e506af7/cpu-accel-preset-kserve-5bf588fdc9-hjg94"
Oct 31 07:44:20.436864 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:44:20.436813    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 07:44:20.475679 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:44:20.475658    4720 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-cpu-accelerator-preset-applies-to-workload-8e506af7/cpu-accel-preset-kserve-5bf588fdc9-hjg94"
Oct 31 07:44:20.896897 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:44:20.896869    4720 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-cpu-accelerator-preset-applies-to-workload-8e506af7/cpu-accel-preset-kserve-5bf588fdc9-hjg94"]
Oct 31 07:44:20.899775 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:44:20.899749    4720 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod274101ad_39fb_4fe4_9227_555be0aa17ff.slice/crio-94f1d98dc65dce9c603a8241380d01640a49cd6dac734fb546cb12263329a7f7 WatchSource:0}: Error finding container 94f1d98dc65dce9c603a8241380d01640a49cd6dac734fb546cb12263329a7f7: Status 404 returned error can't find the container with id 94f1d98dc65dce9c603a8241380d01640a49cd6dac734fb546cb12263329a7f7
Oct 31 07:44:21.622668 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:44:21.622634    4720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-cpu-accelerator-preset-applies-to-workload-8e506af7/cpu-accel-preset-kserve-5bf588fdc9-hjg94" event={"ID":"274101ad-39fb-4fe4-9227-555be0aa17ff","Type":"ContainerStarted","Data":"988725a71a125ae77758108ab88c9e8adb3317075dd688478afb892d286c59ef"}
Oct 31 07:44:21.622668 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:44:21.622671    4720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-cpu-accelerator-preset-applies-to-workload-8e506af7/cpu-accel-preset-kserve-5bf588fdc9-hjg94" event={"ID":"274101ad-39fb-4fe4-9227-555be0aa17ff","Type":"ContainerStarted","Data":"94f1d98dc65dce9c603a8241380d01640a49cd6dac734fb546cb12263329a7f7"}
Oct 31 07:44:25.639824 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:44:25.639793    4720 generic.go:356] "Generic (PLEG): container finished" podID="274101ad-39fb-4fe4-9227-555be0aa17ff" containerID="988725a71a125ae77758108ab88c9e8adb3317075dd688478afb892d286c59ef" exitCode=0
Oct 31 07:44:25.640125 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:44:25.639836    4720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-cpu-accelerator-preset-applies-to-workload-8e506af7/cpu-accel-preset-kserve-5bf588fdc9-hjg94" event={"ID":"274101ad-39fb-4fe4-9227-555be0aa17ff","Type":"ContainerDied","Data":"988725a71a125ae77758108ab88c9e8adb3317075dd688478afb892d286c59ef"}
Oct 31 07:44:25.695818 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:44:25.695792    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 07:44:32.395083 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:44:32.395043    4720 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-cpu-accelerator-preset-applies-to-workload-8e506af7/cpu-accel-preset-kserve-5bf588fdc9-hjg94"]
Oct 31 07:44:32.412224 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:44:32.412160    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 07:44:43.467457 ip-10-0-139-83 kubenswrapper[4720]: E1031 07:44:43.467410    4720 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8m768" containerName="extractor"
Oct 31 07:44:47.535136 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:44:47.535053    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 07:44:47.722615 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:44:47.722589    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 07:44:55.723930 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:44:55.723887    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 07:45:01.335405 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:45:01.335365    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 07:45:13.476771 ip-10-0-139-83 kubenswrapper[4720]: E1031 07:45:13.476723    4720 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8m768" containerName="extractor"
Oct 31 07:45:13.892882 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:45:13.892853    4720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-cpu-accelerator-preset-applies-to-workload-8e506af7/cpu-accel-preset-kserve-5bf588fdc9-hjg94" event={"ID":"274101ad-39fb-4fe4-9227-555be0aa17ff","Type":"ContainerStarted","Data":"469e659fa975840b6aa7bc739e25768c9ff381f68ed7f0a64f5188bd05d3bb4e"}
Oct 31 07:45:14.109214 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:45:14.109150    4720 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-test-cpu-accelerator-preset-applies-to-workload-8e506af7/cpu-accel-preset-kserve-5bf588fdc9-hjg94" podStartSLOduration=2.1871022 podStartE2EDuration="54.109137333s" totalImagesPullingTime="47.335809524s" totalInitContainerRuntime="4.586225609s" isStatefulPod=false podCreationTimestamp="2026-10-31 07:44:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 07:45:14.058827581 +0000 UTC m=+968.189895651" watchObservedRunningTime="2026-10-31 07:45:14.109137333 +0000 UTC m=+968.240205399"
Oct 31 07:45:14.475824 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:45:14.475793    4720 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-cpu-accelerator-preset-applies-to-workload-8e506af7/cpu-accel-preset-kserve-5bf588fdc9-hjg94"
Oct 31 07:45:20.703833 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:45:20.703797    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 07:45:26.021963 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:45:26.021927    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 07:45:28.912351 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:45:28.912273    4720 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-test-cpu-accelerator-preset-applies-to-workload-8e506af7/cpu-accel-preset-kserve-5bf588fdc9-hjg94" podUID="274101ad-39fb-4fe4-9227-555be0aa17ff" containerName="main" containerID="cri-o://469e659fa975840b6aa7bc739e25768c9ff381f68ed7f0a64f5188bd05d3bb4e" gracePeriod=45
Oct 31 07:45:43.460441 ip-10-0-139-83 kubenswrapper[4720]: E1031 07:45:43.460401    4720 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8m768" containerName="extractor"
Oct 31 07:45:56.034589 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:45:56.034563    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 07:46:13.459155 ip-10-0-139-83 kubenswrapper[4720]: E1031 07:46:13.459111    4720 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8m768" containerName="extractor"
Oct 31 07:46:14.159157 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:46:14.159131    4720 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-cpu-accelerator-preset-applies-to-workload-8e506af7_cpu-accel-preset-kserve-5bf588fdc9-hjg94_274101ad-39fb-4fe4-9227-555be0aa17ff/main/0.log"
Oct 31 07:46:14.159527 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:46:14.159507    4720 generic.go:356] "Generic (PLEG): container finished" podID="274101ad-39fb-4fe4-9227-555be0aa17ff" containerID="469e659fa975840b6aa7bc739e25768c9ff381f68ed7f0a64f5188bd05d3bb4e" exitCode=137
Oct 31 07:46:14.159588 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:46:14.159564    4720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-cpu-accelerator-preset-applies-to-workload-8e506af7/cpu-accel-preset-kserve-5bf588fdc9-hjg94" event={"ID":"274101ad-39fb-4fe4-9227-555be0aa17ff","Type":"ContainerDied","Data":"469e659fa975840b6aa7bc739e25768c9ff381f68ed7f0a64f5188bd05d3bb4e"}
Oct 31 07:46:14.550045 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:46:14.550024    4720 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-cpu-accelerator-preset-applies-to-workload-8e506af7_cpu-accel-preset-kserve-5bf588fdc9-hjg94_274101ad-39fb-4fe4-9227-555be0aa17ff/main/0.log"
Oct 31 07:46:14.551362 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:46:14.551345    4720 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-cpu-accelerator-preset-applies-to-workload-8e506af7/cpu-accel-preset-kserve-5bf588fdc9-hjg94"
Oct 31 07:46:14.660495 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:46:14.660471    4720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/274101ad-39fb-4fe4-9227-555be0aa17ff-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/274101ad-39fb-4fe4-9227-555be0aa17ff-model-cache\") pod \"274101ad-39fb-4fe4-9227-555be0aa17ff\" (UID: \"274101ad-39fb-4fe4-9227-555be0aa17ff\") "
Oct 31 07:46:14.660610 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:46:14.660505    4720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/274101ad-39fb-4fe4-9227-555be0aa17ff-home\" (UniqueName: \"kubernetes.io/empty-dir/274101ad-39fb-4fe4-9227-555be0aa17ff-home\") pod \"274101ad-39fb-4fe4-9227-555be0aa17ff\" (UID: \"274101ad-39fb-4fe4-9227-555be0aa17ff\") "
Oct 31 07:46:14.660610 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:46:14.660524    4720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/274101ad-39fb-4fe4-9227-555be0aa17ff-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/274101ad-39fb-4fe4-9227-555be0aa17ff-tmp-dir\") pod \"274101ad-39fb-4fe4-9227-555be0aa17ff\" (UID: \"274101ad-39fb-4fe4-9227-555be0aa17ff\") "
Oct 31 07:46:14.660610 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:46:14.660570    4720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/274101ad-39fb-4fe4-9227-555be0aa17ff-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/274101ad-39fb-4fe4-9227-555be0aa17ff-kserve-provision-location\") pod \"274101ad-39fb-4fe4-9227-555be0aa17ff\" (UID: \"274101ad-39fb-4fe4-9227-555be0aa17ff\") "
Oct 31 07:46:14.660610 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:46:14.660593    4720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/274101ad-39fb-4fe4-9227-555be0aa17ff-dshm\" (UniqueName: \"kubernetes.io/empty-dir/274101ad-39fb-4fe4-9227-555be0aa17ff-dshm\") pod \"274101ad-39fb-4fe4-9227-555be0aa17ff\" (UID: \"274101ad-39fb-4fe4-9227-555be0aa17ff\") "
Oct 31 07:46:14.660814 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:46:14.660624    4720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/274101ad-39fb-4fe4-9227-555be0aa17ff-tls-certs\" (UniqueName: \"kubernetes.io/secret/274101ad-39fb-4fe4-9227-555be0aa17ff-tls-certs\") pod \"274101ad-39fb-4fe4-9227-555be0aa17ff\" (UID: \"274101ad-39fb-4fe4-9227-555be0aa17ff\") "
Oct 31 07:46:14.660814 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:46:14.660659    4720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/274101ad-39fb-4fe4-9227-555be0aa17ff-kube-api-access-rvpr9\" (UniqueName: \"kubernetes.io/projected/274101ad-39fb-4fe4-9227-555be0aa17ff-kube-api-access-rvpr9\") pod \"274101ad-39fb-4fe4-9227-555be0aa17ff\" (UID: \"274101ad-39fb-4fe4-9227-555be0aa17ff\") "
Oct 31 07:46:14.660995 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:46:14.660965    4720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/274101ad-39fb-4fe4-9227-555be0aa17ff-home" pod "274101ad-39fb-4fe4-9227-555be0aa17ff" (UID: "274101ad-39fb-4fe4-9227-555be0aa17ff"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 07:46:14.661121 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:46:14.661099    4720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/274101ad-39fb-4fe4-9227-555be0aa17ff-model-cache" pod "274101ad-39fb-4fe4-9227-555be0aa17ff" (UID: "274101ad-39fb-4fe4-9227-555be0aa17ff"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 07:46:14.662987 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:46:14.662955    4720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/274101ad-39fb-4fe4-9227-555be0aa17ff-kube-api-access-rvpr9" pod "274101ad-39fb-4fe4-9227-555be0aa17ff" (UID: "274101ad-39fb-4fe4-9227-555be0aa17ff"). InnerVolumeSpecName "kube-api-access-rvpr9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 07:46:14.663250 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:46:14.663225    4720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/274101ad-39fb-4fe4-9227-555be0aa17ff-tls-certs" pod "274101ad-39fb-4fe4-9227-555be0aa17ff" (UID: "274101ad-39fb-4fe4-9227-555be0aa17ff"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 07:46:14.663316 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:46:14.663225    4720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/274101ad-39fb-4fe4-9227-555be0aa17ff-dshm" pod "274101ad-39fb-4fe4-9227-555be0aa17ff" (UID: "274101ad-39fb-4fe4-9227-555be0aa17ff"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 07:46:14.672009 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:46:14.671983    4720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/274101ad-39fb-4fe4-9227-555be0aa17ff-tmp-dir" pod "274101ad-39fb-4fe4-9227-555be0aa17ff" (UID: "274101ad-39fb-4fe4-9227-555be0aa17ff"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 07:46:14.713207 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:46:14.713156    4720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/274101ad-39fb-4fe4-9227-555be0aa17ff-kserve-provision-location" pod "274101ad-39fb-4fe4-9227-555be0aa17ff" (UID: "274101ad-39fb-4fe4-9227-555be0aa17ff"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 07:46:14.761375 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:46:14.761350    4720 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/274101ad-39fb-4fe4-9227-555be0aa17ff-tls-certs\") on node \"ip-10-0-139-83.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 07:46:14.761477 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:46:14.761375    4720 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rvpr9\" (UniqueName: \"kubernetes.io/projected/274101ad-39fb-4fe4-9227-555be0aa17ff-kube-api-access-rvpr9\") on node \"ip-10-0-139-83.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 07:46:14.761477 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:46:14.761389    4720 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/274101ad-39fb-4fe4-9227-555be0aa17ff-model-cache\") on node \"ip-10-0-139-83.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 07:46:14.761477 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:46:14.761403    4720 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/274101ad-39fb-4fe4-9227-555be0aa17ff-home\") on node \"ip-10-0-139-83.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 07:46:14.761477 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:46:14.761414    4720 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/274101ad-39fb-4fe4-9227-555be0aa17ff-tmp-dir\") on node \"ip-10-0-139-83.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 07:46:14.761477 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:46:14.761426    4720 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/274101ad-39fb-4fe4-9227-555be0aa17ff-kserve-provision-location\") on node \"ip-10-0-139-83.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 07:46:14.761477 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:46:14.761445    4720 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/274101ad-39fb-4fe4-9227-555be0aa17ff-dshm\") on node \"ip-10-0-139-83.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 07:46:15.165008 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:46:15.164946    4720 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-cpu-accelerator-preset-applies-to-workload-8e506af7_cpu-accel-preset-kserve-5bf588fdc9-hjg94_274101ad-39fb-4fe4-9227-555be0aa17ff/main/0.log"
Oct 31 07:46:15.165347 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:46:15.165306    4720 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-cpu-accelerator-preset-applies-to-workload-8e506af7/cpu-accel-preset-kserve-5bf588fdc9-hjg94"
Oct 31 07:46:15.165347 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:46:15.165318    4720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-cpu-accelerator-preset-applies-to-workload-8e506af7/cpu-accel-preset-kserve-5bf588fdc9-hjg94" event={"ID":"274101ad-39fb-4fe4-9227-555be0aa17ff","Type":"ContainerDied","Data":"94f1d98dc65dce9c603a8241380d01640a49cd6dac734fb546cb12263329a7f7"}
Oct 31 07:46:15.165516 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:46:15.165381    4720 scope.go:122] "RemoveContainer" containerID="469e659fa975840b6aa7bc739e25768c9ff381f68ed7f0a64f5188bd05d3bb4e"
Oct 31 07:46:15.174088 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:46:15.174072    4720 scope.go:122] "RemoveContainer" containerID="988725a71a125ae77758108ab88c9e8adb3317075dd688478afb892d286c59ef"
Oct 31 07:46:15.327194 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:46:15.327171    4720 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-cpu-accelerator-preset-applies-to-workload-8e506af7/cpu-accel-preset-kserve-5bf588fdc9-hjg94"]
Oct 31 07:46:15.376715 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:46:15.376692    4720 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-test-cpu-accelerator-preset-applies-to-workload-8e506af7/cpu-accel-preset-kserve-5bf588fdc9-hjg94"]
Oct 31 07:46:15.491729 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:46:15.491677    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 07:46:15.649507 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:46:15.649480    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 07:46:15.798167 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:46:15.798112    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 07:46:16.024253 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:46:16.024229    4720 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="274101ad-39fb-4fe4-9227-555be0aa17ff" path="/var/lib/kubelet/pods/274101ad-39fb-4fe4-9227-555be0aa17ff/volumes"
Oct 31 07:46:21.040204 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:46:21.040174    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 07:46:21.189500 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:46:21.189471    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 07:46:22.040735 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:46:22.040706    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 07:46:22.205387 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:46:22.205356    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 07:46:23.625547 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:46:23.625518    4720 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-test-cpu-accelerator-preset-user-image-wins-06960b42/cpu-accel-precedence-kserve-7489f44cbc-njf8c"]
Oct 31 07:46:23.625909 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:46:23.625844    4720 cpu_manager.go:414] "containerMap: removing container" podUID="274101ad-39fb-4fe4-9227-555be0aa17ff" containerName="storage-initializer"
Oct 31 07:46:23.625909 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:46:23.625854    4720 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="274101ad-39fb-4fe4-9227-555be0aa17ff" containerName="storage-initializer"
Oct 31 07:46:23.625909 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:46:23.625863    4720 cpu_manager.go:414] "containerMap: removing container" podUID="274101ad-39fb-4fe4-9227-555be0aa17ff" containerName="main"
Oct 31 07:46:23.625909 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:46:23.625868    4720 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="274101ad-39fb-4fe4-9227-555be0aa17ff" containerName="main"
Oct 31 07:46:23.626033 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:46:23.625921    4720 memory_manager.go:365] "RemoveStaleState removing state" podUID="274101ad-39fb-4fe4-9227-555be0aa17ff" containerName="main"
Oct 31 07:46:23.627623 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:46:23.627597    4720 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-cpu-accelerator-preset-user-image-wins-06960b42/cpu-accel-precedence-kserve-7489f44cbc-njf8c"
Oct 31 07:46:23.680844 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:46:23.680809    4720 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-cpu-accelerator-preset-user-image-wins-06960b42/cpu-accel-precedence-kserve-7489f44cbc-njf8c"]
Oct 31 07:46:23.696294 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:46:23.696267    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 07:46:23.723511 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:46:23.723487    4720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/db2b767d-e8f8-4664-9aa4-5d9a1aff0e2f-dshm\") pod \"cpu-accel-precedence-kserve-7489f44cbc-njf8c\" (UID: \"db2b767d-e8f8-4664-9aa4-5d9a1aff0e2f\") " pod="e2e-test-cpu-accelerator-preset-user-image-wins-06960b42/cpu-accel-precedence-kserve-7489f44cbc-njf8c"
Oct 31 07:46:23.723612 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:46:23.723521    4720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/db2b767d-e8f8-4664-9aa4-5d9a1aff0e2f-home\") pod \"cpu-accel-precedence-kserve-7489f44cbc-njf8c\" (UID: \"db2b767d-e8f8-4664-9aa4-5d9a1aff0e2f\") " pod="e2e-test-cpu-accelerator-preset-user-image-wins-06960b42/cpu-accel-precedence-kserve-7489f44cbc-njf8c"
Oct 31 07:46:23.723612 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:46:23.723545    4720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/db2b767d-e8f8-4664-9aa4-5d9a1aff0e2f-kserve-provision-location\") pod \"cpu-accel-precedence-kserve-7489f44cbc-njf8c\" (UID: \"db2b767d-e8f8-4664-9aa4-5d9a1aff0e2f\") " pod="e2e-test-cpu-accelerator-preset-user-image-wins-06960b42/cpu-accel-precedence-kserve-7489f44cbc-njf8c"
Oct 31 07:46:23.723688 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:46:23.723654    4720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/db2b767d-e8f8-4664-9aa4-5d9a1aff0e2f-tls-certs\") pod \"cpu-accel-precedence-kserve-7489f44cbc-njf8c\" (UID: \"db2b767d-e8f8-4664-9aa4-5d9a1aff0e2f\") " pod="e2e-test-cpu-accelerator-preset-user-image-wins-06960b42/cpu-accel-precedence-kserve-7489f44cbc-njf8c"
Oct 31 07:46:23.723717 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:46:23.723686    4720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9fncj\" (UniqueName: \"kubernetes.io/projected/db2b767d-e8f8-4664-9aa4-5d9a1aff0e2f-kube-api-access-9fncj\") pod \"cpu-accel-precedence-kserve-7489f44cbc-njf8c\" (UID: \"db2b767d-e8f8-4664-9aa4-5d9a1aff0e2f\") " pod="e2e-test-cpu-accelerator-preset-user-image-wins-06960b42/cpu-accel-precedence-kserve-7489f44cbc-njf8c"
Oct 31 07:46:23.723717 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:46:23.723707    4720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/db2b767d-e8f8-4664-9aa4-5d9a1aff0e2f-model-cache\") pod \"cpu-accel-precedence-kserve-7489f44cbc-njf8c\" (UID: \"db2b767d-e8f8-4664-9aa4-5d9a1aff0e2f\") " pod="e2e-test-cpu-accelerator-preset-user-image-wins-06960b42/cpu-accel-precedence-kserve-7489f44cbc-njf8c"
Oct 31 07:46:23.723775 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:46:23.723760    4720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/db2b767d-e8f8-4664-9aa4-5d9a1aff0e2f-tmp-dir\") pod \"cpu-accel-precedence-kserve-7489f44cbc-njf8c\" (UID: \"db2b767d-e8f8-4664-9aa4-5d9a1aff0e2f\") " pod="e2e-test-cpu-accelerator-preset-user-image-wins-06960b42/cpu-accel-precedence-kserve-7489f44cbc-njf8c"
Oct 31 07:46:23.728616 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:46:23.728597    4720 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-cpu-accelerator-preset-user-image-wins-06960b42\"/\"seaweedfs-s3-creds\""
Oct 31 07:46:23.728682 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:46:23.728619    4720 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-cpu-accelerator-preset-user-image-wins-06960b42\"/\"openshift-service-ca.crt\""
Oct 31 07:46:23.728682 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:46:23.728625    4720 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-cpu-accelerator-preset-user-image-wins-06960b42\"/\"kube-root-ca.crt\""
Oct 31 07:46:23.728682 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:46:23.728651    4720 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-cpu-accelerator-preset-user-image-wins-06960b42\"/\"default-dockercfg-5jpqt\""
Oct 31 07:46:23.728946 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:46:23.728813    4720 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-cpu-accelerator-preset-user-image-wins-06960b42\"/\"cpu-accel-precedence-kserve-self-signed-certs\""
Oct 31 07:46:23.824962 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:46:23.824941    4720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/db2b767d-e8f8-4664-9aa4-5d9a1aff0e2f-kserve-provision-location\") pod \"cpu-accel-precedence-kserve-7489f44cbc-njf8c\" (UID: \"db2b767d-e8f8-4664-9aa4-5d9a1aff0e2f\") " pod="e2e-test-cpu-accelerator-preset-user-image-wins-06960b42/cpu-accel-precedence-kserve-7489f44cbc-njf8c"
Oct 31 07:46:23.825078 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:46:23.824980    4720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/db2b767d-e8f8-4664-9aa4-5d9a1aff0e2f-tls-certs\") pod \"cpu-accel-precedence-kserve-7489f44cbc-njf8c\" (UID: \"db2b767d-e8f8-4664-9aa4-5d9a1aff0e2f\") " pod="e2e-test-cpu-accelerator-preset-user-image-wins-06960b42/cpu-accel-precedence-kserve-7489f44cbc-njf8c"
Oct 31 07:46:23.825078 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:46:23.825009    4720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9fncj\" (UniqueName: \"kubernetes.io/projected/db2b767d-e8f8-4664-9aa4-5d9a1aff0e2f-kube-api-access-9fncj\") pod \"cpu-accel-precedence-kserve-7489f44cbc-njf8c\" (UID: \"db2b767d-e8f8-4664-9aa4-5d9a1aff0e2f\") " pod="e2e-test-cpu-accelerator-preset-user-image-wins-06960b42/cpu-accel-precedence-kserve-7489f44cbc-njf8c"
Oct 31 07:46:23.825078 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:46:23.825039    4720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/db2b767d-e8f8-4664-9aa4-5d9a1aff0e2f-model-cache\") pod \"cpu-accel-precedence-kserve-7489f44cbc-njf8c\" (UID: \"db2b767d-e8f8-4664-9aa4-5d9a1aff0e2f\") " pod="e2e-test-cpu-accelerator-preset-user-image-wins-06960b42/cpu-accel-precedence-kserve-7489f44cbc-njf8c"
Oct 31 07:46:23.825231 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:46:23.825084    4720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/db2b767d-e8f8-4664-9aa4-5d9a1aff0e2f-tmp-dir\") pod \"cpu-accel-precedence-kserve-7489f44cbc-njf8c\" (UID: \"db2b767d-e8f8-4664-9aa4-5d9a1aff0e2f\") " pod="e2e-test-cpu-accelerator-preset-user-image-wins-06960b42/cpu-accel-precedence-kserve-7489f44cbc-njf8c"
Oct 31 07:46:23.825231 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:46:23.825150    4720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/db2b767d-e8f8-4664-9aa4-5d9a1aff0e2f-dshm\") pod \"cpu-accel-precedence-kserve-7489f44cbc-njf8c\" (UID: \"db2b767d-e8f8-4664-9aa4-5d9a1aff0e2f\") " pod="e2e-test-cpu-accelerator-preset-user-image-wins-06960b42/cpu-accel-precedence-kserve-7489f44cbc-njf8c"
Oct 31 07:46:23.825231 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:46:23.825188    4720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/db2b767d-e8f8-4664-9aa4-5d9a1aff0e2f-home\") pod \"cpu-accel-precedence-kserve-7489f44cbc-njf8c\" (UID: \"db2b767d-e8f8-4664-9aa4-5d9a1aff0e2f\") " pod="e2e-test-cpu-accelerator-preset-user-image-wins-06960b42/cpu-accel-precedence-kserve-7489f44cbc-njf8c"
Oct 31 07:46:23.825409 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:46:23.825390    4720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/db2b767d-e8f8-4664-9aa4-5d9a1aff0e2f-kserve-provision-location\") pod \"cpu-accel-precedence-kserve-7489f44cbc-njf8c\" (UID: \"db2b767d-e8f8-4664-9aa4-5d9a1aff0e2f\") " pod="e2e-test-cpu-accelerator-preset-user-image-wins-06960b42/cpu-accel-precedence-kserve-7489f44cbc-njf8c"
Oct 31 07:46:23.825472 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:46:23.825420    4720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/db2b767d-e8f8-4664-9aa4-5d9a1aff0e2f-model-cache\") pod \"cpu-accel-precedence-kserve-7489f44cbc-njf8c\" (UID: \"db2b767d-e8f8-4664-9aa4-5d9a1aff0e2f\") " pod="e2e-test-cpu-accelerator-preset-user-image-wins-06960b42/cpu-accel-precedence-kserve-7489f44cbc-njf8c"
Oct 31 07:46:23.825537 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:46:23.825519    4720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/db2b767d-e8f8-4664-9aa4-5d9a1aff0e2f-home\") pod \"cpu-accel-precedence-kserve-7489f44cbc-njf8c\" (UID: \"db2b767d-e8f8-4664-9aa4-5d9a1aff0e2f\") " pod="e2e-test-cpu-accelerator-preset-user-image-wins-06960b42/cpu-accel-precedence-kserve-7489f44cbc-njf8c"
Oct 31 07:46:23.825614 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:46:23.825595    4720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/db2b767d-e8f8-4664-9aa4-5d9a1aff0e2f-tmp-dir\") pod \"cpu-accel-precedence-kserve-7489f44cbc-njf8c\" (UID: \"db2b767d-e8f8-4664-9aa4-5d9a1aff0e2f\") " pod="e2e-test-cpu-accelerator-preset-user-image-wins-06960b42/cpu-accel-precedence-kserve-7489f44cbc-njf8c"
Oct 31 07:46:23.827383 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:46:23.827364    4720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/db2b767d-e8f8-4664-9aa4-5d9a1aff0e2f-dshm\") pod \"cpu-accel-precedence-kserve-7489f44cbc-njf8c\" (UID: \"db2b767d-e8f8-4664-9aa4-5d9a1aff0e2f\") " pod="e2e-test-cpu-accelerator-preset-user-image-wins-06960b42/cpu-accel-precedence-kserve-7489f44cbc-njf8c"
Oct 31 07:46:23.827530 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:46:23.827515    4720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/db2b767d-e8f8-4664-9aa4-5d9a1aff0e2f-tls-certs\") pod \"cpu-accel-precedence-kserve-7489f44cbc-njf8c\" (UID: \"db2b767d-e8f8-4664-9aa4-5d9a1aff0e2f\") " pod="e2e-test-cpu-accelerator-preset-user-image-wins-06960b42/cpu-accel-precedence-kserve-7489f44cbc-njf8c"
Oct 31 07:46:23.881286 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:46:23.881236    4720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9fncj\" (UniqueName: \"kubernetes.io/projected/db2b767d-e8f8-4664-9aa4-5d9a1aff0e2f-kube-api-access-9fncj\") pod \"cpu-accel-precedence-kserve-7489f44cbc-njf8c\" (UID: \"db2b767d-e8f8-4664-9aa4-5d9a1aff0e2f\") " pod="e2e-test-cpu-accelerator-preset-user-image-wins-06960b42/cpu-accel-precedence-kserve-7489f44cbc-njf8c"
Oct 31 07:46:23.945309 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:46:23.945288    4720 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-cpu-accelerator-preset-user-image-wins-06960b42/cpu-accel-precedence-kserve-7489f44cbc-njf8c"
Oct 31 07:46:24.366383 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:46:24.366357    4720 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-cpu-accelerator-preset-user-image-wins-06960b42/cpu-accel-precedence-kserve-7489f44cbc-njf8c"]
Oct 31 07:46:24.369145 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:46:24.369117    4720 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb2b767d_e8f8_4664_9aa4_5d9a1aff0e2f.slice/crio-409cc5bccc07a129cf8978c24b91885924ba55270f98c6144c9b844545655e2c WatchSource:0}: Error finding container 409cc5bccc07a129cf8978c24b91885924ba55270f98c6144c9b844545655e2c: Status 404 returned error can't find the container with id 409cc5bccc07a129cf8978c24b91885924ba55270f98c6144c9b844545655e2c
Oct 31 07:46:25.212148 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:46:25.212112    4720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-cpu-accelerator-preset-user-image-wins-06960b42/cpu-accel-precedence-kserve-7489f44cbc-njf8c" event={"ID":"db2b767d-e8f8-4664-9aa4-5d9a1aff0e2f","Type":"ContainerStarted","Data":"f440014f4463a716df8d42e6652f4ea9cbc7b70d91d6436964cc137a66bffa78"}
Oct 31 07:46:25.212533 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:46:25.212156    4720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-cpu-accelerator-preset-user-image-wins-06960b42/cpu-accel-precedence-kserve-7489f44cbc-njf8c" event={"ID":"db2b767d-e8f8-4664-9aa4-5d9a1aff0e2f","Type":"ContainerStarted","Data":"409cc5bccc07a129cf8978c24b91885924ba55270f98c6144c9b844545655e2c"}
Oct 31 07:46:26.360313 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:46:26.360281    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 07:46:30.238383 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:46:30.238350    4720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-cpu-accelerator-preset-user-image-wins-06960b42/cpu-accel-precedence-kserve-7489f44cbc-njf8c" event={"ID":"db2b767d-e8f8-4664-9aa4-5d9a1aff0e2f","Type":"ContainerDied","Data":"f440014f4463a716df8d42e6652f4ea9cbc7b70d91d6436964cc137a66bffa78"}
Oct 31 07:46:30.238737 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:46:30.238349    4720 generic.go:356] "Generic (PLEG): container finished" podID="db2b767d-e8f8-4664-9aa4-5d9a1aff0e2f" containerID="f440014f4463a716df8d42e6652f4ea9cbc7b70d91d6436964cc137a66bffa78" exitCode=0
Oct 31 07:46:30.239729 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:46:30.239716    4720 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 31 07:46:33.278181 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:46:33.278113    4720 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-cpu-accelerator-preset-user-image-wins-06960b42/cpu-accel-precedence-kserve-7489f44cbc-njf8c"]
Oct 31 07:46:33.300503 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:46:33.300475    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 07:46:43.464891 ip-10-0-139-83 kubenswrapper[4720]: E1031 07:46:43.464858    4720 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8m768" containerName="extractor"
Oct 31 07:46:44.823080 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:46:44.823045    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 07:46:45.427637 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:46:45.427607    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 07:46:56.367873 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:46:56.367844    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 07:46:58.288518 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:46:58.288486    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 07:46:58.395403 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:46:58.395361    4720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-cpu-accelerator-preset-user-image-wins-06960b42/cpu-accel-precedence-kserve-7489f44cbc-njf8c" event={"ID":"db2b767d-e8f8-4664-9aa4-5d9a1aff0e2f","Type":"ContainerStarted","Data":"bedc45f5d1c98681b63e0c06513ed91369aec36c6a6b5bd272f45280d27698b6"}
Oct 31 07:46:58.610808 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:46:58.610751    4720 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-test-cpu-accelerator-preset-user-image-wins-06960b42/cpu-accel-precedence-kserve-7489f44cbc-njf8c" podStartSLOduration=2.906702971 podStartE2EDuration="35.610732649s" totalImagesPullingTime="27.800284336s" totalInitContainerRuntime="4.903745342s" isStatefulPod=false podCreationTimestamp="2026-10-31 07:46:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 07:46:58.560937435 +0000 UTC m=+1072.692005523" watchObservedRunningTime="2026-10-31 07:46:58.610732649 +0000 UTC m=+1072.741800721"
Oct 31 07:46:58.945424 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:46:58.945380    4720 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-cpu-accelerator-preset-user-image-wins-06960b42/cpu-accel-precedence-kserve-7489f44cbc-njf8c"
Oct 31 07:47:13.418306 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:47:13.418235    4720 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-test-cpu-accelerator-preset-user-image-wins-06960b42/cpu-accel-precedence-kserve-7489f44cbc-njf8c" podUID="db2b767d-e8f8-4664-9aa4-5d9a1aff0e2f" containerName="main" containerID="cri-o://bedc45f5d1c98681b63e0c06513ed91369aec36c6a6b5bd272f45280d27698b6" gracePeriod=45
Oct 31 07:47:13.461633 ip-10-0-139-83 kubenswrapper[4720]: E1031 07:47:13.461596    4720 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8m768" containerName="extractor"
Oct 31 07:47:21.376089 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:47:21.376049    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 07:47:25.874520 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:47:25.874483    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 07:47:26.023551 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:47:26.023519    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 07:47:36.716262 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:47:36.716224    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 07:47:43.460885 ip-10-0-139-83 kubenswrapper[4720]: E1031 07:47:43.460847    4720 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8m768" containerName="extractor"
Oct 31 07:47:58.664499 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:47:58.664463    4720 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-cpu-accelerator-preset-user-image-wins-06960b42_cpu-accel-precedence-kserve-7489f44cbc-njf8c_db2b767d-e8f8-4664-9aa4-5d9a1aff0e2f/main/0.log"
Oct 31 07:47:58.664928 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:47:58.664827    4720 generic.go:356] "Generic (PLEG): container finished" podID="db2b767d-e8f8-4664-9aa4-5d9a1aff0e2f" containerID="bedc45f5d1c98681b63e0c06513ed91369aec36c6a6b5bd272f45280d27698b6" exitCode=137
Oct 31 07:47:58.664928 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:47:58.664899    4720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-cpu-accelerator-preset-user-image-wins-06960b42/cpu-accel-precedence-kserve-7489f44cbc-njf8c" event={"ID":"db2b767d-e8f8-4664-9aa4-5d9a1aff0e2f","Type":"ContainerDied","Data":"bedc45f5d1c98681b63e0c06513ed91369aec36c6a6b5bd272f45280d27698b6"}
Oct 31 07:47:59.068891 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:47:59.068858    4720 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-cpu-accelerator-preset-user-image-wins-06960b42_cpu-accel-precedence-kserve-7489f44cbc-njf8c_db2b767d-e8f8-4664-9aa4-5d9a1aff0e2f/main/0.log"
Oct 31 07:47:59.070255 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:47:59.070231    4720 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-cpu-accelerator-preset-user-image-wins-06960b42/cpu-accel-precedence-kserve-7489f44cbc-njf8c"
Oct 31 07:47:59.151486 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:47:59.151445    4720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/db2b767d-e8f8-4664-9aa4-5d9a1aff0e2f-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/db2b767d-e8f8-4664-9aa4-5d9a1aff0e2f-tmp-dir\") pod \"db2b767d-e8f8-4664-9aa4-5d9a1aff0e2f\" (UID: \"db2b767d-e8f8-4664-9aa4-5d9a1aff0e2f\") "
Oct 31 07:47:59.151686 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:47:59.151509    4720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/db2b767d-e8f8-4664-9aa4-5d9a1aff0e2f-kube-api-access-9fncj\" (UniqueName: \"kubernetes.io/projected/db2b767d-e8f8-4664-9aa4-5d9a1aff0e2f-kube-api-access-9fncj\") pod \"db2b767d-e8f8-4664-9aa4-5d9a1aff0e2f\" (UID: \"db2b767d-e8f8-4664-9aa4-5d9a1aff0e2f\") "
Oct 31 07:47:59.151686 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:47:59.151540    4720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/db2b767d-e8f8-4664-9aa4-5d9a1aff0e2f-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/db2b767d-e8f8-4664-9aa4-5d9a1aff0e2f-model-cache\") pod \"db2b767d-e8f8-4664-9aa4-5d9a1aff0e2f\" (UID: \"db2b767d-e8f8-4664-9aa4-5d9a1aff0e2f\") "
Oct 31 07:47:59.151686 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:47:59.151584    4720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/db2b767d-e8f8-4664-9aa4-5d9a1aff0e2f-dshm\" (UniqueName: \"kubernetes.io/empty-dir/db2b767d-e8f8-4664-9aa4-5d9a1aff0e2f-dshm\") pod \"db2b767d-e8f8-4664-9aa4-5d9a1aff0e2f\" (UID: \"db2b767d-e8f8-4664-9aa4-5d9a1aff0e2f\") "
Oct 31 07:47:59.151686 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:47:59.151619    4720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/db2b767d-e8f8-4664-9aa4-5d9a1aff0e2f-home\" (UniqueName: \"kubernetes.io/empty-dir/db2b767d-e8f8-4664-9aa4-5d9a1aff0e2f-home\") pod \"db2b767d-e8f8-4664-9aa4-5d9a1aff0e2f\" (UID: \"db2b767d-e8f8-4664-9aa4-5d9a1aff0e2f\") "
Oct 31 07:47:59.151686 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:47:59.151646    4720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/db2b767d-e8f8-4664-9aa4-5d9a1aff0e2f-tls-certs\" (UniqueName: \"kubernetes.io/secret/db2b767d-e8f8-4664-9aa4-5d9a1aff0e2f-tls-certs\") pod \"db2b767d-e8f8-4664-9aa4-5d9a1aff0e2f\" (UID: \"db2b767d-e8f8-4664-9aa4-5d9a1aff0e2f\") "
Oct 31 07:47:59.151686 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:47:59.151682    4720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/db2b767d-e8f8-4664-9aa4-5d9a1aff0e2f-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/db2b767d-e8f8-4664-9aa4-5d9a1aff0e2f-kserve-provision-location\") pod \"db2b767d-e8f8-4664-9aa4-5d9a1aff0e2f\" (UID: \"db2b767d-e8f8-4664-9aa4-5d9a1aff0e2f\") "
Oct 31 07:47:59.151973 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:47:59.151844    4720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db2b767d-e8f8-4664-9aa4-5d9a1aff0e2f-model-cache" pod "db2b767d-e8f8-4664-9aa4-5d9a1aff0e2f" (UID: "db2b767d-e8f8-4664-9aa4-5d9a1aff0e2f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 07:47:59.152025 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:47:59.151974    4720 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/db2b767d-e8f8-4664-9aa4-5d9a1aff0e2f-model-cache\") on node \"ip-10-0-139-83.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 07:47:59.152151 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:47:59.152125    4720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db2b767d-e8f8-4664-9aa4-5d9a1aff0e2f-tmp-dir" pod "db2b767d-e8f8-4664-9aa4-5d9a1aff0e2f" (UID: "db2b767d-e8f8-4664-9aa4-5d9a1aff0e2f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 07:47:59.152347 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:47:59.152292    4720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db2b767d-e8f8-4664-9aa4-5d9a1aff0e2f-home" pod "db2b767d-e8f8-4664-9aa4-5d9a1aff0e2f" (UID: "db2b767d-e8f8-4664-9aa4-5d9a1aff0e2f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 07:47:59.154035 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:47:59.154006    4720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db2b767d-e8f8-4664-9aa4-5d9a1aff0e2f-dshm" pod "db2b767d-e8f8-4664-9aa4-5d9a1aff0e2f" (UID: "db2b767d-e8f8-4664-9aa4-5d9a1aff0e2f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 07:47:59.154481 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:47:59.154457    4720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/db2b767d-e8f8-4664-9aa4-5d9a1aff0e2f-tls-certs" pod "db2b767d-e8f8-4664-9aa4-5d9a1aff0e2f" (UID: "db2b767d-e8f8-4664-9aa4-5d9a1aff0e2f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 07:47:59.154558 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:47:59.154493    4720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/db2b767d-e8f8-4664-9aa4-5d9a1aff0e2f-kube-api-access-9fncj" pod "db2b767d-e8f8-4664-9aa4-5d9a1aff0e2f" (UID: "db2b767d-e8f8-4664-9aa4-5d9a1aff0e2f"). InnerVolumeSpecName "kube-api-access-9fncj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 07:47:59.214755 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:47:59.214656    4720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db2b767d-e8f8-4664-9aa4-5d9a1aff0e2f-kserve-provision-location" pod "db2b767d-e8f8-4664-9aa4-5d9a1aff0e2f" (UID: "db2b767d-e8f8-4664-9aa4-5d9a1aff0e2f"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 07:47:59.253341 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:47:59.253288    4720 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/db2b767d-e8f8-4664-9aa4-5d9a1aff0e2f-dshm\") on node \"ip-10-0-139-83.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 07:47:59.253341 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:47:59.253353    4720 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/db2b767d-e8f8-4664-9aa4-5d9a1aff0e2f-home\") on node \"ip-10-0-139-83.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 07:47:59.253564 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:47:59.253365    4720 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/db2b767d-e8f8-4664-9aa4-5d9a1aff0e2f-tls-certs\") on node \"ip-10-0-139-83.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 07:47:59.253564 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:47:59.253377    4720 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/db2b767d-e8f8-4664-9aa4-5d9a1aff0e2f-kserve-provision-location\") on node \"ip-10-0-139-83.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 07:47:59.253564 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:47:59.253386    4720 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/db2b767d-e8f8-4664-9aa4-5d9a1aff0e2f-tmp-dir\") on node \"ip-10-0-139-83.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 07:47:59.253564 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:47:59.253402    4720 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9fncj\" (UniqueName: \"kubernetes.io/projected/db2b767d-e8f8-4664-9aa4-5d9a1aff0e2f-kube-api-access-9fncj\") on node \"ip-10-0-139-83.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 07:47:59.670811 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:47:59.670785    4720 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-cpu-accelerator-preset-user-image-wins-06960b42_cpu-accel-precedence-kserve-7489f44cbc-njf8c_db2b767d-e8f8-4664-9aa4-5d9a1aff0e2f/main/0.log"
Oct 31 07:47:59.671249 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:47:59.671206    4720 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-cpu-accelerator-preset-user-image-wins-06960b42/cpu-accel-precedence-kserve-7489f44cbc-njf8c"
Oct 31 07:47:59.671249 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:47:59.671221    4720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-cpu-accelerator-preset-user-image-wins-06960b42/cpu-accel-precedence-kserve-7489f44cbc-njf8c" event={"ID":"db2b767d-e8f8-4664-9aa4-5d9a1aff0e2f","Type":"ContainerDied","Data":"409cc5bccc07a129cf8978c24b91885924ba55270f98c6144c9b844545655e2c"}
Oct 31 07:47:59.671320 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:47:59.671271    4720 scope.go:122] "RemoveContainer" containerID="bedc45f5d1c98681b63e0c06513ed91369aec36c6a6b5bd272f45280d27698b6"
Oct 31 07:47:59.680233 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:47:59.680215    4720 scope.go:122] "RemoveContainer" containerID="f440014f4463a716df8d42e6652f4ea9cbc7b70d91d6436964cc137a66bffa78"
Oct 31 07:47:59.833599 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:47:59.833570    4720 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-cpu-accelerator-preset-user-image-wins-06960b42/cpu-accel-precedence-kserve-7489f44cbc-njf8c"]
Oct 31 07:47:59.882948 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:47:59.882910    4720 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-test-cpu-accelerator-preset-user-image-wins-06960b42/cpu-accel-precedence-kserve-7489f44cbc-njf8c"]
Oct 31 07:48:00.024796 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:48:00.024718    4720 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="db2b767d-e8f8-4664-9aa4-5d9a1aff0e2f" path="/var/lib/kubelet/pods/db2b767d-e8f8-4664-9aa4-5d9a1aff0e2f/volumes"
Oct 31 07:48:02.902033 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:48:02.901993    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 07:48:03.067741 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:48:03.067695    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 07:48:06.695460 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:48:06.695427    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 07:48:08.569032 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:48:08.568996    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 07:48:08.724103 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:48:08.724058    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 07:48:11.851290 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:48:11.851253    4720 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpu-accelerator-preset-ser-afc820a4-kserve-66b55b88f9c7nkz"]
Oct 31 07:48:11.851724 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:48:11.851582    4720 cpu_manager.go:414] "containerMap: removing container" podUID="db2b767d-e8f8-4664-9aa4-5d9a1aff0e2f" containerName="main"
Oct 31 07:48:11.851724 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:48:11.851594    4720 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="db2b767d-e8f8-4664-9aa4-5d9a1aff0e2f" containerName="main"
Oct 31 07:48:11.851724 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:48:11.851611    4720 cpu_manager.go:414] "containerMap: removing container" podUID="db2b767d-e8f8-4664-9aa4-5d9a1aff0e2f" containerName="storage-initializer"
Oct 31 07:48:11.851724 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:48:11.851616    4720 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="db2b767d-e8f8-4664-9aa4-5d9a1aff0e2f" containerName="storage-initializer"
Oct 31 07:48:11.851724 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:48:11.851674    4720 memory_manager.go:365] "RemoveStaleState removing state" podUID="db2b767d-e8f8-4664-9aa4-5d9a1aff0e2f" containerName="main"
Oct 31 07:48:11.854309 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:48:11.854293    4720 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpu-accelerator-preset-ser-afc820a4-kserve-66b55b88f9c7nkz"
Oct 31 07:48:11.943626 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:48:11.943586    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 07:48:11.958131 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:48:11.958104    4720 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c\"/\"seaweedfs-s3-creds\""
Oct 31 07:48:11.958131 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:48:11.958104    4720 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c\"/\"openshift-service-ca.crt\""
Oct 31 07:48:11.958321 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:48:11.958133    4720 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c\"/\"odh-kserve-custom-ca-bundle\""
Oct 31 07:48:11.958321 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:48:11.958111    4720 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c\"/\"kube-root-ca.crt\""
Oct 31 07:48:11.958321 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:48:11.958246    4720 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c\"/\"test-cb25a14c9ba423627edeb5388343bd353-kserve-self-signed-certs\""
Oct 31 07:48:11.958321 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:48:11.958311    4720 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c\"/\"default-dockercfg-mfsf4\""
Oct 31 07:48:11.959570 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:48:11.959554    4720 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpu-accelerator-preset-ser-afc820a4-kserve-66b55b88f9c7nkz"]
Oct 31 07:48:11.963484 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:48:11.963468    4720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f99f3d33-fe07-4def-8606-9590c86a8670-home\") pod \"test-cpu-accelerator-preset-ser-afc820a4-kserve-66b55b88f9c7nkz\" (UID: \"f99f3d33-fe07-4def-8606-9590c86a8670\") " pod="e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpu-accelerator-preset-ser-afc820a4-kserve-66b55b88f9c7nkz"
Oct 31 07:48:11.963564 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:48:11.963493    4720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f99f3d33-fe07-4def-8606-9590c86a8670-tls-certs\") pod \"test-cpu-accelerator-preset-ser-afc820a4-kserve-66b55b88f9c7nkz\" (UID: \"f99f3d33-fe07-4def-8606-9590c86a8670\") " pod="e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpu-accelerator-preset-ser-afc820a4-kserve-66b55b88f9c7nkz"
Oct 31 07:48:11.963564 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:48:11.963517    4720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f99f3d33-fe07-4def-8606-9590c86a8670-kserve-provision-location\") pod \"test-cpu-accelerator-preset-ser-afc820a4-kserve-66b55b88f9c7nkz\" (UID: \"f99f3d33-fe07-4def-8606-9590c86a8670\") " pod="e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpu-accelerator-preset-ser-afc820a4-kserve-66b55b88f9c7nkz"
Oct 31 07:48:11.963564 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:48:11.963533    4720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f99f3d33-fe07-4def-8606-9590c86a8670-cabundle-cert\") pod \"test-cpu-accelerator-preset-ser-afc820a4-kserve-66b55b88f9c7nkz\" (UID: \"f99f3d33-fe07-4def-8606-9590c86a8670\") " pod="e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpu-accelerator-preset-ser-afc820a4-kserve-66b55b88f9c7nkz"
Oct 31 07:48:11.963669 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:48:11.963599    4720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jvg2w\" (UniqueName: \"kubernetes.io/projected/f99f3d33-fe07-4def-8606-9590c86a8670-kube-api-access-jvg2w\") pod \"test-cpu-accelerator-preset-ser-afc820a4-kserve-66b55b88f9c7nkz\" (UID: \"f99f3d33-fe07-4def-8606-9590c86a8670\") " pod="e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpu-accelerator-preset-ser-afc820a4-kserve-66b55b88f9c7nkz"
Oct 31 07:48:11.963704 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:48:11.963666    4720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f99f3d33-fe07-4def-8606-9590c86a8670-dshm\") pod \"test-cpu-accelerator-preset-ser-afc820a4-kserve-66b55b88f9c7nkz\" (UID: \"f99f3d33-fe07-4def-8606-9590c86a8670\") " pod="e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpu-accelerator-preset-ser-afc820a4-kserve-66b55b88f9c7nkz"
Oct 31 07:48:11.963704 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:48:11.963687    4720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f99f3d33-fe07-4def-8606-9590c86a8670-model-cache\") pod \"test-cpu-accelerator-preset-ser-afc820a4-kserve-66b55b88f9c7nkz\" (UID: \"f99f3d33-fe07-4def-8606-9590c86a8670\") " pod="e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpu-accelerator-preset-ser-afc820a4-kserve-66b55b88f9c7nkz"
Oct 31 07:48:11.963767 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:48:11.963711    4720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f99f3d33-fe07-4def-8606-9590c86a8670-tmp-dir\") pod \"test-cpu-accelerator-preset-ser-afc820a4-kserve-66b55b88f9c7nkz\" (UID: \"f99f3d33-fe07-4def-8606-9590c86a8670\") " pod="e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpu-accelerator-preset-ser-afc820a4-kserve-66b55b88f9c7nkz"
Oct 31 07:48:12.065068 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:48:12.065032    4720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f99f3d33-fe07-4def-8606-9590c86a8670-dshm\") pod \"test-cpu-accelerator-preset-ser-afc820a4-kserve-66b55b88f9c7nkz\" (UID: \"f99f3d33-fe07-4def-8606-9590c86a8670\") " pod="e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpu-accelerator-preset-ser-afc820a4-kserve-66b55b88f9c7nkz"
Oct 31 07:48:12.065068 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:48:12.065072    4720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f99f3d33-fe07-4def-8606-9590c86a8670-model-cache\") pod \"test-cpu-accelerator-preset-ser-afc820a4-kserve-66b55b88f9c7nkz\" (UID: \"f99f3d33-fe07-4def-8606-9590c86a8670\") " pod="e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpu-accelerator-preset-ser-afc820a4-kserve-66b55b88f9c7nkz"
Oct 31 07:48:12.065404 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:48:12.065106    4720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f99f3d33-fe07-4def-8606-9590c86a8670-tmp-dir\") pod \"test-cpu-accelerator-preset-ser-afc820a4-kserve-66b55b88f9c7nkz\" (UID: \"f99f3d33-fe07-4def-8606-9590c86a8670\") " pod="e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpu-accelerator-preset-ser-afc820a4-kserve-66b55b88f9c7nkz"
Oct 31 07:48:12.065404 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:48:12.065168    4720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f99f3d33-fe07-4def-8606-9590c86a8670-home\") pod \"test-cpu-accelerator-preset-ser-afc820a4-kserve-66b55b88f9c7nkz\" (UID: \"f99f3d33-fe07-4def-8606-9590c86a8670\") " pod="e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpu-accelerator-preset-ser-afc820a4-kserve-66b55b88f9c7nkz"
Oct 31 07:48:12.065404 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:48:12.065195    4720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f99f3d33-fe07-4def-8606-9590c86a8670-tls-certs\") pod \"test-cpu-accelerator-preset-ser-afc820a4-kserve-66b55b88f9c7nkz\" (UID: \"f99f3d33-fe07-4def-8606-9590c86a8670\") " pod="e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpu-accelerator-preset-ser-afc820a4-kserve-66b55b88f9c7nkz"
Oct 31 07:48:12.065404 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:48:12.065228    4720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f99f3d33-fe07-4def-8606-9590c86a8670-kserve-provision-location\") pod \"test-cpu-accelerator-preset-ser-afc820a4-kserve-66b55b88f9c7nkz\" (UID: \"f99f3d33-fe07-4def-8606-9590c86a8670\") " pod="e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpu-accelerator-preset-ser-afc820a4-kserve-66b55b88f9c7nkz"
Oct 31 07:48:12.065404 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:48:12.065252    4720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f99f3d33-fe07-4def-8606-9590c86a8670-cabundle-cert\") pod \"test-cpu-accelerator-preset-ser-afc820a4-kserve-66b55b88f9c7nkz\" (UID: \"f99f3d33-fe07-4def-8606-9590c86a8670\") " pod="e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpu-accelerator-preset-ser-afc820a4-kserve-66b55b88f9c7nkz"
Oct 31 07:48:12.065404 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:48:12.065276    4720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jvg2w\" (UniqueName: \"kubernetes.io/projected/f99f3d33-fe07-4def-8606-9590c86a8670-kube-api-access-jvg2w\") pod \"test-cpu-accelerator-preset-ser-afc820a4-kserve-66b55b88f9c7nkz\" (UID: \"f99f3d33-fe07-4def-8606-9590c86a8670\") " pod="e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpu-accelerator-preset-ser-afc820a4-kserve-66b55b88f9c7nkz"
Oct 31 07:48:12.065702 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:48:12.065609    4720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f99f3d33-fe07-4def-8606-9590c86a8670-model-cache\") pod \"test-cpu-accelerator-preset-ser-afc820a4-kserve-66b55b88f9c7nkz\" (UID: \"f99f3d33-fe07-4def-8606-9590c86a8670\") " pod="e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpu-accelerator-preset-ser-afc820a4-kserve-66b55b88f9c7nkz"
Oct 31 07:48:12.065702 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:48:12.065633    4720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f99f3d33-fe07-4def-8606-9590c86a8670-tmp-dir\") pod \"test-cpu-accelerator-preset-ser-afc820a4-kserve-66b55b88f9c7nkz\" (UID: \"f99f3d33-fe07-4def-8606-9590c86a8670\") " pod="e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpu-accelerator-preset-ser-afc820a4-kserve-66b55b88f9c7nkz"
Oct 31 07:48:12.065702 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:48:12.065681    4720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f99f3d33-fe07-4def-8606-9590c86a8670-home\") pod \"test-cpu-accelerator-preset-ser-afc820a4-kserve-66b55b88f9c7nkz\" (UID: \"f99f3d33-fe07-4def-8606-9590c86a8670\") " pod="e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpu-accelerator-preset-ser-afc820a4-kserve-66b55b88f9c7nkz"
Oct 31 07:48:12.065815 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:48:12.065786    4720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f99f3d33-fe07-4def-8606-9590c86a8670-kserve-provision-location\") pod \"test-cpu-accelerator-preset-ser-afc820a4-kserve-66b55b88f9c7nkz\" (UID: \"f99f3d33-fe07-4def-8606-9590c86a8670\") " pod="e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpu-accelerator-preset-ser-afc820a4-kserve-66b55b88f9c7nkz"
Oct 31 07:48:12.066100 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:48:12.066081    4720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f99f3d33-fe07-4def-8606-9590c86a8670-cabundle-cert\") pod \"test-cpu-accelerator-preset-ser-afc820a4-kserve-66b55b88f9c7nkz\" (UID: \"f99f3d33-fe07-4def-8606-9590c86a8670\") " pod="e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpu-accelerator-preset-ser-afc820a4-kserve-66b55b88f9c7nkz"
Oct 31 07:48:12.068222 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:48:12.068191    4720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f99f3d33-fe07-4def-8606-9590c86a8670-dshm\") pod \"test-cpu-accelerator-preset-ser-afc820a4-kserve-66b55b88f9c7nkz\" (UID: \"f99f3d33-fe07-4def-8606-9590c86a8670\") " pod="e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpu-accelerator-preset-ser-afc820a4-kserve-66b55b88f9c7nkz"
Oct 31 07:48:12.068351 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:48:12.068317    4720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f99f3d33-fe07-4def-8606-9590c86a8670-tls-certs\") pod \"test-cpu-accelerator-preset-ser-afc820a4-kserve-66b55b88f9c7nkz\" (UID: \"f99f3d33-fe07-4def-8606-9590c86a8670\") " pod="e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpu-accelerator-preset-ser-afc820a4-kserve-66b55b88f9c7nkz"
Oct 31 07:48:12.121722 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:48:12.121649    4720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jvg2w\" (UniqueName: \"kubernetes.io/projected/f99f3d33-fe07-4def-8606-9590c86a8670-kube-api-access-jvg2w\") pod \"test-cpu-accelerator-preset-ser-afc820a4-kserve-66b55b88f9c7nkz\" (UID: \"f99f3d33-fe07-4def-8606-9590c86a8670\") " pod="e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpu-accelerator-preset-ser-afc820a4-kserve-66b55b88f9c7nkz"
Oct 31 07:48:12.165684 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:48:12.165631    4720 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpu-accelerator-preset-ser-afc820a4-kserve-66b55b88f9c7nkz"
Oct 31 07:48:12.401195 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:48:12.401115    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 07:48:12.648600 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:48:12.648415    4720 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpu-accelerator-preset-ser-afc820a4-kserve-66b55b88f9c7nkz"]
Oct 31 07:48:12.726450 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:48:12.726419    4720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpu-accelerator-preset-ser-afc820a4-kserve-66b55b88f9c7nkz" event={"ID":"f99f3d33-fe07-4def-8606-9590c86a8670","Type":"ContainerStarted","Data":"757eb4f84c116572e97ea11bfc7e5edd210b8bea79c04f23653618fba608513f"}
Oct 31 07:48:12.726594 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:48:12.726459    4720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpu-accelerator-preset-ser-afc820a4-kserve-66b55b88f9c7nkz" event={"ID":"f99f3d33-fe07-4def-8606-9590c86a8670","Type":"ContainerStarted","Data":"d0be08604bd955b7173e37892039152158eb1f728971f424fb63d769ff1fd857"}
Oct 31 07:48:13.460880 ip-10-0-139-83 kubenswrapper[4720]: E1031 07:48:13.460844    4720 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8m768" containerName="extractor"
Oct 31 07:48:14.051688 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:48:14.051654    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 07:48:14.245000 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:48:14.244970    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 07:48:15.741885 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:48:15.741851    4720 generic.go:356] "Generic (PLEG): container finished" podID="f99f3d33-fe07-4def-8606-9590c86a8670" containerID="757eb4f84c116572e97ea11bfc7e5edd210b8bea79c04f23653618fba608513f" exitCode=0
Oct 31 07:48:15.742281 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:48:15.741924    4720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpu-accelerator-preset-ser-afc820a4-kserve-66b55b88f9c7nkz" event={"ID":"f99f3d33-fe07-4def-8606-9590c86a8670","Type":"ContainerDied","Data":"757eb4f84c116572e97ea11bfc7e5edd210b8bea79c04f23653618fba608513f"}
Oct 31 07:48:16.748114 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:48:16.748080    4720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpu-accelerator-preset-ser-afc820a4-kserve-66b55b88f9c7nkz" event={"ID":"f99f3d33-fe07-4def-8606-9590c86a8670","Type":"ContainerStarted","Data":"a3be95a488fdfd9acad231bb2d1697dd47c03bbffc12f349e2a7332235a2abc5"}
Oct 31 07:48:16.964393 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:48:16.964322    4720 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpu-accelerator-preset-ser-afc820a4-kserve-66b55b88f9c7nkz" podStartSLOduration=3.626221065 podStartE2EDuration="5.964307466s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.338086401s" isStatefulPod=false podCreationTimestamp="2026-10-31 07:48:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 07:48:16.913832814 +0000 UTC m=+1151.044900913" watchObservedRunningTime="2026-10-31 07:48:16.964307466 +0000 UTC m=+1151.095375576"
Oct 31 07:48:17.166579 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:48:17.166541    4720 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpu-accelerator-preset-ser-afc820a4-kserve-66b55b88f9c7nkz"
Oct 31 07:48:20.155284 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:48:20.155248    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 07:48:21.705345 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:48:21.705291    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 07:48:22.166601 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:48:22.166567    4720 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpu-accelerator-preset-ser-afc820a4-kserve-66b55b88f9c7nkz"
Oct 31 07:48:22.168344 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:48:22.168294    4720 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpu-accelerator-preset-ser-afc820a4-kserve-66b55b88f9c7nkz" podUID="f99f3d33-fe07-4def-8606-9590c86a8670" containerName="main" probeResult="failure" output="Get \"https://10.132.0.37:8000/health\": dial tcp 10.132.0.37:8000: connect: connection refused"
Oct 31 07:48:25.668142 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:48:25.668112    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 07:48:25.828793 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:48:25.828758    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 07:48:32.166652 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:48:32.166611    4720 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpu-accelerator-preset-ser-afc820a4-kserve-66b55b88f9c7nkz" podUID="f99f3d33-fe07-4def-8606-9590c86a8670" containerName="main" probeResult="failure" output="Get \"https://10.132.0.37:8000/health\": dial tcp 10.132.0.37:8000: connect: connection refused"
Oct 31 07:48:33.113374 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:48:33.113321    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 07:48:35.639080 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:48:35.639042    4720 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-56f46b8cdd-9hmnc"]
Oct 31 07:48:35.643043 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:48:35.643023    4720 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-default-56f46b8cdd-9hmnc"
Oct 31 07:48:35.658997 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:48:35.658973    4720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4d6179bf-191a-4085-8742-fe4ba603e788-workload-certs\") pod \"openshift-ai-inference-openshift-default-56f46b8cdd-9hmnc\" (UID: \"4d6179bf-191a-4085-8742-fe4ba603e788\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-56f46b8cdd-9hmnc"
Oct 31 07:48:35.659117 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:48:35.659002    4720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d6179bf-191a-4085-8742-fe4ba603e788-workload-socket\") pod \"openshift-ai-inference-openshift-default-56f46b8cdd-9hmnc\" (UID: \"4d6179bf-191a-4085-8742-fe4ba603e788\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-56f46b8cdd-9hmnc"
Oct 31 07:48:35.659117 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:48:35.659020    4720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4d6179bf-191a-4085-8742-fe4ba603e788-istio-envoy\") pod \"openshift-ai-inference-openshift-default-56f46b8cdd-9hmnc\" (UID: \"4d6179bf-191a-4085-8742-fe4ba603e788\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-56f46b8cdd-9hmnc"
Oct 31 07:48:35.659117 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:48:35.659049    4720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4d6179bf-191a-4085-8742-fe4ba603e788-istio-podinfo\") pod \"openshift-ai-inference-openshift-default-56f46b8cdd-9hmnc\" (UID: \"4d6179bf-191a-4085-8742-fe4ba603e788\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-56f46b8cdd-9hmnc"
Oct 31 07:48:35.659289 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:48:35.659144    4720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4d6179bf-191a-4085-8742-fe4ba603e788-istio-data\") pod \"openshift-ai-inference-openshift-default-56f46b8cdd-9hmnc\" (UID: \"4d6179bf-191a-4085-8742-fe4ba603e788\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-56f46b8cdd-9hmnc"
Oct 31 07:48:35.659289 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:48:35.659200    4720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vr8sd\" (UniqueName: \"kubernetes.io/projected/4d6179bf-191a-4085-8742-fe4ba603e788-kube-api-access-vr8sd\") pod \"openshift-ai-inference-openshift-default-56f46b8cdd-9hmnc\" (UID: \"4d6179bf-191a-4085-8742-fe4ba603e788\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-56f46b8cdd-9hmnc"
Oct 31 07:48:35.659289 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:48:35.659240    4720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d6179bf-191a-4085-8742-fe4ba603e788-credential-socket\") pod \"openshift-ai-inference-openshift-default-56f46b8cdd-9hmnc\" (UID: \"4d6179bf-191a-4085-8742-fe4ba603e788\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-56f46b8cdd-9hmnc"
Oct 31 07:48:35.659470 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:48:35.659369    4720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4d6179bf-191a-4085-8742-fe4ba603e788-istio-token\") pod \"openshift-ai-inference-openshift-default-56f46b8cdd-9hmnc\" (UID: \"4d6179bf-191a-4085-8742-fe4ba603e788\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-56f46b8cdd-9hmnc"
Oct 31 07:48:35.659470 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:48:35.659398    4720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4d6179bf-191a-4085-8742-fe4ba603e788-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-default-56f46b8cdd-9hmnc\" (UID: \"4d6179bf-191a-4085-8742-fe4ba603e788\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-56f46b8cdd-9hmnc"
Oct 31 07:48:35.694664 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:48:35.694634    4720 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Oct 31 07:48:35.694819 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:48:35.694803    4720 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"openshift-ai-inference-openshift-default-dockercfg-z29g8\""
Oct 31 07:48:35.760261 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:48:35.760224    4720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4d6179bf-191a-4085-8742-fe4ba603e788-istio-podinfo\") pod \"openshift-ai-inference-openshift-default-56f46b8cdd-9hmnc\" (UID: \"4d6179bf-191a-4085-8742-fe4ba603e788\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-56f46b8cdd-9hmnc"
Oct 31 07:48:35.760442 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:48:35.760272    4720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4d6179bf-191a-4085-8742-fe4ba603e788-istio-data\") pod \"openshift-ai-inference-openshift-default-56f46b8cdd-9hmnc\" (UID: \"4d6179bf-191a-4085-8742-fe4ba603e788\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-56f46b8cdd-9hmnc"
Oct 31 07:48:35.760442 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:48:35.760302    4720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vr8sd\" (UniqueName: \"kubernetes.io/projected/4d6179bf-191a-4085-8742-fe4ba603e788-kube-api-access-vr8sd\") pod \"openshift-ai-inference-openshift-default-56f46b8cdd-9hmnc\" (UID: \"4d6179bf-191a-4085-8742-fe4ba603e788\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-56f46b8cdd-9hmnc"
Oct 31 07:48:35.760442 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:48:35.760375    4720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d6179bf-191a-4085-8742-fe4ba603e788-credential-socket\") pod \"openshift-ai-inference-openshift-default-56f46b8cdd-9hmnc\" (UID: \"4d6179bf-191a-4085-8742-fe4ba603e788\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-56f46b8cdd-9hmnc"
Oct 31 07:48:35.760614 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:48:35.760448    4720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4d6179bf-191a-4085-8742-fe4ba603e788-istio-token\") pod \"openshift-ai-inference-openshift-default-56f46b8cdd-9hmnc\" (UID: \"4d6179bf-191a-4085-8742-fe4ba603e788\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-56f46b8cdd-9hmnc"
Oct 31 07:48:35.760614 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:48:35.760473    4720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4d6179bf-191a-4085-8742-fe4ba603e788-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-default-56f46b8cdd-9hmnc\" (UID: \"4d6179bf-191a-4085-8742-fe4ba603e788\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-56f46b8cdd-9hmnc"
Oct 31 07:48:35.760614 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:48:35.760550    4720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4d6179bf-191a-4085-8742-fe4ba603e788-workload-certs\") pod \"openshift-ai-inference-openshift-default-56f46b8cdd-9hmnc\" (UID: \"4d6179bf-191a-4085-8742-fe4ba603e788\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-56f46b8cdd-9hmnc"
Oct 31 07:48:35.760767 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:48:35.760694    4720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4d6179bf-191a-4085-8742-fe4ba603e788-istio-data\") pod \"openshift-ai-inference-openshift-default-56f46b8cdd-9hmnc\" (UID: \"4d6179bf-191a-4085-8742-fe4ba603e788\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-56f46b8cdd-9hmnc"
Oct 31 07:48:35.760860 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:48:35.760838    4720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d6179bf-191a-4085-8742-fe4ba603e788-credential-socket\") pod \"openshift-ai-inference-openshift-default-56f46b8cdd-9hmnc\" (UID: \"4d6179bf-191a-4085-8742-fe4ba603e788\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-56f46b8cdd-9hmnc"
Oct 31 07:48:35.760954 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:48:35.760925    4720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4d6179bf-191a-4085-8742-fe4ba603e788-workload-certs\") pod \"openshift-ai-inference-openshift-default-56f46b8cdd-9hmnc\" (UID: \"4d6179bf-191a-4085-8742-fe4ba603e788\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-56f46b8cdd-9hmnc"
Oct 31 07:48:35.761240 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:48:35.760576    4720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d6179bf-191a-4085-8742-fe4ba603e788-workload-socket\") pod \"openshift-ai-inference-openshift-default-56f46b8cdd-9hmnc\" (UID: \"4d6179bf-191a-4085-8742-fe4ba603e788\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-56f46b8cdd-9hmnc"
Oct 31 07:48:35.761240 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:48:35.761028    4720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4d6179bf-191a-4085-8742-fe4ba603e788-istio-envoy\") pod \"openshift-ai-inference-openshift-default-56f46b8cdd-9hmnc\" (UID: \"4d6179bf-191a-4085-8742-fe4ba603e788\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-56f46b8cdd-9hmnc"
Oct 31 07:48:35.761240 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:48:35.761102    4720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d6179bf-191a-4085-8742-fe4ba603e788-workload-socket\") pod \"openshift-ai-inference-openshift-default-56f46b8cdd-9hmnc\" (UID: \"4d6179bf-191a-4085-8742-fe4ba603e788\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-56f46b8cdd-9hmnc"
Oct 31 07:48:35.761240 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:48:35.761190    4720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4d6179bf-191a-4085-8742-fe4ba603e788-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-default-56f46b8cdd-9hmnc\" (UID: \"4d6179bf-191a-4085-8742-fe4ba603e788\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-56f46b8cdd-9hmnc"
Oct 31 07:48:35.762966 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:48:35.762939    4720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4d6179bf-191a-4085-8742-fe4ba603e788-istio-podinfo\") pod \"openshift-ai-inference-openshift-default-56f46b8cdd-9hmnc\" (UID: \"4d6179bf-191a-4085-8742-fe4ba603e788\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-56f46b8cdd-9hmnc"
Oct 31 07:48:35.763120 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:48:35.763101    4720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4d6179bf-191a-4085-8742-fe4ba603e788-istio-envoy\") pod \"openshift-ai-inference-openshift-default-56f46b8cdd-9hmnc\" (UID: \"4d6179bf-191a-4085-8742-fe4ba603e788\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-56f46b8cdd-9hmnc"
Oct 31 07:48:35.779742 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:48:35.779715    4720 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-56f46b8cdd-9hmnc"]
Oct 31 07:48:35.829636 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:48:35.829612    4720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vr8sd\" (UniqueName: \"kubernetes.io/projected/4d6179bf-191a-4085-8742-fe4ba603e788-kube-api-access-vr8sd\") pod \"openshift-ai-inference-openshift-default-56f46b8cdd-9hmnc\" (UID: \"4d6179bf-191a-4085-8742-fe4ba603e788\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-56f46b8cdd-9hmnc"
Oct 31 07:48:35.834772 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:48:35.834747    4720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4d6179bf-191a-4085-8742-fe4ba603e788-istio-token\") pod \"openshift-ai-inference-openshift-default-56f46b8cdd-9hmnc\" (UID: \"4d6179bf-191a-4085-8742-fe4ba603e788\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-56f46b8cdd-9hmnc"
Oct 31 07:48:35.955370 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:48:35.955271    4720 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-default-56f46b8cdd-9hmnc"
Oct 31 07:48:36.380675 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:48:36.380650    4720 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-56f46b8cdd-9hmnc"]
Oct 31 07:48:36.382895 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:48:36.382864    4720 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4d6179bf_191a_4085_8742_fe4ba603e788.slice/crio-27ac8be4c0c5c3dc929f07f7a212d95e594ac9498341d7c3ea760a1bf3033fcf WatchSource:0}: Error finding container 27ac8be4c0c5c3dc929f07f7a212d95e594ac9498341d7c3ea760a1bf3033fcf: Status 404 returned error can't find the container with id 27ac8be4c0c5c3dc929f07f7a212d95e594ac9498341d7c3ea760a1bf3033fcf
Oct 31 07:48:36.850507 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:48:36.850466    4720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-56f46b8cdd-9hmnc" event={"ID":"4d6179bf-191a-4085-8742-fe4ba603e788","Type":"ContainerStarted","Data":"27ac8be4c0c5c3dc929f07f7a212d95e594ac9498341d7c3ea760a1bf3033fcf"}
Oct 31 07:48:37.045133 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:48:37.045106    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 07:48:40.089120 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:48:40.089077    4720 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 31 07:48:40.089458 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:48:40.089161    4720 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 31 07:48:40.089458 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:48:40.089197    4720 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 31 07:48:40.901156 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:48:40.901123    4720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-56f46b8cdd-9hmnc" event={"ID":"4d6179bf-191a-4085-8742-fe4ba603e788","Type":"ContainerStarted","Data":"044ba974d734f848bacbca636742c7617e24931e4e790195866362b59f31b875"}
Oct 31 07:48:40.955769 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:48:40.955743    4720 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/openshift-ai-inference-openshift-default-56f46b8cdd-9hmnc"
Oct 31 07:48:41.134207 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:48:41.134155    4720 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/openshift-ai-inference-openshift-default-56f46b8cdd-9hmnc" podStartSLOduration=2.431567372 podStartE2EDuration="6.13414274s" totalImagesPullingTime="3.702575368s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 07:48:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 07:48:41.084104589 +0000 UTC m=+1175.215172661" watchObservedRunningTime="2026-10-31 07:48:41.13414274 +0000 UTC m=+1175.265210812"
Oct 31 07:48:41.957519 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:48:41.957485    4720 patch_prober.go:29] interesting pod/openshift-ai-inference-openshift-default-56f46b8cdd-9hmnc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.38:15021/healthz/ready\": dial tcp 10.132.0.38:15021: connect: connection refused" start-of-body=
Oct 31 07:48:41.957711 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:48:41.957568    4720 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-56f46b8cdd-9hmnc" podUID="4d6179bf-191a-4085-8742-fe4ba603e788" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.38:15021/healthz/ready\": dial tcp 10.132.0.38:15021: connect: connection refused"
Oct 31 07:48:42.166166 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:48:42.166122    4720 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpu-accelerator-preset-ser-afc820a4-kserve-66b55b88f9c7nkz" podUID="f99f3d33-fe07-4def-8606-9590c86a8670" containerName="main" probeResult="failure" output="Get \"https://10.132.0.37:8000/health\": dial tcp 10.132.0.37:8000: connect: connection refused"
Oct 31 07:48:42.956466 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:48:42.956434    4720 patch_prober.go:29] interesting pod/openshift-ai-inference-openshift-default-56f46b8cdd-9hmnc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.38:15021/healthz/ready\": dial tcp 10.132.0.38:15021: connect: connection refused" start-of-body=
Oct 31 07:48:42.956633 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:48:42.956483    4720 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-56f46b8cdd-9hmnc" podUID="4d6179bf-191a-4085-8742-fe4ba603e788" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.38:15021/healthz/ready\": dial tcp 10.132.0.38:15021: connect: connection refused"
Oct 31 07:48:43.461737 ip-10-0-139-83 kubenswrapper[4720]: E1031 07:48:43.461631    4720 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8m768" containerName="extractor"
Oct 31 07:48:43.956004 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:48:43.955975    4720 patch_prober.go:29] interesting pod/openshift-ai-inference-openshift-default-56f46b8cdd-9hmnc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.38:15021/healthz/ready\": dial tcp 10.132.0.38:15021: connect: connection refused" start-of-body=
Oct 31 07:48:43.956173 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:48:43.956030    4720 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-56f46b8cdd-9hmnc" podUID="4d6179bf-191a-4085-8742-fe4ba603e788" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.38:15021/healthz/ready\": dial tcp 10.132.0.38:15021: connect: connection refused"
Oct 31 07:48:44.956210 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:48:44.956168    4720 patch_prober.go:29] interesting pod/openshift-ai-inference-openshift-default-56f46b8cdd-9hmnc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.38:15021/healthz/ready\": dial tcp 10.132.0.38:15021: connect: connection refused" start-of-body=
Oct 31 07:48:44.956594 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:48:44.956242    4720 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-56f46b8cdd-9hmnc" podUID="4d6179bf-191a-4085-8742-fe4ba603e788" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.38:15021/healthz/ready\": dial tcp 10.132.0.38:15021: connect: connection refused"
Oct 31 07:48:45.956429 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:48:45.956393    4720 patch_prober.go:29] interesting pod/openshift-ai-inference-openshift-default-56f46b8cdd-9hmnc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.38:15021/healthz/ready\": dial tcp 10.132.0.38:15021: connect: connection refused" start-of-body=
Oct 31 07:48:45.956907 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:48:45.956460    4720 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-56f46b8cdd-9hmnc" podUID="4d6179bf-191a-4085-8742-fe4ba603e788" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.38:15021/healthz/ready\": dial tcp 10.132.0.38:15021: connect: connection refused"
Oct 31 07:48:46.960481 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:48:46.960452    4720 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/openshift-ai-inference-openshift-default-56f46b8cdd-9hmnc"
Oct 31 07:48:46.960875 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:48:46.960741    4720 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/openshift-ai-inference-openshift-default-56f46b8cdd-9hmnc"
Oct 31 07:48:46.961500 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:48:46.961484    4720 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/openshift-ai-inference-openshift-default-56f46b8cdd-9hmnc"
Oct 31 07:48:52.166653 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:48:52.166611    4720 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpu-accelerator-preset-ser-afc820a4-kserve-66b55b88f9c7nkz" podUID="f99f3d33-fe07-4def-8606-9590c86a8670" containerName="main" probeResult="failure" output="Get \"https://10.132.0.37:8000/health\": dial tcp 10.132.0.37:8000: connect: connection refused"
Oct 31 07:49:02.166713 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:49:02.166665    4720 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpu-accelerator-preset-ser-afc820a4-kserve-66b55b88f9c7nkz" podUID="f99f3d33-fe07-4def-8606-9590c86a8670" containerName="main" probeResult="failure" output="Get \"https://10.132.0.37:8000/health\": dial tcp 10.132.0.37:8000: connect: connection refused"
Oct 31 07:49:06.116517 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:49:06.116485    4720 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-4jszp_b1db0e3b-e85d-4769-8012-cf4b6f740d9d/console-operator/2.log"
Oct 31 07:49:06.119222 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:49:06.119198    4720 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-4jszp_b1db0e3b-e85d-4769-8012-cf4b6f740d9d/console-operator/2.log"
Oct 31 07:49:07.041573 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:49:07.041536    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 07:49:12.166012 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:49:12.165965    4720 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpu-accelerator-preset-ser-afc820a4-kserve-66b55b88f9c7nkz" podUID="f99f3d33-fe07-4def-8606-9590c86a8670" containerName="main" probeResult="failure" output="Get \"https://10.132.0.37:8000/health\": dial tcp 10.132.0.37:8000: connect: connection refused"
Oct 31 07:49:13.461525 ip-10-0-139-83 kubenswrapper[4720]: E1031 07:49:13.461482    4720 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8m768" containerName="extractor"
Oct 31 07:49:22.101598 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:49:22.101521    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 07:49:22.166203 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:49:22.166173    4720 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpu-accelerator-preset-ser-afc820a4-kserve-66b55b88f9c7nkz" podUID="f99f3d33-fe07-4def-8606-9590c86a8670" containerName="main" probeResult="failure" output="Get \"https://10.132.0.37:8000/health\": dial tcp 10.132.0.37:8000: connect: connection refused"
Oct 31 07:49:30.769615 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:49:30.769564    4720 prober.go:122] "Probe failed" probeType="Liveness" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-83.us-west-1.compute.internal" podUID="2183ffc19200ec816330ede9883f3bf6" containerName="haproxy" probeResult="failure" output="Get \"https://172.20.0.1:6443/version\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 07:49:32.176034 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:49:32.175999    4720 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpu-accelerator-preset-ser-afc820a4-kserve-66b55b88f9c7nkz"
Oct 31 07:49:32.183482 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:49:32.183458    4720 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpu-accelerator-preset-ser-afc820a4-kserve-66b55b88f9c7nkz"
Oct 31 07:49:37.354836 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:49:37.354798    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 07:49:39.958087 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:49:39.958057    4720 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpu-accelerator-preset-ser-afc820a4-kserve-66b55b88f9c7nkz"]
Oct 31 07:49:39.975053 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:49:39.975024    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 07:49:40.126616 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:49:40.126557    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 07:49:43.461469 ip-10-0-139-83 kubenswrapper[4720]: E1031 07:49:43.461438    4720 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8m768" containerName="extractor"
Oct 31 07:49:52.002702 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:49:52.002666    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 07:49:54.978303 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:49:54.978269    4720 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpu-accelerator-preset-ser-afc820a4-kserve-66b55b88f9c7nkz" podUID="f99f3d33-fe07-4def-8606-9590c86a8670" containerName="main" containerID="cri-o://a3be95a488fdfd9acad231bb2d1697dd47c03bbffc12f349e2a7332235a2abc5" gracePeriod=45
Oct 31 07:49:55.176898 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:49:55.176857    4720 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpu-accelerator-preset-ser-afc820a4-kserve-66b55b88f9c7nkz" podUID="f99f3d33-fe07-4def-8606-9590c86a8670" containerName="main" probeResult="failure" output="Get \"https://10.132.0.37:8000/health\": dial tcp 10.132.0.37:8000: connect: connection refused"
Oct 31 07:49:55.765543 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:49:55.765514    4720 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpu-accelerator-preset-ser-afc820a4-kserve-66b55b88f9c7nkz"
Oct 31 07:49:55.808855 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:49:55.808830    4720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f99f3d33-fe07-4def-8606-9590c86a8670-tls-certs\" (UniqueName: \"kubernetes.io/secret/f99f3d33-fe07-4def-8606-9590c86a8670-tls-certs\") pod \"f99f3d33-fe07-4def-8606-9590c86a8670\" (UID: \"f99f3d33-fe07-4def-8606-9590c86a8670\") "
Oct 31 07:49:55.808950 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:49:55.808860    4720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f99f3d33-fe07-4def-8606-9590c86a8670-home\" (UniqueName: \"kubernetes.io/empty-dir/f99f3d33-fe07-4def-8606-9590c86a8670-home\") pod \"f99f3d33-fe07-4def-8606-9590c86a8670\" (UID: \"f99f3d33-fe07-4def-8606-9590c86a8670\") "
Oct 31 07:49:55.808950 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:49:55.808881    4720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f99f3d33-fe07-4def-8606-9590c86a8670-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f99f3d33-fe07-4def-8606-9590c86a8670-cabundle-cert\") pod \"f99f3d33-fe07-4def-8606-9590c86a8670\" (UID: \"f99f3d33-fe07-4def-8606-9590c86a8670\") "
Oct 31 07:49:55.808950 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:49:55.808941    4720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f99f3d33-fe07-4def-8606-9590c86a8670-kube-api-access-jvg2w\" (UniqueName: \"kubernetes.io/projected/f99f3d33-fe07-4def-8606-9590c86a8670-kube-api-access-jvg2w\") pod \"f99f3d33-fe07-4def-8606-9590c86a8670\" (UID: \"f99f3d33-fe07-4def-8606-9590c86a8670\") "
Oct 31 07:49:55.809089 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:49:55.808980    4720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f99f3d33-fe07-4def-8606-9590c86a8670-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f99f3d33-fe07-4def-8606-9590c86a8670-tmp-dir\") pod \"f99f3d33-fe07-4def-8606-9590c86a8670\" (UID: \"f99f3d33-fe07-4def-8606-9590c86a8670\") "
Oct 31 07:49:55.809089 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:49:55.809005    4720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f99f3d33-fe07-4def-8606-9590c86a8670-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f99f3d33-fe07-4def-8606-9590c86a8670-model-cache\") pod \"f99f3d33-fe07-4def-8606-9590c86a8670\" (UID: \"f99f3d33-fe07-4def-8606-9590c86a8670\") "
Oct 31 07:49:55.809089 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:49:55.809063    4720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f99f3d33-fe07-4def-8606-9590c86a8670-dshm\" (UniqueName: \"kubernetes.io/empty-dir/f99f3d33-fe07-4def-8606-9590c86a8670-dshm\") pod \"f99f3d33-fe07-4def-8606-9590c86a8670\" (UID: \"f99f3d33-fe07-4def-8606-9590c86a8670\") "
Oct 31 07:49:55.809235 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:49:55.809115    4720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f99f3d33-fe07-4def-8606-9590c86a8670-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f99f3d33-fe07-4def-8606-9590c86a8670-kserve-provision-location\") pod \"f99f3d33-fe07-4def-8606-9590c86a8670\" (UID: \"f99f3d33-fe07-4def-8606-9590c86a8670\") "
Oct 31 07:49:55.809320 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:49:55.809299    4720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f99f3d33-fe07-4def-8606-9590c86a8670-home" pod "f99f3d33-fe07-4def-8606-9590c86a8670" (UID: "f99f3d33-fe07-4def-8606-9590c86a8670"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 07:49:55.809422 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:49:55.809304    4720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f99f3d33-fe07-4def-8606-9590c86a8670-cabundle-cert" pod "f99f3d33-fe07-4def-8606-9590c86a8670" (UID: "f99f3d33-fe07-4def-8606-9590c86a8670"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 07:49:55.809745 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:49:55.809708    4720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f99f3d33-fe07-4def-8606-9590c86a8670-model-cache" pod "f99f3d33-fe07-4def-8606-9590c86a8670" (UID: "f99f3d33-fe07-4def-8606-9590c86a8670"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 07:49:55.811165 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:49:55.811139    4720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f99f3d33-fe07-4def-8606-9590c86a8670-tls-certs" pod "f99f3d33-fe07-4def-8606-9590c86a8670" (UID: "f99f3d33-fe07-4def-8606-9590c86a8670"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 07:49:55.811466 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:49:55.811443    4720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f99f3d33-fe07-4def-8606-9590c86a8670-kube-api-access-jvg2w" pod "f99f3d33-fe07-4def-8606-9590c86a8670" (UID: "f99f3d33-fe07-4def-8606-9590c86a8670"). InnerVolumeSpecName "kube-api-access-jvg2w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 07:49:55.811820 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:49:55.811786    4720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f99f3d33-fe07-4def-8606-9590c86a8670-dshm" pod "f99f3d33-fe07-4def-8606-9590c86a8670" (UID: "f99f3d33-fe07-4def-8606-9590c86a8670"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 07:49:55.819685 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:49:55.819657    4720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f99f3d33-fe07-4def-8606-9590c86a8670-tmp-dir" pod "f99f3d33-fe07-4def-8606-9590c86a8670" (UID: "f99f3d33-fe07-4def-8606-9590c86a8670"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 07:49:55.832687 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:49:55.832639    4720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f99f3d33-fe07-4def-8606-9590c86a8670-kserve-provision-location" pod "f99f3d33-fe07-4def-8606-9590c86a8670" (UID: "f99f3d33-fe07-4def-8606-9590c86a8670"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 07:49:55.909839 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:49:55.909818    4720 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f99f3d33-fe07-4def-8606-9590c86a8670-kserve-provision-location\") on node \"ip-10-0-139-83.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 07:49:55.909944 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:49:55.909840    4720 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f99f3d33-fe07-4def-8606-9590c86a8670-tls-certs\") on node \"ip-10-0-139-83.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 07:49:55.909944 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:49:55.909853    4720 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f99f3d33-fe07-4def-8606-9590c86a8670-home\") on node \"ip-10-0-139-83.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 07:49:55.909944 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:49:55.909864    4720 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f99f3d33-fe07-4def-8606-9590c86a8670-cabundle-cert\") on node \"ip-10-0-139-83.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 07:49:55.909944 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:49:55.909876    4720 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jvg2w\" (UniqueName: \"kubernetes.io/projected/f99f3d33-fe07-4def-8606-9590c86a8670-kube-api-access-jvg2w\") on node \"ip-10-0-139-83.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 07:49:55.909944 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:49:55.909889    4720 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f99f3d33-fe07-4def-8606-9590c86a8670-tmp-dir\") on node \"ip-10-0-139-83.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 07:49:55.909944 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:49:55.909901    4720 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f99f3d33-fe07-4def-8606-9590c86a8670-model-cache\") on node \"ip-10-0-139-83.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 07:49:55.909944 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:49:55.909912    4720 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f99f3d33-fe07-4def-8606-9590c86a8670-dshm\") on node \"ip-10-0-139-83.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 07:49:56.233067 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:49:56.233000    4720 generic.go:356] "Generic (PLEG): container finished" podID="f99f3d33-fe07-4def-8606-9590c86a8670" containerID="a3be95a488fdfd9acad231bb2d1697dd47c03bbffc12f349e2a7332235a2abc5" exitCode=0
Oct 31 07:49:56.233467 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:49:56.233069    4720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpu-accelerator-preset-ser-afc820a4-kserve-66b55b88f9c7nkz" event={"ID":"f99f3d33-fe07-4def-8606-9590c86a8670","Type":"ContainerDied","Data":"a3be95a488fdfd9acad231bb2d1697dd47c03bbffc12f349e2a7332235a2abc5"}
Oct 31 07:49:56.233467 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:49:56.233096    4720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpu-accelerator-preset-ser-afc820a4-kserve-66b55b88f9c7nkz" event={"ID":"f99f3d33-fe07-4def-8606-9590c86a8670","Type":"ContainerDied","Data":"d0be08604bd955b7173e37892039152158eb1f728971f424fb63d769ff1fd857"}
Oct 31 07:49:56.233467 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:49:56.233106    4720 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpu-accelerator-preset-ser-afc820a4-kserve-66b55b88f9c7nkz"
Oct 31 07:49:56.233467 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:49:56.233114    4720 scope.go:122] "RemoveContainer" containerID="a3be95a488fdfd9acad231bb2d1697dd47c03bbffc12f349e2a7332235a2abc5"
Oct 31 07:49:56.241518 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:49:56.241479    4720 scope.go:122] "RemoveContainer" containerID="757eb4f84c116572e97ea11bfc7e5edd210b8bea79c04f23653618fba608513f"
Oct 31 07:49:56.249188 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:49:56.249170    4720 scope.go:122] "RemoveContainer" containerID="a3be95a488fdfd9acad231bb2d1697dd47c03bbffc12f349e2a7332235a2abc5"
Oct 31 07:49:56.249477 ip-10-0-139-83 kubenswrapper[4720]: E1031 07:49:56.249455    4720 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a3be95a488fdfd9acad231bb2d1697dd47c03bbffc12f349e2a7332235a2abc5\": container with ID starting with a3be95a488fdfd9acad231bb2d1697dd47c03bbffc12f349e2a7332235a2abc5 not found: ID does not exist" containerID="a3be95a488fdfd9acad231bb2d1697dd47c03bbffc12f349e2a7332235a2abc5"
Oct 31 07:49:56.249575 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:49:56.249483    4720 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a3be95a488fdfd9acad231bb2d1697dd47c03bbffc12f349e2a7332235a2abc5"} err="failed to get container status \"a3be95a488fdfd9acad231bb2d1697dd47c03bbffc12f349e2a7332235a2abc5\": rpc error: code = NotFound desc = could not find container \"a3be95a488fdfd9acad231bb2d1697dd47c03bbffc12f349e2a7332235a2abc5\": container with ID starting with a3be95a488fdfd9acad231bb2d1697dd47c03bbffc12f349e2a7332235a2abc5 not found: ID does not exist"
Oct 31 07:49:56.249575 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:49:56.249500    4720 scope.go:122] "RemoveContainer" containerID="757eb4f84c116572e97ea11bfc7e5edd210b8bea79c04f23653618fba608513f"
Oct 31 07:49:56.249765 ip-10-0-139-83 kubenswrapper[4720]: E1031 07:49:56.249747    4720 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"757eb4f84c116572e97ea11bfc7e5edd210b8bea79c04f23653618fba608513f\": container with ID starting with 757eb4f84c116572e97ea11bfc7e5edd210b8bea79c04f23653618fba608513f not found: ID does not exist" containerID="757eb4f84c116572e97ea11bfc7e5edd210b8bea79c04f23653618fba608513f"
Oct 31 07:49:56.249809 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:49:56.249770    4720 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"757eb4f84c116572e97ea11bfc7e5edd210b8bea79c04f23653618fba608513f"} err="failed to get container status \"757eb4f84c116572e97ea11bfc7e5edd210b8bea79c04f23653618fba608513f\": rpc error: code = NotFound desc = could not find container \"757eb4f84c116572e97ea11bfc7e5edd210b8bea79c04f23653618fba608513f\": container with ID starting with 757eb4f84c116572e97ea11bfc7e5edd210b8bea79c04f23653618fba608513f not found: ID does not exist"
Oct 31 07:49:56.394945 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:49:56.394920    4720 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpu-accelerator-preset-ser-afc820a4-kserve-66b55b88f9c7nkz"]
Oct 31 07:49:56.444286 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:49:56.444263    4720 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpu-accelerator-preset-ser-afc820a4-kserve-66b55b88f9c7nkz"]
Oct 31 07:49:58.025747 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:49:58.025716    4720 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f99f3d33-fe07-4def-8606-9590c86a8670" path="/var/lib/kubelet/pods/f99f3d33-fe07-4def-8606-9590c86a8670/volumes"
Oct 31 07:50:07.375030 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:50:07.374996    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 07:50:13.460841 ip-10-0-139-83 kubenswrapper[4720]: E1031 07:50:13.460810    4720 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8m768" containerName="extractor"
Oct 31 07:50:22.381799 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:50:22.381767    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 07:50:26.389113 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:50:26.389079    4720 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/periodic-gathering-lmblb-khh8v"]
Oct 31 07:50:26.389540 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:50:26.389394    4720 cpu_manager.go:414] "containerMap: removing container" podUID="f99f3d33-fe07-4def-8606-9590c86a8670" containerName="main"
Oct 31 07:50:26.389540 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:50:26.389405    4720 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f99f3d33-fe07-4def-8606-9590c86a8670" containerName="main"
Oct 31 07:50:26.389540 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:50:26.389437    4720 cpu_manager.go:414] "containerMap: removing container" podUID="f99f3d33-fe07-4def-8606-9590c86a8670" containerName="storage-initializer"
Oct 31 07:50:26.389540 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:50:26.389441    4720 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f99f3d33-fe07-4def-8606-9590c86a8670" containerName="storage-initializer"
Oct 31 07:50:26.389540 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:50:26.389499    4720 memory_manager.go:365] "RemoveStaleState removing state" podUID="f99f3d33-fe07-4def-8606-9590c86a8670" containerName="main"
Oct 31 07:50:26.392260 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:50:26.392243    4720 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-lmblb-khh8v"
Oct 31 07:50:26.448484 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:50:26.448460    4720 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-lmblb-khh8v"]
Oct 31 07:50:26.495678 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:50:26.495658    4720 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-pkrx7\""
Oct 31 07:50:26.532905 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:50:26.532878    4720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/287cc66c-4032-4bfa-a316-250ce6cb7d34-service-ca-bundle\") pod \"periodic-gathering-lmblb-khh8v\" (UID: \"287cc66c-4032-4bfa-a316-250ce6cb7d34\") " pod="openshift-insights/periodic-gathering-lmblb-khh8v"
Oct 31 07:50:26.533022 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:50:26.532920    4720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/287cc66c-4032-4bfa-a316-250ce6cb7d34-trusted-ca-bundle\") pod \"periodic-gathering-lmblb-khh8v\" (UID: \"287cc66c-4032-4bfa-a316-250ce6cb7d34\") " pod="openshift-insights/periodic-gathering-lmblb-khh8v"
Oct 31 07:50:26.533022 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:50:26.532946    4720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8z59p\" (UniqueName: \"kubernetes.io/projected/287cc66c-4032-4bfa-a316-250ce6cb7d34-kube-api-access-8z59p\") pod \"periodic-gathering-lmblb-khh8v\" (UID: \"287cc66c-4032-4bfa-a316-250ce6cb7d34\") " pod="openshift-insights/periodic-gathering-lmblb-khh8v"
Oct 31 07:50:26.533104 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:50:26.533057    4720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/287cc66c-4032-4bfa-a316-250ce6cb7d34-archives-path\") pod \"periodic-gathering-lmblb-khh8v\" (UID: \"287cc66c-4032-4bfa-a316-250ce6cb7d34\") " pod="openshift-insights/periodic-gathering-lmblb-khh8v"
Oct 31 07:50:26.633546 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:50:26.633518    4720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/287cc66c-4032-4bfa-a316-250ce6cb7d34-service-ca-bundle\") pod \"periodic-gathering-lmblb-khh8v\" (UID: \"287cc66c-4032-4bfa-a316-250ce6cb7d34\") " pod="openshift-insights/periodic-gathering-lmblb-khh8v"
Oct 31 07:50:26.633644 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:50:26.633561    4720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/287cc66c-4032-4bfa-a316-250ce6cb7d34-trusted-ca-bundle\") pod \"periodic-gathering-lmblb-khh8v\" (UID: \"287cc66c-4032-4bfa-a316-250ce6cb7d34\") " pod="openshift-insights/periodic-gathering-lmblb-khh8v"
Oct 31 07:50:26.633644 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:50:26.633584    4720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8z59p\" (UniqueName: \"kubernetes.io/projected/287cc66c-4032-4bfa-a316-250ce6cb7d34-kube-api-access-8z59p\") pod \"periodic-gathering-lmblb-khh8v\" (UID: \"287cc66c-4032-4bfa-a316-250ce6cb7d34\") " pod="openshift-insights/periodic-gathering-lmblb-khh8v"
Oct 31 07:50:26.633740 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:50:26.633717    4720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/287cc66c-4032-4bfa-a316-250ce6cb7d34-archives-path\") pod \"periodic-gathering-lmblb-khh8v\" (UID: \"287cc66c-4032-4bfa-a316-250ce6cb7d34\") " pod="openshift-insights/periodic-gathering-lmblb-khh8v"
Oct 31 07:50:26.634052 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:50:26.634031    4720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/287cc66c-4032-4bfa-a316-250ce6cb7d34-archives-path\") pod \"periodic-gathering-lmblb-khh8v\" (UID: \"287cc66c-4032-4bfa-a316-250ce6cb7d34\") " pod="openshift-insights/periodic-gathering-lmblb-khh8v"
Oct 31 07:50:26.634318 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:50:26.634297    4720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/287cc66c-4032-4bfa-a316-250ce6cb7d34-service-ca-bundle\") pod \"periodic-gathering-lmblb-khh8v\" (UID: \"287cc66c-4032-4bfa-a316-250ce6cb7d34\") " pod="openshift-insights/periodic-gathering-lmblb-khh8v"
Oct 31 07:50:26.634399 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:50:26.634384    4720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/287cc66c-4032-4bfa-a316-250ce6cb7d34-trusted-ca-bundle\") pod \"periodic-gathering-lmblb-khh8v\" (UID: \"287cc66c-4032-4bfa-a316-250ce6cb7d34\") " pod="openshift-insights/periodic-gathering-lmblb-khh8v"
Oct 31 07:50:26.689775 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:50:26.689723    4720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8z59p\" (UniqueName: \"kubernetes.io/projected/287cc66c-4032-4bfa-a316-250ce6cb7d34-kube-api-access-8z59p\") pod \"periodic-gathering-lmblb-khh8v\" (UID: \"287cc66c-4032-4bfa-a316-250ce6cb7d34\") " pod="openshift-insights/periodic-gathering-lmblb-khh8v"
Oct 31 07:50:26.703531 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:50:26.703514    4720 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-lmblb-khh8v"
Oct 31 07:50:27.072139 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:50:27.072069    4720 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-lmblb-khh8v"]
Oct 31 07:50:27.075576 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:50:27.075551    4720 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod287cc66c_4032_4bfa_a316_250ce6cb7d34.slice/crio-3589e7fec2623fab48dfd0a4710933ac1d2345bd2a0ccdf05898caee3aaae1d7 WatchSource:0}: Error finding container 3589e7fec2623fab48dfd0a4710933ac1d2345bd2a0ccdf05898caee3aaae1d7: Status 404 returned error can't find the container with id 3589e7fec2623fab48dfd0a4710933ac1d2345bd2a0ccdf05898caee3aaae1d7
Oct 31 07:50:27.365734 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:50:27.365702    4720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-lmblb-khh8v" event={"ID":"287cc66c-4032-4bfa-a316-250ce6cb7d34","Type":"ContainerStarted","Data":"afba0824f8629581551348a3c4a97eaff8c37e92d81071f1295eb1c3b8c733fd"}
Oct 31 07:50:27.365904 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:50:27.365751    4720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-lmblb-khh8v" event={"ID":"287cc66c-4032-4bfa-a316-250ce6cb7d34","Type":"ContainerStarted","Data":"3589e7fec2623fab48dfd0a4710933ac1d2345bd2a0ccdf05898caee3aaae1d7"}
Oct 31 07:50:27.478371 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:50:27.478312    4720 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/periodic-gathering-lmblb-khh8v" podStartSLOduration=1.478299696 podStartE2EDuration="1.478299696s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 07:50:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 07:50:27.476930302 +0000 UTC m=+1281.607998372" watchObservedRunningTime="2026-10-31 07:50:27.478299696 +0000 UTC m=+1281.609367766"
Oct 31 07:50:29.470654 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:50:29.470602    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 07:50:29.623815 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:50:29.623786    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 07:50:30.119002 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:50:30.118970    4720 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-599dcb8f76lf46m"]
Oct 31 07:50:30.122275 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:50:30.122259    4720 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-599dcb8f76lf46m"
Oct 31 07:50:30.225753 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:50:30.225725    4720 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-aae51b37\"/\"kube-root-ca.crt\""
Oct 31 07:50:30.225753 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:50:30.225726    4720 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-aae51b37\"/\"openshift-service-ca.crt\""
Oct 31 07:50:30.225943 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:50:30.225867    4720 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-aae51b37\"/\"default-dockercfg-fz4zj\""
Oct 31 07:50:30.225943 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:50:30.225867    4720 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-aae51b37\"/\"llmisv431d7e30f13554296364403a5b6365f5-kserve-self-signed-certs\""
Oct 31 07:50:30.227210 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:50:30.227191    4720 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-599dcb8f76lf46m"]
Oct 31 07:50:30.239822 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:50:30.239801    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 07:50:30.259949 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:50:30.259928    4720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ae3fd7c0-d7e7-4623-be62-233560d64000-home\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-599dcb8f76lf46m\" (UID: \"ae3fd7c0-d7e7-4623-be62-233560d64000\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-599dcb8f76lf46m"
Oct 31 07:50:30.260042 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:50:30.259966    4720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ae3fd7c0-d7e7-4623-be62-233560d64000-model-cache\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-599dcb8f76lf46m\" (UID: \"ae3fd7c0-d7e7-4623-be62-233560d64000\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-599dcb8f76lf46m"
Oct 31 07:50:30.260042 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:50:30.259992    4720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ae3fd7c0-d7e7-4623-be62-233560d64000-dshm\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-599dcb8f76lf46m\" (UID: \"ae3fd7c0-d7e7-4623-be62-233560d64000\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-599dcb8f76lf46m"
Oct 31 07:50:30.260140 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:50:30.260117    4720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ae3fd7c0-d7e7-4623-be62-233560d64000-tls-certs\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-599dcb8f76lf46m\" (UID: \"ae3fd7c0-d7e7-4623-be62-233560d64000\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-599dcb8f76lf46m"
Oct 31 07:50:30.260178 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:50:30.260151    4720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z4jqr\" (UniqueName: \"kubernetes.io/projected/ae3fd7c0-d7e7-4623-be62-233560d64000-kube-api-access-z4jqr\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-599dcb8f76lf46m\" (UID: \"ae3fd7c0-d7e7-4623-be62-233560d64000\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-599dcb8f76lf46m"
Oct 31 07:50:30.260243 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:50:30.260230    4720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ae3fd7c0-d7e7-4623-be62-233560d64000-tmp-dir\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-599dcb8f76lf46m\" (UID: \"ae3fd7c0-d7e7-4623-be62-233560d64000\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-599dcb8f76lf46m"
Oct 31 07:50:30.361220 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:50:30.361195    4720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ae3fd7c0-d7e7-4623-be62-233560d64000-model-cache\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-599dcb8f76lf46m\" (UID: \"ae3fd7c0-d7e7-4623-be62-233560d64000\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-599dcb8f76lf46m"
Oct 31 07:50:30.361387 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:50:30.361239    4720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ae3fd7c0-d7e7-4623-be62-233560d64000-dshm\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-599dcb8f76lf46m\" (UID: \"ae3fd7c0-d7e7-4623-be62-233560d64000\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-599dcb8f76lf46m"
Oct 31 07:50:30.361387 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:50:30.361261    4720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ae3fd7c0-d7e7-4623-be62-233560d64000-tls-certs\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-599dcb8f76lf46m\" (UID: \"ae3fd7c0-d7e7-4623-be62-233560d64000\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-599dcb8f76lf46m"
Oct 31 07:50:30.361387 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:50:30.361278    4720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z4jqr\" (UniqueName: \"kubernetes.io/projected/ae3fd7c0-d7e7-4623-be62-233560d64000-kube-api-access-z4jqr\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-599dcb8f76lf46m\" (UID: \"ae3fd7c0-d7e7-4623-be62-233560d64000\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-599dcb8f76lf46m"
Oct 31 07:50:30.361563 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:50:30.361406    4720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ae3fd7c0-d7e7-4623-be62-233560d64000-tmp-dir\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-599dcb8f76lf46m\" (UID: \"ae3fd7c0-d7e7-4623-be62-233560d64000\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-599dcb8f76lf46m"
Oct 31 07:50:30.361563 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:50:30.361452    4720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ae3fd7c0-d7e7-4623-be62-233560d64000-home\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-599dcb8f76lf46m\" (UID: \"ae3fd7c0-d7e7-4623-be62-233560d64000\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-599dcb8f76lf46m"
Oct 31 07:50:30.361691 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:50:30.361671    4720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ae3fd7c0-d7e7-4623-be62-233560d64000-model-cache\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-599dcb8f76lf46m\" (UID: \"ae3fd7c0-d7e7-4623-be62-233560d64000\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-599dcb8f76lf46m"
Oct 31 07:50:30.361757 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:50:30.361735    4720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ae3fd7c0-d7e7-4623-be62-233560d64000-tmp-dir\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-599dcb8f76lf46m\" (UID: \"ae3fd7c0-d7e7-4623-be62-233560d64000\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-599dcb8f76lf46m"
Oct 31 07:50:30.361791 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:50:30.361765    4720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ae3fd7c0-d7e7-4623-be62-233560d64000-home\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-599dcb8f76lf46m\" (UID: \"ae3fd7c0-d7e7-4623-be62-233560d64000\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-599dcb8f76lf46m"
Oct 31 07:50:30.363686 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:50:30.363664    4720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ae3fd7c0-d7e7-4623-be62-233560d64000-dshm\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-599dcb8f76lf46m\" (UID: \"ae3fd7c0-d7e7-4623-be62-233560d64000\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-599dcb8f76lf46m"
Oct 31 07:50:30.363837 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:50:30.363821    4720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ae3fd7c0-d7e7-4623-be62-233560d64000-tls-certs\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-599dcb8f76lf46m\" (UID: \"ae3fd7c0-d7e7-4623-be62-233560d64000\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-599dcb8f76lf46m"
Oct 31 07:50:30.417677 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:50:30.417619    4720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z4jqr\" (UniqueName: \"kubernetes.io/projected/ae3fd7c0-d7e7-4623-be62-233560d64000-kube-api-access-z4jqr\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-599dcb8f76lf46m\" (UID: \"ae3fd7c0-d7e7-4623-be62-233560d64000\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-599dcb8f76lf46m"
Oct 31 07:50:30.433643 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:50:30.433622    4720 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-599dcb8f76lf46m"
Oct 31 07:50:30.686399 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:50:30.686313    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 07:50:30.800716 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:50:30.800692    4720 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-599dcb8f76lf46m"]
Oct 31 07:50:30.804857 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:50:30.804826    4720 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae3fd7c0_d7e7_4623_be62_233560d64000.slice/crio-796ce4c549e2016e98538ae4f2d6d77f1183e78ca537caac9f28b16b8513d4f0 WatchSource:0}: Error finding container 796ce4c549e2016e98538ae4f2d6d77f1183e78ca537caac9f28b16b8513d4f0: Status 404 returned error can't find the container with id 796ce4c549e2016e98538ae4f2d6d77f1183e78ca537caac9f28b16b8513d4f0
Oct 31 07:50:31.384959 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:50:31.384916    4720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-599dcb8f76lf46m" event={"ID":"ae3fd7c0-d7e7-4623-be62-233560d64000","Type":"ContainerStarted","Data":"796ce4c549e2016e98538ae4f2d6d77f1183e78ca537caac9f28b16b8513d4f0"}
Oct 31 07:50:34.403883 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:50:34.403849    4720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-599dcb8f76lf46m" event={"ID":"ae3fd7c0-d7e7-4623-be62-233560d64000","Type":"ContainerStarted","Data":"0b2b83f63ffacddf405ef1d22822a75ef52ad504480a544ed77369977cba22cd"}
Oct 31 07:50:34.434629 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:50:34.434605    4720 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-599dcb8f76lf46m"
Oct 31 07:50:34.517363 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:50:34.517280    4720 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-599dcb8f76lf46m" podStartSLOduration=1.40704551 podStartE2EDuration="4.517262566s" totalImagesPullingTime="3.110217056s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 07:50:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 07:50:34.516409774 +0000 UTC m=+1288.647477843" watchObservedRunningTime="2026-10-31 07:50:34.517262566 +0000 UTC m=+1288.648330636"
Oct 31 07:50:35.629750 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:50:35.629718    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 07:50:37.732764 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:50:37.732731    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 07:50:40.434627 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:50:40.434591    4720 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-599dcb8f76lf46m"
Oct 31 07:50:40.446554 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:50:40.446529    4720 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-599dcb8f76lf46m"
Oct 31 07:50:41.036269 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:50:41.036242    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 07:50:41.189603 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:50:41.189574    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 07:50:41.444208 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:50:41.444179    4720 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-599dcb8f76lf46m"
Oct 31 07:50:43.460887 ip-10-0-139-83 kubenswrapper[4720]: E1031 07:50:43.460853    4720 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8m768" containerName="extractor"
Oct 31 07:50:46.553596 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:50:46.553528    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 07:50:51.584471 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:50:51.584443    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 07:50:51.665177 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:50:51.665149    4720 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-56f46b8cdd-gd6lh"]
Oct 31 07:50:51.670241 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:50:51.670214    4720 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-default-56f46b8cdd-gd6lh"
Oct 31 07:50:51.714016 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:50:51.713994    4720 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-56f46b8cdd-fwl5k"]
Oct 31 07:50:51.717199 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:50:51.717184    4720 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-default-56f46b8cdd-fwl5k"
Oct 31 07:50:51.753835 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:50:51.753807    4720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a4330a4d-4a3f-40ea-b824-5b5c8c1fb0ee-workload-certs\") pod \"openshift-ai-inference-openshift-default-56f46b8cdd-gd6lh\" (UID: \"a4330a4d-4a3f-40ea-b824-5b5c8c1fb0ee\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-56f46b8cdd-gd6lh"
Oct 31 07:50:51.753955 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:50:51.753845    4720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a4330a4d-4a3f-40ea-b824-5b5c8c1fb0ee-istio-token\") pod \"openshift-ai-inference-openshift-default-56f46b8cdd-gd6lh\" (UID: \"a4330a4d-4a3f-40ea-b824-5b5c8c1fb0ee\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-56f46b8cdd-gd6lh"
Oct 31 07:50:51.753955 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:50:51.753928    4720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d133cdef-3e19-49da-86bd-1e75e4191ab2-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-default-56f46b8cdd-fwl5k\" (UID: \"d133cdef-3e19-49da-86bd-1e75e4191ab2\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-56f46b8cdd-fwl5k"
Oct 31 07:50:51.754068 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:50:51.753974    4720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a4330a4d-4a3f-40ea-b824-5b5c8c1fb0ee-istio-envoy\") pod \"openshift-ai-inference-openshift-default-56f46b8cdd-gd6lh\" (UID: \"a4330a4d-4a3f-40ea-b824-5b5c8c1fb0ee\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-56f46b8cdd-gd6lh"
Oct 31 07:50:51.754068 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:50:51.753995    4720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d133cdef-3e19-49da-86bd-1e75e4191ab2-credential-socket\") pod \"openshift-ai-inference-openshift-default-56f46b8cdd-fwl5k\" (UID: \"d133cdef-3e19-49da-86bd-1e75e4191ab2\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-56f46b8cdd-fwl5k"
Oct 31 07:50:51.754170 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:50:51.754067    4720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a4330a4d-4a3f-40ea-b824-5b5c8c1fb0ee-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-default-56f46b8cdd-gd6lh\" (UID: \"a4330a4d-4a3f-40ea-b824-5b5c8c1fb0ee\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-56f46b8cdd-gd6lh"
Oct 31 07:50:51.754170 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:50:51.754089    4720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d133cdef-3e19-49da-86bd-1e75e4191ab2-workload-socket\") pod \"openshift-ai-inference-openshift-default-56f46b8cdd-fwl5k\" (UID: \"d133cdef-3e19-49da-86bd-1e75e4191ab2\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-56f46b8cdd-fwl5k"
Oct 31 07:50:51.754170 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:50:51.754149    4720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d133cdef-3e19-49da-86bd-1e75e4191ab2-istio-data\") pod \"openshift-ai-inference-openshift-default-56f46b8cdd-fwl5k\" (UID: \"d133cdef-3e19-49da-86bd-1e75e4191ab2\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-56f46b8cdd-fwl5k"
Oct 31 07:50:51.754312 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:50:51.754179    4720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d133cdef-3e19-49da-86bd-1e75e4191ab2-istio-token\") pod \"openshift-ai-inference-openshift-default-56f46b8cdd-fwl5k\" (UID: \"d133cdef-3e19-49da-86bd-1e75e4191ab2\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-56f46b8cdd-fwl5k"
Oct 31 07:50:51.754312 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:50:51.754252    4720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4330a4d-4a3f-40ea-b824-5b5c8c1fb0ee-workload-socket\") pod \"openshift-ai-inference-openshift-default-56f46b8cdd-gd6lh\" (UID: \"a4330a4d-4a3f-40ea-b824-5b5c8c1fb0ee\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-56f46b8cdd-gd6lh"
Oct 31 07:50:51.754312 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:50:51.754268    4720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d133cdef-3e19-49da-86bd-1e75e4191ab2-workload-certs\") pod \"openshift-ai-inference-openshift-default-56f46b8cdd-fwl5k\" (UID: \"d133cdef-3e19-49da-86bd-1e75e4191ab2\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-56f46b8cdd-fwl5k"
Oct 31 07:50:51.754312 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:50:51.754290    4720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a4330a4d-4a3f-40ea-b824-5b5c8c1fb0ee-istio-podinfo\") pod \"openshift-ai-inference-openshift-default-56f46b8cdd-gd6lh\" (UID: \"a4330a4d-4a3f-40ea-b824-5b5c8c1fb0ee\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-56f46b8cdd-gd6lh"
Oct 31 07:50:51.754535 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:50:51.754370    4720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4330a4d-4a3f-40ea-b824-5b5c8c1fb0ee-credential-socket\") pod \"openshift-ai-inference-openshift-default-56f46b8cdd-gd6lh\" (UID: \"a4330a4d-4a3f-40ea-b824-5b5c8c1fb0ee\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-56f46b8cdd-gd6lh"
Oct 31 07:50:51.754535 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:50:51.754387    4720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d133cdef-3e19-49da-86bd-1e75e4191ab2-istio-podinfo\") pod \"openshift-ai-inference-openshift-default-56f46b8cdd-fwl5k\" (UID: \"d133cdef-3e19-49da-86bd-1e75e4191ab2\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-56f46b8cdd-fwl5k"
Oct 31 07:50:51.754535 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:50:51.754409    4720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a4330a4d-4a3f-40ea-b824-5b5c8c1fb0ee-istio-data\") pod \"openshift-ai-inference-openshift-default-56f46b8cdd-gd6lh\" (UID: \"a4330a4d-4a3f-40ea-b824-5b5c8c1fb0ee\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-56f46b8cdd-gd6lh"
Oct 31 07:50:51.754535 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:50:51.754453    4720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-79qv7\" (UniqueName: \"kubernetes.io/projected/d133cdef-3e19-49da-86bd-1e75e4191ab2-kube-api-access-79qv7\") pod \"openshift-ai-inference-openshift-default-56f46b8cdd-fwl5k\" (UID: \"d133cdef-3e19-49da-86bd-1e75e4191ab2\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-56f46b8cdd-fwl5k"
Oct 31 07:50:51.754535 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:50:51.754474    4720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vxpk2\" (UniqueName: \"kubernetes.io/projected/a4330a4d-4a3f-40ea-b824-5b5c8c1fb0ee-kube-api-access-vxpk2\") pod \"openshift-ai-inference-openshift-default-56f46b8cdd-gd6lh\" (UID: \"a4330a4d-4a3f-40ea-b824-5b5c8c1fb0ee\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-56f46b8cdd-gd6lh"
Oct 31 07:50:51.754535 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:50:51.754490    4720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d133cdef-3e19-49da-86bd-1e75e4191ab2-istio-envoy\") pod \"openshift-ai-inference-openshift-default-56f46b8cdd-fwl5k\" (UID: \"d133cdef-3e19-49da-86bd-1e75e4191ab2\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-56f46b8cdd-fwl5k"
Oct 31 07:50:51.768075 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:50:51.768054    4720 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-56f46b8cdd-gd6lh"]
Oct 31 07:50:51.854904 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:50:51.854850    4720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a4330a4d-4a3f-40ea-b824-5b5c8c1fb0ee-istio-envoy\") pod \"openshift-ai-inference-openshift-default-56f46b8cdd-gd6lh\" (UID: \"a4330a4d-4a3f-40ea-b824-5b5c8c1fb0ee\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-56f46b8cdd-gd6lh"
Oct 31 07:50:51.854904 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:50:51.854878    4720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d133cdef-3e19-49da-86bd-1e75e4191ab2-credential-socket\") pod \"openshift-ai-inference-openshift-default-56f46b8cdd-fwl5k\" (UID: \"d133cdef-3e19-49da-86bd-1e75e4191ab2\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-56f46b8cdd-fwl5k"
Oct 31 07:50:51.854904 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:50:51.854897    4720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a4330a4d-4a3f-40ea-b824-5b5c8c1fb0ee-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-default-56f46b8cdd-gd6lh\" (UID: \"a4330a4d-4a3f-40ea-b824-5b5c8c1fb0ee\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-56f46b8cdd-gd6lh"
Oct 31 07:50:51.855122 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:50:51.855018    4720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d133cdef-3e19-49da-86bd-1e75e4191ab2-workload-socket\") pod \"openshift-ai-inference-openshift-default-56f46b8cdd-fwl5k\" (UID: \"d133cdef-3e19-49da-86bd-1e75e4191ab2\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-56f46b8cdd-fwl5k"
Oct 31 07:50:51.855122 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:50:51.855055    4720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d133cdef-3e19-49da-86bd-1e75e4191ab2-istio-data\") pod \"openshift-ai-inference-openshift-default-56f46b8cdd-fwl5k\" (UID: \"d133cdef-3e19-49da-86bd-1e75e4191ab2\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-56f46b8cdd-fwl5k"
Oct 31 07:50:51.855122 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:50:51.855079    4720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d133cdef-3e19-49da-86bd-1e75e4191ab2-istio-token\") pod \"openshift-ai-inference-openshift-default-56f46b8cdd-fwl5k\" (UID: \"d133cdef-3e19-49da-86bd-1e75e4191ab2\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-56f46b8cdd-fwl5k"
Oct 31 07:50:51.855361 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:50:51.855135    4720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4330a4d-4a3f-40ea-b824-5b5c8c1fb0ee-workload-socket\") pod \"openshift-ai-inference-openshift-default-56f46b8cdd-gd6lh\" (UID: \"a4330a4d-4a3f-40ea-b824-5b5c8c1fb0ee\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-56f46b8cdd-gd6lh"
Oct 31 07:50:51.855361 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:50:51.855166    4720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d133cdef-3e19-49da-86bd-1e75e4191ab2-workload-certs\") pod \"openshift-ai-inference-openshift-default-56f46b8cdd-fwl5k\" (UID: \"d133cdef-3e19-49da-86bd-1e75e4191ab2\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-56f46b8cdd-fwl5k"
Oct 31 07:50:51.855361 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:50:51.855205    4720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a4330a4d-4a3f-40ea-b824-5b5c8c1fb0ee-istio-podinfo\") pod \"openshift-ai-inference-openshift-default-56f46b8cdd-gd6lh\" (UID: \"a4330a4d-4a3f-40ea-b824-5b5c8c1fb0ee\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-56f46b8cdd-gd6lh"
Oct 31 07:50:51.855361 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:50:51.855241    4720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4330a4d-4a3f-40ea-b824-5b5c8c1fb0ee-credential-socket\") pod \"openshift-ai-inference-openshift-default-56f46b8cdd-gd6lh\" (UID: \"a4330a4d-4a3f-40ea-b824-5b5c8c1fb0ee\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-56f46b8cdd-gd6lh"
Oct 31 07:50:51.855361 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:50:51.855268    4720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d133cdef-3e19-49da-86bd-1e75e4191ab2-istio-podinfo\") pod \"openshift-ai-inference-openshift-default-56f46b8cdd-fwl5k\" (UID: \"d133cdef-3e19-49da-86bd-1e75e4191ab2\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-56f46b8cdd-fwl5k"
Oct 31 07:50:51.855361 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:50:51.855272    4720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d133cdef-3e19-49da-86bd-1e75e4191ab2-credential-socket\") pod \"openshift-ai-inference-openshift-default-56f46b8cdd-fwl5k\" (UID: \"d133cdef-3e19-49da-86bd-1e75e4191ab2\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-56f46b8cdd-fwl5k"
Oct 31 07:50:51.855361 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:50:51.855352    4720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a4330a4d-4a3f-40ea-b824-5b5c8c1fb0ee-istio-data\") pod \"openshift-ai-inference-openshift-default-56f46b8cdd-gd6lh\" (UID: \"a4330a4d-4a3f-40ea-b824-5b5c8c1fb0ee\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-56f46b8cdd-gd6lh"
Oct 31 07:50:51.855717 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:50:51.855385    4720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-79qv7\" (UniqueName: \"kubernetes.io/projected/d133cdef-3e19-49da-86bd-1e75e4191ab2-kube-api-access-79qv7\") pod \"openshift-ai-inference-openshift-default-56f46b8cdd-fwl5k\" (UID: \"d133cdef-3e19-49da-86bd-1e75e4191ab2\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-56f46b8cdd-fwl5k"
Oct 31 07:50:51.855717 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:50:51.855424    4720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vxpk2\" (UniqueName: \"kubernetes.io/projected/a4330a4d-4a3f-40ea-b824-5b5c8c1fb0ee-kube-api-access-vxpk2\") pod \"openshift-ai-inference-openshift-default-56f46b8cdd-gd6lh\" (UID: \"a4330a4d-4a3f-40ea-b824-5b5c8c1fb0ee\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-56f46b8cdd-gd6lh"
Oct 31 07:50:51.855717 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:50:51.855452    4720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d133cdef-3e19-49da-86bd-1e75e4191ab2-istio-envoy\") pod \"openshift-ai-inference-openshift-default-56f46b8cdd-fwl5k\" (UID: \"d133cdef-3e19-49da-86bd-1e75e4191ab2\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-56f46b8cdd-fwl5k"
Oct 31 07:50:51.855717 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:50:51.855456    4720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d133cdef-3e19-49da-86bd-1e75e4191ab2-istio-data\") pod \"openshift-ai-inference-openshift-default-56f46b8cdd-fwl5k\" (UID: \"d133cdef-3e19-49da-86bd-1e75e4191ab2\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-56f46b8cdd-fwl5k"
Oct 31 07:50:51.855717 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:50:51.855492    4720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a4330a4d-4a3f-40ea-b824-5b5c8c1fb0ee-workload-certs\") pod \"openshift-ai-inference-openshift-default-56f46b8cdd-gd6lh\" (UID: \"a4330a4d-4a3f-40ea-b824-5b5c8c1fb0ee\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-56f46b8cdd-gd6lh"
Oct 31 07:50:51.855717 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:50:51.855518    4720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a4330a4d-4a3f-40ea-b824-5b5c8c1fb0ee-istio-token\") pod \"openshift-ai-inference-openshift-default-56f46b8cdd-gd6lh\" (UID: \"a4330a4d-4a3f-40ea-b824-5b5c8c1fb0ee\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-56f46b8cdd-gd6lh"
Oct 31 07:50:51.855717 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:50:51.855552    4720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d133cdef-3e19-49da-86bd-1e75e4191ab2-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-default-56f46b8cdd-fwl5k\" (UID: \"d133cdef-3e19-49da-86bd-1e75e4191ab2\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-56f46b8cdd-fwl5k"
Oct 31 07:50:51.856216 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:50:51.855515    4720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4330a4d-4a3f-40ea-b824-5b5c8c1fb0ee-workload-socket\") pod \"openshift-ai-inference-openshift-default-56f46b8cdd-gd6lh\" (UID: \"a4330a4d-4a3f-40ea-b824-5b5c8c1fb0ee\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-56f46b8cdd-gd6lh"
Oct 31 07:50:51.856216 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:50:51.855819    4720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a4330a4d-4a3f-40ea-b824-5b5c8c1fb0ee-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-default-56f46b8cdd-gd6lh\" (UID: \"a4330a4d-4a3f-40ea-b824-5b5c8c1fb0ee\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-56f46b8cdd-gd6lh"
Oct 31 07:50:51.856216 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:50:51.855882    4720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a4330a4d-4a3f-40ea-b824-5b5c8c1fb0ee-istio-data\") pod \"openshift-ai-inference-openshift-default-56f46b8cdd-gd6lh\" (UID: \"a4330a4d-4a3f-40ea-b824-5b5c8c1fb0ee\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-56f46b8cdd-gd6lh"
Oct 31 07:50:51.856216 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:50:51.855906    4720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a4330a4d-4a3f-40ea-b824-5b5c8c1fb0ee-workload-certs\") pod \"openshift-ai-inference-openshift-default-56f46b8cdd-gd6lh\" (UID: \"a4330a4d-4a3f-40ea-b824-5b5c8c1fb0ee\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-56f46b8cdd-gd6lh"
Oct 31 07:50:51.856216 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:50:51.856046    4720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d133cdef-3e19-49da-86bd-1e75e4191ab2-workload-certs\") pod \"openshift-ai-inference-openshift-default-56f46b8cdd-fwl5k\" (UID: \"d133cdef-3e19-49da-86bd-1e75e4191ab2\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-56f46b8cdd-fwl5k"
Oct 31 07:50:51.856216 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:50:51.856136    4720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4330a4d-4a3f-40ea-b824-5b5c8c1fb0ee-credential-socket\") pod \"openshift-ai-inference-openshift-default-56f46b8cdd-gd6lh\" (UID: \"a4330a4d-4a3f-40ea-b824-5b5c8c1fb0ee\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-56f46b8cdd-gd6lh"
Oct 31 07:50:51.856548 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:50:51.856321    4720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d133cdef-3e19-49da-86bd-1e75e4191ab2-workload-socket\") pod \"openshift-ai-inference-openshift-default-56f46b8cdd-fwl5k\" (UID: \"d133cdef-3e19-49da-86bd-1e75e4191ab2\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-56f46b8cdd-fwl5k"
Oct 31 07:50:51.856603 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:50:51.856563    4720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d133cdef-3e19-49da-86bd-1e75e4191ab2-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-default-56f46b8cdd-fwl5k\" (UID: \"d133cdef-3e19-49da-86bd-1e75e4191ab2\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-56f46b8cdd-fwl5k"
Oct 31 07:50:51.857549 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:50:51.857526    4720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a4330a4d-4a3f-40ea-b824-5b5c8c1fb0ee-istio-envoy\") pod \"openshift-ai-inference-openshift-default-56f46b8cdd-gd6lh\" (UID: \"a4330a4d-4a3f-40ea-b824-5b5c8c1fb0ee\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-56f46b8cdd-gd6lh"
Oct 31 07:50:51.857797 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:50:51.857780    4720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d133cdef-3e19-49da-86bd-1e75e4191ab2-istio-podinfo\") pod \"openshift-ai-inference-openshift-default-56f46b8cdd-fwl5k\" (UID: \"d133cdef-3e19-49da-86bd-1e75e4191ab2\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-56f46b8cdd-fwl5k"
Oct 31 07:50:51.859027 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:50:51.859002    4720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d133cdef-3e19-49da-86bd-1e75e4191ab2-istio-envoy\") pod \"openshift-ai-inference-openshift-default-56f46b8cdd-fwl5k\" (UID: \"d133cdef-3e19-49da-86bd-1e75e4191ab2\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-56f46b8cdd-fwl5k"
Oct 31 07:50:51.859211 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:50:51.859194    4720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a4330a4d-4a3f-40ea-b824-5b5c8c1fb0ee-istio-podinfo\") pod \"openshift-ai-inference-openshift-default-56f46b8cdd-gd6lh\" (UID: \"a4330a4d-4a3f-40ea-b824-5b5c8c1fb0ee\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-56f46b8cdd-gd6lh"
Oct 31 07:50:51.868904 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:50:51.868883    4720 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-56f46b8cdd-fwl5k"]
Oct 31 07:50:51.914954 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:50:51.914926    4720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a4330a4d-4a3f-40ea-b824-5b5c8c1fb0ee-istio-token\") pod \"openshift-ai-inference-openshift-default-56f46b8cdd-gd6lh\" (UID: \"a4330a4d-4a3f-40ea-b824-5b5c8c1fb0ee\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-56f46b8cdd-gd6lh"
Oct 31 07:50:51.915118 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:50:51.915100    4720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vxpk2\" (UniqueName: \"kubernetes.io/projected/a4330a4d-4a3f-40ea-b824-5b5c8c1fb0ee-kube-api-access-vxpk2\") pod \"openshift-ai-inference-openshift-default-56f46b8cdd-gd6lh\" (UID: \"a4330a4d-4a3f-40ea-b824-5b5c8c1fb0ee\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-56f46b8cdd-gd6lh"
Oct 31 07:50:51.915471 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:50:51.915448    4720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-79qv7\" (UniqueName: \"kubernetes.io/projected/d133cdef-3e19-49da-86bd-1e75e4191ab2-kube-api-access-79qv7\") pod \"openshift-ai-inference-openshift-default-56f46b8cdd-fwl5k\" (UID: \"d133cdef-3e19-49da-86bd-1e75e4191ab2\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-56f46b8cdd-fwl5k"
Oct 31 07:50:51.915637 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:50:51.915620    4720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d133cdef-3e19-49da-86bd-1e75e4191ab2-istio-token\") pod \"openshift-ai-inference-openshift-default-56f46b8cdd-fwl5k\" (UID: \"d133cdef-3e19-49da-86bd-1e75e4191ab2\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-56f46b8cdd-fwl5k"
Oct 31 07:50:51.983948 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:50:51.983922    4720 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-default-56f46b8cdd-gd6lh"
Oct 31 07:50:52.028932 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:50:52.028906    4720 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-default-56f46b8cdd-fwl5k"
Oct 31 07:50:52.247869 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:50:52.247774    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 07:50:52.411689 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:50:52.411667    4720 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-56f46b8cdd-gd6lh"]
Oct 31 07:50:52.414970 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:50:52.414940    4720 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda4330a4d_4a3f_40ea_b824_5b5c8c1fb0ee.slice/crio-a2959305e3ffa192d9470ef39cff9858554bc2afeffe68c7f89160a3b5e6634e WatchSource:0}: Error finding container a2959305e3ffa192d9470ef39cff9858554bc2afeffe68c7f89160a3b5e6634e: Status 404 returned error can't find the container with id a2959305e3ffa192d9470ef39cff9858554bc2afeffe68c7f89160a3b5e6634e
Oct 31 07:50:52.417442 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:50:52.417407    4720 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 31 07:50:52.417543 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:50:52.417472    4720 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 31 07:50:52.417543 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:50:52.417506    4720 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 31 07:50:52.464984 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:50:52.464950    4720 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-56f46b8cdd-fwl5k"]
Oct 31 07:50:52.473089 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:50:52.473061    4720 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 31 07:50:52.473192 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:50:52.473132    4720 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 31 07:50:52.473192 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:50:52.473166    4720 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 31 07:50:52.482496 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:50:52.482471    4720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-56f46b8cdd-fwl5k" event={"ID":"d133cdef-3e19-49da-86bd-1e75e4191ab2","Type":"ContainerStarted","Data":"eb3dbaad1158afa9a31224758dcb5dc1254ab7f28fb999331f4e8ce2e963c1f8"}
Oct 31 07:50:52.484537 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:50:52.484512    4720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-56f46b8cdd-gd6lh" event={"ID":"a4330a4d-4a3f-40ea-b824-5b5c8c1fb0ee","Type":"ContainerStarted","Data":"a2959305e3ffa192d9470ef39cff9858554bc2afeffe68c7f89160a3b5e6634e"}
Oct 31 07:50:53.525317 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:50:53.525278    4720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-56f46b8cdd-gd6lh" event={"ID":"a4330a4d-4a3f-40ea-b824-5b5c8c1fb0ee","Type":"ContainerStarted","Data":"7cf00be1f39377820191da1c0b6c3f34d6cdf182db37676709ded2c2518ddcc4"}
Oct 31 07:50:53.528638 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:50:53.528610    4720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-56f46b8cdd-fwl5k" event={"ID":"d133cdef-3e19-49da-86bd-1e75e4191ab2","Type":"ContainerStarted","Data":"8122081ffda3101946c6f766ddfda39e79bcb9ad7dbb5994551822a606b7019f"}
Oct 31 07:50:53.815192 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:50:53.815075    4720 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/openshift-ai-inference-openshift-default-56f46b8cdd-fwl5k" podStartSLOduration=2.815052922 podStartE2EDuration="2.815052922s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 07:50:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 07:50:53.814462423 +0000 UTC m=+1307.945530493" watchObservedRunningTime="2026-10-31 07:50:53.815052922 +0000 UTC m=+1307.946120992"
Oct 31 07:50:53.817045 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:50:53.816992    4720 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/openshift-ai-inference-openshift-default-56f46b8cdd-gd6lh" podStartSLOduration=2.816980345 podStartE2EDuration="2.816980345s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 07:50:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 07:50:53.698399505 +0000 UTC m=+1307.829467579" watchObservedRunningTime="2026-10-31 07:50:53.816980345 +0000 UTC m=+1307.948048416"
Oct 31 07:50:53.991354 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:50:53.985708    4720 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/openshift-ai-inference-openshift-default-56f46b8cdd-gd6lh"
Oct 31 07:50:53.991354 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:50:53.989449    4720 patch_prober.go:29] interesting pod/openshift-ai-inference-openshift-default-56f46b8cdd-gd6lh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.41:15021/healthz/ready\": dial tcp 10.132.0.41:15021: connect: connection refused" start-of-body=
Oct 31 07:50:53.991354 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:50:53.989503    4720 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-56f46b8cdd-gd6lh" podUID="a4330a4d-4a3f-40ea-b824-5b5c8c1fb0ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.41:15021/healthz/ready\": dial tcp 10.132.0.41:15021: connect: connection refused"
Oct 31 07:50:54.029761 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:50:54.029684    4720 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/openshift-ai-inference-openshift-default-56f46b8cdd-fwl5k"
Oct 31 07:50:54.030864 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:50:54.030836    4720 patch_prober.go:29] interesting pod/openshift-ai-inference-openshift-default-56f46b8cdd-fwl5k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.42:15021/healthz/ready\": dial tcp 10.132.0.42:15021: connect: connection refused" start-of-body=
Oct 31 07:50:54.030987 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:50:54.030908    4720 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-56f46b8cdd-fwl5k" podUID="d133cdef-3e19-49da-86bd-1e75e4191ab2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.42:15021/healthz/ready\": dial tcp 10.132.0.42:15021: connect: connection refused"
Oct 31 07:50:54.984783 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:50:54.984750    4720 patch_prober.go:29] interesting pod/openshift-ai-inference-openshift-default-56f46b8cdd-gd6lh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.41:15021/healthz/ready\": dial tcp 10.132.0.41:15021: connect: connection refused" start-of-body=
Oct 31 07:50:54.985367 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:50:54.985318    4720 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-56f46b8cdd-gd6lh" podUID="a4330a4d-4a3f-40ea-b824-5b5c8c1fb0ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.41:15021/healthz/ready\": dial tcp 10.132.0.41:15021: connect: connection refused"
Oct 31 07:50:55.029941 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:50:55.029907    4720 patch_prober.go:29] interesting pod/openshift-ai-inference-openshift-default-56f46b8cdd-fwl5k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.42:15021/healthz/ready\": dial tcp 10.132.0.42:15021: connect: connection refused" start-of-body=
Oct 31 07:50:55.030201 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:50:55.030153    4720 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-56f46b8cdd-fwl5k" podUID="d133cdef-3e19-49da-86bd-1e75e4191ab2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.42:15021/healthz/ready\": dial tcp 10.132.0.42:15021: connect: connection refused"
Oct 31 07:50:55.989352 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:50:55.988705    4720 patch_prober.go:29] interesting pod/openshift-ai-inference-openshift-default-56f46b8cdd-gd6lh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.41:15021/healthz/ready\": dial tcp 10.132.0.41:15021: connect: connection refused" start-of-body=
Oct 31 07:50:55.989352 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:50:55.988765    4720 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-56f46b8cdd-gd6lh" podUID="a4330a4d-4a3f-40ea-b824-5b5c8c1fb0ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.41:15021/healthz/ready\": dial tcp 10.132.0.41:15021: connect: connection refused"
Oct 31 07:50:56.030784 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:50:56.030741    4720 patch_prober.go:29] interesting pod/openshift-ai-inference-openshift-default-56f46b8cdd-fwl5k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.42:15021/healthz/ready\": dial tcp 10.132.0.42:15021: connect: connection refused" start-of-body=
Oct 31 07:50:56.031047 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:50:56.031019    4720 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-56f46b8cdd-fwl5k" podUID="d133cdef-3e19-49da-86bd-1e75e4191ab2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.42:15021/healthz/ready\": dial tcp 10.132.0.42:15021: connect: connection refused"
Oct 31 07:50:56.988351 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:50:56.987711    4720 patch_prober.go:29] interesting pod/openshift-ai-inference-openshift-default-56f46b8cdd-gd6lh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.41:15021/healthz/ready\": dial tcp 10.132.0.41:15021: connect: connection refused" start-of-body=
Oct 31 07:50:56.988351 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:50:56.987771    4720 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-56f46b8cdd-gd6lh" podUID="a4330a4d-4a3f-40ea-b824-5b5c8c1fb0ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.41:15021/healthz/ready\": dial tcp 10.132.0.41:15021: connect: connection refused"
Oct 31 07:50:57.029847 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:50:57.029812    4720 patch_prober.go:29] interesting pod/openshift-ai-inference-openshift-default-56f46b8cdd-fwl5k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.42:15021/healthz/ready\": dial tcp 10.132.0.42:15021: connect: connection refused" start-of-body=
Oct 31 07:50:57.030242 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:50:57.029871    4720 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-56f46b8cdd-fwl5k" podUID="d133cdef-3e19-49da-86bd-1e75e4191ab2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.42:15021/healthz/ready\": dial tcp 10.132.0.42:15021: connect: connection refused"
Oct 31 07:50:57.995355 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:50:57.993460    4720 patch_prober.go:29] interesting pod/openshift-ai-inference-openshift-default-56f46b8cdd-gd6lh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.41:15021/healthz/ready\": dial tcp 10.132.0.41:15021: connect: connection refused" start-of-body=
Oct 31 07:50:57.995355 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:50:57.993527    4720 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-56f46b8cdd-gd6lh" podUID="a4330a4d-4a3f-40ea-b824-5b5c8c1fb0ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.41:15021/healthz/ready\": dial tcp 10.132.0.41:15021: connect: connection refused"
Oct 31 07:50:58.029679 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:50:58.029649    4720 patch_prober.go:29] interesting pod/openshift-ai-inference-openshift-default-56f46b8cdd-fwl5k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.42:15021/healthz/ready\": dial tcp 10.132.0.42:15021: connect: connection refused" start-of-body=
Oct 31 07:50:58.029835 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:50:58.029695    4720 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-56f46b8cdd-fwl5k" podUID="d133cdef-3e19-49da-86bd-1e75e4191ab2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.42:15021/healthz/ready\": dial tcp 10.132.0.42:15021: connect: connection refused"
Oct 31 07:50:58.988431 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:50:58.988404    4720 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/openshift-ai-inference-openshift-default-56f46b8cdd-gd6lh"
Oct 31 07:50:58.988833 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:50:58.988677    4720 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/openshift-ai-inference-openshift-default-56f46b8cdd-gd6lh"
Oct 31 07:50:58.989276 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:50:58.989261    4720 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/openshift-ai-inference-openshift-default-56f46b8cdd-gd6lh"
Oct 31 07:50:59.032682 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:50:59.032656    4720 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/openshift-ai-inference-openshift-default-56f46b8cdd-fwl5k"
Oct 31 07:50:59.032938 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:50:59.032921    4720 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/openshift-ai-inference-openshift-default-56f46b8cdd-fwl5k"
Oct 31 07:50:59.033640 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:50:59.033623    4720 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/openshift-ai-inference-openshift-default-56f46b8cdd-fwl5k"
Oct 31 07:51:07.713587 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:51:07.713556    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 07:51:08.444011 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:51:08.443976    4720 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-599dcb8f76lf46m"]
Oct 31 07:51:08.454750 ip-10-0-139-83 kubenswrapper[4720]: E1031 07:51:08.454713    4720 log.go:32] "ExecSync cmd from runtime service failed" err="rpc error: code = Unknown desc = command error: , stdout: , stderr: , exit code -1" containerID="0b2b83f63ffacddf405ef1d22822a75ef52ad504480a544ed77369977cba22cd" cmd=["/bin/sleep","15"]
Oct 31 07:51:08.454899 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:51:08.454821    4720 handlers.go:85] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-599dcb8f76lf46m" message="" err="rpc error: code = Unknown desc = command error: , stdout: , stderr: , exit code -1"
Oct 31 07:51:08.454899 ip-10-0-139-83 kubenswrapper[4720]: E1031 07:51:08.454838    4720 kuberuntime_container.go:780] "PreStop hook failed" err="rpc error: code = Unknown desc = command error: , stdout: , stderr: , exit code -1" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-599dcb8f76lf46m" podUID="ae3fd7c0-d7e7-4623-be62-233560d64000" containerName="main" containerID="cri-o://0b2b83f63ffacddf405ef1d22822a75ef52ad504480a544ed77369977cba22cd"
Oct 31 07:51:08.454899 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:51:08.454882    4720 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-599dcb8f76lf46m" podUID="ae3fd7c0-d7e7-4623-be62-233560d64000" containerName="main" containerID="cri-o://0b2b83f63ffacddf405ef1d22822a75ef52ad504480a544ed77369977cba22cd" gracePeriod=60
Oct 31 07:51:08.505870 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:51:08.505844    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 07:51:08.682793 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:51:08.682700    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 07:51:08.830117 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:51:08.830093    4720 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-599dcb8f76lf46m"
Oct 31 07:51:08.837885 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:51:08.837864    4720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ae3fd7c0-d7e7-4623-be62-233560d64000-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ae3fd7c0-d7e7-4623-be62-233560d64000-model-cache\") pod \"ae3fd7c0-d7e7-4623-be62-233560d64000\" (UID: \"ae3fd7c0-d7e7-4623-be62-233560d64000\") "
Oct 31 07:51:08.837994 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:51:08.837903    4720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ae3fd7c0-d7e7-4623-be62-233560d64000-home\" (UniqueName: \"kubernetes.io/empty-dir/ae3fd7c0-d7e7-4623-be62-233560d64000-home\") pod \"ae3fd7c0-d7e7-4623-be62-233560d64000\" (UID: \"ae3fd7c0-d7e7-4623-be62-233560d64000\") "
Oct 31 07:51:08.837994 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:51:08.837937    4720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ae3fd7c0-d7e7-4623-be62-233560d64000-dshm\" (UniqueName: \"kubernetes.io/empty-dir/ae3fd7c0-d7e7-4623-be62-233560d64000-dshm\") pod \"ae3fd7c0-d7e7-4623-be62-233560d64000\" (UID: \"ae3fd7c0-d7e7-4623-be62-233560d64000\") "
Oct 31 07:51:08.838115 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:51:08.838002    4720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ae3fd7c0-d7e7-4623-be62-233560d64000-kube-api-access-z4jqr\" (UniqueName: \"kubernetes.io/projected/ae3fd7c0-d7e7-4623-be62-233560d64000-kube-api-access-z4jqr\") pod \"ae3fd7c0-d7e7-4623-be62-233560d64000\" (UID: \"ae3fd7c0-d7e7-4623-be62-233560d64000\") "
Oct 31 07:51:08.838115 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:51:08.838035    4720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ae3fd7c0-d7e7-4623-be62-233560d64000-tls-certs\" (UniqueName: \"kubernetes.io/secret/ae3fd7c0-d7e7-4623-be62-233560d64000-tls-certs\") pod \"ae3fd7c0-d7e7-4623-be62-233560d64000\" (UID: \"ae3fd7c0-d7e7-4623-be62-233560d64000\") "
Oct 31 07:51:08.838115 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:51:08.838077    4720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ae3fd7c0-d7e7-4623-be62-233560d64000-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ae3fd7c0-d7e7-4623-be62-233560d64000-tmp-dir\") pod \"ae3fd7c0-d7e7-4623-be62-233560d64000\" (UID: \"ae3fd7c0-d7e7-4623-be62-233560d64000\") "
Oct 31 07:51:08.838244 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:51:08.838104    4720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae3fd7c0-d7e7-4623-be62-233560d64000-model-cache" pod "ae3fd7c0-d7e7-4623-be62-233560d64000" (UID: "ae3fd7c0-d7e7-4623-be62-233560d64000"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 07:51:08.838244 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:51:08.838207    4720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae3fd7c0-d7e7-4623-be62-233560d64000-home" pod "ae3fd7c0-d7e7-4623-be62-233560d64000" (UID: "ae3fd7c0-d7e7-4623-be62-233560d64000"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 07:51:08.838410 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:51:08.838314    4720 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ae3fd7c0-d7e7-4623-be62-233560d64000-model-cache\") on node \"ip-10-0-139-83.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 07:51:08.838410 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:51:08.838359    4720 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ae3fd7c0-d7e7-4623-be62-233560d64000-home\") on node \"ip-10-0-139-83.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 07:51:08.838519 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:51:08.838421    4720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae3fd7c0-d7e7-4623-be62-233560d64000-tmp-dir" pod "ae3fd7c0-d7e7-4623-be62-233560d64000" (UID: "ae3fd7c0-d7e7-4623-be62-233560d64000"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 07:51:08.840642 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:51:08.840609    4720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ae3fd7c0-d7e7-4623-be62-233560d64000-tls-certs" pod "ae3fd7c0-d7e7-4623-be62-233560d64000" (UID: "ae3fd7c0-d7e7-4623-be62-233560d64000"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 07:51:08.840728 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:51:08.840711    4720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae3fd7c0-d7e7-4623-be62-233560d64000-dshm" pod "ae3fd7c0-d7e7-4623-be62-233560d64000" (UID: "ae3fd7c0-d7e7-4623-be62-233560d64000"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 07:51:08.840808 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:51:08.840746    4720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ae3fd7c0-d7e7-4623-be62-233560d64000-kube-api-access-z4jqr" pod "ae3fd7c0-d7e7-4623-be62-233560d64000" (UID: "ae3fd7c0-d7e7-4623-be62-233560d64000"). InnerVolumeSpecName "kube-api-access-z4jqr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 07:51:08.938722 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:51:08.938695    4720 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z4jqr\" (UniqueName: \"kubernetes.io/projected/ae3fd7c0-d7e7-4623-be62-233560d64000-kube-api-access-z4jqr\") on node \"ip-10-0-139-83.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 07:51:08.938722 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:51:08.938719    4720 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ae3fd7c0-d7e7-4623-be62-233560d64000-tls-certs\") on node \"ip-10-0-139-83.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 07:51:08.938873 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:51:08.938729    4720 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ae3fd7c0-d7e7-4623-be62-233560d64000-tmp-dir\") on node \"ip-10-0-139-83.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 07:51:08.938873 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:51:08.938738    4720 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ae3fd7c0-d7e7-4623-be62-233560d64000-dshm\") on node \"ip-10-0-139-83.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 07:51:09.616440 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:51:09.616404    4720 generic.go:356] "Generic (PLEG): container finished" podID="ae3fd7c0-d7e7-4623-be62-233560d64000" containerID="0b2b83f63ffacddf405ef1d22822a75ef52ad504480a544ed77369977cba22cd" exitCode=0
Oct 31 07:51:09.616619 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:51:09.616452    4720 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-599dcb8f76lf46m"
Oct 31 07:51:09.616619 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:51:09.616453    4720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-599dcb8f76lf46m" event={"ID":"ae3fd7c0-d7e7-4623-be62-233560d64000","Type":"ContainerDied","Data":"0b2b83f63ffacddf405ef1d22822a75ef52ad504480a544ed77369977cba22cd"}
Oct 31 07:51:09.616619 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:51:09.616570    4720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-599dcb8f76lf46m" event={"ID":"ae3fd7c0-d7e7-4623-be62-233560d64000","Type":"ContainerDied","Data":"796ce4c549e2016e98538ae4f2d6d77f1183e78ca537caac9f28b16b8513d4f0"}
Oct 31 07:51:09.616619 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:51:09.616598    4720 scope.go:122] "RemoveContainer" containerID="0b2b83f63ffacddf405ef1d22822a75ef52ad504480a544ed77369977cba22cd"
Oct 31 07:51:09.625971 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:51:09.625954    4720 scope.go:122] "RemoveContainer" containerID="0b2b83f63ffacddf405ef1d22822a75ef52ad504480a544ed77369977cba22cd"
Oct 31 07:51:09.626244 ip-10-0-139-83 kubenswrapper[4720]: E1031 07:51:09.626226    4720 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0b2b83f63ffacddf405ef1d22822a75ef52ad504480a544ed77369977cba22cd\": container with ID starting with 0b2b83f63ffacddf405ef1d22822a75ef52ad504480a544ed77369977cba22cd not found: ID does not exist" containerID="0b2b83f63ffacddf405ef1d22822a75ef52ad504480a544ed77369977cba22cd"
Oct 31 07:51:09.626287 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:51:09.626254    4720 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0b2b83f63ffacddf405ef1d22822a75ef52ad504480a544ed77369977cba22cd"} err="failed to get container status \"0b2b83f63ffacddf405ef1d22822a75ef52ad504480a544ed77369977cba22cd\": rpc error: code = NotFound desc = could not find container \"0b2b83f63ffacddf405ef1d22822a75ef52ad504480a544ed77369977cba22cd\": container with ID starting with 0b2b83f63ffacddf405ef1d22822a75ef52ad504480a544ed77369977cba22cd not found: ID does not exist"
Oct 31 07:51:09.776935 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:51:09.776909    4720 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-599dcb8f76lf46m"]
Oct 31 07:51:09.826165 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:51:09.826141    4720 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-599dcb8f76lf46m"]
Oct 31 07:51:10.025729 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:51:10.025661    4720 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ae3fd7c0-d7e7-4623-be62-233560d64000" path="/var/lib/kubelet/pods/ae3fd7c0-d7e7-4623-be62-233560d64000/volumes"
Oct 31 07:51:13.459024 ip-10-0-139-83 kubenswrapper[4720]: E1031 07:51:13.458988    4720 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8m768" containerName="extractor"
Oct 31 07:51:22.720501 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:51:22.720456    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 07:51:31.652970 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:51:31.652932    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 07:51:32.260035 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:51:32.260002    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 07:51:38.070702 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:51:38.070669    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 07:51:43.460710 ip-10-0-139-83 kubenswrapper[4720]: E1031 07:51:43.460668    4720 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8m768" containerName="extractor"
Oct 31 07:51:48.790496 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:51:48.790455    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 07:51:49.405052 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:51:49.405018    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 07:51:57.073914 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:51:57.073819    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 07:52:00.843440 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:52:00.843408    4720 generic.go:356] "Generic (PLEG): container finished" podID="287cc66c-4032-4bfa-a316-250ce6cb7d34" containerID="afba0824f8629581551348a3c4a97eaff8c37e92d81071f1295eb1c3b8c733fd" exitCode=0
Oct 31 07:52:00.843799 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:52:00.843484    4720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-lmblb-khh8v" event={"ID":"287cc66c-4032-4bfa-a316-250ce6cb7d34","Type":"ContainerDied","Data":"afba0824f8629581551348a3c4a97eaff8c37e92d81071f1295eb1c3b8c733fd"}
Oct 31 07:52:01.986315 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:52:01.986272    4720 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-lmblb-khh8v"
Oct 31 07:52:02.175612 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:52:02.175524    4720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/287cc66c-4032-4bfa-a316-250ce6cb7d34-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/287cc66c-4032-4bfa-a316-250ce6cb7d34-service-ca-bundle\") pod \"287cc66c-4032-4bfa-a316-250ce6cb7d34\" (UID: \"287cc66c-4032-4bfa-a316-250ce6cb7d34\") "
Oct 31 07:52:02.175612 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:52:02.175610    4720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/287cc66c-4032-4bfa-a316-250ce6cb7d34-archives-path\" (UniqueName: \"kubernetes.io/empty-dir/287cc66c-4032-4bfa-a316-250ce6cb7d34-archives-path\") pod \"287cc66c-4032-4bfa-a316-250ce6cb7d34\" (UID: \"287cc66c-4032-4bfa-a316-250ce6cb7d34\") "
Oct 31 07:52:02.175852 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:52:02.175643    4720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/287cc66c-4032-4bfa-a316-250ce6cb7d34-kube-api-access-8z59p\" (UniqueName: \"kubernetes.io/projected/287cc66c-4032-4bfa-a316-250ce6cb7d34-kube-api-access-8z59p\") pod \"287cc66c-4032-4bfa-a316-250ce6cb7d34\" (UID: \"287cc66c-4032-4bfa-a316-250ce6cb7d34\") "
Oct 31 07:52:02.175852 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:52:02.175687    4720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/287cc66c-4032-4bfa-a316-250ce6cb7d34-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/287cc66c-4032-4bfa-a316-250ce6cb7d34-trusted-ca-bundle\") pod \"287cc66c-4032-4bfa-a316-250ce6cb7d34\" (UID: \"287cc66c-4032-4bfa-a316-250ce6cb7d34\") "
Oct 31 07:52:02.176003 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:52:02.175975    4720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/287cc66c-4032-4bfa-a316-250ce6cb7d34-service-ca-bundle" pod "287cc66c-4032-4bfa-a316-250ce6cb7d34" (UID: "287cc66c-4032-4bfa-a316-250ce6cb7d34"). InnerVolumeSpecName "service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 07:52:02.176056 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:52:02.176002    4720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/287cc66c-4032-4bfa-a316-250ce6cb7d34-archives-path" pod "287cc66c-4032-4bfa-a316-250ce6cb7d34" (UID: "287cc66c-4032-4bfa-a316-250ce6cb7d34"). InnerVolumeSpecName "archives-path". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 07:52:02.176105 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:52:02.176084    4720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/287cc66c-4032-4bfa-a316-250ce6cb7d34-trusted-ca-bundle" pod "287cc66c-4032-4bfa-a316-250ce6cb7d34" (UID: "287cc66c-4032-4bfa-a316-250ce6cb7d34"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 07:52:02.177987 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:52:02.177962    4720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/287cc66c-4032-4bfa-a316-250ce6cb7d34-kube-api-access-8z59p" pod "287cc66c-4032-4bfa-a316-250ce6cb7d34" (UID: "287cc66c-4032-4bfa-a316-250ce6cb7d34"). InnerVolumeSpecName "kube-api-access-8z59p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 07:52:02.276717 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:52:02.276681    4720 reconciler_common.go:299] "Volume detached for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/287cc66c-4032-4bfa-a316-250ce6cb7d34-service-ca-bundle\") on node \"ip-10-0-139-83.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 07:52:02.276717 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:52:02.276711    4720 reconciler_common.go:299] "Volume detached for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/287cc66c-4032-4bfa-a316-250ce6cb7d34-archives-path\") on node \"ip-10-0-139-83.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 07:52:02.276717 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:52:02.276724    4720 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8z59p\" (UniqueName: \"kubernetes.io/projected/287cc66c-4032-4bfa-a316-250ce6cb7d34-kube-api-access-8z59p\") on node \"ip-10-0-139-83.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 07:52:02.276956 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:52:02.276738    4720 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/287cc66c-4032-4bfa-a316-250ce6cb7d34-trusted-ca-bundle\") on node \"ip-10-0-139-83.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 07:52:02.854026 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:52:02.853988    4720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-lmblb-khh8v" event={"ID":"287cc66c-4032-4bfa-a316-250ce6cb7d34","Type":"ContainerDied","Data":"3589e7fec2623fab48dfd0a4710933ac1d2345bd2a0ccdf05898caee3aaae1d7"}
Oct 31 07:52:02.854026 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:52:02.854013    4720 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-lmblb-khh8v"
Oct 31 07:52:02.854026 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:52:02.854034    4720 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3589e7fec2623fab48dfd0a4710933ac1d2345bd2a0ccdf05898caee3aaae1d7"
Oct 31 07:52:03.086768 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:52:03.086737    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 07:52:03.869053 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:52:03.869019    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 07:52:04.011750 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:52:04.011720    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 07:52:08.051118 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:52:08.051084    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 07:52:08.782971 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:52:08.782932    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 07:52:08.934826 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:52:08.934789    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 07:52:13.462303 ip-10-0-139-83 kubenswrapper[4720]: E1031 07:52:13.462259    4720 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8m768" containerName="extractor"
Oct 31 07:52:34.862349 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:52:34.862292    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 07:52:38.415254 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:52:38.415218    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 07:52:43.462625 ip-10-0-139-83 kubenswrapper[4720]: E1031 07:52:43.462586    4720 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8m768" containerName="extractor"
Oct 31 07:52:47.308754 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:52:47.308715    4720 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7d8f9d9664-mzd96"]
Oct 31 07:52:47.309159 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:52:47.309141    4720 cpu_manager.go:414] "containerMap: removing container" podUID="ae3fd7c0-d7e7-4623-be62-233560d64000" containerName="main"
Oct 31 07:52:47.309203 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:52:47.309162    4720 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ae3fd7c0-d7e7-4623-be62-233560d64000" containerName="main"
Oct 31 07:52:47.309203 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:52:47.309175    4720 cpu_manager.go:414] "containerMap: removing container" podUID="287cc66c-4032-4bfa-a316-250ce6cb7d34" containerName="insights-gathering"
Oct 31 07:52:47.309203 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:52:47.309180    4720 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="287cc66c-4032-4bfa-a316-250ce6cb7d34" containerName="insights-gathering"
Oct 31 07:52:47.309295 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:52:47.309255    4720 memory_manager.go:365] "RemoveStaleState removing state" podUID="ae3fd7c0-d7e7-4623-be62-233560d64000" containerName="main"
Oct 31 07:52:47.309295 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:52:47.309262    4720 memory_manager.go:365] "RemoveStaleState removing state" podUID="287cc66c-4032-4bfa-a316-250ce6cb7d34" containerName="insights-gathering"
Oct 31 07:52:47.312252 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:52:47.312238    4720 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7d8f9d9664-mzd96"
Oct 31 07:52:47.413566 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:52:47.413532    4720 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-flow-control-smoke-4f8df83c\"/\"kube-root-ca.crt\""
Oct 31 07:52:47.413566 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:52:47.413532    4720 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-flow-control-smoke-4f8df83c\"/\"default-dockercfg-kq8rt\""
Oct 31 07:52:47.413566 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:52:47.413532    4720 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-flow-control-smoke-4f8df83c\"/\"fc-smoke-test-kserve-self-signed-certs\""
Oct 31 07:52:47.413807 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:52:47.413676    4720 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-flow-control-smoke-4f8df83c\"/\"openshift-service-ca.crt\""
Oct 31 07:52:47.415287 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:52:47.415268    4720 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7d8f9d9664-mzd96"]
Oct 31 07:52:47.429228 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:52:47.429196    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 07:52:47.456480 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:52:47.456445    4720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/15d71d9b-0872-4e06-85f1-66a9a3f8eadb-home\") pod \"fc-smoke-test-kserve-7d8f9d9664-mzd96\" (UID: \"15d71d9b-0872-4e06-85f1-66a9a3f8eadb\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7d8f9d9664-mzd96"
Oct 31 07:52:47.456622 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:52:47.456509    4720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/15d71d9b-0872-4e06-85f1-66a9a3f8eadb-dshm\") pod \"fc-smoke-test-kserve-7d8f9d9664-mzd96\" (UID: \"15d71d9b-0872-4e06-85f1-66a9a3f8eadb\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7d8f9d9664-mzd96"
Oct 31 07:52:47.456622 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:52:47.456607    4720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/15d71d9b-0872-4e06-85f1-66a9a3f8eadb-model-cache\") pod \"fc-smoke-test-kserve-7d8f9d9664-mzd96\" (UID: \"15d71d9b-0872-4e06-85f1-66a9a3f8eadb\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7d8f9d9664-mzd96"
Oct 31 07:52:47.456748 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:52:47.456728    4720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9l2tz\" (UniqueName: \"kubernetes.io/projected/15d71d9b-0872-4e06-85f1-66a9a3f8eadb-kube-api-access-9l2tz\") pod \"fc-smoke-test-kserve-7d8f9d9664-mzd96\" (UID: \"15d71d9b-0872-4e06-85f1-66a9a3f8eadb\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7d8f9d9664-mzd96"
Oct 31 07:52:47.456807 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:52:47.456786    4720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/15d71d9b-0872-4e06-85f1-66a9a3f8eadb-tmp-dir\") pod \"fc-smoke-test-kserve-7d8f9d9664-mzd96\" (UID: \"15d71d9b-0872-4e06-85f1-66a9a3f8eadb\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7d8f9d9664-mzd96"
Oct 31 07:52:47.456880 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:52:47.456812    4720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/15d71d9b-0872-4e06-85f1-66a9a3f8eadb-tls-certs\") pod \"fc-smoke-test-kserve-7d8f9d9664-mzd96\" (UID: \"15d71d9b-0872-4e06-85f1-66a9a3f8eadb\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7d8f9d9664-mzd96"
Oct 31 07:52:47.557597 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:52:47.557557    4720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/15d71d9b-0872-4e06-85f1-66a9a3f8eadb-tmp-dir\") pod \"fc-smoke-test-kserve-7d8f9d9664-mzd96\" (UID: \"15d71d9b-0872-4e06-85f1-66a9a3f8eadb\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7d8f9d9664-mzd96"
Oct 31 07:52:47.557597 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:52:47.557596    4720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/15d71d9b-0872-4e06-85f1-66a9a3f8eadb-tls-certs\") pod \"fc-smoke-test-kserve-7d8f9d9664-mzd96\" (UID: \"15d71d9b-0872-4e06-85f1-66a9a3f8eadb\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7d8f9d9664-mzd96"
Oct 31 07:52:47.557832 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:52:47.557644    4720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/15d71d9b-0872-4e06-85f1-66a9a3f8eadb-home\") pod \"fc-smoke-test-kserve-7d8f9d9664-mzd96\" (UID: \"15d71d9b-0872-4e06-85f1-66a9a3f8eadb\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7d8f9d9664-mzd96"
Oct 31 07:52:47.557832 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:52:47.557686    4720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/15d71d9b-0872-4e06-85f1-66a9a3f8eadb-dshm\") pod \"fc-smoke-test-kserve-7d8f9d9664-mzd96\" (UID: \"15d71d9b-0872-4e06-85f1-66a9a3f8eadb\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7d8f9d9664-mzd96"
Oct 31 07:52:47.557832 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:52:47.557735    4720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/15d71d9b-0872-4e06-85f1-66a9a3f8eadb-model-cache\") pod \"fc-smoke-test-kserve-7d8f9d9664-mzd96\" (UID: \"15d71d9b-0872-4e06-85f1-66a9a3f8eadb\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7d8f9d9664-mzd96"
Oct 31 07:52:47.557832 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:52:47.557768    4720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9l2tz\" (UniqueName: \"kubernetes.io/projected/15d71d9b-0872-4e06-85f1-66a9a3f8eadb-kube-api-access-9l2tz\") pod \"fc-smoke-test-kserve-7d8f9d9664-mzd96\" (UID: \"15d71d9b-0872-4e06-85f1-66a9a3f8eadb\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7d8f9d9664-mzd96"
Oct 31 07:52:47.558055 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:52:47.558029    4720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/15d71d9b-0872-4e06-85f1-66a9a3f8eadb-tmp-dir\") pod \"fc-smoke-test-kserve-7d8f9d9664-mzd96\" (UID: \"15d71d9b-0872-4e06-85f1-66a9a3f8eadb\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7d8f9d9664-mzd96"
Oct 31 07:52:47.558124 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:52:47.558083    4720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/15d71d9b-0872-4e06-85f1-66a9a3f8eadb-home\") pod \"fc-smoke-test-kserve-7d8f9d9664-mzd96\" (UID: \"15d71d9b-0872-4e06-85f1-66a9a3f8eadb\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7d8f9d9664-mzd96"
Oct 31 07:52:47.558178 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:52:47.558134    4720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/15d71d9b-0872-4e06-85f1-66a9a3f8eadb-model-cache\") pod \"fc-smoke-test-kserve-7d8f9d9664-mzd96\" (UID: \"15d71d9b-0872-4e06-85f1-66a9a3f8eadb\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7d8f9d9664-mzd96"
Oct 31 07:52:47.560190 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:52:47.560131    4720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/15d71d9b-0872-4e06-85f1-66a9a3f8eadb-dshm\") pod \"fc-smoke-test-kserve-7d8f9d9664-mzd96\" (UID: \"15d71d9b-0872-4e06-85f1-66a9a3f8eadb\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7d8f9d9664-mzd96"
Oct 31 07:52:47.560430 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:52:47.560411    4720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/15d71d9b-0872-4e06-85f1-66a9a3f8eadb-tls-certs\") pod \"fc-smoke-test-kserve-7d8f9d9664-mzd96\" (UID: \"15d71d9b-0872-4e06-85f1-66a9a3f8eadb\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7d8f9d9664-mzd96"
Oct 31 07:52:47.616153 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:52:47.616119    4720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9l2tz\" (UniqueName: \"kubernetes.io/projected/15d71d9b-0872-4e06-85f1-66a9a3f8eadb-kube-api-access-9l2tz\") pod \"fc-smoke-test-kserve-7d8f9d9664-mzd96\" (UID: \"15d71d9b-0872-4e06-85f1-66a9a3f8eadb\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7d8f9d9664-mzd96"
Oct 31 07:52:47.629559 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:52:47.629529    4720 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7d8f9d9664-mzd96"
Oct 31 07:52:47.871777 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:52:47.871738    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 07:52:48.017494 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:52:48.017462    4720 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7d8f9d9664-mzd96"]
Oct 31 07:52:48.020781 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:52:48.020752    4720 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod15d71d9b_0872_4e06_85f1_66a9a3f8eadb.slice/crio-dba2d9b9d5f0bf863da09d313c3be8d0b355b2698fe1bd8a03235032b7645f06 WatchSource:0}: Error finding container dba2d9b9d5f0bf863da09d313c3be8d0b355b2698fe1bd8a03235032b7645f06: Status 404 returned error can't find the container with id dba2d9b9d5f0bf863da09d313c3be8d0b355b2698fe1bd8a03235032b7645f06
Oct 31 07:52:48.057022 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:52:48.056995    4720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7d8f9d9664-mzd96" event={"ID":"15d71d9b-0872-4e06-85f1-66a9a3f8eadb","Type":"ContainerStarted","Data":"dba2d9b9d5f0bf863da09d313c3be8d0b355b2698fe1bd8a03235032b7645f06"}
Oct 31 07:52:49.063410 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:52:49.063369    4720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7d8f9d9664-mzd96" event={"ID":"15d71d9b-0872-4e06-85f1-66a9a3f8eadb","Type":"ContainerStarted","Data":"155ee11fd6dab07538f65c6fb7f4bac1756f01b8292d01aa77364ff9ab240d57"}
Oct 31 07:52:49.176813 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:52:49.176754    4720 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7d8f9d9664-mzd96" podStartSLOduration=2.176737268 podStartE2EDuration="2.176737268s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 07:52:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 07:52:49.176110144 +0000 UTC m=+1423.307178208" watchObservedRunningTime="2026-10-31 07:52:49.176737268 +0000 UTC m=+1423.307805339"
Oct 31 07:52:49.629804 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:52:49.629767    4720 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7d8f9d9664-mzd96"
Oct 31 07:52:54.533355 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:52:54.532444    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 07:52:57.630230 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:52:57.630191    4720 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7d8f9d9664-mzd96"
Oct 31 07:52:57.642792 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:52:57.642767    4720 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7d8f9d9664-mzd96"
Oct 31 07:52:58.171675 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:52:58.171646    4720 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7d8f9d9664-mzd96"
Oct 31 07:53:08.382594 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:53:08.382533    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 07:53:13.460813 ip-10-0-139-83 kubenswrapper[4720]: E1031 07:53:13.460773    4720 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8m768" containerName="extractor"
Oct 31 07:53:21.200133 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:53:21.200101    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 07:53:21.345643 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:53:21.345588    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 07:53:30.883867 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:53:30.883832    4720 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7d8f9d9664-mzd96"]
Oct 31 07:53:30.895579 ip-10-0-139-83 kubenswrapper[4720]: E1031 07:53:30.895536    4720 log.go:32] "ExecSync cmd from runtime service failed" err="rpc error: code = Unknown desc = command error: , stdout: , stderr: , exit code -1" containerID="155ee11fd6dab07538f65c6fb7f4bac1756f01b8292d01aa77364ff9ab240d57" cmd=["/bin/sleep","15"]
Oct 31 07:53:30.895737 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:53:30.895599    4720 handlers.go:85] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7d8f9d9664-mzd96" message="" err="rpc error: code = Unknown desc = command error: , stdout: , stderr: , exit code -1"
Oct 31 07:53:30.895737 ip-10-0-139-83 kubenswrapper[4720]: E1031 07:53:30.895616    4720 kuberuntime_container.go:780] "PreStop hook failed" err="rpc error: code = Unknown desc = command error: , stdout: , stderr: , exit code -1" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7d8f9d9664-mzd96" podUID="15d71d9b-0872-4e06-85f1-66a9a3f8eadb" containerName="main" containerID="cri-o://155ee11fd6dab07538f65c6fb7f4bac1756f01b8292d01aa77364ff9ab240d57"
Oct 31 07:53:30.895737 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:53:30.895655    4720 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7d8f9d9664-mzd96" podUID="15d71d9b-0872-4e06-85f1-66a9a3f8eadb" containerName="main" containerID="cri-o://155ee11fd6dab07538f65c6fb7f4bac1756f01b8292d01aa77364ff9ab240d57" gracePeriod=60
Oct 31 07:53:30.917736 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:53:30.917705    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 07:53:31.068712 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:53:31.068671    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 07:53:31.161648 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:53:31.161566    4720 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7d8f9d9664-mzd96" podUID="15d71d9b-0872-4e06-85f1-66a9a3f8eadb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.43:8000/health\": dial tcp 10.132.0.43:8000: connect: connection refused"
Oct 31 07:53:31.315038 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:53:31.315005    4720 generic.go:356] "Generic (PLEG): container finished" podID="15d71d9b-0872-4e06-85f1-66a9a3f8eadb" containerID="155ee11fd6dab07538f65c6fb7f4bac1756f01b8292d01aa77364ff9ab240d57" exitCode=0
Oct 31 07:53:31.315201 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:53:31.315079    4720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7d8f9d9664-mzd96" event={"ID":"15d71d9b-0872-4e06-85f1-66a9a3f8eadb","Type":"ContainerDied","Data":"155ee11fd6dab07538f65c6fb7f4bac1756f01b8292d01aa77364ff9ab240d57"}
Oct 31 07:53:31.638658 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:53:31.638635    4720 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7d8f9d9664-mzd96"
Oct 31 07:53:31.697557 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:53:31.697527    4720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/15d71d9b-0872-4e06-85f1-66a9a3f8eadb-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/15d71d9b-0872-4e06-85f1-66a9a3f8eadb-tmp-dir\") pod \"15d71d9b-0872-4e06-85f1-66a9a3f8eadb\" (UID: \"15d71d9b-0872-4e06-85f1-66a9a3f8eadb\") "
Oct 31 07:53:31.697705 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:53:31.697565    4720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/15d71d9b-0872-4e06-85f1-66a9a3f8eadb-tls-certs\" (UniqueName: \"kubernetes.io/secret/15d71d9b-0872-4e06-85f1-66a9a3f8eadb-tls-certs\") pod \"15d71d9b-0872-4e06-85f1-66a9a3f8eadb\" (UID: \"15d71d9b-0872-4e06-85f1-66a9a3f8eadb\") "
Oct 31 07:53:31.697705 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:53:31.697675    4720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/15d71d9b-0872-4e06-85f1-66a9a3f8eadb-home\" (UniqueName: \"kubernetes.io/empty-dir/15d71d9b-0872-4e06-85f1-66a9a3f8eadb-home\") pod \"15d71d9b-0872-4e06-85f1-66a9a3f8eadb\" (UID: \"15d71d9b-0872-4e06-85f1-66a9a3f8eadb\") "
Oct 31 07:53:31.697791 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:53:31.697706    4720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/15d71d9b-0872-4e06-85f1-66a9a3f8eadb-dshm\" (UniqueName: \"kubernetes.io/empty-dir/15d71d9b-0872-4e06-85f1-66a9a3f8eadb-dshm\") pod \"15d71d9b-0872-4e06-85f1-66a9a3f8eadb\" (UID: \"15d71d9b-0872-4e06-85f1-66a9a3f8eadb\") "
Oct 31 07:53:31.697791 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:53:31.697739    4720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/15d71d9b-0872-4e06-85f1-66a9a3f8eadb-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/15d71d9b-0872-4e06-85f1-66a9a3f8eadb-model-cache\") pod \"15d71d9b-0872-4e06-85f1-66a9a3f8eadb\" (UID: \"15d71d9b-0872-4e06-85f1-66a9a3f8eadb\") "
Oct 31 07:53:31.697889 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:53:31.697794    4720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/15d71d9b-0872-4e06-85f1-66a9a3f8eadb-kube-api-access-9l2tz\" (UniqueName: \"kubernetes.io/projected/15d71d9b-0872-4e06-85f1-66a9a3f8eadb-kube-api-access-9l2tz\") pod \"15d71d9b-0872-4e06-85f1-66a9a3f8eadb\" (UID: \"15d71d9b-0872-4e06-85f1-66a9a3f8eadb\") "
Oct 31 07:53:31.697889 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:53:31.697821    4720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/15d71d9b-0872-4e06-85f1-66a9a3f8eadb-tmp-dir" pod "15d71d9b-0872-4e06-85f1-66a9a3f8eadb" (UID: "15d71d9b-0872-4e06-85f1-66a9a3f8eadb"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 07:53:31.697992 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:53:31.697949    4720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/15d71d9b-0872-4e06-85f1-66a9a3f8eadb-home" pod "15d71d9b-0872-4e06-85f1-66a9a3f8eadb" (UID: "15d71d9b-0872-4e06-85f1-66a9a3f8eadb"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 07:53:31.698070 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:53:31.698052    4720 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/15d71d9b-0872-4e06-85f1-66a9a3f8eadb-home\") on node \"ip-10-0-139-83.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 07:53:31.698133 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:53:31.698076    4720 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/15d71d9b-0872-4e06-85f1-66a9a3f8eadb-tmp-dir\") on node \"ip-10-0-139-83.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 07:53:31.698133 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:53:31.698091    4720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/15d71d9b-0872-4e06-85f1-66a9a3f8eadb-model-cache" pod "15d71d9b-0872-4e06-85f1-66a9a3f8eadb" (UID: "15d71d9b-0872-4e06-85f1-66a9a3f8eadb"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 07:53:31.699985 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:53:31.699964    4720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/15d71d9b-0872-4e06-85f1-66a9a3f8eadb-dshm" pod "15d71d9b-0872-4e06-85f1-66a9a3f8eadb" (UID: "15d71d9b-0872-4e06-85f1-66a9a3f8eadb"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 07:53:31.700081 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:53:31.700068    4720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/15d71d9b-0872-4e06-85f1-66a9a3f8eadb-tls-certs" pod "15d71d9b-0872-4e06-85f1-66a9a3f8eadb" (UID: "15d71d9b-0872-4e06-85f1-66a9a3f8eadb"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 07:53:31.700122 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:53:31.700108    4720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/15d71d9b-0872-4e06-85f1-66a9a3f8eadb-kube-api-access-9l2tz" pod "15d71d9b-0872-4e06-85f1-66a9a3f8eadb" (UID: "15d71d9b-0872-4e06-85f1-66a9a3f8eadb"). InnerVolumeSpecName "kube-api-access-9l2tz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 07:53:31.799286 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:53:31.799212    4720 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/15d71d9b-0872-4e06-85f1-66a9a3f8eadb-dshm\") on node \"ip-10-0-139-83.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 07:53:31.799286 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:53:31.799240    4720 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/15d71d9b-0872-4e06-85f1-66a9a3f8eadb-model-cache\") on node \"ip-10-0-139-83.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 07:53:31.799286 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:53:31.799257    4720 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9l2tz\" (UniqueName: \"kubernetes.io/projected/15d71d9b-0872-4e06-85f1-66a9a3f8eadb-kube-api-access-9l2tz\") on node \"ip-10-0-139-83.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 07:53:31.799286 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:53:31.799268    4720 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/15d71d9b-0872-4e06-85f1-66a9a3f8eadb-tls-certs\") on node \"ip-10-0-139-83.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 07:53:32.320718 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:53:32.320694    4720 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7d8f9d9664-mzd96"
Oct 31 07:53:32.320718 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:53:32.320704    4720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7d8f9d9664-mzd96" event={"ID":"15d71d9b-0872-4e06-85f1-66a9a3f8eadb","Type":"ContainerDied","Data":"dba2d9b9d5f0bf863da09d313c3be8d0b355b2698fe1bd8a03235032b7645f06"}
Oct 31 07:53:32.321248 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:53:32.320757    4720 scope.go:122] "RemoveContainer" containerID="155ee11fd6dab07538f65c6fb7f4bac1756f01b8292d01aa77364ff9ab240d57"
Oct 31 07:53:32.482166 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:53:32.482133    4720 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7d8f9d9664-mzd96"]
Oct 31 07:53:32.489564 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:53:32.489535    4720 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7d8f9d9664-mzd96"]
Oct 31 07:53:33.392544 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:53:33.392507    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 07:53:34.025586 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:53:34.025548    4720 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="15d71d9b-0872-4e06-85f1-66a9a3f8eadb" path="/var/lib/kubelet/pods/15d71d9b-0872-4e06-85f1-66a9a3f8eadb/volumes"
Oct 31 07:53:38.739802 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:53:38.739763    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 07:53:43.460432 ip-10-0-139-83 kubenswrapper[4720]: E1031 07:53:43.460394    4720 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8m768" containerName="extractor"
Oct 31 07:54:06.146078 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:54:06.146008    4720 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-4jszp_b1db0e3b-e85d-4769-8012-cf4b6f740d9d/console-operator/2.log"
Oct 31 07:54:06.148586 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:54:06.148233    4720 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-4jszp_b1db0e3b-e85d-4769-8012-cf4b6f740d9d/console-operator/2.log"
Oct 31 07:54:08.989268 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:54:08.989235    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 07:54:09.152064 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:54:09.152040    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 07:54:13.461304 ip-10-0-139-83 kubenswrapper[4720]: E1031 07:54:13.461261    4720 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8m768" containerName="extractor"
Oct 31 07:54:18.725206 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:54:18.725175    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 07:54:20.003591 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:54:20.003561    4720 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-767f9954b4-pc9s2"]
Oct 31 07:54:20.003956 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:54:20.003915    4720 cpu_manager.go:414] "containerMap: removing container" podUID="15d71d9b-0872-4e06-85f1-66a9a3f8eadb" containerName="main"
Oct 31 07:54:20.003956 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:54:20.003926    4720 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="15d71d9b-0872-4e06-85f1-66a9a3f8eadb" containerName="main"
Oct 31 07:54:20.004026 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:54:20.003991    4720 memory_manager.go:365] "RemoveStaleState removing state" podUID="15d71d9b-0872-4e06-85f1-66a9a3f8eadb" containerName="main"
Oct 31 07:54:20.008051 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:54:20.008035    4720 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-767f9954b4-pc9s2"
Oct 31 07:54:20.077210 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:54:20.077179    4720 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-767f9954b4-pc9s2"]
Oct 31 07:54:20.091087 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:54:20.091058    4720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5cfa8db7-5b83-400c-8c50-81eeaeb33210-model-cache\") pod \"scheduler-configmap-ref-test-kserve-767f9954b4-pc9s2\" (UID: \"5cfa8db7-5b83-400c-8c50-81eeaeb33210\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-767f9954b4-pc9s2"
Oct 31 07:54:20.091200 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:54:20.091105    4720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5cfa8db7-5b83-400c-8c50-81eeaeb33210-home\") pod \"scheduler-configmap-ref-test-kserve-767f9954b4-pc9s2\" (UID: \"5cfa8db7-5b83-400c-8c50-81eeaeb33210\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-767f9954b4-pc9s2"
Oct 31 07:54:20.091200 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:54:20.091123    4720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-48fqw\" (UniqueName: \"kubernetes.io/projected/5cfa8db7-5b83-400c-8c50-81eeaeb33210-kube-api-access-48fqw\") pod \"scheduler-configmap-ref-test-kserve-767f9954b4-pc9s2\" (UID: \"5cfa8db7-5b83-400c-8c50-81eeaeb33210\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-767f9954b4-pc9s2"
Oct 31 07:54:20.091309 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:54:20.091284    4720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5cfa8db7-5b83-400c-8c50-81eeaeb33210-dshm\") pod \"scheduler-configmap-ref-test-kserve-767f9954b4-pc9s2\" (UID: \"5cfa8db7-5b83-400c-8c50-81eeaeb33210\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-767f9954b4-pc9s2"
Oct 31 07:54:20.091389 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:54:20.091376    4720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5cfa8db7-5b83-400c-8c50-81eeaeb33210-tmp-dir\") pod \"scheduler-configmap-ref-test-kserve-767f9954b4-pc9s2\" (UID: \"5cfa8db7-5b83-400c-8c50-81eeaeb33210\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-767f9954b4-pc9s2"
Oct 31 07:54:20.091425 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:54:20.091411    4720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5cfa8db7-5b83-400c-8c50-81eeaeb33210-tls-certs\") pod \"scheduler-configmap-ref-test-kserve-767f9954b4-pc9s2\" (UID: \"5cfa8db7-5b83-400c-8c50-81eeaeb33210\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-767f9954b4-pc9s2"
Oct 31 07:54:20.124787 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:54:20.124763    4720 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-15d6126c\"/\"kube-root-ca.crt\""
Oct 31 07:54:20.124787 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:54:20.124774    4720 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-15d6126c\"/\"scheduler-configmap-ref-test-kserve-self-signed-certs\""
Oct 31 07:54:20.124958 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:54:20.124891    4720 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-15d6126c\"/\"default-dockercfg-5cq6q\""
Oct 31 07:54:20.124958 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:54:20.124887    4720 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-15d6126c\"/\"openshift-service-ca.crt\""
Oct 31 07:54:20.172865 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:54:20.172835    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 07:54:20.191950 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:54:20.191927    4720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5cfa8db7-5b83-400c-8c50-81eeaeb33210-tls-certs\") pod \"scheduler-configmap-ref-test-kserve-767f9954b4-pc9s2\" (UID: \"5cfa8db7-5b83-400c-8c50-81eeaeb33210\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-767f9954b4-pc9s2"
Oct 31 07:54:20.192050 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:54:20.191969    4720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5cfa8db7-5b83-400c-8c50-81eeaeb33210-model-cache\") pod \"scheduler-configmap-ref-test-kserve-767f9954b4-pc9s2\" (UID: \"5cfa8db7-5b83-400c-8c50-81eeaeb33210\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-767f9954b4-pc9s2"
Oct 31 07:54:20.192050 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:54:20.192005    4720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5cfa8db7-5b83-400c-8c50-81eeaeb33210-home\") pod \"scheduler-configmap-ref-test-kserve-767f9954b4-pc9s2\" (UID: \"5cfa8db7-5b83-400c-8c50-81eeaeb33210\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-767f9954b4-pc9s2"
Oct 31 07:54:20.192050 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:54:20.192022    4720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-48fqw\" (UniqueName: \"kubernetes.io/projected/5cfa8db7-5b83-400c-8c50-81eeaeb33210-kube-api-access-48fqw\") pod \"scheduler-configmap-ref-test-kserve-767f9954b4-pc9s2\" (UID: \"5cfa8db7-5b83-400c-8c50-81eeaeb33210\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-767f9954b4-pc9s2"
Oct 31 07:54:20.192050 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:54:20.192048    4720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5cfa8db7-5b83-400c-8c50-81eeaeb33210-dshm\") pod \"scheduler-configmap-ref-test-kserve-767f9954b4-pc9s2\" (UID: \"5cfa8db7-5b83-400c-8c50-81eeaeb33210\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-767f9954b4-pc9s2"
Oct 31 07:54:20.192238 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:54:20.192077    4720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5cfa8db7-5b83-400c-8c50-81eeaeb33210-tmp-dir\") pod \"scheduler-configmap-ref-test-kserve-767f9954b4-pc9s2\" (UID: \"5cfa8db7-5b83-400c-8c50-81eeaeb33210\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-767f9954b4-pc9s2"
Oct 31 07:54:20.192468 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:54:20.192436    4720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5cfa8db7-5b83-400c-8c50-81eeaeb33210-home\") pod \"scheduler-configmap-ref-test-kserve-767f9954b4-pc9s2\" (UID: \"5cfa8db7-5b83-400c-8c50-81eeaeb33210\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-767f9954b4-pc9s2"
Oct 31 07:54:20.192552 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:54:20.192464    4720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5cfa8db7-5b83-400c-8c50-81eeaeb33210-tmp-dir\") pod \"scheduler-configmap-ref-test-kserve-767f9954b4-pc9s2\" (UID: \"5cfa8db7-5b83-400c-8c50-81eeaeb33210\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-767f9954b4-pc9s2"
Oct 31 07:54:20.192552 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:54:20.192489    4720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5cfa8db7-5b83-400c-8c50-81eeaeb33210-model-cache\") pod \"scheduler-configmap-ref-test-kserve-767f9954b4-pc9s2\" (UID: \"5cfa8db7-5b83-400c-8c50-81eeaeb33210\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-767f9954b4-pc9s2"
Oct 31 07:54:20.194456 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:54:20.194437    4720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5cfa8db7-5b83-400c-8c50-81eeaeb33210-dshm\") pod \"scheduler-configmap-ref-test-kserve-767f9954b4-pc9s2\" (UID: \"5cfa8db7-5b83-400c-8c50-81eeaeb33210\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-767f9954b4-pc9s2"
Oct 31 07:54:20.194692 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:54:20.194677    4720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5cfa8db7-5b83-400c-8c50-81eeaeb33210-tls-certs\") pod \"scheduler-configmap-ref-test-kserve-767f9954b4-pc9s2\" (UID: \"5cfa8db7-5b83-400c-8c50-81eeaeb33210\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-767f9954b4-pc9s2"
Oct 31 07:54:20.248222 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:54:20.248198    4720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-48fqw\" (UniqueName: \"kubernetes.io/projected/5cfa8db7-5b83-400c-8c50-81eeaeb33210-kube-api-access-48fqw\") pod \"scheduler-configmap-ref-test-kserve-767f9954b4-pc9s2\" (UID: \"5cfa8db7-5b83-400c-8c50-81eeaeb33210\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-767f9954b4-pc9s2"
Oct 31 07:54:20.320563 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:54:20.320494    4720 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-767f9954b4-pc9s2"
Oct 31 07:54:20.328763 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:54:20.328737    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 07:54:20.500365 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:54:20.500269    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 07:54:20.654367 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:54:20.654313    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 07:54:20.691515 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:54:20.691483    4720 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-767f9954b4-pc9s2"]
Oct 31 07:54:20.694987 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:54:20.694960    4720 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5cfa8db7_5b83_400c_8c50_81eeaeb33210.slice/crio-5d10c00984ba4951f27976059d0410d6481285be59221bb447d3f454cf4bc49e WatchSource:0}: Error finding container 5d10c00984ba4951f27976059d0410d6481285be59221bb447d3f454cf4bc49e: Status 404 returned error can't find the container with id 5d10c00984ba4951f27976059d0410d6481285be59221bb447d3f454cf4bc49e
Oct 31 07:54:21.543238 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:54:21.543203    4720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-767f9954b4-pc9s2" event={"ID":"5cfa8db7-5b83-400c-8c50-81eeaeb33210","Type":"ContainerStarted","Data":"8363f624d40266e33e0c7053b17d08fde4b4c33195d0b034683259ed0de30bdb"}
Oct 31 07:54:21.543728 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:54:21.543258    4720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-767f9954b4-pc9s2" event={"ID":"5cfa8db7-5b83-400c-8c50-81eeaeb33210","Type":"ContainerStarted","Data":"5d10c00984ba4951f27976059d0410d6481285be59221bb447d3f454cf4bc49e"}
Oct 31 07:54:21.657475 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:54:21.657415    4720 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-767f9954b4-pc9s2" podStartSLOduration=2.657402094 podStartE2EDuration="2.657402094s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 07:54:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 07:54:21.65649724 +0000 UTC m=+1515.787565313" watchObservedRunningTime="2026-10-31 07:54:21.657402094 +0000 UTC m=+1515.788470164"
Oct 31 07:54:22.321432 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:54:22.321403    4720 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-767f9954b4-pc9s2"
Oct 31 07:54:24.926730 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:54:24.926694    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 07:54:26.604352 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:54:26.603728    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 07:54:29.799948 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:54:29.799903    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 07:54:30.267750 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:54:30.267707    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 07:54:30.321538 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:54:30.321502    4720 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-767f9954b4-pc9s2"
Oct 31 07:54:30.334062 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:54:30.334040    4720 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-767f9954b4-pc9s2"
Oct 31 07:54:30.627723 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:54:30.627694    4720 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-767f9954b4-pc9s2"
Oct 31 07:54:31.863463 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:54:31.863427    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 07:54:32.018876 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:54:32.018842    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 07:54:33.731164 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:54:33.731095    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 07:54:36.807120 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:54:36.807082    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 07:54:39.079201 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:54:39.079166    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 07:54:43.460593 ip-10-0-139-83 kubenswrapper[4720]: E1031 07:54:43.460561    4720 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8m768" containerName="extractor"
Oct 31 07:54:48.461369 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:54:48.461315    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 07:54:58.876529 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:54:58.876496    4720 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-767f9954b4-pc9s2"]
Oct 31 07:54:58.886397 ip-10-0-139-83 kubenswrapper[4720]: E1031 07:54:58.886358    4720 log.go:32] "ExecSync cmd from runtime service failed" err="rpc error: code = Unknown desc = command error: , stdout: , stderr: , exit code -1" containerID="8363f624d40266e33e0c7053b17d08fde4b4c33195d0b034683259ed0de30bdb" cmd=["/bin/sleep","15"]
Oct 31 07:54:58.886560 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:54:58.886417    4720 handlers.go:85] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-767f9954b4-pc9s2" message="" err="rpc error: code = Unknown desc = command error: , stdout: , stderr: , exit code -1"
Oct 31 07:54:58.886560 ip-10-0-139-83 kubenswrapper[4720]: E1031 07:54:58.886431    4720 kuberuntime_container.go:780] "PreStop hook failed" err="rpc error: code = Unknown desc = command error: , stdout: , stderr: , exit code -1" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-767f9954b4-pc9s2" podUID="5cfa8db7-5b83-400c-8c50-81eeaeb33210" containerName="main" containerID="cri-o://8363f624d40266e33e0c7053b17d08fde4b4c33195d0b034683259ed0de30bdb"
Oct 31 07:54:58.886560 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:54:58.886474    4720 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-767f9954b4-pc9s2" podUID="5cfa8db7-5b83-400c-8c50-81eeaeb33210" containerName="main" containerID="cri-o://8363f624d40266e33e0c7053b17d08fde4b4c33195d0b034683259ed0de30bdb" gracePeriod=60
Oct 31 07:54:58.983022 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:54:58.982982    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 07:54:59.174738 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:54:59.174649    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 07:54:59.639515 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:54:59.639492    4720 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-767f9954b4-pc9s2"
Oct 31 07:54:59.695083 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:54:59.695060    4720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5cfa8db7-5b83-400c-8c50-81eeaeb33210-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5cfa8db7-5b83-400c-8c50-81eeaeb33210-model-cache\") pod \"5cfa8db7-5b83-400c-8c50-81eeaeb33210\" (UID: \"5cfa8db7-5b83-400c-8c50-81eeaeb33210\") "
Oct 31 07:54:59.695217 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:54:59.695092    4720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5cfa8db7-5b83-400c-8c50-81eeaeb33210-home\" (UniqueName: \"kubernetes.io/empty-dir/5cfa8db7-5b83-400c-8c50-81eeaeb33210-home\") pod \"5cfa8db7-5b83-400c-8c50-81eeaeb33210\" (UID: \"5cfa8db7-5b83-400c-8c50-81eeaeb33210\") "
Oct 31 07:54:59.695217 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:54:59.695113    4720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5cfa8db7-5b83-400c-8c50-81eeaeb33210-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5cfa8db7-5b83-400c-8c50-81eeaeb33210-tmp-dir\") pod \"5cfa8db7-5b83-400c-8c50-81eeaeb33210\" (UID: \"5cfa8db7-5b83-400c-8c50-81eeaeb33210\") "
Oct 31 07:54:59.695217 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:54:59.695147    4720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5cfa8db7-5b83-400c-8c50-81eeaeb33210-dshm\" (UniqueName: \"kubernetes.io/empty-dir/5cfa8db7-5b83-400c-8c50-81eeaeb33210-dshm\") pod \"5cfa8db7-5b83-400c-8c50-81eeaeb33210\" (UID: \"5cfa8db7-5b83-400c-8c50-81eeaeb33210\") "
Oct 31 07:54:59.695217 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:54:59.695167    4720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5cfa8db7-5b83-400c-8c50-81eeaeb33210-tls-certs\" (UniqueName: \"kubernetes.io/secret/5cfa8db7-5b83-400c-8c50-81eeaeb33210-tls-certs\") pod \"5cfa8db7-5b83-400c-8c50-81eeaeb33210\" (UID: \"5cfa8db7-5b83-400c-8c50-81eeaeb33210\") "
Oct 31 07:54:59.695217 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:54:59.695213    4720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5cfa8db7-5b83-400c-8c50-81eeaeb33210-kube-api-access-48fqw\" (UniqueName: \"kubernetes.io/projected/5cfa8db7-5b83-400c-8c50-81eeaeb33210-kube-api-access-48fqw\") pod \"5cfa8db7-5b83-400c-8c50-81eeaeb33210\" (UID: \"5cfa8db7-5b83-400c-8c50-81eeaeb33210\") "
Oct 31 07:54:59.695498 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:54:59.695393    4720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5cfa8db7-5b83-400c-8c50-81eeaeb33210-home" pod "5cfa8db7-5b83-400c-8c50-81eeaeb33210" (UID: "5cfa8db7-5b83-400c-8c50-81eeaeb33210"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 07:54:59.695498 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:54:59.695407    4720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5cfa8db7-5b83-400c-8c50-81eeaeb33210-model-cache" pod "5cfa8db7-5b83-400c-8c50-81eeaeb33210" (UID: "5cfa8db7-5b83-400c-8c50-81eeaeb33210"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 07:54:59.695498 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:54:59.695422    4720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5cfa8db7-5b83-400c-8c50-81eeaeb33210-tmp-dir" pod "5cfa8db7-5b83-400c-8c50-81eeaeb33210" (UID: "5cfa8db7-5b83-400c-8c50-81eeaeb33210"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 07:54:59.695638 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:54:59.695550    4720 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5cfa8db7-5b83-400c-8c50-81eeaeb33210-model-cache\") on node \"ip-10-0-139-83.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 07:54:59.695638 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:54:59.695570    4720 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5cfa8db7-5b83-400c-8c50-81eeaeb33210-home\") on node \"ip-10-0-139-83.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 07:54:59.695638 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:54:59.695609    4720 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5cfa8db7-5b83-400c-8c50-81eeaeb33210-tmp-dir\") on node \"ip-10-0-139-83.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 07:54:59.697545 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:54:59.697519    4720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5cfa8db7-5b83-400c-8c50-81eeaeb33210-dshm" pod "5cfa8db7-5b83-400c-8c50-81eeaeb33210" (UID: "5cfa8db7-5b83-400c-8c50-81eeaeb33210"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 07:54:59.697643 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:54:59.697552    4720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5cfa8db7-5b83-400c-8c50-81eeaeb33210-tls-certs" pod "5cfa8db7-5b83-400c-8c50-81eeaeb33210" (UID: "5cfa8db7-5b83-400c-8c50-81eeaeb33210"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 07:54:59.697643 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:54:59.697609    4720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5cfa8db7-5b83-400c-8c50-81eeaeb33210-kube-api-access-48fqw" pod "5cfa8db7-5b83-400c-8c50-81eeaeb33210" (UID: "5cfa8db7-5b83-400c-8c50-81eeaeb33210"). InnerVolumeSpecName "kube-api-access-48fqw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 07:54:59.753961 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:54:59.753882    4720 generic.go:356] "Generic (PLEG): container finished" podID="5cfa8db7-5b83-400c-8c50-81eeaeb33210" containerID="8363f624d40266e33e0c7053b17d08fde4b4c33195d0b034683259ed0de30bdb" exitCode=0
Oct 31 07:54:59.754098 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:54:59.753969    4720 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-767f9954b4-pc9s2"
Oct 31 07:54:59.754098 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:54:59.753968    4720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-767f9954b4-pc9s2" event={"ID":"5cfa8db7-5b83-400c-8c50-81eeaeb33210","Type":"ContainerDied","Data":"8363f624d40266e33e0c7053b17d08fde4b4c33195d0b034683259ed0de30bdb"}
Oct 31 07:54:59.754182 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:54:59.754118    4720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-767f9954b4-pc9s2" event={"ID":"5cfa8db7-5b83-400c-8c50-81eeaeb33210","Type":"ContainerDied","Data":"5d10c00984ba4951f27976059d0410d6481285be59221bb447d3f454cf4bc49e"}
Oct 31 07:54:59.754182 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:54:59.754147    4720 scope.go:122] "RemoveContainer" containerID="8363f624d40266e33e0c7053b17d08fde4b4c33195d0b034683259ed0de30bdb"
Oct 31 07:54:59.764732 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:54:59.764716    4720 scope.go:122] "RemoveContainer" containerID="8363f624d40266e33e0c7053b17d08fde4b4c33195d0b034683259ed0de30bdb"
Oct 31 07:54:59.764970 ip-10-0-139-83 kubenswrapper[4720]: E1031 07:54:59.764955    4720 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8363f624d40266e33e0c7053b17d08fde4b4c33195d0b034683259ed0de30bdb\": container with ID starting with 8363f624d40266e33e0c7053b17d08fde4b4c33195d0b034683259ed0de30bdb not found: ID does not exist" containerID="8363f624d40266e33e0c7053b17d08fde4b4c33195d0b034683259ed0de30bdb"
Oct 31 07:54:59.765022 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:54:59.764979    4720 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8363f624d40266e33e0c7053b17d08fde4b4c33195d0b034683259ed0de30bdb"} err="failed to get container status \"8363f624d40266e33e0c7053b17d08fde4b4c33195d0b034683259ed0de30bdb\": rpc error: code = NotFound desc = could not find container \"8363f624d40266e33e0c7053b17d08fde4b4c33195d0b034683259ed0de30bdb\": container with ID starting with 8363f624d40266e33e0c7053b17d08fde4b4c33195d0b034683259ed0de30bdb not found: ID does not exist"
Oct 31 07:54:59.796617 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:54:59.796592    4720 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5cfa8db7-5b83-400c-8c50-81eeaeb33210-dshm\") on node \"ip-10-0-139-83.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 07:54:59.796617 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:54:59.796612    4720 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5cfa8db7-5b83-400c-8c50-81eeaeb33210-tls-certs\") on node \"ip-10-0-139-83.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 07:54:59.796617 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:54:59.796620    4720 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-48fqw\" (UniqueName: \"kubernetes.io/projected/5cfa8db7-5b83-400c-8c50-81eeaeb33210-kube-api-access-48fqw\") on node \"ip-10-0-139-83.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 07:54:59.914722 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:54:59.914683    4720 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-767f9954b4-pc9s2"]
Oct 31 07:54:59.916840 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:54:59.916817    4720 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-767f9954b4-pc9s2"]
Oct 31 07:55:00.025418 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:55:00.025306    4720 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5cfa8db7-5b83-400c-8c50-81eeaeb33210" path="/var/lib/kubelet/pods/5cfa8db7-5b83-400c-8c50-81eeaeb33210/volumes"
Oct 31 07:55:00.617580 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:55:00.617543    4720 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-767f9954b4-pc9s2" podUID="5cfa8db7-5b83-400c-8c50-81eeaeb33210" containerName="main" probeResult="failure" output="Get \"https://10.132.0.44:8000/health\": context deadline exceeded"
Oct 31 07:55:04.093999 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:55:04.093973    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 07:55:13.422797 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:55:13.422765    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 07:55:13.460470 ip-10-0-139-83 kubenswrapper[4720]: E1031 07:55:13.460438    4720 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8m768" containerName="extractor"
Oct 31 07:55:13.572892 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:55:13.572856    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 07:55:19.056493 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:55:19.056421    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 07:55:30.769502 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:55:30.769459    4720 prober.go:122] "Probe failed" probeType="Liveness" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-83.us-west-1.compute.internal" podUID="2183ffc19200ec816330ede9883f3bf6" containerName="haproxy" probeResult="failure" output="Get \"https://172.20.0.1:6443/version\": context deadline exceeded"
Oct 31 07:55:34.070496 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:55:34.070465    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 07:55:43.459996 ip-10-0-139-83 kubenswrapper[4720]: E1031 07:55:43.459959    4720 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8m768" containerName="extractor"
Oct 31 07:55:46.981153 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:55:46.981120    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 07:55:47.206955 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:55:47.206927    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 07:55:47.818350 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:55:47.818296    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 07:55:49.423399 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:55:49.423365    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 07:55:52.765320 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:55:52.765282    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 07:55:58.161275 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:55:58.161238    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 07:55:58.308218 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:55:58.308185    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 07:56:13.460009 ip-10-0-139-83 kubenswrapper[4720]: E1031 07:56:13.459971    4720 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8m768" containerName="extractor"
Oct 31 07:56:19.397869 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:56:19.397830    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 07:56:34.406303 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:56:34.406267    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 07:56:43.461146 ip-10-0-139-83 kubenswrapper[4720]: E1031 07:56:43.461107    4720 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8m768" containerName="extractor"
Oct 31 07:56:49.758947 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:56:49.758869    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 07:57:13.460430 ip-10-0-139-83 kubenswrapper[4720]: E1031 07:57:13.460348    4720 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8m768" containerName="extractor"
Oct 31 07:57:19.728992 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:57:19.728954    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 07:57:21.549982 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:57:21.549951    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 07:57:21.713026 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:57:21.712993    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 07:57:21.868933 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:57:21.868905    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 07:57:22.247594 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:57:22.247561    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 07:57:22.449723 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:57:22.449693    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 07:57:26.191084 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:57:26.191053    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 07:57:27.634168 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:57:27.634135    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 07:57:31.305578 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:57:31.305546    4720 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-87cb66947-2297z"]
Oct 31 07:57:31.305945 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:57:31.305852    4720 cpu_manager.go:414] "containerMap: removing container" podUID="5cfa8db7-5b83-400c-8c50-81eeaeb33210" containerName="main"
Oct 31 07:57:31.305945 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:57:31.305862    4720 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5cfa8db7-5b83-400c-8c50-81eeaeb33210" containerName="main"
Oct 31 07:57:31.305945 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:57:31.305936    4720 memory_manager.go:365] "RemoveStaleState removing state" podUID="5cfa8db7-5b83-400c-8c50-81eeaeb33210" containerName="main"
Oct 31 07:57:31.308847 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:57:31.308831    4720 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-87cb66947-2297z"
Oct 31 07:57:31.410159 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:57:31.410122    4720 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-gateway-section-name-propagation-49b5edd9\"/\"gw-sec0c69dceeb48768325d1a53a749e65786-kserve-self-signed-certs\""
Oct 31 07:57:31.410321 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:57:31.410189    4720 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-gateway-section-name-propagation-49b5edd9\"/\"openshift-service-ca.crt\""
Oct 31 07:57:31.410321 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:57:31.410223    4720 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-gateway-section-name-propagation-49b5edd9\"/\"kube-root-ca.crt\""
Oct 31 07:57:31.411847 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:57:31.411828    4720 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-87cb66947-2297z"]
Oct 31 07:57:31.411915 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:57:31.411877    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 07:57:31.415929 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:57:31.415912    4720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/afb766a3-c5c7-4fd1-9376-825f4045155d-tmp-dir\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-87cb66947-2297z\" (UID: \"afb766a3-c5c7-4fd1-9376-825f4045155d\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-87cb66947-2297z"
Oct 31 07:57:31.415981 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:57:31.415951    4720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/afb766a3-c5c7-4fd1-9376-825f4045155d-tls-certs\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-87cb66947-2297z\" (UID: \"afb766a3-c5c7-4fd1-9376-825f4045155d\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-87cb66947-2297z"
Oct 31 07:57:31.416027 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:57:31.415982    4720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/afb766a3-c5c7-4fd1-9376-825f4045155d-dshm\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-87cb66947-2297z\" (UID: \"afb766a3-c5c7-4fd1-9376-825f4045155d\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-87cb66947-2297z"
Oct 31 07:57:31.416106 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:57:31.416092    4720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/afb766a3-c5c7-4fd1-9376-825f4045155d-model-cache\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-87cb66947-2297z\" (UID: \"afb766a3-c5c7-4fd1-9376-825f4045155d\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-87cb66947-2297z"
Oct 31 07:57:31.416140 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:57:31.416127    4720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/afb766a3-c5c7-4fd1-9376-825f4045155d-home\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-87cb66947-2297z\" (UID: \"afb766a3-c5c7-4fd1-9376-825f4045155d\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-87cb66947-2297z"
Oct 31 07:57:31.416173 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:57:31.416145    4720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kkslq\" (UniqueName: \"kubernetes.io/projected/afb766a3-c5c7-4fd1-9376-825f4045155d-kube-api-access-kkslq\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-87cb66947-2297z\" (UID: \"afb766a3-c5c7-4fd1-9376-825f4045155d\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-87cb66947-2297z"
Oct 31 07:57:31.460878 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:57:31.460852    4720 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-gateway-section-name-propagation-49b5edd9\"/\"default-dockercfg-lhw4k\""
Oct 31 07:57:31.516678 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:57:31.516654    4720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/afb766a3-c5c7-4fd1-9376-825f4045155d-tls-certs\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-87cb66947-2297z\" (UID: \"afb766a3-c5c7-4fd1-9376-825f4045155d\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-87cb66947-2297z"
Oct 31 07:57:31.516779 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:57:31.516687    4720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/afb766a3-c5c7-4fd1-9376-825f4045155d-dshm\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-87cb66947-2297z\" (UID: \"afb766a3-c5c7-4fd1-9376-825f4045155d\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-87cb66947-2297z"
Oct 31 07:57:31.516779 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:57:31.516728    4720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/afb766a3-c5c7-4fd1-9376-825f4045155d-model-cache\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-87cb66947-2297z\" (UID: \"afb766a3-c5c7-4fd1-9376-825f4045155d\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-87cb66947-2297z"
Oct 31 07:57:31.516779 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:57:31.516747    4720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/afb766a3-c5c7-4fd1-9376-825f4045155d-home\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-87cb66947-2297z\" (UID: \"afb766a3-c5c7-4fd1-9376-825f4045155d\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-87cb66947-2297z"
Oct 31 07:57:31.516779 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:57:31.516762    4720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kkslq\" (UniqueName: \"kubernetes.io/projected/afb766a3-c5c7-4fd1-9376-825f4045155d-kube-api-access-kkslq\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-87cb66947-2297z\" (UID: \"afb766a3-c5c7-4fd1-9376-825f4045155d\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-87cb66947-2297z"
Oct 31 07:57:31.516972 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:57:31.516845    4720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/afb766a3-c5c7-4fd1-9376-825f4045155d-tmp-dir\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-87cb66947-2297z\" (UID: \"afb766a3-c5c7-4fd1-9376-825f4045155d\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-87cb66947-2297z"
Oct 31 07:57:31.517210 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:57:31.517179    4720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/afb766a3-c5c7-4fd1-9376-825f4045155d-tmp-dir\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-87cb66947-2297z\" (UID: \"afb766a3-c5c7-4fd1-9376-825f4045155d\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-87cb66947-2297z"
Oct 31 07:57:31.517210 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:57:31.517200    4720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/afb766a3-c5c7-4fd1-9376-825f4045155d-home\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-87cb66947-2297z\" (UID: \"afb766a3-c5c7-4fd1-9376-825f4045155d\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-87cb66947-2297z"
Oct 31 07:57:31.517367 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:57:31.517298    4720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/afb766a3-c5c7-4fd1-9376-825f4045155d-model-cache\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-87cb66947-2297z\" (UID: \"afb766a3-c5c7-4fd1-9376-825f4045155d\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-87cb66947-2297z"
Oct 31 07:57:31.519163 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:57:31.519123    4720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/afb766a3-c5c7-4fd1-9376-825f4045155d-dshm\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-87cb66947-2297z\" (UID: \"afb766a3-c5c7-4fd1-9376-825f4045155d\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-87cb66947-2297z"
Oct 31 07:57:31.519547 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:57:31.519524    4720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/afb766a3-c5c7-4fd1-9376-825f4045155d-tls-certs\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-87cb66947-2297z\" (UID: \"afb766a3-c5c7-4fd1-9376-825f4045155d\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-87cb66947-2297z"
Oct 31 07:57:31.572827 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:57:31.572761    4720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kkslq\" (UniqueName: \"kubernetes.io/projected/afb766a3-c5c7-4fd1-9376-825f4045155d-kube-api-access-kkslq\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-87cb66947-2297z\" (UID: \"afb766a3-c5c7-4fd1-9376-825f4045155d\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-87cb66947-2297z"
Oct 31 07:57:31.620482 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:57:31.620461    4720 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-87cb66947-2297z"
Oct 31 07:57:31.992958 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:57:31.992931    4720 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-87cb66947-2297z"]
Oct 31 07:57:31.996309 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:57:31.996282    4720 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podafb766a3_c5c7_4fd1_9376_825f4045155d.slice/crio-ef132a6bd097121c099661533952cbbff1da9e8c74d14f4cf4cade76aa138a9a WatchSource:0}: Error finding container ef132a6bd097121c099661533952cbbff1da9e8c74d14f4cf4cade76aa138a9a: Status 404 returned error can't find the container with id ef132a6bd097121c099661533952cbbff1da9e8c74d14f4cf4cade76aa138a9a
Oct 31 07:57:32.414091 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:57:32.414058    4720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-87cb66947-2297z" event={"ID":"afb766a3-c5c7-4fd1-9376-825f4045155d","Type":"ContainerStarted","Data":"6da90a755e8f62682b696d574fcfa4e76f85c651d004ec73aacecde80a617603"}
Oct 31 07:57:32.414541 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:57:32.414102    4720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-87cb66947-2297z" event={"ID":"afb766a3-c5c7-4fd1-9376-825f4045155d","Type":"ContainerStarted","Data":"ef132a6bd097121c099661533952cbbff1da9e8c74d14f4cf4cade76aa138a9a"}
Oct 31 07:57:32.528844 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:57:32.528785    4720 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-87cb66947-2297z" podStartSLOduration=1.528752757 podStartE2EDuration="1.528752757s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 07:57:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 07:57:32.527070565 +0000 UTC m=+1706.658138637" watchObservedRunningTime="2026-10-31 07:57:32.528752757 +0000 UTC m=+1706.659820828"
Oct 31 07:57:32.621591 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:57:32.621565    4720 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-87cb66947-2297z"
Oct 31 07:57:33.198998 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:57:33.198965    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 07:57:33.348886 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:57:33.348854    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 07:57:33.777155 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:57:33.777118    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 07:57:33.942806 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:57:33.942778    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 07:57:34.107965 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:57:34.107938    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 07:57:34.478529 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:57:34.478491    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 07:57:34.624292 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:57:34.624261    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 07:57:34.776732 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:57:34.776648    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 07:57:37.967312 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:57:37.967277    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 07:57:38.713222 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:57:38.713191    4720 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-87cb66947-2297z"]
Oct 31 07:57:38.723357 ip-10-0-139-83 kubenswrapper[4720]: E1031 07:57:38.723301    4720 log.go:32] "ExecSync cmd from runtime service failed" err="rpc error: code = Unknown desc = command error: , stdout: , stderr: , exit code -1" containerID="6da90a755e8f62682b696d574fcfa4e76f85c651d004ec73aacecde80a617603" cmd=["/bin/sleep","15"]
Oct 31 07:57:38.723529 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:57:38.723375    4720 handlers.go:85] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-87cb66947-2297z" message="" err="rpc error: code = Unknown desc = command error: , stdout: , stderr: , exit code -1"
Oct 31 07:57:38.723529 ip-10-0-139-83 kubenswrapper[4720]: E1031 07:57:38.723392    4720 kuberuntime_container.go:780] "PreStop hook failed" err="rpc error: code = Unknown desc = command error: , stdout: , stderr: , exit code -1" pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-87cb66947-2297z" podUID="afb766a3-c5c7-4fd1-9376-825f4045155d" containerName="main" containerID="cri-o://6da90a755e8f62682b696d574fcfa4e76f85c651d004ec73aacecde80a617603"
Oct 31 07:57:38.723529 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:57:38.723478    4720 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-87cb66947-2297z" podUID="afb766a3-c5c7-4fd1-9376-825f4045155d" containerName="main" containerID="cri-o://6da90a755e8f62682b696d574fcfa4e76f85c651d004ec73aacecde80a617603" gracePeriod=60
Oct 31 07:57:38.814737 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:57:38.814707    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 07:57:38.967688 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:57:38.967637    4720 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-87cb66947-2297z"
Oct 31 07:57:39.073084 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:57:39.073053    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 07:57:39.079843 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:57:39.079809    4720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/afb766a3-c5c7-4fd1-9376-825f4045155d-tls-certs\" (UniqueName: \"kubernetes.io/secret/afb766a3-c5c7-4fd1-9376-825f4045155d-tls-certs\") pod \"afb766a3-c5c7-4fd1-9376-825f4045155d\" (UID: \"afb766a3-c5c7-4fd1-9376-825f4045155d\") "
Oct 31 07:57:39.080051 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:57:39.080027    4720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/afb766a3-c5c7-4fd1-9376-825f4045155d-kube-api-access-kkslq\" (UniqueName: \"kubernetes.io/projected/afb766a3-c5c7-4fd1-9376-825f4045155d-kube-api-access-kkslq\") pod \"afb766a3-c5c7-4fd1-9376-825f4045155d\" (UID: \"afb766a3-c5c7-4fd1-9376-825f4045155d\") "
Oct 31 07:57:39.080357 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:57:39.080093    4720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/afb766a3-c5c7-4fd1-9376-825f4045155d-home\" (UniqueName: \"kubernetes.io/empty-dir/afb766a3-c5c7-4fd1-9376-825f4045155d-home\") pod \"afb766a3-c5c7-4fd1-9376-825f4045155d\" (UID: \"afb766a3-c5c7-4fd1-9376-825f4045155d\") "
Oct 31 07:57:39.080502 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:57:39.080474    4720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/afb766a3-c5c7-4fd1-9376-825f4045155d-home" pod "afb766a3-c5c7-4fd1-9376-825f4045155d" (UID: "afb766a3-c5c7-4fd1-9376-825f4045155d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 07:57:39.080578 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:57:39.080484    4720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/afb766a3-c5c7-4fd1-9376-825f4045155d-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/afb766a3-c5c7-4fd1-9376-825f4045155d-model-cache\") pod \"afb766a3-c5c7-4fd1-9376-825f4045155d\" (UID: \"afb766a3-c5c7-4fd1-9376-825f4045155d\") "
Oct 31 07:57:39.080638 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:57:39.080613    4720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/afb766a3-c5c7-4fd1-9376-825f4045155d-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/afb766a3-c5c7-4fd1-9376-825f4045155d-tmp-dir\") pod \"afb766a3-c5c7-4fd1-9376-825f4045155d\" (UID: \"afb766a3-c5c7-4fd1-9376-825f4045155d\") "
Oct 31 07:57:39.080709 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:57:39.080638    4720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/afb766a3-c5c7-4fd1-9376-825f4045155d-model-cache" pod "afb766a3-c5c7-4fd1-9376-825f4045155d" (UID: "afb766a3-c5c7-4fd1-9376-825f4045155d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 07:57:39.080709 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:57:39.080657    4720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/afb766a3-c5c7-4fd1-9376-825f4045155d-dshm\" (UniqueName: \"kubernetes.io/empty-dir/afb766a3-c5c7-4fd1-9376-825f4045155d-dshm\") pod \"afb766a3-c5c7-4fd1-9376-825f4045155d\" (UID: \"afb766a3-c5c7-4fd1-9376-825f4045155d\") "
Oct 31 07:57:39.081016 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:57:39.080988    4720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/afb766a3-c5c7-4fd1-9376-825f4045155d-tmp-dir" pod "afb766a3-c5c7-4fd1-9376-825f4045155d" (UID: "afb766a3-c5c7-4fd1-9376-825f4045155d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 07:57:39.081165 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:57:39.081148    4720 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/afb766a3-c5c7-4fd1-9376-825f4045155d-home\") on node \"ip-10-0-139-83.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 07:57:39.081228 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:57:39.081172    4720 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/afb766a3-c5c7-4fd1-9376-825f4045155d-model-cache\") on node \"ip-10-0-139-83.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 07:57:39.081228 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:57:39.081189    4720 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/afb766a3-c5c7-4fd1-9376-825f4045155d-tmp-dir\") on node \"ip-10-0-139-83.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 07:57:39.082626 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:57:39.082600    4720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/afb766a3-c5c7-4fd1-9376-825f4045155d-kube-api-access-kkslq" pod "afb766a3-c5c7-4fd1-9376-825f4045155d" (UID: "afb766a3-c5c7-4fd1-9376-825f4045155d"). InnerVolumeSpecName "kube-api-access-kkslq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 07:57:39.082937 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:57:39.082911    4720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/afb766a3-c5c7-4fd1-9376-825f4045155d-tls-certs" pod "afb766a3-c5c7-4fd1-9376-825f4045155d" (UID: "afb766a3-c5c7-4fd1-9376-825f4045155d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 07:57:39.083224 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:57:39.083205    4720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/afb766a3-c5c7-4fd1-9376-825f4045155d-dshm" pod "afb766a3-c5c7-4fd1-9376-825f4045155d" (UID: "afb766a3-c5c7-4fd1-9376-825f4045155d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 07:57:39.181592 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:57:39.181567    4720 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kkslq\" (UniqueName: \"kubernetes.io/projected/afb766a3-c5c7-4fd1-9376-825f4045155d-kube-api-access-kkslq\") on node \"ip-10-0-139-83.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 07:57:39.181592 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:57:39.181591    4720 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/afb766a3-c5c7-4fd1-9376-825f4045155d-dshm\") on node \"ip-10-0-139-83.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 07:57:39.181729 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:57:39.181601    4720 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/afb766a3-c5c7-4fd1-9376-825f4045155d-tls-certs\") on node \"ip-10-0-139-83.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 07:57:39.451927 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:57:39.451893    4720 generic.go:356] "Generic (PLEG): container finished" podID="afb766a3-c5c7-4fd1-9376-825f4045155d" containerID="6da90a755e8f62682b696d574fcfa4e76f85c651d004ec73aacecde80a617603" exitCode=0
Oct 31 07:57:39.452073 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:57:39.451938    4720 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-87cb66947-2297z"
Oct 31 07:57:39.452073 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:57:39.451978    4720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-87cb66947-2297z" event={"ID":"afb766a3-c5c7-4fd1-9376-825f4045155d","Type":"ContainerDied","Data":"6da90a755e8f62682b696d574fcfa4e76f85c651d004ec73aacecde80a617603"}
Oct 31 07:57:39.452073 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:57:39.452028    4720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-87cb66947-2297z" event={"ID":"afb766a3-c5c7-4fd1-9376-825f4045155d","Type":"ContainerDied","Data":"ef132a6bd097121c099661533952cbbff1da9e8c74d14f4cf4cade76aa138a9a"}
Oct 31 07:57:39.452073 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:57:39.452049    4720 scope.go:122] "RemoveContainer" containerID="6da90a755e8f62682b696d574fcfa4e76f85c651d004ec73aacecde80a617603"
Oct 31 07:57:39.461550 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:57:39.461529    4720 scope.go:122] "RemoveContainer" containerID="6da90a755e8f62682b696d574fcfa4e76f85c651d004ec73aacecde80a617603"
Oct 31 07:57:39.461807 ip-10-0-139-83 kubenswrapper[4720]: E1031 07:57:39.461786    4720 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6da90a755e8f62682b696d574fcfa4e76f85c651d004ec73aacecde80a617603\": container with ID starting with 6da90a755e8f62682b696d574fcfa4e76f85c651d004ec73aacecde80a617603 not found: ID does not exist" containerID="6da90a755e8f62682b696d574fcfa4e76f85c651d004ec73aacecde80a617603"
Oct 31 07:57:39.461910 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:57:39.461819    4720 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6da90a755e8f62682b696d574fcfa4e76f85c651d004ec73aacecde80a617603"} err="failed to get container status \"6da90a755e8f62682b696d574fcfa4e76f85c651d004ec73aacecde80a617603\": rpc error: code = NotFound desc = could not find container \"6da90a755e8f62682b696d574fcfa4e76f85c651d004ec73aacecde80a617603\": container with ID starting with 6da90a755e8f62682b696d574fcfa4e76f85c651d004ec73aacecde80a617603 not found: ID does not exist"
Oct 31 07:57:39.615166 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:57:39.615138    4720 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-87cb66947-2297z"]
Oct 31 07:57:39.664610 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:57:39.664581    4720 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-87cb66947-2297z"]
Oct 31 07:57:39.827289 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:57:39.827193    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 07:57:40.025842 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:57:40.025800    4720 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="afb766a3-c5c7-4fd1-9376-825f4045155d" path="/var/lib/kubelet/pods/afb766a3-c5c7-4fd1-9376-825f4045155d/volumes"
Oct 31 07:57:41.560222 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:57:41.560180    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 07:57:41.719207 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:57:41.719172    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 07:57:43.460759 ip-10-0-139-83 kubenswrapper[4720]: E1031 07:57:43.460720    4720 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8m768" containerName="extractor"
Oct 31 07:57:46.587373 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:57:46.587320    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 07:57:47.794104 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:57:47.794071    4720 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-65ddcfbf9c-rqjzs"]
Oct 31 07:57:47.794659 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:57:47.794638    4720 cpu_manager.go:414] "containerMap: removing container" podUID="afb766a3-c5c7-4fd1-9376-825f4045155d" containerName="main"
Oct 31 07:57:47.794740 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:57:47.794661    4720 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="afb766a3-c5c7-4fd1-9376-825f4045155d" containerName="main"
Oct 31 07:57:47.794805 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:57:47.794790    4720 memory_manager.go:365] "RemoveStaleState removing state" podUID="afb766a3-c5c7-4fd1-9376-825f4045155d" containerName="main"
Oct 31 07:57:47.806254 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:57:47.806227    4720 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-65ddcfbf9c-rqjzs"
Oct 31 07:57:47.844218 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:57:47.844185    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 07:57:47.848698 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:57:47.848676    4720 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-65ddcfbf9c-rqjzs"]
Oct 31 07:57:47.896546 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:57:47.896521    4720 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-enabled-requires-token-d4ed2c08\"/\"kube-root-ca.crt\""
Oct 31 07:57:47.896546 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:57:47.896546    4720 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-enabled-requires-token-d4ed2c08\"/\"openshift-service-ca.crt\""
Oct 31 07:57:47.896732 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:57:47.896524    4720 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-enabled-requires-token-d4ed2c08\"/\"auth-enabled-test-kserve-self-signed-certs\""
Oct 31 07:57:47.896732 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:57:47.896524    4720 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-enabled-requires-token-d4ed2c08\"/\"seaweedfs-s3-creds\""
Oct 31 07:57:47.896732 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:57:47.896583    4720 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-enabled-requires-token-d4ed2c08\"/\"odh-kserve-custom-ca-bundle\""
Oct 31 07:57:47.896732 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:57:47.896589    4720 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-enabled-requires-token-d4ed2c08\"/\"default-dockercfg-b9hzd\""
Oct 31 07:57:47.952383 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:57:47.952306    4720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/76078419-24f6-433c-a8c4-e6d2823fa3c1-model-cache\") pod \"auth-enabled-test-kserve-65ddcfbf9c-rqjzs\" (UID: \"76078419-24f6-433c-a8c4-e6d2823fa3c1\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-65ddcfbf9c-rqjzs"
Oct 31 07:57:47.952516 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:57:47.952442    4720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/76078419-24f6-433c-a8c4-e6d2823fa3c1-kserve-provision-location\") pod \"auth-enabled-test-kserve-65ddcfbf9c-rqjzs\" (UID: \"76078419-24f6-433c-a8c4-e6d2823fa3c1\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-65ddcfbf9c-rqjzs"
Oct 31 07:57:47.952582 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:57:47.952532    4720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/76078419-24f6-433c-a8c4-e6d2823fa3c1-home\") pod \"auth-enabled-test-kserve-65ddcfbf9c-rqjzs\" (UID: \"76078419-24f6-433c-a8c4-e6d2823fa3c1\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-65ddcfbf9c-rqjzs"
Oct 31 07:57:47.952642 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:57:47.952607    4720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m6tng\" (UniqueName: \"kubernetes.io/projected/76078419-24f6-433c-a8c4-e6d2823fa3c1-kube-api-access-m6tng\") pod \"auth-enabled-test-kserve-65ddcfbf9c-rqjzs\" (UID: \"76078419-24f6-433c-a8c4-e6d2823fa3c1\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-65ddcfbf9c-rqjzs"
Oct 31 07:57:47.952698 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:57:47.952654    4720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/76078419-24f6-433c-a8c4-e6d2823fa3c1-tmp-dir\") pod \"auth-enabled-test-kserve-65ddcfbf9c-rqjzs\" (UID: \"76078419-24f6-433c-a8c4-e6d2823fa3c1\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-65ddcfbf9c-rqjzs"
Oct 31 07:57:47.952746 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:57:47.952694    4720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/76078419-24f6-433c-a8c4-e6d2823fa3c1-tls-certs\") pod \"auth-enabled-test-kserve-65ddcfbf9c-rqjzs\" (UID: \"76078419-24f6-433c-a8c4-e6d2823fa3c1\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-65ddcfbf9c-rqjzs"
Oct 31 07:57:47.952792 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:57:47.952777    4720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/76078419-24f6-433c-a8c4-e6d2823fa3c1-dshm\") pod \"auth-enabled-test-kserve-65ddcfbf9c-rqjzs\" (UID: \"76078419-24f6-433c-a8c4-e6d2823fa3c1\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-65ddcfbf9c-rqjzs"
Oct 31 07:57:47.952845 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:57:47.952827    4720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/76078419-24f6-433c-a8c4-e6d2823fa3c1-cabundle-cert\") pod \"auth-enabled-test-kserve-65ddcfbf9c-rqjzs\" (UID: \"76078419-24f6-433c-a8c4-e6d2823fa3c1\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-65ddcfbf9c-rqjzs"
Oct 31 07:57:48.053467 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:57:48.053409    4720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m6tng\" (UniqueName: \"kubernetes.io/projected/76078419-24f6-433c-a8c4-e6d2823fa3c1-kube-api-access-m6tng\") pod \"auth-enabled-test-kserve-65ddcfbf9c-rqjzs\" (UID: \"76078419-24f6-433c-a8c4-e6d2823fa3c1\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-65ddcfbf9c-rqjzs"
Oct 31 07:57:48.053467 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:57:48.053451    4720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/76078419-24f6-433c-a8c4-e6d2823fa3c1-tmp-dir\") pod \"auth-enabled-test-kserve-65ddcfbf9c-rqjzs\" (UID: \"76078419-24f6-433c-a8c4-e6d2823fa3c1\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-65ddcfbf9c-rqjzs"
Oct 31 07:57:48.053635 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:57:48.053476    4720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/76078419-24f6-433c-a8c4-e6d2823fa3c1-tls-certs\") pod \"auth-enabled-test-kserve-65ddcfbf9c-rqjzs\" (UID: \"76078419-24f6-433c-a8c4-e6d2823fa3c1\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-65ddcfbf9c-rqjzs"
Oct 31 07:57:48.053635 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:57:48.053494    4720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/76078419-24f6-433c-a8c4-e6d2823fa3c1-dshm\") pod \"auth-enabled-test-kserve-65ddcfbf9c-rqjzs\" (UID: \"76078419-24f6-433c-a8c4-e6d2823fa3c1\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-65ddcfbf9c-rqjzs"
Oct 31 07:57:48.053635 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:57:48.053515    4720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/76078419-24f6-433c-a8c4-e6d2823fa3c1-cabundle-cert\") pod \"auth-enabled-test-kserve-65ddcfbf9c-rqjzs\" (UID: \"76078419-24f6-433c-a8c4-e6d2823fa3c1\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-65ddcfbf9c-rqjzs"
Oct 31 07:57:48.053799 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:57:48.053629    4720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/76078419-24f6-433c-a8c4-e6d2823fa3c1-model-cache\") pod \"auth-enabled-test-kserve-65ddcfbf9c-rqjzs\" (UID: \"76078419-24f6-433c-a8c4-e6d2823fa3c1\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-65ddcfbf9c-rqjzs"
Oct 31 07:57:48.053799 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:57:48.053689    4720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/76078419-24f6-433c-a8c4-e6d2823fa3c1-kserve-provision-location\") pod \"auth-enabled-test-kserve-65ddcfbf9c-rqjzs\" (UID: \"76078419-24f6-433c-a8c4-e6d2823fa3c1\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-65ddcfbf9c-rqjzs"
Oct 31 07:57:48.053799 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:57:48.053749    4720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/76078419-24f6-433c-a8c4-e6d2823fa3c1-home\") pod \"auth-enabled-test-kserve-65ddcfbf9c-rqjzs\" (UID: \"76078419-24f6-433c-a8c4-e6d2823fa3c1\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-65ddcfbf9c-rqjzs"
Oct 31 07:57:48.053943 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:57:48.053902    4720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/76078419-24f6-433c-a8c4-e6d2823fa3c1-tmp-dir\") pod \"auth-enabled-test-kserve-65ddcfbf9c-rqjzs\" (UID: \"76078419-24f6-433c-a8c4-e6d2823fa3c1\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-65ddcfbf9c-rqjzs"
Oct 31 07:57:48.054031 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:57:48.054008    4720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/76078419-24f6-433c-a8c4-e6d2823fa3c1-model-cache\") pod \"auth-enabled-test-kserve-65ddcfbf9c-rqjzs\" (UID: \"76078419-24f6-433c-a8c4-e6d2823fa3c1\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-65ddcfbf9c-rqjzs"
Oct 31 07:57:48.054093 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:57:48.054067    4720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/76078419-24f6-433c-a8c4-e6d2823fa3c1-kserve-provision-location\") pod \"auth-enabled-test-kserve-65ddcfbf9c-rqjzs\" (UID: \"76078419-24f6-433c-a8c4-e6d2823fa3c1\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-65ddcfbf9c-rqjzs"
Oct 31 07:57:48.054174 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:57:48.054156    4720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/76078419-24f6-433c-a8c4-e6d2823fa3c1-home\") pod \"auth-enabled-test-kserve-65ddcfbf9c-rqjzs\" (UID: \"76078419-24f6-433c-a8c4-e6d2823fa3c1\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-65ddcfbf9c-rqjzs"
Oct 31 07:57:48.054244 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:57:48.054228    4720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/76078419-24f6-433c-a8c4-e6d2823fa3c1-cabundle-cert\") pod \"auth-enabled-test-kserve-65ddcfbf9c-rqjzs\" (UID: \"76078419-24f6-433c-a8c4-e6d2823fa3c1\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-65ddcfbf9c-rqjzs"
Oct 31 07:57:48.055963 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:57:48.055943    4720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/76078419-24f6-433c-a8c4-e6d2823fa3c1-dshm\") pod \"auth-enabled-test-kserve-65ddcfbf9c-rqjzs\" (UID: \"76078419-24f6-433c-a8c4-e6d2823fa3c1\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-65ddcfbf9c-rqjzs"
Oct 31 07:57:48.056256 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:57:48.056239    4720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/76078419-24f6-433c-a8c4-e6d2823fa3c1-tls-certs\") pod \"auth-enabled-test-kserve-65ddcfbf9c-rqjzs\" (UID: \"76078419-24f6-433c-a8c4-e6d2823fa3c1\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-65ddcfbf9c-rqjzs"
Oct 31 07:57:48.109945 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:57:48.109923    4720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m6tng\" (UniqueName: \"kubernetes.io/projected/76078419-24f6-433c-a8c4-e6d2823fa3c1-kube-api-access-m6tng\") pod \"auth-enabled-test-kserve-65ddcfbf9c-rqjzs\" (UID: \"76078419-24f6-433c-a8c4-e6d2823fa3c1\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-65ddcfbf9c-rqjzs"
Oct 31 07:57:48.121774 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:57:48.121757    4720 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-65ddcfbf9c-rqjzs"
Oct 31 07:57:48.271941 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:57:48.271910    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 07:57:48.421982 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:57:48.421942    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 07:57:48.595956 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:57:48.595787    4720 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-65ddcfbf9c-rqjzs"]
Oct 31 07:57:48.598546 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:57:48.598520    4720 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod76078419_24f6_433c_a8c4_e6d2823fa3c1.slice/crio-092dde506177299f4714612183c0fbca893084dc121eb8fa9af6759043497c06 WatchSource:0}: Error finding container 092dde506177299f4714612183c0fbca893084dc121eb8fa9af6759043497c06: Status 404 returned error can't find the container with id 092dde506177299f4714612183c0fbca893084dc121eb8fa9af6759043497c06
Oct 31 07:57:49.502266 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:57:49.502225    4720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-65ddcfbf9c-rqjzs" event={"ID":"76078419-24f6-433c-a8c4-e6d2823fa3c1","Type":"ContainerStarted","Data":"fc89a4e7350c54e24a6d69ceb1876b6960cf6fb13798677613f067b46b1d9b5b"}
Oct 31 07:57:49.502748 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:57:49.502293    4720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-65ddcfbf9c-rqjzs" event={"ID":"76078419-24f6-433c-a8c4-e6d2823fa3c1","Type":"ContainerStarted","Data":"092dde506177299f4714612183c0fbca893084dc121eb8fa9af6759043497c06"}
Oct 31 07:57:50.115352 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:57:50.114503    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 07:57:51.545934 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:57:51.545891    4720 generic.go:356] "Generic (PLEG): container finished" podID="76078419-24f6-433c-a8c4-e6d2823fa3c1" containerID="fc89a4e7350c54e24a6d69ceb1876b6960cf6fb13798677613f067b46b1d9b5b" exitCode=0
Oct 31 07:57:51.546360 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:57:51.545965    4720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-65ddcfbf9c-rqjzs" event={"ID":"76078419-24f6-433c-a8c4-e6d2823fa3c1","Type":"ContainerDied","Data":"fc89a4e7350c54e24a6d69ceb1876b6960cf6fb13798677613f067b46b1d9b5b"}
Oct 31 07:57:52.558838 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:57:52.558803    4720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-65ddcfbf9c-rqjzs" event={"ID":"76078419-24f6-433c-a8c4-e6d2823fa3c1","Type":"ContainerStarted","Data":"f6d4fb2944ec42e85dad6cc3ce55b24afbc7febd0165b08a85d85a4280b5c870"}
Oct 31 07:57:52.787072 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:57:52.787014    4720 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-65ddcfbf9c-rqjzs" podStartSLOduration=3.145338525 podStartE2EDuration="5.786993113s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.641654588s" isStatefulPod=false podCreationTimestamp="2026-10-31 07:57:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 07:57:52.731961047 +0000 UTC m=+1726.863029120" watchObservedRunningTime="2026-10-31 07:57:52.786993113 +0000 UTC m=+1726.918061187"
Oct 31 07:57:53.125469 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:57:53.125431    4720 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-65ddcfbf9c-rqjzs"
Oct 31 07:57:54.916437 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:57:54.916401    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 07:57:58.122711 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:57:58.122675    4720 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-65ddcfbf9c-rqjzs"
Oct 31 07:57:58.124066 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:57:58.124041    4720 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-65ddcfbf9c-rqjzs" podUID="76078419-24f6-433c-a8c4-e6d2823fa3c1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.46:8000/health\": dial tcp 10.132.0.46:8000: connect: connection refused"
Oct 31 07:57:58.347180 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:57:58.347150    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 07:57:58.554794 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:57:58.554715    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 07:57:59.871671 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:57:59.871627    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 07:58:00.967202 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:58:00.967166    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 07:58:04.044441 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:58:04.044406    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 07:58:04.391498 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:58:04.391469    4720 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-84865b64j4n6"]
Oct 31 07:58:04.393746 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:58:04.393727    4720 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-84865b64j4n6"
Oct 31 07:58:04.482645 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:58:04.482613    4720 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-84865b64j4n6"]
Oct 31 07:58:04.500101 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:58:04.500068    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 07:58:04.554082 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:58:04.554053    4720 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-2541cd65\"/\"kube-root-ca.crt\""
Oct 31 07:58:04.554082 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:58:04.554051    4720 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-2541cd65\"/\"openshift-service-ca.crt\""
Oct 31 07:58:04.554391 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:58:04.554178    4720 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-2541cd65\"/\"default-dockercfg-h92jf\""
Oct 31 07:58:04.554391 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:58:04.554271    4720 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-2541cd65\"/\"scheduler-ha-replicas-test-epp-sa-dockercfg-8v9s4\""
Oct 31 07:58:04.554391 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:58:04.554366    4720 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-2541cd65\"/\"scheduler-ha-replicas-test-kserve-self-signed-certs\""
Oct 31 07:58:04.580667 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:58:04.580643    4720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qdpzf\" (UniqueName: \"kubernetes.io/projected/c036a258-67b1-48e2-8975-9f9479bdf3cd-kube-api-access-qdpzf\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-84865b64j4n6\" (UID: \"c036a258-67b1-48e2-8975-9f9479bdf3cd\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-84865b64j4n6"
Oct 31 07:58:04.580840 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:58:04.580818    4720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c036a258-67b1-48e2-8975-9f9479bdf3cd-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-84865b64j4n6\" (UID: \"c036a258-67b1-48e2-8975-9f9479bdf3cd\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-84865b64j4n6"
Oct 31 07:58:04.682248 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:58:04.682176    4720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c036a258-67b1-48e2-8975-9f9479bdf3cd-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-84865b64j4n6\" (UID: \"c036a258-67b1-48e2-8975-9f9479bdf3cd\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-84865b64j4n6"
Oct 31 07:58:04.682248 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:58:04.682233    4720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qdpzf\" (UniqueName: \"kubernetes.io/projected/c036a258-67b1-48e2-8975-9f9479bdf3cd-kube-api-access-qdpzf\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-84865b64j4n6\" (UID: \"c036a258-67b1-48e2-8975-9f9479bdf3cd\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-84865b64j4n6"
Oct 31 07:58:04.684877 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:58:04.684852    4720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c036a258-67b1-48e2-8975-9f9479bdf3cd-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-84865b64j4n6\" (UID: \"c036a258-67b1-48e2-8975-9f9479bdf3cd\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-84865b64j4n6"
Oct 31 07:58:04.738199 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:58:04.738170    4720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qdpzf\" (UniqueName: \"kubernetes.io/projected/c036a258-67b1-48e2-8975-9f9479bdf3cd-kube-api-access-qdpzf\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-84865b64j4n6\" (UID: \"c036a258-67b1-48e2-8975-9f9479bdf3cd\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-84865b64j4n6"
Oct 31 07:58:05.005008 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:58:05.004918    4720 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-84865b64j4n6"
Oct 31 07:58:05.374194 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:58:05.374159    4720 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-84865b64j4n6"]
Oct 31 07:58:05.378157 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:58:05.378129    4720 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc036a258_67b1_48e2_8975_9f9479bdf3cd.slice/crio-d4fd43a92d7271549b1333ceab83ffe127511862df68c8d327fa26f7757d2ca4 WatchSource:0}: Error finding container d4fd43a92d7271549b1333ceab83ffe127511862df68c8d327fa26f7757d2ca4: Status 404 returned error can't find the container with id d4fd43a92d7271549b1333ceab83ffe127511862df68c8d327fa26f7757d2ca4
Oct 31 07:58:05.380789 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:58:05.380771    4720 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 31 07:58:05.634822 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:58:05.634748    4720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-84865b64j4n6" event={"ID":"c036a258-67b1-48e2-8975-9f9479bdf3cd","Type":"ContainerStarted","Data":"d4fd43a92d7271549b1333ceab83ffe127511862df68c8d327fa26f7757d2ca4"}
Oct 31 07:58:07.661922 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:58:07.661827    4720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-84865b64j4n6" event={"ID":"c036a258-67b1-48e2-8975-9f9479bdf3cd","Type":"ContainerStarted","Data":"b38ee0bbc8e193df8a0f4680602a0645478b9140017f476668d4a4c39ab27e70"}
Oct 31 07:58:07.662315 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:58:07.661952    4720 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-84865b64j4n6"
Oct 31 07:58:07.775500 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:58:07.775434    4720 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-84865b64j4n6" podStartSLOduration=1.78970237 podStartE2EDuration="3.775414927s" totalImagesPullingTime="1.985712557s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 07:58:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 07:58:07.773611131 +0000 UTC m=+1741.904679205" watchObservedRunningTime="2026-10-31 07:58:07.775414927 +0000 UTC m=+1741.906483002"
Oct 31 07:58:08.093377 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:58:08.093320    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 07:58:08.122635 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:58:08.122593    4720 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-65ddcfbf9c-rqjzs" podUID="76078419-24f6-433c-a8c4-e6d2823fa3c1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.46:8000/health\": dial tcp 10.132.0.46:8000: connect: connection refused"
Oct 31 07:58:08.646378 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:58:08.646276    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 07:58:11.480189 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:58:11.480152    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 07:58:12.170177 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:58:12.170146    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 07:58:13.460262 ip-10-0-139-83 kubenswrapper[4720]: E1031 07:58:13.460221    4720 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8m768" containerName="extractor"
Oct 31 07:58:15.156263 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:58:15.156189    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 07:58:15.302070 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:58:15.302031    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 07:58:18.122461 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:58:18.122424    4720 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-65ddcfbf9c-rqjzs" podUID="76078419-24f6-433c-a8c4-e6d2823fa3c1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.46:8000/health\": dial tcp 10.132.0.46:8000: connect: connection refused"
Oct 31 07:58:20.085606 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:58:20.085571    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 07:58:24.074914 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:58:24.074879    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 07:58:28.122466 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:58:28.122414    4720 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-65ddcfbf9c-rqjzs" podUID="76078419-24f6-433c-a8c4-e6d2823fa3c1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.46:8000/health\": dial tcp 10.132.0.46:8000: connect: connection refused"
Oct 31 07:58:31.225413 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:58:31.225355    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 07:58:31.388577 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:58:31.388543    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 07:58:35.082209 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:58:35.082131    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 07:58:38.122403 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:58:38.122354    4720 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-65ddcfbf9c-rqjzs" podUID="76078419-24f6-433c-a8c4-e6d2823fa3c1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.46:8000/health\": dial tcp 10.132.0.46:8000: connect: connection refused"
Oct 31 07:58:38.668079 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:58:38.668042    4720 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-84865b64j4n6" podUID="c036a258-67b1-48e2-8975-9f9479bdf3cd" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 07:58:39.619298 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:58:39.619268    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 07:58:43.460881 ip-10-0-139-83 kubenswrapper[4720]: E1031 07:58:43.460841    4720 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8m768" containerName="extractor"
Oct 31 07:58:43.528395 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:58:43.528364    4720 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-84865b64j4n6"]
Oct 31 07:58:43.530798 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:58:43.530752    4720 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-84865b64j4n6" podUID="c036a258-67b1-48e2-8975-9f9479bdf3cd" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 07:58:43.556755 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:58:43.556700    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 07:58:43.710991 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:58:43.710910    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 07:58:47.553504 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:58:47.553468    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 07:58:48.122596 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:58:48.122553    4720 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-65ddcfbf9c-rqjzs" podUID="76078419-24f6-433c-a8c4-e6d2823fa3c1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.46:8000/health\": dial tcp 10.132.0.46:8000: connect: connection refused"
Oct 31 07:58:53.531034 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:58:53.530994    4720 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-84865b64j4n6" podUID="c036a258-67b1-48e2-8975-9f9479bdf3cd" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 07:58:58.122483 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:58:58.122442    4720 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-65ddcfbf9c-rqjzs" podUID="76078419-24f6-433c-a8c4-e6d2823fa3c1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.46:8000/health\": dial tcp 10.132.0.46:8000: connect: connection refused"
Oct 31 07:58:58.549184 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:58:58.549098    4720 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-84865b64j4n6" podUID="c036a258-67b1-48e2-8975-9f9479bdf3cd" containerName="main" containerID="cri-o://b38ee0bbc8e193df8a0f4680602a0645478b9140017f476668d4a4c39ab27e70" gracePeriod=45
Oct 31 07:59:00.449154 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:59:00.449112    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 07:59:03.530976 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:59:03.530933    4720 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-84865b64j4n6" podUID="c036a258-67b1-48e2-8975-9f9479bdf3cd" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 07:59:06.176054 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:59:06.175946    4720 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-4jszp_b1db0e3b-e85d-4769-8012-cf4b6f740d9d/console-operator/2.log"
Oct 31 07:59:06.183655 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:59:06.183636    4720 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-4jszp_b1db0e3b-e85d-4769-8012-cf4b6f740d9d/console-operator/2.log"
Oct 31 07:59:08.122606 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:59:08.122571    4720 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-65ddcfbf9c-rqjzs" podUID="76078419-24f6-433c-a8c4-e6d2823fa3c1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.46:8000/health\": dial tcp 10.132.0.46:8000: connect: connection refused"
Oct 31 07:59:13.460651 ip-10-0-139-83 kubenswrapper[4720]: E1031 07:59:13.460615    4720 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8m768" containerName="extractor"
Oct 31 07:59:13.531174 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:59:13.531132    4720 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-84865b64j4n6" podUID="c036a258-67b1-48e2-8975-9f9479bdf3cd" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 07:59:18.123142 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:59:18.123103    4720 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-65ddcfbf9c-rqjzs" podUID="76078419-24f6-433c-a8c4-e6d2823fa3c1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.46:8000/health\": dial tcp 10.132.0.46:8000: connect: connection refused"
Oct 31 07:59:18.858757 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:59:18.858719    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 07:59:20.392115 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:59:20.392082    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 07:59:23.531179 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:59:23.531145    4720 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-84865b64j4n6" podUID="c036a258-67b1-48e2-8975-9f9479bdf3cd" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 07:59:28.122302 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:59:28.122262    4720 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-65ddcfbf9c-rqjzs" podUID="76078419-24f6-433c-a8c4-e6d2823fa3c1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.46:8000/health\": dial tcp 10.132.0.46:8000: connect: connection refused"
Oct 31 07:59:28.815520 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:59:28.815461    4720 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-84865b64j4n6"
Oct 31 07:59:28.892609 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:59:28.892577    4720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c036a258-67b1-48e2-8975-9f9479bdf3cd-kube-api-access-qdpzf\" (UniqueName: \"kubernetes.io/projected/c036a258-67b1-48e2-8975-9f9479bdf3cd-kube-api-access-qdpzf\") pod \"c036a258-67b1-48e2-8975-9f9479bdf3cd\" (UID: \"c036a258-67b1-48e2-8975-9f9479bdf3cd\") "
Oct 31 07:59:28.892774 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:59:28.892671    4720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c036a258-67b1-48e2-8975-9f9479bdf3cd-tls-certs\" (UniqueName: \"kubernetes.io/secret/c036a258-67b1-48e2-8975-9f9479bdf3cd-tls-certs\") pod \"c036a258-67b1-48e2-8975-9f9479bdf3cd\" (UID: \"c036a258-67b1-48e2-8975-9f9479bdf3cd\") "
Oct 31 07:59:28.894938 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:59:28.894910    4720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c036a258-67b1-48e2-8975-9f9479bdf3cd-kube-api-access-qdpzf" pod "c036a258-67b1-48e2-8975-9f9479bdf3cd" (UID: "c036a258-67b1-48e2-8975-9f9479bdf3cd"). InnerVolumeSpecName "kube-api-access-qdpzf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 07:59:28.894938 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:59:28.894910    4720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c036a258-67b1-48e2-8975-9f9479bdf3cd-tls-certs" pod "c036a258-67b1-48e2-8975-9f9479bdf3cd" (UID: "c036a258-67b1-48e2-8975-9f9479bdf3cd"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 07:59:28.993543 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:59:28.993510    4720 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c036a258-67b1-48e2-8975-9f9479bdf3cd-tls-certs\") on node \"ip-10-0-139-83.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 07:59:28.993543 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:59:28.993537    4720 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qdpzf\" (UniqueName: \"kubernetes.io/projected/c036a258-67b1-48e2-8975-9f9479bdf3cd-kube-api-access-qdpzf\") on node \"ip-10-0-139-83.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 07:59:29.052211 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:59:29.052178    4720 generic.go:356] "Generic (PLEG): container finished" podID="c036a258-67b1-48e2-8975-9f9479bdf3cd" containerID="b38ee0bbc8e193df8a0f4680602a0645478b9140017f476668d4a4c39ab27e70" exitCode=0
Oct 31 07:59:29.052404 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:59:29.052225    4720 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-84865b64j4n6"
Oct 31 07:59:29.052404 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:59:29.052278    4720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-84865b64j4n6" event={"ID":"c036a258-67b1-48e2-8975-9f9479bdf3cd","Type":"ContainerDied","Data":"b38ee0bbc8e193df8a0f4680602a0645478b9140017f476668d4a4c39ab27e70"}
Oct 31 07:59:29.052404 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:59:29.052360    4720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-84865b64j4n6" event={"ID":"c036a258-67b1-48e2-8975-9f9479bdf3cd","Type":"ContainerDied","Data":"d4fd43a92d7271549b1333ceab83ffe127511862df68c8d327fa26f7757d2ca4"}
Oct 31 07:59:29.052404 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:59:29.052396    4720 scope.go:122] "RemoveContainer" containerID="b38ee0bbc8e193df8a0f4680602a0645478b9140017f476668d4a4c39ab27e70"
Oct 31 07:59:29.061781 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:59:29.061766    4720 scope.go:122] "RemoveContainer" containerID="b38ee0bbc8e193df8a0f4680602a0645478b9140017f476668d4a4c39ab27e70"
Oct 31 07:59:29.062072 ip-10-0-139-83 kubenswrapper[4720]: E1031 07:59:29.062049    4720 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b38ee0bbc8e193df8a0f4680602a0645478b9140017f476668d4a4c39ab27e70\": container with ID starting with b38ee0bbc8e193df8a0f4680602a0645478b9140017f476668d4a4c39ab27e70 not found: ID does not exist" containerID="b38ee0bbc8e193df8a0f4680602a0645478b9140017f476668d4a4c39ab27e70"
Oct 31 07:59:29.062178 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:59:29.062079    4720 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b38ee0bbc8e193df8a0f4680602a0645478b9140017f476668d4a4c39ab27e70"} err="failed to get container status \"b38ee0bbc8e193df8a0f4680602a0645478b9140017f476668d4a4c39ab27e70\": rpc error: code = NotFound desc = could not find container \"b38ee0bbc8e193df8a0f4680602a0645478b9140017f476668d4a4c39ab27e70\": container with ID starting with b38ee0bbc8e193df8a0f4680602a0645478b9140017f476668d4a4c39ab27e70 not found: ID does not exist"
Oct 31 07:59:29.214253 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:59:29.214217    4720 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-84865b64j4n6"]
Oct 31 07:59:29.216153 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:59:29.216129    4720 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-84865b64j4n6"]
Oct 31 07:59:30.026831 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:59:30.026796    4720 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c036a258-67b1-48e2-8975-9f9479bdf3cd" path="/var/lib/kubelet/pods/c036a258-67b1-48e2-8975-9f9479bdf3cd/volumes"
Oct 31 07:59:31.237280 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:59:31.237246    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 07:59:32.942074 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:59:32.942040    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 07:59:38.089630 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:59:38.089598    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 07:59:38.131957 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:59:38.131933    4720 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-65ddcfbf9c-rqjzs"
Oct 31 07:59:38.139313 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:59:38.139292    4720 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-65ddcfbf9c-rqjzs"
Oct 31 07:59:39.660015 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:59:39.659980    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 07:59:39.992747 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:59:39.992668    4720 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-bd5gqsr5"]
Oct 31 07:59:39.993023 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:59:39.993011    4720 cpu_manager.go:414] "containerMap: removing container" podUID="c036a258-67b1-48e2-8975-9f9479bdf3cd" containerName="main"
Oct 31 07:59:39.993064 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:59:39.993024    4720 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c036a258-67b1-48e2-8975-9f9479bdf3cd" containerName="main"
Oct 31 07:59:39.993116 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:59:39.993108    4720 memory_manager.go:365] "RemoveStaleState removing state" podUID="c036a258-67b1-48e2-8975-9f9479bdf3cd" containerName="main"
Oct 31 07:59:39.997411 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:59:39.997394    4720 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-bd5gqsr5"
Oct 31 07:59:40.053570 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:59:40.053542    4720 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-bd5gqsr5"]
Oct 31 07:59:40.095089 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:59:40.095057    4720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3ddd1869-5999-4fbd-983c-e7b25c410c2f-tls-certs\") pod \"scheduler-custom-template-test-kserve-router-scheduler-bd5gqsr5\" (UID: \"3ddd1869-5999-4fbd-983c-e7b25c410c2f\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-bd5gqsr5"
Oct 31 07:59:40.095244 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:59:40.095107    4720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/3ddd1869-5999-4fbd-983c-e7b25c410c2f-tokenizer-cache\") pod \"scheduler-custom-template-test-kserve-router-scheduler-bd5gqsr5\" (UID: \"3ddd1869-5999-4fbd-983c-e7b25c410c2f\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-bd5gqsr5"
Oct 31 07:59:40.095244 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:59:40.095124    4720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tlkkt\" (UniqueName: \"kubernetes.io/projected/3ddd1869-5999-4fbd-983c-e7b25c410c2f-kube-api-access-tlkkt\") pod \"scheduler-custom-template-test-kserve-router-scheduler-bd5gqsr5\" (UID: \"3ddd1869-5999-4fbd-983c-e7b25c410c2f\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-bd5gqsr5"
Oct 31 07:59:40.095346 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:59:40.095274    4720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cachi2-cache\" (UniqueName: \"kubernetes.io/empty-dir/3ddd1869-5999-4fbd-983c-e7b25c410c2f-cachi2-cache\") pod \"scheduler-custom-template-test-kserve-router-scheduler-bd5gqsr5\" (UID: \"3ddd1869-5999-4fbd-983c-e7b25c410c2f\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-bd5gqsr5"
Oct 31 07:59:40.099936 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:59:40.099913    4720 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-35377f35\"/\"kube-root-ca.crt\""
Oct 31 07:59:40.100037 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:59:40.099953    4720 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-35377f35\"/\"openshift-service-ca.crt\""
Oct 31 07:59:40.100037 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:59:40.099976    4720 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-35377f35\"/\"default-dockercfg-zfb55\""
Oct 31 07:59:40.100037 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:59:40.100002    4720 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-35377f35\"/\"scheduler-custom-template-test-kserve-self-signed-certs\""
Oct 31 07:59:40.100037 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:59:40.100031    4720 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-35377f35\"/\"scheduler-custom-template-test-epp-sa-dockercfg-b9sqq\""
Oct 31 07:59:40.115992 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:59:40.115955    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 07:59:40.196410 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:59:40.196376    4720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/3ddd1869-5999-4fbd-983c-e7b25c410c2f-tokenizer-cache\") pod \"scheduler-custom-template-test-kserve-router-scheduler-bd5gqsr5\" (UID: \"3ddd1869-5999-4fbd-983c-e7b25c410c2f\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-bd5gqsr5"
Oct 31 07:59:40.196410 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:59:40.196414    4720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tlkkt\" (UniqueName: \"kubernetes.io/projected/3ddd1869-5999-4fbd-983c-e7b25c410c2f-kube-api-access-tlkkt\") pod \"scheduler-custom-template-test-kserve-router-scheduler-bd5gqsr5\" (UID: \"3ddd1869-5999-4fbd-983c-e7b25c410c2f\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-bd5gqsr5"
Oct 31 07:59:40.196620 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:59:40.196464    4720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cachi2-cache\" (UniqueName: \"kubernetes.io/empty-dir/3ddd1869-5999-4fbd-983c-e7b25c410c2f-cachi2-cache\") pod \"scheduler-custom-template-test-kserve-router-scheduler-bd5gqsr5\" (UID: \"3ddd1869-5999-4fbd-983c-e7b25c410c2f\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-bd5gqsr5"
Oct 31 07:59:40.196620 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:59:40.196545    4720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3ddd1869-5999-4fbd-983c-e7b25c410c2f-tls-certs\") pod \"scheduler-custom-template-test-kserve-router-scheduler-bd5gqsr5\" (UID: \"3ddd1869-5999-4fbd-983c-e7b25c410c2f\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-bd5gqsr5"
Oct 31 07:59:40.196849 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:59:40.196825    4720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/3ddd1869-5999-4fbd-983c-e7b25c410c2f-tokenizer-cache\") pod \"scheduler-custom-template-test-kserve-router-scheduler-bd5gqsr5\" (UID: \"3ddd1869-5999-4fbd-983c-e7b25c410c2f\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-bd5gqsr5"
Oct 31 07:59:40.196919 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:59:40.196872    4720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cachi2-cache\" (UniqueName: \"kubernetes.io/empty-dir/3ddd1869-5999-4fbd-983c-e7b25c410c2f-cachi2-cache\") pod \"scheduler-custom-template-test-kserve-router-scheduler-bd5gqsr5\" (UID: \"3ddd1869-5999-4fbd-983c-e7b25c410c2f\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-bd5gqsr5"
Oct 31 07:59:40.199016 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:59:40.198999    4720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3ddd1869-5999-4fbd-983c-e7b25c410c2f-tls-certs\") pod \"scheduler-custom-template-test-kserve-router-scheduler-bd5gqsr5\" (UID: \"3ddd1869-5999-4fbd-983c-e7b25c410c2f\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-bd5gqsr5"
Oct 31 07:59:40.253994 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:59:40.253937    4720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tlkkt\" (UniqueName: \"kubernetes.io/projected/3ddd1869-5999-4fbd-983c-e7b25c410c2f-kube-api-access-tlkkt\") pod \"scheduler-custom-template-test-kserve-router-scheduler-bd5gqsr5\" (UID: \"3ddd1869-5999-4fbd-983c-e7b25c410c2f\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-bd5gqsr5"
Oct 31 07:59:40.309389 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:59:40.309359    4720 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-bd5gqsr5"
Oct 31 07:59:40.677556 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:59:40.677527    4720 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-bd5gqsr5"]
Oct 31 07:59:40.683946 ip-10-0-139-83 kubenswrapper[4720]: W1031 07:59:40.683911    4720 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ddd1869_5999_4fbd_983c_e7b25c410c2f.slice/crio-dea757c85dc90b55bd4ccfbd8e554459684b8f9247b7006eea7dfdeb3bae48d6 WatchSource:0}: Error finding container dea757c85dc90b55bd4ccfbd8e554459684b8f9247b7006eea7dfdeb3bae48d6: Status 404 returned error can't find the container with id dea757c85dc90b55bd4ccfbd8e554459684b8f9247b7006eea7dfdeb3bae48d6
Oct 31 07:59:41.113592 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:59:41.113558    4720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-bd5gqsr5" event={"ID":"3ddd1869-5999-4fbd-983c-e7b25c410c2f","Type":"ContainerStarted","Data":"7a3698bab69f98380c63acfde8b0c99076b19a8f2ef5f42b009de5c12c215317"}
Oct 31 07:59:41.113787 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:59:41.113614    4720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-bd5gqsr5" event={"ID":"3ddd1869-5999-4fbd-983c-e7b25c410c2f","Type":"ContainerStarted","Data":"dea757c85dc90b55bd4ccfbd8e554459684b8f9247b7006eea7dfdeb3bae48d6"}
Oct 31 07:59:41.113787 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:59:41.113663    4720 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-bd5gqsr5"
Oct 31 07:59:41.225968 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:59:41.225919    4720 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-bd5gqsr5" podStartSLOduration=2.225906231 podStartE2EDuration="2.225906231s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 07:59:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 07:59:41.224669789 +0000 UTC m=+1835.355737858" watchObservedRunningTime="2026-10-31 07:59:41.225906231 +0000 UTC m=+1835.356974298"
Oct 31 07:59:43.463611 ip-10-0-139-83 kubenswrapper[4720]: E1031 07:59:43.463566    4720 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8m768" containerName="extractor"
Oct 31 07:59:43.635896 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:59:43.635845    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 07:59:43.794525 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:59:43.794446    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 07:59:45.422154 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:59:45.422054    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 07:59:49.690974 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:59:49.690937    4720 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-65ddcfbf9c-rqjzs"]
Oct 31 07:59:49.719779 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:59:49.719738    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 07:59:49.870180 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:59:49.870146    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 07:59:52.218222 ip-10-0-139-83 kubenswrapper[4720]: I1031 07:59:52.218175    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 08:00:00.802411 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:00:00.802379    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 08:00:04.713729 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:00:04.713686    4720 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-65ddcfbf9c-rqjzs" podUID="76078419-24f6-433c-a8c4-e6d2823fa3c1" containerName="main" containerID="cri-o://f6d4fb2944ec42e85dad6cc3ce55b24afbc7febd0165b08a85d85a4280b5c870" gracePeriod=45
Oct 31 08:00:05.132923 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:00:05.132885    4720 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-65ddcfbf9c-rqjzs" podUID="76078419-24f6-433c-a8c4-e6d2823fa3c1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.46:8000/health\": dial tcp 10.132.0.46:8000: connect: connection refused"
Oct 31 08:00:05.354632 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:00:05.354606    4720 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-65ddcfbf9c-rqjzs"
Oct 31 08:00:05.513162 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:00:05.513102    4720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/76078419-24f6-433c-a8c4-e6d2823fa3c1-home\" (UniqueName: \"kubernetes.io/empty-dir/76078419-24f6-433c-a8c4-e6d2823fa3c1-home\") pod \"76078419-24f6-433c-a8c4-e6d2823fa3c1\" (UID: \"76078419-24f6-433c-a8c4-e6d2823fa3c1\") "
Oct 31 08:00:05.513162 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:00:05.513152    4720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/76078419-24f6-433c-a8c4-e6d2823fa3c1-tls-certs\" (UniqueName: \"kubernetes.io/secret/76078419-24f6-433c-a8c4-e6d2823fa3c1-tls-certs\") pod \"76078419-24f6-433c-a8c4-e6d2823fa3c1\" (UID: \"76078419-24f6-433c-a8c4-e6d2823fa3c1\") "
Oct 31 08:00:05.513378 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:00:05.513186    4720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/76078419-24f6-433c-a8c4-e6d2823fa3c1-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/76078419-24f6-433c-a8c4-e6d2823fa3c1-kserve-provision-location\") pod \"76078419-24f6-433c-a8c4-e6d2823fa3c1\" (UID: \"76078419-24f6-433c-a8c4-e6d2823fa3c1\") "
Oct 31 08:00:05.513378 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:00:05.513207    4720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/76078419-24f6-433c-a8c4-e6d2823fa3c1-dshm\" (UniqueName: \"kubernetes.io/empty-dir/76078419-24f6-433c-a8c4-e6d2823fa3c1-dshm\") pod \"76078419-24f6-433c-a8c4-e6d2823fa3c1\" (UID: \"76078419-24f6-433c-a8c4-e6d2823fa3c1\") "
Oct 31 08:00:05.513378 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:00:05.513243    4720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/76078419-24f6-433c-a8c4-e6d2823fa3c1-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/76078419-24f6-433c-a8c4-e6d2823fa3c1-model-cache\") pod \"76078419-24f6-433c-a8c4-e6d2823fa3c1\" (UID: \"76078419-24f6-433c-a8c4-e6d2823fa3c1\") "
Oct 31 08:00:05.513378 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:00:05.513264    4720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/76078419-24f6-433c-a8c4-e6d2823fa3c1-kube-api-access-m6tng\" (UniqueName: \"kubernetes.io/projected/76078419-24f6-433c-a8c4-e6d2823fa3c1-kube-api-access-m6tng\") pod \"76078419-24f6-433c-a8c4-e6d2823fa3c1\" (UID: \"76078419-24f6-433c-a8c4-e6d2823fa3c1\") "
Oct 31 08:00:05.513378 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:00:05.513296    4720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/76078419-24f6-433c-a8c4-e6d2823fa3c1-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/76078419-24f6-433c-a8c4-e6d2823fa3c1-tmp-dir\") pod \"76078419-24f6-433c-a8c4-e6d2823fa3c1\" (UID: \"76078419-24f6-433c-a8c4-e6d2823fa3c1\") "
Oct 31 08:00:05.513643 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:00:05.513381    4720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/76078419-24f6-433c-a8c4-e6d2823fa3c1-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/76078419-24f6-433c-a8c4-e6d2823fa3c1-cabundle-cert\") pod \"76078419-24f6-433c-a8c4-e6d2823fa3c1\" (UID: \"76078419-24f6-433c-a8c4-e6d2823fa3c1\") "
Oct 31 08:00:05.513643 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:00:05.513617    4720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76078419-24f6-433c-a8c4-e6d2823fa3c1-model-cache" pod "76078419-24f6-433c-a8c4-e6d2823fa3c1" (UID: "76078419-24f6-433c-a8c4-e6d2823fa3c1"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 08:00:05.513821 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:00:05.513802    4720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76078419-24f6-433c-a8c4-e6d2823fa3c1-home" pod "76078419-24f6-433c-a8c4-e6d2823fa3c1" (UID: "76078419-24f6-433c-a8c4-e6d2823fa3c1"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 08:00:05.514002 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:00:05.513922    4720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/76078419-24f6-433c-a8c4-e6d2823fa3c1-cabundle-cert" pod "76078419-24f6-433c-a8c4-e6d2823fa3c1" (UID: "76078419-24f6-433c-a8c4-e6d2823fa3c1"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 08:00:05.515707 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:00:05.515679    4720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/76078419-24f6-433c-a8c4-e6d2823fa3c1-tls-certs" pod "76078419-24f6-433c-a8c4-e6d2823fa3c1" (UID: "76078419-24f6-433c-a8c4-e6d2823fa3c1"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 08:00:05.515825 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:00:05.515723    4720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76078419-24f6-433c-a8c4-e6d2823fa3c1-dshm" pod "76078419-24f6-433c-a8c4-e6d2823fa3c1" (UID: "76078419-24f6-433c-a8c4-e6d2823fa3c1"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 08:00:05.516007 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:00:05.515988    4720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/76078419-24f6-433c-a8c4-e6d2823fa3c1-kube-api-access-m6tng" pod "76078419-24f6-433c-a8c4-e6d2823fa3c1" (UID: "76078419-24f6-433c-a8c4-e6d2823fa3c1"). InnerVolumeSpecName "kube-api-access-m6tng". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 08:00:05.532095 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:00:05.532065    4720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76078419-24f6-433c-a8c4-e6d2823fa3c1-tmp-dir" pod "76078419-24f6-433c-a8c4-e6d2823fa3c1" (UID: "76078419-24f6-433c-a8c4-e6d2823fa3c1"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 08:00:05.540958 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:00:05.540938    4720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76078419-24f6-433c-a8c4-e6d2823fa3c1-kserve-provision-location" pod "76078419-24f6-433c-a8c4-e6d2823fa3c1" (UID: "76078419-24f6-433c-a8c4-e6d2823fa3c1"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 08:00:05.614218 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:00:05.614194    4720 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/76078419-24f6-433c-a8c4-e6d2823fa3c1-model-cache\") on node \"ip-10-0-139-83.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 08:00:05.614218 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:00:05.614215    4720 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m6tng\" (UniqueName: \"kubernetes.io/projected/76078419-24f6-433c-a8c4-e6d2823fa3c1-kube-api-access-m6tng\") on node \"ip-10-0-139-83.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 08:00:05.614366 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:00:05.614225    4720 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/76078419-24f6-433c-a8c4-e6d2823fa3c1-tmp-dir\") on node \"ip-10-0-139-83.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 08:00:05.614366 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:00:05.614234    4720 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/76078419-24f6-433c-a8c4-e6d2823fa3c1-cabundle-cert\") on node \"ip-10-0-139-83.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 08:00:05.614366 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:00:05.614242    4720 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/76078419-24f6-433c-a8c4-e6d2823fa3c1-home\") on node \"ip-10-0-139-83.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 08:00:05.614366 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:00:05.614250    4720 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/76078419-24f6-433c-a8c4-e6d2823fa3c1-tls-certs\") on node \"ip-10-0-139-83.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 08:00:05.614366 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:00:05.614258    4720 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/76078419-24f6-433c-a8c4-e6d2823fa3c1-kserve-provision-location\") on node \"ip-10-0-139-83.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 08:00:05.614366 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:00:05.614266    4720 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/76078419-24f6-433c-a8c4-e6d2823fa3c1-dshm\") on node \"ip-10-0-139-83.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 08:00:06.231283 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:00:06.231249    4720 generic.go:356] "Generic (PLEG): container finished" podID="76078419-24f6-433c-a8c4-e6d2823fa3c1" containerID="f6d4fb2944ec42e85dad6cc3ce55b24afbc7febd0165b08a85d85a4280b5c870" exitCode=0
Oct 31 08:00:06.231623 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:00:06.231293    4720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-65ddcfbf9c-rqjzs" event={"ID":"76078419-24f6-433c-a8c4-e6d2823fa3c1","Type":"ContainerDied","Data":"f6d4fb2944ec42e85dad6cc3ce55b24afbc7febd0165b08a85d85a4280b5c870"}
Oct 31 08:00:06.231623 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:00:06.231306    4720 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-65ddcfbf9c-rqjzs"
Oct 31 08:00:06.231623 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:00:06.231372    4720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-65ddcfbf9c-rqjzs" event={"ID":"76078419-24f6-433c-a8c4-e6d2823fa3c1","Type":"ContainerDied","Data":"092dde506177299f4714612183c0fbca893084dc121eb8fa9af6759043497c06"}
Oct 31 08:00:06.231623 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:00:06.231402    4720 scope.go:122] "RemoveContainer" containerID="f6d4fb2944ec42e85dad6cc3ce55b24afbc7febd0165b08a85d85a4280b5c870"
Oct 31 08:00:06.241769 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:00:06.240354    4720 scope.go:122] "RemoveContainer" containerID="fc89a4e7350c54e24a6d69ceb1876b6960cf6fb13798677613f067b46b1d9b5b"
Oct 31 08:00:06.249363 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:00:06.249318    4720 scope.go:122] "RemoveContainer" containerID="f6d4fb2944ec42e85dad6cc3ce55b24afbc7febd0165b08a85d85a4280b5c870"
Oct 31 08:00:06.249608 ip-10-0-139-83 kubenswrapper[4720]: E1031 08:00:06.249591    4720 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f6d4fb2944ec42e85dad6cc3ce55b24afbc7febd0165b08a85d85a4280b5c870\": container with ID starting with f6d4fb2944ec42e85dad6cc3ce55b24afbc7febd0165b08a85d85a4280b5c870 not found: ID does not exist" containerID="f6d4fb2944ec42e85dad6cc3ce55b24afbc7febd0165b08a85d85a4280b5c870"
Oct 31 08:00:06.249649 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:00:06.249618    4720 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f6d4fb2944ec42e85dad6cc3ce55b24afbc7febd0165b08a85d85a4280b5c870"} err="failed to get container status \"f6d4fb2944ec42e85dad6cc3ce55b24afbc7febd0165b08a85d85a4280b5c870\": rpc error: code = NotFound desc = could not find container \"f6d4fb2944ec42e85dad6cc3ce55b24afbc7febd0165b08a85d85a4280b5c870\": container with ID starting with f6d4fb2944ec42e85dad6cc3ce55b24afbc7febd0165b08a85d85a4280b5c870 not found: ID does not exist"
Oct 31 08:00:06.249649 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:00:06.249635    4720 scope.go:122] "RemoveContainer" containerID="fc89a4e7350c54e24a6d69ceb1876b6960cf6fb13798677613f067b46b1d9b5b"
Oct 31 08:00:06.249841 ip-10-0-139-83 kubenswrapper[4720]: E1031 08:00:06.249825    4720 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fc89a4e7350c54e24a6d69ceb1876b6960cf6fb13798677613f067b46b1d9b5b\": container with ID starting with fc89a4e7350c54e24a6d69ceb1876b6960cf6fb13798677613f067b46b1d9b5b not found: ID does not exist" containerID="fc89a4e7350c54e24a6d69ceb1876b6960cf6fb13798677613f067b46b1d9b5b"
Oct 31 08:00:06.249877 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:00:06.249846    4720 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fc89a4e7350c54e24a6d69ceb1876b6960cf6fb13798677613f067b46b1d9b5b"} err="failed to get container status \"fc89a4e7350c54e24a6d69ceb1876b6960cf6fb13798677613f067b46b1d9b5b\": rpc error: code = NotFound desc = could not find container \"fc89a4e7350c54e24a6d69ceb1876b6960cf6fb13798677613f067b46b1d9b5b\": container with ID starting with fc89a4e7350c54e24a6d69ceb1876b6960cf6fb13798677613f067b46b1d9b5b not found: ID does not exist"
Oct 31 08:00:06.443022 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:00:06.442994    4720 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-65ddcfbf9c-rqjzs"]
Oct 31 08:00:06.492419 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:00:06.492390    4720 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-65ddcfbf9c-rqjzs"]
Oct 31 08:00:08.025044 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:00:08.025015    4720 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="76078419-24f6-433c-a8c4-e6d2823fa3c1" path="/var/lib/kubelet/pods/76078419-24f6-433c-a8c4-e6d2823fa3c1/volumes"
Oct 31 08:00:12.120001 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:00:12.119973    4720 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-bd5gqsr5"
Oct 31 08:00:13.459607 ip-10-0-139-83 kubenswrapper[4720]: E1031 08:00:13.459576    4720 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8m768" containerName="extractor"
Oct 31 08:00:18.638168 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:00:18.638131    4720 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-bd5gqsr5"]
Oct 31 08:00:18.692804 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:00:18.692778    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 08:00:18.841582 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:00:18.841548    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 08:00:20.731258 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:00:20.731224    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 08:00:22.119621 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:00:22.119575    4720 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-bd5gqsr5" podUID="3ddd1869-5999-4fbd-983c-e7b25c410c2f" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 08:00:32.119572 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:00:32.119534    4720 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-bd5gqsr5" podUID="3ddd1869-5999-4fbd-983c-e7b25c410c2f" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 08:00:33.658571 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:00:33.658528    4720 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-bd5gqsr5" podUID="3ddd1869-5999-4fbd-983c-e7b25c410c2f" containerName="main" containerID="cri-o://7a3698bab69f98380c63acfde8b0c99076b19a8f2ef5f42b009de5c12c215317" gracePeriod=45
Oct 31 08:00:42.119534 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:00:42.119492    4720 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-bd5gqsr5" podUID="3ddd1869-5999-4fbd-983c-e7b25c410c2f" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 08:00:42.119939 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:00:42.119600    4720 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-bd5gqsr5"
Oct 31 08:00:43.460559 ip-10-0-139-83 kubenswrapper[4720]: E1031 08:00:43.460523    4720 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8m768" containerName="extractor"
Oct 31 08:00:45.757023 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:00:45.756964    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 08:00:52.119669 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:00:52.119628    4720 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-bd5gqsr5" podUID="3ddd1869-5999-4fbd-983c-e7b25c410c2f" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 08:01:01.131863 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:01:01.131827    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 08:01:02.119491 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:01:02.119458    4720 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-bd5gqsr5" podUID="3ddd1869-5999-4fbd-983c-e7b25c410c2f" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 08:01:03.915218 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:01:03.915189    4720 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-bd5gqsr5"
Oct 31 08:01:04.094563 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:01:04.094527    4720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3ddd1869-5999-4fbd-983c-e7b25c410c2f-tls-certs\" (UniqueName: \"kubernetes.io/secret/3ddd1869-5999-4fbd-983c-e7b25c410c2f-tls-certs\") pod \"3ddd1869-5999-4fbd-983c-e7b25c410c2f\" (UID: \"3ddd1869-5999-4fbd-983c-e7b25c410c2f\") "
Oct 31 08:01:04.094733 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:01:04.094592    4720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3ddd1869-5999-4fbd-983c-e7b25c410c2f-kube-api-access-tlkkt\" (UniqueName: \"kubernetes.io/projected/3ddd1869-5999-4fbd-983c-e7b25c410c2f-kube-api-access-tlkkt\") pod \"3ddd1869-5999-4fbd-983c-e7b25c410c2f\" (UID: \"3ddd1869-5999-4fbd-983c-e7b25c410c2f\") "
Oct 31 08:01:04.094733 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:01:04.094695    4720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3ddd1869-5999-4fbd-983c-e7b25c410c2f-tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/3ddd1869-5999-4fbd-983c-e7b25c410c2f-tokenizer-cache\") pod \"3ddd1869-5999-4fbd-983c-e7b25c410c2f\" (UID: \"3ddd1869-5999-4fbd-983c-e7b25c410c2f\") "
Oct 31 08:01:04.094855 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:01:04.094805    4720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3ddd1869-5999-4fbd-983c-e7b25c410c2f-cachi2-cache\" (UniqueName: \"kubernetes.io/empty-dir/3ddd1869-5999-4fbd-983c-e7b25c410c2f-cachi2-cache\") pod \"3ddd1869-5999-4fbd-983c-e7b25c410c2f\" (UID: \"3ddd1869-5999-4fbd-983c-e7b25c410c2f\") "
Oct 31 08:01:04.094976 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:01:04.094949    4720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ddd1869-5999-4fbd-983c-e7b25c410c2f-tokenizer-cache" pod "3ddd1869-5999-4fbd-983c-e7b25c410c2f" (UID: "3ddd1869-5999-4fbd-983c-e7b25c410c2f"). InnerVolumeSpecName "tokenizer-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 08:01:04.095049 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:01:04.095028    4720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ddd1869-5999-4fbd-983c-e7b25c410c2f-cachi2-cache" pod "3ddd1869-5999-4fbd-983c-e7b25c410c2f" (UID: "3ddd1869-5999-4fbd-983c-e7b25c410c2f"). InnerVolumeSpecName "cachi2-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 08:01:04.095110 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:01:04.095098    4720 reconciler_common.go:299] "Volume detached for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/3ddd1869-5999-4fbd-983c-e7b25c410c2f-tokenizer-cache\") on node \"ip-10-0-139-83.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 08:01:04.096822 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:01:04.096796    4720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3ddd1869-5999-4fbd-983c-e7b25c410c2f-kube-api-access-tlkkt" pod "3ddd1869-5999-4fbd-983c-e7b25c410c2f" (UID: "3ddd1869-5999-4fbd-983c-e7b25c410c2f"). InnerVolumeSpecName "kube-api-access-tlkkt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 08:01:04.096933 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:01:04.096850    4720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3ddd1869-5999-4fbd-983c-e7b25c410c2f-tls-certs" pod "3ddd1869-5999-4fbd-983c-e7b25c410c2f" (UID: "3ddd1869-5999-4fbd-983c-e7b25c410c2f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 08:01:04.195489 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:01:04.195465    4720 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tlkkt\" (UniqueName: \"kubernetes.io/projected/3ddd1869-5999-4fbd-983c-e7b25c410c2f-kube-api-access-tlkkt\") on node \"ip-10-0-139-83.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 08:01:04.195587 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:01:04.195491    4720 reconciler_common.go:299] "Volume detached for volume \"cachi2-cache\" (UniqueName: \"kubernetes.io/empty-dir/3ddd1869-5999-4fbd-983c-e7b25c410c2f-cachi2-cache\") on node \"ip-10-0-139-83.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 08:01:04.195587 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:01:04.195504    4720 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3ddd1869-5999-4fbd-983c-e7b25c410c2f-tls-certs\") on node \"ip-10-0-139-83.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 08:01:04.490321 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:01:04.490243    4720 generic.go:356] "Generic (PLEG): container finished" podID="3ddd1869-5999-4fbd-983c-e7b25c410c2f" containerID="7a3698bab69f98380c63acfde8b0c99076b19a8f2ef5f42b009de5c12c215317" exitCode=0
Oct 31 08:01:04.490321 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:01:04.490289    4720 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-bd5gqsr5"
Oct 31 08:01:04.490564 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:01:04.490347    4720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-bd5gqsr5" event={"ID":"3ddd1869-5999-4fbd-983c-e7b25c410c2f","Type":"ContainerDied","Data":"7a3698bab69f98380c63acfde8b0c99076b19a8f2ef5f42b009de5c12c215317"}
Oct 31 08:01:04.490564 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:01:04.490405    4720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-bd5gqsr5" event={"ID":"3ddd1869-5999-4fbd-983c-e7b25c410c2f","Type":"ContainerDied","Data":"dea757c85dc90b55bd4ccfbd8e554459684b8f9247b7006eea7dfdeb3bae48d6"}
Oct 31 08:01:04.490564 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:01:04.490437    4720 scope.go:122] "RemoveContainer" containerID="7a3698bab69f98380c63acfde8b0c99076b19a8f2ef5f42b009de5c12c215317"
Oct 31 08:01:04.499657 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:01:04.499638    4720 scope.go:122] "RemoveContainer" containerID="7a3698bab69f98380c63acfde8b0c99076b19a8f2ef5f42b009de5c12c215317"
Oct 31 08:01:04.499907 ip-10-0-139-83 kubenswrapper[4720]: E1031 08:01:04.499887    4720 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7a3698bab69f98380c63acfde8b0c99076b19a8f2ef5f42b009de5c12c215317\": container with ID starting with 7a3698bab69f98380c63acfde8b0c99076b19a8f2ef5f42b009de5c12c215317 not found: ID does not exist" containerID="7a3698bab69f98380c63acfde8b0c99076b19a8f2ef5f42b009de5c12c215317"
Oct 31 08:01:04.499955 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:01:04.499916    4720 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7a3698bab69f98380c63acfde8b0c99076b19a8f2ef5f42b009de5c12c215317"} err="failed to get container status \"7a3698bab69f98380c63acfde8b0c99076b19a8f2ef5f42b009de5c12c215317\": rpc error: code = NotFound desc = could not find container \"7a3698bab69f98380c63acfde8b0c99076b19a8f2ef5f42b009de5c12c215317\": container with ID starting with 7a3698bab69f98380c63acfde8b0c99076b19a8f2ef5f42b009de5c12c215317 not found: ID does not exist"
Oct 31 08:01:04.652780 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:01:04.652746    4720 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-bd5gqsr5"]
Oct 31 08:01:04.702304 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:01:04.702273    4720 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-bd5gqsr5"]
Oct 31 08:01:06.025212 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:01:06.025179    4720 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3ddd1869-5999-4fbd-983c-e7b25c410c2f" path="/var/lib/kubelet/pods/3ddd1869-5999-4fbd-983c-e7b25c410c2f/volumes"
Oct 31 08:01:06.817891 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:01:06.817860    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 08:01:06.977630 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:01:06.977600    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 08:01:12.416859 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:01:12.416829    4720 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-8468d8f4d9-gg242"]
Oct 31 08:01:12.417242 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:01:12.417155    4720 cpu_manager.go:414] "containerMap: removing container" podUID="76078419-24f6-433c-a8c4-e6d2823fa3c1" containerName="storage-initializer"
Oct 31 08:01:12.417242 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:01:12.417166    4720 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="76078419-24f6-433c-a8c4-e6d2823fa3c1" containerName="storage-initializer"
Oct 31 08:01:12.417242 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:01:12.417179    4720 cpu_manager.go:414] "containerMap: removing container" podUID="76078419-24f6-433c-a8c4-e6d2823fa3c1" containerName="main"
Oct 31 08:01:12.417242 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:01:12.417184    4720 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="76078419-24f6-433c-a8c4-e6d2823fa3c1" containerName="main"
Oct 31 08:01:12.417242 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:01:12.417201    4720 cpu_manager.go:414] "containerMap: removing container" podUID="3ddd1869-5999-4fbd-983c-e7b25c410c2f" containerName="main"
Oct 31 08:01:12.417242 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:01:12.417206    4720 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3ddd1869-5999-4fbd-983c-e7b25c410c2f" containerName="main"
Oct 31 08:01:12.417456 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:01:12.417263    4720 memory_manager.go:365] "RemoveStaleState removing state" podUID="3ddd1869-5999-4fbd-983c-e7b25c410c2f" containerName="main"
Oct 31 08:01:12.417456 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:01:12.417273    4720 memory_manager.go:365] "RemoveStaleState removing state" podUID="76078419-24f6-433c-a8c4-e6d2823fa3c1" containerName="main"
Oct 31 08:01:12.421765 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:01:12.421748    4720 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-8468d8f4d9-gg242"
Oct 31 08:01:12.462847 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:01:12.462821    4720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p7hjw\" (UniqueName: \"kubernetes.io/projected/93de269e-e665-4ae7-89e0-e0d0e1ae7cc0-kube-api-access-p7hjw\") pod \"scheduler-v06-pd-migration-test-kserve-8468d8f4d9-gg242\" (UID: \"93de269e-e665-4ae7-89e0-e0d0e1ae7cc0\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-8468d8f4d9-gg242"
Oct 31 08:01:12.462997 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:01:12.462857    4720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/93de269e-e665-4ae7-89e0-e0d0e1ae7cc0-tmp-dir\") pod \"scheduler-v06-pd-migration-test-kserve-8468d8f4d9-gg242\" (UID: \"93de269e-e665-4ae7-89e0-e0d0e1ae7cc0\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-8468d8f4d9-gg242"
Oct 31 08:01:12.462997 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:01:12.462875    4720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/93de269e-e665-4ae7-89e0-e0d0e1ae7cc0-home\") pod \"scheduler-v06-pd-migration-test-kserve-8468d8f4d9-gg242\" (UID: \"93de269e-e665-4ae7-89e0-e0d0e1ae7cc0\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-8468d8f4d9-gg242"
Oct 31 08:01:12.462997 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:01:12.462891    4720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/93de269e-e665-4ae7-89e0-e0d0e1ae7cc0-model-cache\") pod \"scheduler-v06-pd-migration-test-kserve-8468d8f4d9-gg242\" (UID: \"93de269e-e665-4ae7-89e0-e0d0e1ae7cc0\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-8468d8f4d9-gg242"
Oct 31 08:01:12.463168 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:01:12.463080    4720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/93de269e-e665-4ae7-89e0-e0d0e1ae7cc0-dshm\") pod \"scheduler-v06-pd-migration-test-kserve-8468d8f4d9-gg242\" (UID: \"93de269e-e665-4ae7-89e0-e0d0e1ae7cc0\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-8468d8f4d9-gg242"
Oct 31 08:01:12.463168 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:01:12.463164    4720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/93de269e-e665-4ae7-89e0-e0d0e1ae7cc0-tls-certs\") pod \"scheduler-v06-pd-migration-test-kserve-8468d8f4d9-gg242\" (UID: \"93de269e-e665-4ae7-89e0-e0d0e1ae7cc0\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-8468d8f4d9-gg242"
Oct 31 08:01:12.525621 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:01:12.525589    4720 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-9c386aa9\"/\"default-dockercfg-d8tqr\""
Oct 31 08:01:12.525774 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:01:12.525722    4720 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-9c386aa9\"/\"scheduler-v06-pd-migration-test-kserve-self-signed-certs\""
Oct 31 08:01:12.525828 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:01:12.525800    4720 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-9c386aa9\"/\"scheduler-v06-pd-migration-test-kserve-dockercfg-n7rj9\""
Oct 31 08:01:12.525879 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:01:12.525865    4720 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-9c386aa9\"/\"kube-root-ca.crt\""
Oct 31 08:01:12.525952 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:01:12.525940    4720 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-9c386aa9\"/\"openshift-service-ca.crt\""
Oct 31 08:01:12.527853 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:01:12.527833    4720 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-8468d8f4d9-gg242"]
Oct 31 08:01:12.563661 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:01:12.563627    4720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/93de269e-e665-4ae7-89e0-e0d0e1ae7cc0-dshm\") pod \"scheduler-v06-pd-migration-test-kserve-8468d8f4d9-gg242\" (UID: \"93de269e-e665-4ae7-89e0-e0d0e1ae7cc0\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-8468d8f4d9-gg242"
Oct 31 08:01:12.563809 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:01:12.563681    4720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/93de269e-e665-4ae7-89e0-e0d0e1ae7cc0-tls-certs\") pod \"scheduler-v06-pd-migration-test-kserve-8468d8f4d9-gg242\" (UID: \"93de269e-e665-4ae7-89e0-e0d0e1ae7cc0\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-8468d8f4d9-gg242"
Oct 31 08:01:12.563809 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:01:12.563705    4720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p7hjw\" (UniqueName: \"kubernetes.io/projected/93de269e-e665-4ae7-89e0-e0d0e1ae7cc0-kube-api-access-p7hjw\") pod \"scheduler-v06-pd-migration-test-kserve-8468d8f4d9-gg242\" (UID: \"93de269e-e665-4ae7-89e0-e0d0e1ae7cc0\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-8468d8f4d9-gg242"
Oct 31 08:01:12.563809 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:01:12.563727    4720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/93de269e-e665-4ae7-89e0-e0d0e1ae7cc0-tmp-dir\") pod \"scheduler-v06-pd-migration-test-kserve-8468d8f4d9-gg242\" (UID: \"93de269e-e665-4ae7-89e0-e0d0e1ae7cc0\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-8468d8f4d9-gg242"
Oct 31 08:01:12.563809 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:01:12.563743    4720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/93de269e-e665-4ae7-89e0-e0d0e1ae7cc0-home\") pod \"scheduler-v06-pd-migration-test-kserve-8468d8f4d9-gg242\" (UID: \"93de269e-e665-4ae7-89e0-e0d0e1ae7cc0\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-8468d8f4d9-gg242"
Oct 31 08:01:12.563809 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:01:12.563758    4720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/93de269e-e665-4ae7-89e0-e0d0e1ae7cc0-model-cache\") pod \"scheduler-v06-pd-migration-test-kserve-8468d8f4d9-gg242\" (UID: \"93de269e-e665-4ae7-89e0-e0d0e1ae7cc0\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-8468d8f4d9-gg242"
Oct 31 08:01:12.564169 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:01:12.564146    4720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/93de269e-e665-4ae7-89e0-e0d0e1ae7cc0-home\") pod \"scheduler-v06-pd-migration-test-kserve-8468d8f4d9-gg242\" (UID: \"93de269e-e665-4ae7-89e0-e0d0e1ae7cc0\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-8468d8f4d9-gg242"
Oct 31 08:01:12.564271 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:01:12.564167    4720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/93de269e-e665-4ae7-89e0-e0d0e1ae7cc0-tmp-dir\") pod \"scheduler-v06-pd-migration-test-kserve-8468d8f4d9-gg242\" (UID: \"93de269e-e665-4ae7-89e0-e0d0e1ae7cc0\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-8468d8f4d9-gg242"
Oct 31 08:01:12.564271 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:01:12.564208    4720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/93de269e-e665-4ae7-89e0-e0d0e1ae7cc0-model-cache\") pod \"scheduler-v06-pd-migration-test-kserve-8468d8f4d9-gg242\" (UID: \"93de269e-e665-4ae7-89e0-e0d0e1ae7cc0\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-8468d8f4d9-gg242"
Oct 31 08:01:12.566045 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:01:12.566022    4720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/93de269e-e665-4ae7-89e0-e0d0e1ae7cc0-dshm\") pod \"scheduler-v06-pd-migration-test-kserve-8468d8f4d9-gg242\" (UID: \"93de269e-e665-4ae7-89e0-e0d0e1ae7cc0\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-8468d8f4d9-gg242"
Oct 31 08:01:12.566404 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:01:12.566387    4720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/93de269e-e665-4ae7-89e0-e0d0e1ae7cc0-tls-certs\") pod \"scheduler-v06-pd-migration-test-kserve-8468d8f4d9-gg242\" (UID: \"93de269e-e665-4ae7-89e0-e0d0e1ae7cc0\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-8468d8f4d9-gg242"
Oct 31 08:01:12.596806 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:01:12.596776    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 08:01:12.620042 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:01:12.620018    4720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p7hjw\" (UniqueName: \"kubernetes.io/projected/93de269e-e665-4ae7-89e0-e0d0e1ae7cc0-kube-api-access-p7hjw\") pod \"scheduler-v06-pd-migration-test-kserve-8468d8f4d9-gg242\" (UID: \"93de269e-e665-4ae7-89e0-e0d0e1ae7cc0\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-8468d8f4d9-gg242"
Oct 31 08:01:12.736137 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:01:12.736050    4720 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-8468d8f4d9-gg242"
Oct 31 08:01:13.105178 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:01:13.105142    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 08:01:13.159204 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:01:13.159174    4720 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-8468d8f4d9-gg242"]
Oct 31 08:01:13.162774 ip-10-0-139-83 kubenswrapper[4720]: W1031 08:01:13.162749    4720 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod93de269e_e665_4ae7_89e0_e0d0e1ae7cc0.slice/crio-28130b769ed5af2f8bf476fbc66406e01526d1cc7569ecdd2bffe47fed8bccb6 WatchSource:0}: Error finding container 28130b769ed5af2f8bf476fbc66406e01526d1cc7569ecdd2bffe47fed8bccb6: Status 404 returned error can't find the container with id 28130b769ed5af2f8bf476fbc66406e01526d1cc7569ecdd2bffe47fed8bccb6
Oct 31 08:01:13.465678 ip-10-0-139-83 kubenswrapper[4720]: E1031 08:01:13.465589    4720 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8m768" containerName="extractor"
Oct 31 08:01:13.535993 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:01:13.535956    4720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-8468d8f4d9-gg242" event={"ID":"93de269e-e665-4ae7-89e0-e0d0e1ae7cc0","Type":"ContainerStarted","Data":"28130b769ed5af2f8bf476fbc66406e01526d1cc7569ecdd2bffe47fed8bccb6"}
Oct 31 08:01:13.561407 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:01:13.561375    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 08:01:16.798247 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:01:16.797979    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 08:01:19.066784 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:01:19.066696    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 08:01:19.227133 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:01:19.227102    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 08:01:19.566608 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:01:19.566514    4720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-8468d8f4d9-gg242" event={"ID":"93de269e-e665-4ae7-89e0-e0d0e1ae7cc0","Type":"ContainerStarted","Data":"603b32c96d2e7b36176fe989b44ccca355fefb44e3ec92c92125c20077c7be79"}
Oct 31 08:01:19.566822 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:01:19.566647    4720 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-8468d8f4d9-gg242"
Oct 31 08:01:20.604351 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:01:20.603441    4720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-8468d8f4d9-gg242" event={"ID":"93de269e-e665-4ae7-89e0-e0d0e1ae7cc0","Type":"ContainerStarted","Data":"34649321d12fcd222c2b2c84e987d532d5e31c253ee7a26e7cc7f344dd148263"}
Oct 31 08:01:20.726352 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:01:20.725759    4720 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-8468d8f4d9-gg242" podStartSLOduration=3.12492829 podStartE2EDuration="8.72573984s" totalImagesPullingTime="5.60081155s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 08:01:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 08:01:20.724760118 +0000 UTC m=+1934.855828191" watchObservedRunningTime="2026-10-31 08:01:20.72573984 +0000 UTC m=+1934.856807912"
Oct 31 08:01:20.739533 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:01:20.739485    4720 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-8468d8f4d9-gg242"
Oct 31 08:01:21.100354 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:01:21.097255    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 08:01:22.736676 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:01:22.736638    4720 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-8468d8f4d9-gg242"
Oct 31 08:01:22.749063 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:01:22.749040    4720 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-8468d8f4d9-gg242"
Oct 31 08:01:23.644163 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:01:23.644133    4720 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-8468d8f4d9-gg242"
Oct 31 08:01:35.655639 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:01:35.655607    4720 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-8468d8f4d9-gg242"
Oct 31 08:01:43.461834 ip-10-0-139-83 kubenswrapper[4720]: E1031 08:01:43.461789    4720 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8m768" containerName="extractor"
Oct 31 08:01:55.416808 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:01:55.416769    4720 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-8468d8f4d9-gg242"]
Oct 31 08:01:55.429494 ip-10-0-139-83 kubenswrapper[4720]: E1031 08:01:55.429413    4720 log.go:32] "ExecSync cmd from runtime service failed" err="rpc error: code = Unknown desc = command error: , stdout: , stderr: , exit code -1" containerID="34649321d12fcd222c2b2c84e987d532d5e31c253ee7a26e7cc7f344dd148263" cmd=["/bin/sleep","15"]
Oct 31 08:01:55.429657 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:01:55.429552    4720 handlers.go:85] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-8468d8f4d9-gg242" message="" err="rpc error: code = Unknown desc = command error: , stdout: , stderr: , exit code -1"
Oct 31 08:01:55.429657 ip-10-0-139-83 kubenswrapper[4720]: E1031 08:01:55.429587    4720 kuberuntime_container.go:780] "PreStop hook failed" err="rpc error: code = Unknown desc = command error: , stdout: , stderr: , exit code -1" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-8468d8f4d9-gg242" podUID="93de269e-e665-4ae7-89e0-e0d0e1ae7cc0" containerName="main" containerID="cri-o://34649321d12fcd222c2b2c84e987d532d5e31c253ee7a26e7cc7f344dd148263"
Oct 31 08:01:55.429729 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:01:55.429700    4720 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-8468d8f4d9-gg242" podUID="93de269e-e665-4ae7-89e0-e0d0e1ae7cc0" containerName="main" containerID="cri-o://34649321d12fcd222c2b2c84e987d532d5e31c253ee7a26e7cc7f344dd148263" gracePeriod=60
Oct 31 08:01:55.455071 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:01:55.455035    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 08:01:55.603018 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:01:55.602987    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 08:01:55.633921 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:01:55.633885    4720 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-8468d8f4d9-gg242" podUID="93de269e-e665-4ae7-89e0-e0d0e1ae7cc0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8001/health\": dial tcp 10.132.0.49:8001: connect: connection refused"
Oct 31 08:01:55.642011 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:01:55.641959    4720 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-8468d8f4d9-gg242" podUID="93de269e-e665-4ae7-89e0-e0d0e1ae7cc0" containerName="llm-d-routing-sidecar" containerID="cri-o://603b32c96d2e7b36176fe989b44ccca355fefb44e3ec92c92125c20077c7be79" gracePeriod=60
Oct 31 08:01:55.646933 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:01:55.646903    4720 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-8468d8f4d9-gg242" podUID="93de269e-e665-4ae7-89e0-e0d0e1ae7cc0" containerName="llm-d-routing-sidecar" probeResult="failure" output="Get \"https://10.132.0.49:8000/health\": EOF"
Oct 31 08:01:55.780851 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:01:55.780817    4720 generic.go:356] "Generic (PLEG): container finished" podID="93de269e-e665-4ae7-89e0-e0d0e1ae7cc0" containerID="34649321d12fcd222c2b2c84e987d532d5e31c253ee7a26e7cc7f344dd148263" exitCode=0
Oct 31 08:01:55.780851 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:01:55.780844    4720 generic.go:356] "Generic (PLEG): container finished" podID="93de269e-e665-4ae7-89e0-e0d0e1ae7cc0" containerID="603b32c96d2e7b36176fe989b44ccca355fefb44e3ec92c92125c20077c7be79" exitCode=0
Oct 31 08:01:55.781099 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:01:55.780884    4720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-8468d8f4d9-gg242" event={"ID":"93de269e-e665-4ae7-89e0-e0d0e1ae7cc0","Type":"ContainerDied","Data":"34649321d12fcd222c2b2c84e987d532d5e31c253ee7a26e7cc7f344dd148263"}
Oct 31 08:01:55.781099 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:01:55.780963    4720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-8468d8f4d9-gg242" event={"ID":"93de269e-e665-4ae7-89e0-e0d0e1ae7cc0","Type":"ContainerDied","Data":"603b32c96d2e7b36176fe989b44ccca355fefb44e3ec92c92125c20077c7be79"}
Oct 31 08:01:55.897668 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:01:55.897640    4720 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-8468d8f4d9-gg242"
Oct 31 08:01:56.039084 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:01:56.038989    4720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/93de269e-e665-4ae7-89e0-e0d0e1ae7cc0-dshm\" (UniqueName: \"kubernetes.io/empty-dir/93de269e-e665-4ae7-89e0-e0d0e1ae7cc0-dshm\") pod \"93de269e-e665-4ae7-89e0-e0d0e1ae7cc0\" (UID: \"93de269e-e665-4ae7-89e0-e0d0e1ae7cc0\") "
Oct 31 08:01:56.039084 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:01:56.039052    4720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/93de269e-e665-4ae7-89e0-e0d0e1ae7cc0-kube-api-access-p7hjw\" (UniqueName: \"kubernetes.io/projected/93de269e-e665-4ae7-89e0-e0d0e1ae7cc0-kube-api-access-p7hjw\") pod \"93de269e-e665-4ae7-89e0-e0d0e1ae7cc0\" (UID: \"93de269e-e665-4ae7-89e0-e0d0e1ae7cc0\") "
Oct 31 08:01:56.039348 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:01:56.039092    4720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/93de269e-e665-4ae7-89e0-e0d0e1ae7cc0-home\" (UniqueName: \"kubernetes.io/empty-dir/93de269e-e665-4ae7-89e0-e0d0e1ae7cc0-home\") pod \"93de269e-e665-4ae7-89e0-e0d0e1ae7cc0\" (UID: \"93de269e-e665-4ae7-89e0-e0d0e1ae7cc0\") "
Oct 31 08:01:56.039348 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:01:56.039111    4720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/93de269e-e665-4ae7-89e0-e0d0e1ae7cc0-tls-certs\" (UniqueName: \"kubernetes.io/secret/93de269e-e665-4ae7-89e0-e0d0e1ae7cc0-tls-certs\") pod \"93de269e-e665-4ae7-89e0-e0d0e1ae7cc0\" (UID: \"93de269e-e665-4ae7-89e0-e0d0e1ae7cc0\") "
Oct 31 08:01:56.039348 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:01:56.039165    4720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/93de269e-e665-4ae7-89e0-e0d0e1ae7cc0-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/93de269e-e665-4ae7-89e0-e0d0e1ae7cc0-model-cache\") pod \"93de269e-e665-4ae7-89e0-e0d0e1ae7cc0\" (UID: \"93de269e-e665-4ae7-89e0-e0d0e1ae7cc0\") "
Oct 31 08:01:56.039348 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:01:56.039193    4720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/93de269e-e665-4ae7-89e0-e0d0e1ae7cc0-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/93de269e-e665-4ae7-89e0-e0d0e1ae7cc0-tmp-dir\") pod \"93de269e-e665-4ae7-89e0-e0d0e1ae7cc0\" (UID: \"93de269e-e665-4ae7-89e0-e0d0e1ae7cc0\") "
Oct 31 08:01:56.039570 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:01:56.039434    4720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/93de269e-e665-4ae7-89e0-e0d0e1ae7cc0-home" pod "93de269e-e665-4ae7-89e0-e0d0e1ae7cc0" (UID: "93de269e-e665-4ae7-89e0-e0d0e1ae7cc0"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 08:01:56.039570 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:01:56.039449    4720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/93de269e-e665-4ae7-89e0-e0d0e1ae7cc0-model-cache" pod "93de269e-e665-4ae7-89e0-e0d0e1ae7cc0" (UID: "93de269e-e665-4ae7-89e0-e0d0e1ae7cc0"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 08:01:56.039570 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:01:56.039547    4720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/93de269e-e665-4ae7-89e0-e0d0e1ae7cc0-tmp-dir" pod "93de269e-e665-4ae7-89e0-e0d0e1ae7cc0" (UID: "93de269e-e665-4ae7-89e0-e0d0e1ae7cc0"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 08:01:56.041534 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:01:56.041500    4720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/93de269e-e665-4ae7-89e0-e0d0e1ae7cc0-dshm" pod "93de269e-e665-4ae7-89e0-e0d0e1ae7cc0" (UID: "93de269e-e665-4ae7-89e0-e0d0e1ae7cc0"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 08:01:56.041656 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:01:56.041553    4720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/93de269e-e665-4ae7-89e0-e0d0e1ae7cc0-tls-certs" pod "93de269e-e665-4ae7-89e0-e0d0e1ae7cc0" (UID: "93de269e-e665-4ae7-89e0-e0d0e1ae7cc0"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 08:01:56.041656 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:01:56.041588    4720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/93de269e-e665-4ae7-89e0-e0d0e1ae7cc0-kube-api-access-p7hjw" pod "93de269e-e665-4ae7-89e0-e0d0e1ae7cc0" (UID: "93de269e-e665-4ae7-89e0-e0d0e1ae7cc0"). InnerVolumeSpecName "kube-api-access-p7hjw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 08:01:56.089227 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:01:56.089193    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 08:01:56.140940 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:01:56.140904    4720 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/93de269e-e665-4ae7-89e0-e0d0e1ae7cc0-model-cache\") on node \"ip-10-0-139-83.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 08:01:56.140940 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:01:56.140932    4720 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/93de269e-e665-4ae7-89e0-e0d0e1ae7cc0-tmp-dir\") on node \"ip-10-0-139-83.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 08:01:56.140940 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:01:56.140942    4720 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/93de269e-e665-4ae7-89e0-e0d0e1ae7cc0-dshm\") on node \"ip-10-0-139-83.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 08:01:56.140940 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:01:56.140950    4720 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p7hjw\" (UniqueName: \"kubernetes.io/projected/93de269e-e665-4ae7-89e0-e0d0e1ae7cc0-kube-api-access-p7hjw\") on node \"ip-10-0-139-83.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 08:01:56.141214 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:01:56.140959    4720 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/93de269e-e665-4ae7-89e0-e0d0e1ae7cc0-home\") on node \"ip-10-0-139-83.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 08:01:56.141214 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:01:56.140968    4720 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/93de269e-e665-4ae7-89e0-e0d0e1ae7cc0-tls-certs\") on node \"ip-10-0-139-83.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 08:01:56.787241 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:01:56.787211    4720 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-8468d8f4d9-gg242"
Oct 31 08:01:56.787737 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:01:56.787211    4720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-8468d8f4d9-gg242" event={"ID":"93de269e-e665-4ae7-89e0-e0d0e1ae7cc0","Type":"ContainerDied","Data":"28130b769ed5af2f8bf476fbc66406e01526d1cc7569ecdd2bffe47fed8bccb6"}
Oct 31 08:01:56.787737 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:01:56.787375    4720 scope.go:122] "RemoveContainer" containerID="34649321d12fcd222c2b2c84e987d532d5e31c253ee7a26e7cc7f344dd148263"
Oct 31 08:01:56.797058 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:01:56.797039    4720 scope.go:122] "RemoveContainer" containerID="603b32c96d2e7b36176fe989b44ccca355fefb44e3ec92c92125c20077c7be79"
Oct 31 08:01:56.998026 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:01:56.997988    4720 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-8468d8f4d9-gg242"]
Oct 31 08:01:57.047255 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:01:57.047176    4720 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-8468d8f4d9-gg242"]
Oct 31 08:01:58.025370 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:01:58.025319    4720 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="93de269e-e665-4ae7-89e0-e0d0e1ae7cc0" path="/var/lib/kubelet/pods/93de269e-e665-4ae7-89e0-e0d0e1ae7cc0/volumes"
Oct 31 08:02:00.084625 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:02:00.084591    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 08:02:00.242043 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:02:00.242000    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 08:02:01.463573 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:02:01.463537    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 08:02:06.161224 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:02:06.161196    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 08:02:11.742481 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:02:11.742439    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 08:02:11.891908 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:02:11.891875    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 08:02:13.461703 ip-10-0-139-83 kubenswrapper[4720]: E1031 08:02:13.461665    4720 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8m768" containerName="extractor"
Oct 31 08:02:31.443213 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:02:31.443178    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 08:02:43.461412 ip-10-0-139-83 kubenswrapper[4720]: E1031 08:02:43.461370    4720 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8m768" containerName="extractor"
Oct 31 08:02:44.869033 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:02:44.868999    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 08:02:45.019064 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:02:45.019032    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 08:02:51.006994 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:02:51.006957    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 08:02:52.736282 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:02:52.736249    4720 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-bc5ffbf5b-cjc72"]
Oct 31 08:02:52.736681 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:02:52.736596    4720 cpu_manager.go:414] "containerMap: removing container" podUID="93de269e-e665-4ae7-89e0-e0d0e1ae7cc0" containerName="llm-d-routing-sidecar"
Oct 31 08:02:52.736681 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:02:52.736607    4720 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="93de269e-e665-4ae7-89e0-e0d0e1ae7cc0" containerName="llm-d-routing-sidecar"
Oct 31 08:02:52.736681 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:02:52.736618    4720 cpu_manager.go:414] "containerMap: removing container" podUID="93de269e-e665-4ae7-89e0-e0d0e1ae7cc0" containerName="main"
Oct 31 08:02:52.736681 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:02:52.736623    4720 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="93de269e-e665-4ae7-89e0-e0d0e1ae7cc0" containerName="main"
Oct 31 08:02:52.736810 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:02:52.736703    4720 memory_manager.go:365] "RemoveStaleState removing state" podUID="93de269e-e665-4ae7-89e0-e0d0e1ae7cc0" containerName="llm-d-routing-sidecar"
Oct 31 08:02:52.736810 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:02:52.736711    4720 memory_manager.go:365] "RemoveStaleState removing state" podUID="93de269e-e665-4ae7-89e0-e0d0e1ae7cc0" containerName="main"
Oct 31 08:02:52.740149 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:02:52.740132    4720 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-bc5ffbf5b-cjc72"
Oct 31 08:02:52.792644 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:02:52.792616    4720 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-40b85f95\"/\"kube-root-ca.crt\""
Oct 31 08:02:52.832627 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:02:52.832597    4720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4e9f63cd-0961-4be1-a649-a11a4ed84278-tls-certs\") pod \"scheduler-v06-threshold-migration-test-kserve-bc5ffbf5b-cjc72\" (UID: \"4e9f63cd-0961-4be1-a649-a11a4ed84278\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-bc5ffbf5b-cjc72"
Oct 31 08:02:52.832773 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:02:52.832650    4720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4e9f63cd-0961-4be1-a649-a11a4ed84278-dshm\") pod \"scheduler-v06-threshold-migration-test-kserve-bc5ffbf5b-cjc72\" (UID: \"4e9f63cd-0961-4be1-a649-a11a4ed84278\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-bc5ffbf5b-cjc72"
Oct 31 08:02:52.832773 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:02:52.832725    4720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4e9f63cd-0961-4be1-a649-a11a4ed84278-tmp-dir\") pod \"scheduler-v06-threshold-migration-test-kserve-bc5ffbf5b-cjc72\" (UID: \"4e9f63cd-0961-4be1-a649-a11a4ed84278\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-bc5ffbf5b-cjc72"
Oct 31 08:02:52.832869 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:02:52.832776    4720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4e9f63cd-0961-4be1-a649-a11a4ed84278-model-cache\") pod \"scheduler-v06-threshold-migration-test-kserve-bc5ffbf5b-cjc72\" (UID: \"4e9f63cd-0961-4be1-a649-a11a4ed84278\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-bc5ffbf5b-cjc72"
Oct 31 08:02:52.832869 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:02:52.832810    4720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4e9f63cd-0961-4be1-a649-a11a4ed84278-home\") pod \"scheduler-v06-threshold-migration-test-kserve-bc5ffbf5b-cjc72\" (UID: \"4e9f63cd-0961-4be1-a649-a11a4ed84278\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-bc5ffbf5b-cjc72"
Oct 31 08:02:52.832952 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:02:52.832867    4720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cfz55\" (UniqueName: \"kubernetes.io/projected/4e9f63cd-0961-4be1-a649-a11a4ed84278-kube-api-access-cfz55\") pod \"scheduler-v06-threshold-migration-test-kserve-bc5ffbf5b-cjc72\" (UID: \"4e9f63cd-0961-4be1-a649-a11a4ed84278\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-bc5ffbf5b-cjc72"
Oct 31 08:02:52.841976 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:02:52.841948    4720 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-40b85f95\"/\"default-dockercfg-5gg8s\""
Oct 31 08:02:52.842180 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:02:52.842014    4720 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-40b85f95\"/\"scheduler-v06-threshold-migration-test-kserve-self-signed-certs\""
Oct 31 08:02:52.842441 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:02:52.842421    4720 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-40b85f95\"/\"scheduler-v06-threshold-migration-test-kserve-dockercfg-t9g6v\""
Oct 31 08:02:52.842612 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:02:52.842592    4720 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-40b85f95\"/\"openshift-service-ca.crt\""
Oct 31 08:02:52.844233 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:02:52.844213    4720 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-bc5ffbf5b-cjc72"]
Oct 31 08:02:52.910760 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:02:52.910734    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 08:02:52.934130 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:02:52.934101    4720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4e9f63cd-0961-4be1-a649-a11a4ed84278-tls-certs\") pod \"scheduler-v06-threshold-migration-test-kserve-bc5ffbf5b-cjc72\" (UID: \"4e9f63cd-0961-4be1-a649-a11a4ed84278\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-bc5ffbf5b-cjc72"
Oct 31 08:02:52.934255 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:02:52.934149    4720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4e9f63cd-0961-4be1-a649-a11a4ed84278-dshm\") pod \"scheduler-v06-threshold-migration-test-kserve-bc5ffbf5b-cjc72\" (UID: \"4e9f63cd-0961-4be1-a649-a11a4ed84278\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-bc5ffbf5b-cjc72"
Oct 31 08:02:52.934255 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:02:52.934181    4720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4e9f63cd-0961-4be1-a649-a11a4ed84278-tmp-dir\") pod \"scheduler-v06-threshold-migration-test-kserve-bc5ffbf5b-cjc72\" (UID: \"4e9f63cd-0961-4be1-a649-a11a4ed84278\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-bc5ffbf5b-cjc72"
Oct 31 08:02:52.934255 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:02:52.934219    4720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4e9f63cd-0961-4be1-a649-a11a4ed84278-model-cache\") pod \"scheduler-v06-threshold-migration-test-kserve-bc5ffbf5b-cjc72\" (UID: \"4e9f63cd-0961-4be1-a649-a11a4ed84278\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-bc5ffbf5b-cjc72"
Oct 31 08:02:52.934479 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:02:52.934258    4720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4e9f63cd-0961-4be1-a649-a11a4ed84278-home\") pod \"scheduler-v06-threshold-migration-test-kserve-bc5ffbf5b-cjc72\" (UID: \"4e9f63cd-0961-4be1-a649-a11a4ed84278\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-bc5ffbf5b-cjc72"
Oct 31 08:02:52.934479 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:02:52.934297    4720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cfz55\" (UniqueName: \"kubernetes.io/projected/4e9f63cd-0961-4be1-a649-a11a4ed84278-kube-api-access-cfz55\") pod \"scheduler-v06-threshold-migration-test-kserve-bc5ffbf5b-cjc72\" (UID: \"4e9f63cd-0961-4be1-a649-a11a4ed84278\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-bc5ffbf5b-cjc72"
Oct 31 08:02:52.934679 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:02:52.934662    4720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4e9f63cd-0961-4be1-a649-a11a4ed84278-home\") pod \"scheduler-v06-threshold-migration-test-kserve-bc5ffbf5b-cjc72\" (UID: \"4e9f63cd-0961-4be1-a649-a11a4ed84278\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-bc5ffbf5b-cjc72"
Oct 31 08:02:52.934753 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:02:52.934728    4720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4e9f63cd-0961-4be1-a649-a11a4ed84278-tmp-dir\") pod \"scheduler-v06-threshold-migration-test-kserve-bc5ffbf5b-cjc72\" (UID: \"4e9f63cd-0961-4be1-a649-a11a4ed84278\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-bc5ffbf5b-cjc72"
Oct 31 08:02:52.934791 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:02:52.934765    4720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4e9f63cd-0961-4be1-a649-a11a4ed84278-model-cache\") pod \"scheduler-v06-threshold-migration-test-kserve-bc5ffbf5b-cjc72\" (UID: \"4e9f63cd-0961-4be1-a649-a11a4ed84278\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-bc5ffbf5b-cjc72"
Oct 31 08:02:52.936657 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:02:52.936637    4720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4e9f63cd-0961-4be1-a649-a11a4ed84278-dshm\") pod \"scheduler-v06-threshold-migration-test-kserve-bc5ffbf5b-cjc72\" (UID: \"4e9f63cd-0961-4be1-a649-a11a4ed84278\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-bc5ffbf5b-cjc72"
Oct 31 08:02:52.936794 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:02:52.936778    4720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4e9f63cd-0961-4be1-a649-a11a4ed84278-tls-certs\") pod \"scheduler-v06-threshold-migration-test-kserve-bc5ffbf5b-cjc72\" (UID: \"4e9f63cd-0961-4be1-a649-a11a4ed84278\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-bc5ffbf5b-cjc72"
Oct 31 08:02:52.990341 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:02:52.990263    4720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cfz55\" (UniqueName: \"kubernetes.io/projected/4e9f63cd-0961-4be1-a649-a11a4ed84278-kube-api-access-cfz55\") pod \"scheduler-v06-threshold-migration-test-kserve-bc5ffbf5b-cjc72\" (UID: \"4e9f63cd-0961-4be1-a649-a11a4ed84278\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-bc5ffbf5b-cjc72"
Oct 31 08:02:53.054342 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:02:53.054306    4720 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-bc5ffbf5b-cjc72"
Oct 31 08:02:53.329359 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:02:53.329249    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 08:02:53.479061 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:02:53.479018    4720 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-bc5ffbf5b-cjc72"]
Oct 31 08:02:53.482182 ip-10-0-139-83 kubenswrapper[4720]: W1031 08:02:53.482150    4720 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e9f63cd_0961_4be1_a649_a11a4ed84278.slice/crio-086dafd3672cf6b4ceca95c04bf8ee22cdf3967f4e341ccab87367d02b14aea3 WatchSource:0}: Error finding container 086dafd3672cf6b4ceca95c04bf8ee22cdf3967f4e341ccab87367d02b14aea3: Status 404 returned error can't find the container with id 086dafd3672cf6b4ceca95c04bf8ee22cdf3967f4e341ccab87367d02b14aea3
Oct 31 08:02:54.045968 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:02:54.045925    4720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-bc5ffbf5b-cjc72" event={"ID":"4e9f63cd-0961-4be1-a649-a11a4ed84278","Type":"ContainerStarted","Data":"249bc467b0c6aac534df72678f30f1ad9453f1e74feac0c33493e1e3d6e8773e"}
Oct 31 08:02:54.046429 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:02:54.045988    4720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-bc5ffbf5b-cjc72" event={"ID":"4e9f63cd-0961-4be1-a649-a11a4ed84278","Type":"ContainerStarted","Data":"086dafd3672cf6b4ceca95c04bf8ee22cdf3967f4e341ccab87367d02b14aea3"}
Oct 31 08:02:54.046429 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:02:54.046088    4720 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-bc5ffbf5b-cjc72"
Oct 31 08:02:55.053283 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:02:55.053252    4720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-bc5ffbf5b-cjc72" event={"ID":"4e9f63cd-0961-4be1-a649-a11a4ed84278","Type":"ContainerStarted","Data":"f68d97bd19f830e401a67bab8a322bf7c5df7da90add32ada7b33b5bcf4e0064"}
Oct 31 08:02:55.054485 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:02:55.054466    4720 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-bc5ffbf5b-cjc72"
Oct 31 08:02:55.168910 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:02:55.168683    4720 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-bc5ffbf5b-cjc72" podStartSLOduration=3.168665887 podStartE2EDuration="3.168665887s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 08:02:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 08:02:55.166580012 +0000 UTC m=+2029.297648311" watchObservedRunningTime="2026-10-31 08:02:55.168665887 +0000 UTC m=+2029.299733958"
Oct 31 08:02:56.376575 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:02:56.376535    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 08:02:56.528015 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:02:56.527974    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 08:03:01.884386 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:03:01.884314    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 08:03:03.055195 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:03:03.055157    4720 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-bc5ffbf5b-cjc72"
Oct 31 08:03:03.056426 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:03:03.056394    4720 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-bc5ffbf5b-cjc72" podUID="4e9f63cd-0961-4be1-a649-a11a4ed84278" containerName="llm-d-routing-sidecar" probeResult="failure" output="Get \"https://10.132.0.50:8000/health\": dial tcp 10.132.0.50:8000: connect: connection refused"
Oct 31 08:03:03.056533 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:03:03.056423    4720 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-bc5ffbf5b-cjc72" podUID="4e9f63cd-0961-4be1-a649-a11a4ed84278" containerName="llm-d-routing-sidecar" probeResult="failure" output="Get \"https://10.132.0.50:8000/health\": dial tcp 10.132.0.50:8000: connect: connection refused"
Oct 31 08:03:03.067343 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:03:03.067304    4720 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-bc5ffbf5b-cjc72"
Oct 31 08:03:03.129149 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:03:03.129110    4720 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-bc5ffbf5b-cjc72" podUID="4e9f63cd-0961-4be1-a649-a11a4ed84278" containerName="llm-d-routing-sidecar" probeResult="failure" output="Get \"https://10.132.0.50:8000/health\": dial tcp 10.132.0.50:8000: connect: connection refused"
Oct 31 08:03:03.139920 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:03:03.139900    4720 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-bc5ffbf5b-cjc72"
Oct 31 08:03:03.244898 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:03:03.244871    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 08:03:04.133495 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:03:04.133457    4720 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-bc5ffbf5b-cjc72" podUID="4e9f63cd-0961-4be1-a649-a11a4ed84278" containerName="llm-d-routing-sidecar" probeResult="failure" output="Get \"https://10.132.0.50:8000/health\": dial tcp 10.132.0.50:8000: connect: connection refused"
Oct 31 08:03:13.461901 ip-10-0-139-83 kubenswrapper[4720]: E1031 08:03:13.461861    4720 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8m768" containerName="extractor"
Oct 31 08:03:14.148522 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:03:14.148494    4720 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-bc5ffbf5b-cjc72"
Oct 31 08:03:14.358374 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:03:14.358341    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 08:03:14.505722 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:03:14.505634    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 08:03:31.822933 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:03:31.822890    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 08:03:32.829197 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:03:32.829159    4720 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-bc5ffbf5b-cjc72"]
Oct 31 08:03:32.839230 ip-10-0-139-83 kubenswrapper[4720]: E1031 08:03:32.839111    4720 log.go:32] "ExecSync cmd from runtime service failed" err="rpc error: code = Unknown desc = command error: , stdout: , stderr: , exit code -1" containerID="f68d97bd19f830e401a67bab8a322bf7c5df7da90add32ada7b33b5bcf4e0064" cmd=["/bin/sleep","15"]
Oct 31 08:03:32.839507 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:03:32.839473    4720 handlers.go:85] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-bc5ffbf5b-cjc72" message="" err="rpc error: code = Unknown desc = command error: , stdout: , stderr: , exit code -1"
Oct 31 08:03:32.839629 ip-10-0-139-83 kubenswrapper[4720]: E1031 08:03:32.839508    4720 kuberuntime_container.go:780] "PreStop hook failed" err="rpc error: code = Unknown desc = command error: , stdout: , stderr: , exit code -1" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-bc5ffbf5b-cjc72" podUID="4e9f63cd-0961-4be1-a649-a11a4ed84278" containerName="main" containerID="cri-o://f68d97bd19f830e401a67bab8a322bf7c5df7da90add32ada7b33b5bcf4e0064"
Oct 31 08:03:32.839629 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:03:32.839564    4720 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-bc5ffbf5b-cjc72" podUID="4e9f63cd-0961-4be1-a649-a11a4ed84278" containerName="main" containerID="cri-o://f68d97bd19f830e401a67bab8a322bf7c5df7da90add32ada7b33b5bcf4e0064" gracePeriod=60
Oct 31 08:03:32.861186 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:03:32.861158    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 08:03:33.019437 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:03:33.019395    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 08:03:33.130105 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:03:33.130069    4720 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-bc5ffbf5b-cjc72" podUID="4e9f63cd-0961-4be1-a649-a11a4ed84278" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8001/health\": dial tcp 10.132.0.50:8001: connect: connection refused"
Oct 31 08:03:33.262644 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:03:33.262613    4720 generic.go:356] "Generic (PLEG): container finished" podID="4e9f63cd-0961-4be1-a649-a11a4ed84278" containerID="f68d97bd19f830e401a67bab8a322bf7c5df7da90add32ada7b33b5bcf4e0064" exitCode=0
Oct 31 08:03:33.262808 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:03:33.262690    4720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-bc5ffbf5b-cjc72" event={"ID":"4e9f63cd-0961-4be1-a649-a11a4ed84278","Type":"ContainerDied","Data":"f68d97bd19f830e401a67bab8a322bf7c5df7da90add32ada7b33b5bcf4e0064"}
Oct 31 08:03:33.453600 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:03:33.453569    4720 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-bc5ffbf5b-cjc72" podUID="4e9f63cd-0961-4be1-a649-a11a4ed84278" containerName="llm-d-routing-sidecar" containerID="cri-o://249bc467b0c6aac534df72678f30f1ad9453f1e74feac0c33493e1e3d6e8773e" gracePeriod=60
Oct 31 08:03:33.710689 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:03:33.710629    4720 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-bc5ffbf5b-cjc72"
Oct 31 08:03:33.731856 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:03:33.731831    4720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4e9f63cd-0961-4be1-a649-a11a4ed84278-tls-certs\" (UniqueName: \"kubernetes.io/secret/4e9f63cd-0961-4be1-a649-a11a4ed84278-tls-certs\") pod \"4e9f63cd-0961-4be1-a649-a11a4ed84278\" (UID: \"4e9f63cd-0961-4be1-a649-a11a4ed84278\") "
Oct 31 08:03:33.731987 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:03:33.731868    4720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4e9f63cd-0961-4be1-a649-a11a4ed84278-home\" (UniqueName: \"kubernetes.io/empty-dir/4e9f63cd-0961-4be1-a649-a11a4ed84278-home\") pod \"4e9f63cd-0961-4be1-a649-a11a4ed84278\" (UID: \"4e9f63cd-0961-4be1-a649-a11a4ed84278\") "
Oct 31 08:03:33.731987 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:03:33.731893    4720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4e9f63cd-0961-4be1-a649-a11a4ed84278-dshm\" (UniqueName: \"kubernetes.io/empty-dir/4e9f63cd-0961-4be1-a649-a11a4ed84278-dshm\") pod \"4e9f63cd-0961-4be1-a649-a11a4ed84278\" (UID: \"4e9f63cd-0961-4be1-a649-a11a4ed84278\") "
Oct 31 08:03:33.731987 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:03:33.731942    4720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4e9f63cd-0961-4be1-a649-a11a4ed84278-kube-api-access-cfz55\" (UniqueName: \"kubernetes.io/projected/4e9f63cd-0961-4be1-a649-a11a4ed84278-kube-api-access-cfz55\") pod \"4e9f63cd-0961-4be1-a649-a11a4ed84278\" (UID: \"4e9f63cd-0961-4be1-a649-a11a4ed84278\") "
Oct 31 08:03:33.731987 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:03:33.731966    4720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4e9f63cd-0961-4be1-a649-a11a4ed84278-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4e9f63cd-0961-4be1-a649-a11a4ed84278-model-cache\") pod \"4e9f63cd-0961-4be1-a649-a11a4ed84278\" (UID: \"4e9f63cd-0961-4be1-a649-a11a4ed84278\") "
Oct 31 08:03:33.732209 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:03:33.732011    4720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4e9f63cd-0961-4be1-a649-a11a4ed84278-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4e9f63cd-0961-4be1-a649-a11a4ed84278-tmp-dir\") pod \"4e9f63cd-0961-4be1-a649-a11a4ed84278\" (UID: \"4e9f63cd-0961-4be1-a649-a11a4ed84278\") "
Oct 31 08:03:33.732209 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:03:33.732169    4720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e9f63cd-0961-4be1-a649-a11a4ed84278-home" pod "4e9f63cd-0961-4be1-a649-a11a4ed84278" (UID: "4e9f63cd-0961-4be1-a649-a11a4ed84278"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 08:03:33.732312 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:03:33.732281    4720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e9f63cd-0961-4be1-a649-a11a4ed84278-model-cache" pod "4e9f63cd-0961-4be1-a649-a11a4ed84278" (UID: "4e9f63cd-0961-4be1-a649-a11a4ed84278"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 08:03:33.732312 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:03:33.732290    4720 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4e9f63cd-0961-4be1-a649-a11a4ed84278-home\") on node \"ip-10-0-139-83.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 08:03:33.732425 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:03:33.732377    4720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e9f63cd-0961-4be1-a649-a11a4ed84278-tmp-dir" pod "4e9f63cd-0961-4be1-a649-a11a4ed84278" (UID: "4e9f63cd-0961-4be1-a649-a11a4ed84278"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 08:03:33.734296 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:03:33.734269    4720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e9f63cd-0961-4be1-a649-a11a4ed84278-dshm" pod "4e9f63cd-0961-4be1-a649-a11a4ed84278" (UID: "4e9f63cd-0961-4be1-a649-a11a4ed84278"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 08:03:33.734454 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:03:33.734314    4720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e9f63cd-0961-4be1-a649-a11a4ed84278-kube-api-access-cfz55" pod "4e9f63cd-0961-4be1-a649-a11a4ed84278" (UID: "4e9f63cd-0961-4be1-a649-a11a4ed84278"). InnerVolumeSpecName "kube-api-access-cfz55". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 08:03:33.734454 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:03:33.734363    4720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4e9f63cd-0961-4be1-a649-a11a4ed84278-tls-certs" pod "4e9f63cd-0961-4be1-a649-a11a4ed84278" (UID: "4e9f63cd-0961-4be1-a649-a11a4ed84278"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 08:03:33.832740 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:03:33.832716    4720 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4e9f63cd-0961-4be1-a649-a11a4ed84278-tls-certs\") on node \"ip-10-0-139-83.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 08:03:33.832740 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:03:33.832739    4720 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4e9f63cd-0961-4be1-a649-a11a4ed84278-dshm\") on node \"ip-10-0-139-83.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 08:03:33.833089 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:03:33.832749    4720 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cfz55\" (UniqueName: \"kubernetes.io/projected/4e9f63cd-0961-4be1-a649-a11a4ed84278-kube-api-access-cfz55\") on node \"ip-10-0-139-83.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 08:03:33.833089 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:03:33.832759    4720 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4e9f63cd-0961-4be1-a649-a11a4ed84278-model-cache\") on node \"ip-10-0-139-83.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 08:03:33.833089 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:03:33.832787    4720 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4e9f63cd-0961-4be1-a649-a11a4ed84278-tmp-dir\") on node \"ip-10-0-139-83.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 08:03:34.269425 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:03:34.269391    4720 generic.go:356] "Generic (PLEG): container finished" podID="4e9f63cd-0961-4be1-a649-a11a4ed84278" containerID="249bc467b0c6aac534df72678f30f1ad9453f1e74feac0c33493e1e3d6e8773e" exitCode=0
Oct 31 08:03:34.269584 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:03:34.269443    4720 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-bc5ffbf5b-cjc72"
Oct 31 08:03:34.269584 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:03:34.269476    4720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-bc5ffbf5b-cjc72" event={"ID":"4e9f63cd-0961-4be1-a649-a11a4ed84278","Type":"ContainerDied","Data":"249bc467b0c6aac534df72678f30f1ad9453f1e74feac0c33493e1e3d6e8773e"}
Oct 31 08:03:34.269584 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:03:34.269534    4720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-bc5ffbf5b-cjc72" event={"ID":"4e9f63cd-0961-4be1-a649-a11a4ed84278","Type":"ContainerDied","Data":"086dafd3672cf6b4ceca95c04bf8ee22cdf3967f4e341ccab87367d02b14aea3"}
Oct 31 08:03:34.269584 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:03:34.269561    4720 scope.go:122] "RemoveContainer" containerID="f68d97bd19f830e401a67bab8a322bf7c5df7da90add32ada7b33b5bcf4e0064"
Oct 31 08:03:34.278460 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:03:34.278435    4720 scope.go:122] "RemoveContainer" containerID="249bc467b0c6aac534df72678f30f1ad9453f1e74feac0c33493e1e3d6e8773e"
Oct 31 08:03:34.286478 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:03:34.286463    4720 scope.go:122] "RemoveContainer" containerID="f68d97bd19f830e401a67bab8a322bf7c5df7da90add32ada7b33b5bcf4e0064"
Oct 31 08:03:34.286735 ip-10-0-139-83 kubenswrapper[4720]: E1031 08:03:34.286717    4720 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f68d97bd19f830e401a67bab8a322bf7c5df7da90add32ada7b33b5bcf4e0064\": container with ID starting with f68d97bd19f830e401a67bab8a322bf7c5df7da90add32ada7b33b5bcf4e0064 not found: ID does not exist" containerID="f68d97bd19f830e401a67bab8a322bf7c5df7da90add32ada7b33b5bcf4e0064"
Oct 31 08:03:34.286804 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:03:34.286750    4720 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f68d97bd19f830e401a67bab8a322bf7c5df7da90add32ada7b33b5bcf4e0064"} err="failed to get container status \"f68d97bd19f830e401a67bab8a322bf7c5df7da90add32ada7b33b5bcf4e0064\": rpc error: code = NotFound desc = could not find container \"f68d97bd19f830e401a67bab8a322bf7c5df7da90add32ada7b33b5bcf4e0064\": container with ID starting with f68d97bd19f830e401a67bab8a322bf7c5df7da90add32ada7b33b5bcf4e0064 not found: ID does not exist"
Oct 31 08:03:34.286804 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:03:34.286778    4720 scope.go:122] "RemoveContainer" containerID="249bc467b0c6aac534df72678f30f1ad9453f1e74feac0c33493e1e3d6e8773e"
Oct 31 08:03:34.287024 ip-10-0-139-83 kubenswrapper[4720]: E1031 08:03:34.287005    4720 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"249bc467b0c6aac534df72678f30f1ad9453f1e74feac0c33493e1e3d6e8773e\": container with ID starting with 249bc467b0c6aac534df72678f30f1ad9453f1e74feac0c33493e1e3d6e8773e not found: ID does not exist" containerID="249bc467b0c6aac534df72678f30f1ad9453f1e74feac0c33493e1e3d6e8773e"
Oct 31 08:03:34.287066 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:03:34.287031    4720 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"249bc467b0c6aac534df72678f30f1ad9453f1e74feac0c33493e1e3d6e8773e"} err="failed to get container status \"249bc467b0c6aac534df72678f30f1ad9453f1e74feac0c33493e1e3d6e8773e\": rpc error: code = NotFound desc = could not find container \"249bc467b0c6aac534df72678f30f1ad9453f1e74feac0c33493e1e3d6e8773e\": container with ID starting with 249bc467b0c6aac534df72678f30f1ad9453f1e74feac0c33493e1e3d6e8773e not found: ID does not exist"
Oct 31 08:03:34.430593 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:03:34.430554    4720 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-bc5ffbf5b-cjc72"]
Oct 31 08:03:34.479942 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:03:34.479910    4720 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-bc5ffbf5b-cjc72"]
Oct 31 08:03:36.025545 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:03:36.025511    4720 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4e9f63cd-0961-4be1-a649-a11a4ed84278" path="/var/lib/kubelet/pods/4e9f63cd-0961-4be1-a649-a11a4ed84278/volumes"
Oct 31 08:03:43.461269 ip-10-0-139-83 kubenswrapper[4720]: E1031 08:03:43.461226    4720 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8m768" containerName="extractor"
Oct 31 08:04:06.205977 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:04:06.205870    4720 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-4jszp_b1db0e3b-e85d-4769-8012-cf4b6f740d9d/console-operator/2.log"
Oct 31 08:04:06.213864 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:04:06.213841    4720 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-4jszp_b1db0e3b-e85d-4769-8012-cf4b6f740d9d/console-operator/2.log"
Oct 31 08:04:12.233195 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:04:12.233161    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 08:04:13.462344 ip-10-0-139-83 kubenswrapper[4720]: E1031 08:04:13.462287    4720 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8m768" containerName="extractor"
Oct 31 08:04:16.775373 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:04:16.775264    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 08:04:17.372590 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:04:17.372557    4720 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-679758fddf-hh2nc"]
Oct 31 08:04:17.372935 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:04:17.372923    4720 cpu_manager.go:414] "containerMap: removing container" podUID="4e9f63cd-0961-4be1-a649-a11a4ed84278" containerName="llm-d-routing-sidecar"
Oct 31 08:04:17.372980 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:04:17.372937    4720 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4e9f63cd-0961-4be1-a649-a11a4ed84278" containerName="llm-d-routing-sidecar"
Oct 31 08:04:17.372980 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:04:17.372963    4720 cpu_manager.go:414] "containerMap: removing container" podUID="4e9f63cd-0961-4be1-a649-a11a4ed84278" containerName="main"
Oct 31 08:04:17.372980 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:04:17.372968    4720 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4e9f63cd-0961-4be1-a649-a11a4ed84278" containerName="main"
Oct 31 08:04:17.373076 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:04:17.373020    4720 memory_manager.go:365] "RemoveStaleState removing state" podUID="4e9f63cd-0961-4be1-a649-a11a4ed84278" containerName="llm-d-routing-sidecar"
Oct 31 08:04:17.373076 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:04:17.373032    4720 memory_manager.go:365] "RemoveStaleState removing state" podUID="4e9f63cd-0961-4be1-a649-a11a4ed84278" containerName="main"
Oct 31 08:04:17.377322 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:04:17.377305    4720 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-679758fddf-hh2nc"
Oct 31 08:04:17.403709 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:04:17.403672    4720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/94456f54-4f33-4f9c-9e5a-1125d3955ef7-tls-certs\") pod \"auth-invalid-token-test-kserve-679758fddf-hh2nc\" (UID: \"94456f54-4f33-4f9c-9e5a-1125d3955ef7\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-679758fddf-hh2nc"
Oct 31 08:04:17.403851 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:04:17.403710    4720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2cwgs\" (UniqueName: \"kubernetes.io/projected/94456f54-4f33-4f9c-9e5a-1125d3955ef7-kube-api-access-2cwgs\") pod \"auth-invalid-token-test-kserve-679758fddf-hh2nc\" (UID: \"94456f54-4f33-4f9c-9e5a-1125d3955ef7\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-679758fddf-hh2nc"
Oct 31 08:04:17.403851 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:04:17.403819    4720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/94456f54-4f33-4f9c-9e5a-1125d3955ef7-model-cache\") pod \"auth-invalid-token-test-kserve-679758fddf-hh2nc\" (UID: \"94456f54-4f33-4f9c-9e5a-1125d3955ef7\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-679758fddf-hh2nc"
Oct 31 08:04:17.403973 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:04:17.403886    4720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/94456f54-4f33-4f9c-9e5a-1125d3955ef7-kserve-provision-location\") pod \"auth-invalid-token-test-kserve-679758fddf-hh2nc\" (UID: \"94456f54-4f33-4f9c-9e5a-1125d3955ef7\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-679758fddf-hh2nc"
Oct 31 08:04:17.403973 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:04:17.403920    4720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/94456f54-4f33-4f9c-9e5a-1125d3955ef7-home\") pod \"auth-invalid-token-test-kserve-679758fddf-hh2nc\" (UID: \"94456f54-4f33-4f9c-9e5a-1125d3955ef7\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-679758fddf-hh2nc"
Oct 31 08:04:17.404087 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:04:17.403988    4720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/94456f54-4f33-4f9c-9e5a-1125d3955ef7-cabundle-cert\") pod \"auth-invalid-token-test-kserve-679758fddf-hh2nc\" (UID: \"94456f54-4f33-4f9c-9e5a-1125d3955ef7\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-679758fddf-hh2nc"
Oct 31 08:04:17.404087 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:04:17.404058    4720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/94456f54-4f33-4f9c-9e5a-1125d3955ef7-dshm\") pod \"auth-invalid-token-test-kserve-679758fddf-hh2nc\" (UID: \"94456f54-4f33-4f9c-9e5a-1125d3955ef7\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-679758fddf-hh2nc"
Oct 31 08:04:17.404178 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:04:17.404092    4720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/94456f54-4f33-4f9c-9e5a-1125d3955ef7-tmp-dir\") pod \"auth-invalid-token-test-kserve-679758fddf-hh2nc\" (UID: \"94456f54-4f33-4f9c-9e5a-1125d3955ef7\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-679758fddf-hh2nc"
Oct 31 08:04:17.427952 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:04:17.427928    4720 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-679758fddf-hh2nc"]
Oct 31 08:04:17.475555 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:04:17.475524    4720 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6\"/\"seaweedfs-s3-creds\""
Oct 31 08:04:17.475757 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:04:17.475575    4720 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6\"/\"openshift-service-ca.crt\""
Oct 31 08:04:17.475757 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:04:17.475572    4720 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6\"/\"odh-kserve-custom-ca-bundle\""
Oct 31 08:04:17.475757 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:04:17.475594    4720 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6\"/\"kube-root-ca.crt\""
Oct 31 08:04:17.475757 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:04:17.475667    4720 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6\"/\"default-dockercfg-tw6x8\""
Oct 31 08:04:17.475943 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:04:17.475851    4720 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6\"/\"auth-invalid-token-test-kserve-self-signed-certs\""
Oct 31 08:04:17.504988 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:04:17.504964    4720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/94456f54-4f33-4f9c-9e5a-1125d3955ef7-dshm\") pod \"auth-invalid-token-test-kserve-679758fddf-hh2nc\" (UID: \"94456f54-4f33-4f9c-9e5a-1125d3955ef7\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-679758fddf-hh2nc"
Oct 31 08:04:17.505107 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:04:17.504991    4720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/94456f54-4f33-4f9c-9e5a-1125d3955ef7-tmp-dir\") pod \"auth-invalid-token-test-kserve-679758fddf-hh2nc\" (UID: \"94456f54-4f33-4f9c-9e5a-1125d3955ef7\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-679758fddf-hh2nc"
Oct 31 08:04:17.505107 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:04:17.505025    4720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/94456f54-4f33-4f9c-9e5a-1125d3955ef7-tls-certs\") pod \"auth-invalid-token-test-kserve-679758fddf-hh2nc\" (UID: \"94456f54-4f33-4f9c-9e5a-1125d3955ef7\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-679758fddf-hh2nc"
Oct 31 08:04:17.505107 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:04:17.505049    4720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2cwgs\" (UniqueName: \"kubernetes.io/projected/94456f54-4f33-4f9c-9e5a-1125d3955ef7-kube-api-access-2cwgs\") pod \"auth-invalid-token-test-kserve-679758fddf-hh2nc\" (UID: \"94456f54-4f33-4f9c-9e5a-1125d3955ef7\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-679758fddf-hh2nc"
Oct 31 08:04:17.505274 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:04:17.505108    4720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/94456f54-4f33-4f9c-9e5a-1125d3955ef7-model-cache\") pod \"auth-invalid-token-test-kserve-679758fddf-hh2nc\" (UID: \"94456f54-4f33-4f9c-9e5a-1125d3955ef7\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-679758fddf-hh2nc"
Oct 31 08:04:17.505274 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:04:17.505134    4720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/94456f54-4f33-4f9c-9e5a-1125d3955ef7-kserve-provision-location\") pod \"auth-invalid-token-test-kserve-679758fddf-hh2nc\" (UID: \"94456f54-4f33-4f9c-9e5a-1125d3955ef7\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-679758fddf-hh2nc"
Oct 31 08:04:17.505274 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:04:17.505162    4720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/94456f54-4f33-4f9c-9e5a-1125d3955ef7-home\") pod \"auth-invalid-token-test-kserve-679758fddf-hh2nc\" (UID: \"94456f54-4f33-4f9c-9e5a-1125d3955ef7\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-679758fddf-hh2nc"
Oct 31 08:04:17.505274 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:04:17.505205    4720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/94456f54-4f33-4f9c-9e5a-1125d3955ef7-cabundle-cert\") pod \"auth-invalid-token-test-kserve-679758fddf-hh2nc\" (UID: \"94456f54-4f33-4f9c-9e5a-1125d3955ef7\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-679758fddf-hh2nc"
Oct 31 08:04:17.505593 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:04:17.505563    4720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/94456f54-4f33-4f9c-9e5a-1125d3955ef7-model-cache\") pod \"auth-invalid-token-test-kserve-679758fddf-hh2nc\" (UID: \"94456f54-4f33-4f9c-9e5a-1125d3955ef7\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-679758fddf-hh2nc"
Oct 31 08:04:17.505710 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:04:17.505601    4720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/94456f54-4f33-4f9c-9e5a-1125d3955ef7-tmp-dir\") pod \"auth-invalid-token-test-kserve-679758fddf-hh2nc\" (UID: \"94456f54-4f33-4f9c-9e5a-1125d3955ef7\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-679758fddf-hh2nc"
Oct 31 08:04:17.505710 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:04:17.505686    4720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/94456f54-4f33-4f9c-9e5a-1125d3955ef7-home\") pod \"auth-invalid-token-test-kserve-679758fddf-hh2nc\" (UID: \"94456f54-4f33-4f9c-9e5a-1125d3955ef7\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-679758fddf-hh2nc"
Oct 31 08:04:17.505809 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:04:17.505716    4720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/94456f54-4f33-4f9c-9e5a-1125d3955ef7-kserve-provision-location\") pod \"auth-invalid-token-test-kserve-679758fddf-hh2nc\" (UID: \"94456f54-4f33-4f9c-9e5a-1125d3955ef7\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-679758fddf-hh2nc"
Oct 31 08:04:17.505933 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:04:17.505917    4720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/94456f54-4f33-4f9c-9e5a-1125d3955ef7-cabundle-cert\") pod \"auth-invalid-token-test-kserve-679758fddf-hh2nc\" (UID: \"94456f54-4f33-4f9c-9e5a-1125d3955ef7\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-679758fddf-hh2nc"
Oct 31 08:04:17.508012 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:04:17.507987    4720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/94456f54-4f33-4f9c-9e5a-1125d3955ef7-dshm\") pod \"auth-invalid-token-test-kserve-679758fddf-hh2nc\" (UID: \"94456f54-4f33-4f9c-9e5a-1125d3955ef7\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-679758fddf-hh2nc"
Oct 31 08:04:17.508238 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:04:17.508222    4720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/94456f54-4f33-4f9c-9e5a-1125d3955ef7-tls-certs\") pod \"auth-invalid-token-test-kserve-679758fddf-hh2nc\" (UID: \"94456f54-4f33-4f9c-9e5a-1125d3955ef7\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-679758fddf-hh2nc"
Oct 31 08:04:17.561277 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:04:17.561249    4720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2cwgs\" (UniqueName: \"kubernetes.io/projected/94456f54-4f33-4f9c-9e5a-1125d3955ef7-kube-api-access-2cwgs\") pod \"auth-invalid-token-test-kserve-679758fddf-hh2nc\" (UID: \"94456f54-4f33-4f9c-9e5a-1125d3955ef7\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-679758fddf-hh2nc"
Oct 31 08:04:17.692068 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:04:17.691977    4720 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-679758fddf-hh2nc"
Oct 31 08:04:18.120786 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:04:18.120746    4720 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-679758fddf-hh2nc"]
Oct 31 08:04:18.123599 ip-10-0-139-83 kubenswrapper[4720]: W1031 08:04:18.123571    4720 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod94456f54_4f33_4f9c_9e5a_1125d3955ef7.slice/crio-039efbfe82d52682deb101be2be3548c884148fe2dbd529aa47cef80146c137e WatchSource:0}: Error finding container 039efbfe82d52682deb101be2be3548c884148fe2dbd529aa47cef80146c137e: Status 404 returned error can't find the container with id 039efbfe82d52682deb101be2be3548c884148fe2dbd529aa47cef80146c137e
Oct 31 08:04:18.483238 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:04:18.483148    4720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-679758fddf-hh2nc" event={"ID":"94456f54-4f33-4f9c-9e5a-1125d3955ef7","Type":"ContainerStarted","Data":"766fd28b174a193dfc46e254f315f56339365167195e90da7fe24726b6c85c8e"}
Oct 31 08:04:18.483406 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:04:18.483254    4720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-679758fddf-hh2nc" event={"ID":"94456f54-4f33-4f9c-9e5a-1125d3955ef7","Type":"ContainerStarted","Data":"039efbfe82d52682deb101be2be3548c884148fe2dbd529aa47cef80146c137e"}
Oct 31 08:04:20.090901 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:04:20.090816    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 08:04:20.293677 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:04:20.293637    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 08:04:20.496232 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:04:20.496197    4720 generic.go:356] "Generic (PLEG): container finished" podID="94456f54-4f33-4f9c-9e5a-1125d3955ef7" containerID="766fd28b174a193dfc46e254f315f56339365167195e90da7fe24726b6c85c8e" exitCode=0
Oct 31 08:04:20.496399 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:04:20.496271    4720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-679758fddf-hh2nc" event={"ID":"94456f54-4f33-4f9c-9e5a-1125d3955ef7","Type":"ContainerDied","Data":"766fd28b174a193dfc46e254f315f56339365167195e90da7fe24726b6c85c8e"}
Oct 31 08:04:21.503105 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:04:21.503067    4720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-679758fddf-hh2nc" event={"ID":"94456f54-4f33-4f9c-9e5a-1125d3955ef7","Type":"ContainerStarted","Data":"b1955df30e6d77c634fa4bb113921a52d10d3464a89d19c9db3ac0a1af0b9b5e"}
Oct 31 08:04:21.692734 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:04:21.692701    4720 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-679758fddf-hh2nc"
Oct 31 08:04:21.721421 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:04:21.721361    4720 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-679758fddf-hh2nc" podStartSLOduration=2.482038637 podStartE2EDuration="4.721340756s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.239302119s" isStatefulPod=false podCreationTimestamp="2026-10-31 08:04:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 08:04:21.67119495 +0000 UTC m=+2115.802263043" watchObservedRunningTime="2026-10-31 08:04:21.721340756 +0000 UTC m=+2115.852408822"
Oct 31 08:04:24.100618 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:04:24.100576    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 08:04:24.505113 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:04:24.505032    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 08:04:26.566860 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:04:26.566822    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 08:04:27.692936 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:04:27.692899    4720 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-679758fddf-hh2nc"
Oct 31 08:04:27.694846 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:04:27.694818    4720 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-679758fddf-hh2nc" podUID="94456f54-4f33-4f9c-9e5a-1125d3955ef7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/health\": dial tcp 10.132.0.51:8000: connect: connection refused"
Oct 31 08:04:28.605353 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:04:28.602597    4720 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-79d7476f4b-9dq2q"]
Oct 31 08:04:28.609371 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:04:28.608310    4720 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-79d7476f4b-9dq2q"
Oct 31 08:04:28.630732 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:04:28.627763    4720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/26959502-cb50-4bc4-9bb0-56966ad5a7f6-kserve-provision-location\") pod \"tokenizer-clean-path-test-kserve-79d7476f4b-9dq2q\" (UID: \"26959502-cb50-4bc4-9bb0-56966ad5a7f6\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-79d7476f4b-9dq2q"
Oct 31 08:04:28.630732 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:04:28.627818    4720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/26959502-cb50-4bc4-9bb0-56966ad5a7f6-dshm\") pod \"tokenizer-clean-path-test-kserve-79d7476f4b-9dq2q\" (UID: \"26959502-cb50-4bc4-9bb0-56966ad5a7f6\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-79d7476f4b-9dq2q"
Oct 31 08:04:28.630732 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:04:28.627870    4720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/26959502-cb50-4bc4-9bb0-56966ad5a7f6-home\") pod \"tokenizer-clean-path-test-kserve-79d7476f4b-9dq2q\" (UID: \"26959502-cb50-4bc4-9bb0-56966ad5a7f6\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-79d7476f4b-9dq2q"
Oct 31 08:04:28.630732 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:04:28.627901    4720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/26959502-cb50-4bc4-9bb0-56966ad5a7f6-tls-certs\") pod \"tokenizer-clean-path-test-kserve-79d7476f4b-9dq2q\" (UID: \"26959502-cb50-4bc4-9bb0-56966ad5a7f6\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-79d7476f4b-9dq2q"
Oct 31 08:04:28.630732 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:04:28.627956    4720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/26959502-cb50-4bc4-9bb0-56966ad5a7f6-cabundle-cert\") pod \"tokenizer-clean-path-test-kserve-79d7476f4b-9dq2q\" (UID: \"26959502-cb50-4bc4-9bb0-56966ad5a7f6\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-79d7476f4b-9dq2q"
Oct 31 08:04:28.630732 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:04:28.627981    4720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-df6mm\" (UniqueName: \"kubernetes.io/projected/26959502-cb50-4bc4-9bb0-56966ad5a7f6-kube-api-access-df6mm\") pod \"tokenizer-clean-path-test-kserve-79d7476f4b-9dq2q\" (UID: \"26959502-cb50-4bc4-9bb0-56966ad5a7f6\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-79d7476f4b-9dq2q"
Oct 31 08:04:28.630732 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:04:28.628033    4720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/26959502-cb50-4bc4-9bb0-56966ad5a7f6-model-cache\") pod \"tokenizer-clean-path-test-kserve-79d7476f4b-9dq2q\" (UID: \"26959502-cb50-4bc4-9bb0-56966ad5a7f6\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-79d7476f4b-9dq2q"
Oct 31 08:04:28.630732 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:04:28.628069    4720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/26959502-cb50-4bc4-9bb0-56966ad5a7f6-tmp-dir\") pod \"tokenizer-clean-path-test-kserve-79d7476f4b-9dq2q\" (UID: \"26959502-cb50-4bc4-9bb0-56966ad5a7f6\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-79d7476f4b-9dq2q"
Oct 31 08:04:28.662351 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:04:28.661727    4720 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"default-dockercfg-6t9jp\""
Oct 31 08:04:28.664435 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:04:28.662843    4720 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"seaweedfs-s3-creds\""
Oct 31 08:04:28.741389 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:04:28.714399    4720 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-79d7476f4b-9dq2q"]
Oct 31 08:04:28.741389 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:04:28.714606    4720 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"tokenizer-clean-path-test-kserve-self-signed-certs\""
Oct 31 08:04:28.741389 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:04:28.714739    4720 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"kube-root-ca.crt\""
Oct 31 08:04:28.741389 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:04:28.714827    4720 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"openshift-service-ca.crt\""
Oct 31 08:04:28.741389 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:04:28.714887    4720 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"odh-kserve-custom-ca-bundle\""
Oct 31 08:04:28.749372 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:04:28.748974    4720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/26959502-cb50-4bc4-9bb0-56966ad5a7f6-kserve-provision-location\") pod \"tokenizer-clean-path-test-kserve-79d7476f4b-9dq2q\" (UID: \"26959502-cb50-4bc4-9bb0-56966ad5a7f6\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-79d7476f4b-9dq2q"
Oct 31 08:04:28.749372 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:04:28.749046    4720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/26959502-cb50-4bc4-9bb0-56966ad5a7f6-dshm\") pod \"tokenizer-clean-path-test-kserve-79d7476f4b-9dq2q\" (UID: \"26959502-cb50-4bc4-9bb0-56966ad5a7f6\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-79d7476f4b-9dq2q"
Oct 31 08:04:28.749372 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:04:28.749103    4720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/26959502-cb50-4bc4-9bb0-56966ad5a7f6-home\") pod \"tokenizer-clean-path-test-kserve-79d7476f4b-9dq2q\" (UID: \"26959502-cb50-4bc4-9bb0-56966ad5a7f6\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-79d7476f4b-9dq2q"
Oct 31 08:04:28.749372 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:04:28.749137    4720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/26959502-cb50-4bc4-9bb0-56966ad5a7f6-tls-certs\") pod \"tokenizer-clean-path-test-kserve-79d7476f4b-9dq2q\" (UID: \"26959502-cb50-4bc4-9bb0-56966ad5a7f6\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-79d7476f4b-9dq2q"
Oct 31 08:04:28.749372 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:04:28.749172    4720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/26959502-cb50-4bc4-9bb0-56966ad5a7f6-cabundle-cert\") pod \"tokenizer-clean-path-test-kserve-79d7476f4b-9dq2q\" (UID: \"26959502-cb50-4bc4-9bb0-56966ad5a7f6\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-79d7476f4b-9dq2q"
Oct 31 08:04:28.749372 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:04:28.749197    4720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-df6mm\" (UniqueName: \"kubernetes.io/projected/26959502-cb50-4bc4-9bb0-56966ad5a7f6-kube-api-access-df6mm\") pod \"tokenizer-clean-path-test-kserve-79d7476f4b-9dq2q\" (UID: \"26959502-cb50-4bc4-9bb0-56966ad5a7f6\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-79d7476f4b-9dq2q"
Oct 31 08:04:28.749372 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:04:28.749255    4720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/26959502-cb50-4bc4-9bb0-56966ad5a7f6-model-cache\") pod \"tokenizer-clean-path-test-kserve-79d7476f4b-9dq2q\" (UID: \"26959502-cb50-4bc4-9bb0-56966ad5a7f6\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-79d7476f4b-9dq2q"
Oct 31 08:04:28.749372 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:04:28.749291    4720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/26959502-cb50-4bc4-9bb0-56966ad5a7f6-tmp-dir\") pod \"tokenizer-clean-path-test-kserve-79d7476f4b-9dq2q\" (UID: \"26959502-cb50-4bc4-9bb0-56966ad5a7f6\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-79d7476f4b-9dq2q"
Oct 31 08:04:28.749921 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:04:28.749800    4720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/26959502-cb50-4bc4-9bb0-56966ad5a7f6-tmp-dir\") pod \"tokenizer-clean-path-test-kserve-79d7476f4b-9dq2q\" (UID: \"26959502-cb50-4bc4-9bb0-56966ad5a7f6\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-79d7476f4b-9dq2q"
Oct 31 08:04:28.750165 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:04:28.750067    4720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/26959502-cb50-4bc4-9bb0-56966ad5a7f6-kserve-provision-location\") pod \"tokenizer-clean-path-test-kserve-79d7476f4b-9dq2q\" (UID: \"26959502-cb50-4bc4-9bb0-56966ad5a7f6\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-79d7476f4b-9dq2q"
Oct 31 08:04:28.760714 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:04:28.760595    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 08:04:28.808354 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:04:28.781191    4720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/26959502-cb50-4bc4-9bb0-56966ad5a7f6-cabundle-cert\") pod \"tokenizer-clean-path-test-kserve-79d7476f4b-9dq2q\" (UID: \"26959502-cb50-4bc4-9bb0-56966ad5a7f6\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-79d7476f4b-9dq2q"
Oct 31 08:04:28.816445 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:04:28.781846    4720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/26959502-cb50-4bc4-9bb0-56966ad5a7f6-home\") pod \"tokenizer-clean-path-test-kserve-79d7476f4b-9dq2q\" (UID: \"26959502-cb50-4bc4-9bb0-56966ad5a7f6\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-79d7476f4b-9dq2q"
Oct 31 08:04:28.829358 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:04:28.782491    4720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/26959502-cb50-4bc4-9bb0-56966ad5a7f6-model-cache\") pod \"tokenizer-clean-path-test-kserve-79d7476f4b-9dq2q\" (UID: \"26959502-cb50-4bc4-9bb0-56966ad5a7f6\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-79d7476f4b-9dq2q"
Oct 31 08:04:28.833354 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:04:28.813411    4720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/26959502-cb50-4bc4-9bb0-56966ad5a7f6-dshm\") pod \"tokenizer-clean-path-test-kserve-79d7476f4b-9dq2q\" (UID: \"26959502-cb50-4bc4-9bb0-56966ad5a7f6\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-79d7476f4b-9dq2q"
Oct 31 08:04:28.863091 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:04:28.861027    4720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/26959502-cb50-4bc4-9bb0-56966ad5a7f6-tls-certs\") pod \"tokenizer-clean-path-test-kserve-79d7476f4b-9dq2q\" (UID: \"26959502-cb50-4bc4-9bb0-56966ad5a7f6\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-79d7476f4b-9dq2q"
Oct 31 08:04:28.955353 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:04:28.951940    4720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-df6mm\" (UniqueName: \"kubernetes.io/projected/26959502-cb50-4bc4-9bb0-56966ad5a7f6-kube-api-access-df6mm\") pod \"tokenizer-clean-path-test-kserve-79d7476f4b-9dq2q\" (UID: \"26959502-cb50-4bc4-9bb0-56966ad5a7f6\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-79d7476f4b-9dq2q"
Oct 31 08:04:28.976538 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:04:28.976502    4720 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-79d7476f4b-9dq2q"
Oct 31 08:04:29.261431 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:04:29.258374    4720 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-6bbcf98bf7-7sk52"]
Oct 31 08:04:29.264652 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:04:29.264623    4720 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-6bbcf98bf7-7sk52"
Oct 31 08:04:29.320355 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:04:29.316405    4720 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-6bbcf98bf7-7sk52"]
Oct 31 08:04:29.375197 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:04:29.375155    4720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a2d24fa7-975c-4976-8420-0e861eb88c30-cabundle-cert\") pod \"tokenizer-clean-path-test-tokenizer-6bbcf98bf7-7sk52\" (UID: \"a2d24fa7-975c-4976-8420-0e861eb88c30\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-6bbcf98bf7-7sk52"
Oct 31 08:04:29.375509 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:04:29.375486    4720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a2d24fa7-975c-4976-8420-0e861eb88c30-tls-certs\") pod \"tokenizer-clean-path-test-tokenizer-6bbcf98bf7-7sk52\" (UID: \"a2d24fa7-975c-4976-8420-0e861eb88c30\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-6bbcf98bf7-7sk52"
Oct 31 08:04:29.375630 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:04:29.375616    4720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a2d24fa7-975c-4976-8420-0e861eb88c30-tmp-dir\") pod \"tokenizer-clean-path-test-tokenizer-6bbcf98bf7-7sk52\" (UID: \"a2d24fa7-975c-4976-8420-0e861eb88c30\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-6bbcf98bf7-7sk52"
Oct 31 08:04:29.375762 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:04:29.375746    4720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a2d24fa7-975c-4976-8420-0e861eb88c30-kserve-provision-location\") pod \"tokenizer-clean-path-test-tokenizer-6bbcf98bf7-7sk52\" (UID: \"a2d24fa7-975c-4976-8420-0e861eb88c30\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-6bbcf98bf7-7sk52"
Oct 31 08:04:29.566959 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:04:29.566857    4720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a2d24fa7-975c-4976-8420-0e861eb88c30-cabundle-cert\") pod \"tokenizer-clean-path-test-tokenizer-6bbcf98bf7-7sk52\" (UID: \"a2d24fa7-975c-4976-8420-0e861eb88c30\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-6bbcf98bf7-7sk52"
Oct 31 08:04:29.567262 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:04:29.567239    4720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a2d24fa7-975c-4976-8420-0e861eb88c30-tls-certs\") pod \"tokenizer-clean-path-test-tokenizer-6bbcf98bf7-7sk52\" (UID: \"a2d24fa7-975c-4976-8420-0e861eb88c30\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-6bbcf98bf7-7sk52"
Oct 31 08:04:29.567437 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:04:29.567419    4720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a2d24fa7-975c-4976-8420-0e861eb88c30-tmp-dir\") pod \"tokenizer-clean-path-test-tokenizer-6bbcf98bf7-7sk52\" (UID: \"a2d24fa7-975c-4976-8420-0e861eb88c30\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-6bbcf98bf7-7sk52"
Oct 31 08:04:29.567595 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:04:29.567578    4720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a2d24fa7-975c-4976-8420-0e861eb88c30-kserve-provision-location\") pod \"tokenizer-clean-path-test-tokenizer-6bbcf98bf7-7sk52\" (UID: \"a2d24fa7-975c-4976-8420-0e861eb88c30\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-6bbcf98bf7-7sk52"
Oct 31 08:04:29.568162 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:04:29.568135    4720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a2d24fa7-975c-4976-8420-0e861eb88c30-kserve-provision-location\") pod \"tokenizer-clean-path-test-tokenizer-6bbcf98bf7-7sk52\" (UID: \"a2d24fa7-975c-4976-8420-0e861eb88c30\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-6bbcf98bf7-7sk52"
Oct 31 08:04:29.568385 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:04:29.568355    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 08:04:29.607493 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:04:29.568776    4720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a2d24fa7-975c-4976-8420-0e861eb88c30-tmp-dir\") pod \"tokenizer-clean-path-test-tokenizer-6bbcf98bf7-7sk52\" (UID: \"a2d24fa7-975c-4976-8420-0e861eb88c30\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-6bbcf98bf7-7sk52"
Oct 31 08:04:29.632355 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:04:29.608431    4720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a2d24fa7-975c-4976-8420-0e861eb88c30-cabundle-cert\") pod \"tokenizer-clean-path-test-tokenizer-6bbcf98bf7-7sk52\" (UID: \"a2d24fa7-975c-4976-8420-0e861eb88c30\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-6bbcf98bf7-7sk52"
Oct 31 08:04:29.701492 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:04:29.701436    4720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a2d24fa7-975c-4976-8420-0e861eb88c30-tls-certs\") pod \"tokenizer-clean-path-test-tokenizer-6bbcf98bf7-7sk52\" (UID: \"a2d24fa7-975c-4976-8420-0e861eb88c30\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-6bbcf98bf7-7sk52"
Oct 31 08:04:29.756364 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:04:29.753403    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 08:04:29.842440 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:04:29.841942    4720 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-79d7476f4b-9dq2q"]
Oct 31 08:04:29.968360 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:04:29.966608    4720 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-6bbcf98bf7-7sk52"
Oct 31 08:04:30.852359 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:04:30.849392    4720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-79d7476f4b-9dq2q" event={"ID":"26959502-cb50-4bc4-9bb0-56966ad5a7f6","Type":"ContainerStarted","Data":"1337e45cec97ff6b1876d05cbbc47f3575638457ebc0166320c0cbe3994bbf7f"}
Oct 31 08:04:30.852359 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:04:30.849564    4720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-79d7476f4b-9dq2q" event={"ID":"26959502-cb50-4bc4-9bb0-56966ad5a7f6","Type":"ContainerStarted","Data":"ed6dc702b5e6920ea3ca6e6be5c41976f9e0f08aa70055b21c88f17c478931c4"}
Oct 31 08:04:31.092354 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:04:31.092088    4720 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-6bbcf98bf7-7sk52"]
Oct 31 08:04:31.113086 ip-10-0-139-83 kubenswrapper[4720]: W1031 08:04:31.112986    4720 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda2d24fa7_975c_4976_8420_0e861eb88c30.slice/crio-1dd088ad151c0747e102fc99933f47d94a5734721557000ca0a61970805cfe2a WatchSource:0}: Error finding container 1dd088ad151c0747e102fc99933f47d94a5734721557000ca0a61970805cfe2a: Status 404 returned error can't find the container with id 1dd088ad151c0747e102fc99933f47d94a5734721557000ca0a61970805cfe2a
Oct 31 08:04:31.888353 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:04:31.888285    4720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-6bbcf98bf7-7sk52" event={"ID":"a2d24fa7-975c-4976-8420-0e861eb88c30","Type":"ContainerStarted","Data":"208848e7b8b84d3a1523d4587d16d480f6e0c2db6d4f44a28be6efc93d334f2f"}
Oct 31 08:04:31.888866 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:04:31.888389    4720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-6bbcf98bf7-7sk52" event={"ID":"a2d24fa7-975c-4976-8420-0e861eb88c30","Type":"ContainerStarted","Data":"1dd088ad151c0747e102fc99933f47d94a5734721557000ca0a61970805cfe2a"}
Oct 31 08:04:32.136957 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:04:32.136922    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 08:04:32.294411 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:04:32.293150    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 08:04:34.951768 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:04:34.951721    4720 generic.go:356] "Generic (PLEG): container finished" podID="26959502-cb50-4bc4-9bb0-56966ad5a7f6" containerID="1337e45cec97ff6b1876d05cbbc47f3575638457ebc0166320c0cbe3994bbf7f" exitCode=0
Oct 31 08:04:34.952344 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:04:34.951793    4720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-79d7476f4b-9dq2q" event={"ID":"26959502-cb50-4bc4-9bb0-56966ad5a7f6","Type":"ContainerDied","Data":"1337e45cec97ff6b1876d05cbbc47f3575638457ebc0166320c0cbe3994bbf7f"}
Oct 31 08:04:35.958886 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:04:35.958847    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 08:04:35.960487 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:04:35.960457    4720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-79d7476f4b-9dq2q" event={"ID":"26959502-cb50-4bc4-9bb0-56966ad5a7f6","Type":"ContainerStarted","Data":"0fa4febce910182c67a8a435c9ee7cf780562024e90c0ab984b0ffc2b8fc89f0"}
Oct 31 08:04:35.962274 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:04:35.962252    4720 generic.go:356] "Generic (PLEG): container finished" podID="a2d24fa7-975c-4976-8420-0e861eb88c30" containerID="208848e7b8b84d3a1523d4587d16d480f6e0c2db6d4f44a28be6efc93d334f2f" exitCode=0
Oct 31 08:04:35.962399 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:04:35.962306    4720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-6bbcf98bf7-7sk52" event={"ID":"a2d24fa7-975c-4976-8420-0e861eb88c30","Type":"ContainerDied","Data":"208848e7b8b84d3a1523d4587d16d480f6e0c2db6d4f44a28be6efc93d334f2f"}
Oct 31 08:04:35.981003 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:04:35.980977    4720 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-79d7476f4b-9dq2q"
Oct 31 08:04:36.079815 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:04:36.079749    4720 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-79d7476f4b-9dq2q" podStartSLOduration=4.019288006 podStartE2EDuration="8.079728601s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.060440595s" isStatefulPod=false podCreationTimestamp="2026-10-31 08:04:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 08:04:36.074927249 +0000 UTC m=+2130.205995321" watchObservedRunningTime="2026-10-31 08:04:36.079728601 +0000 UTC m=+2130.210796674"
Oct 31 08:04:36.970628 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:04:36.970591    4720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-6bbcf98bf7-7sk52" event={"ID":"a2d24fa7-975c-4976-8420-0e861eb88c30","Type":"ContainerStarted","Data":"3a51649f98752e97d4c8117ef86a5293e212df21f1baeea60ad5d857e03b1d81"}
Oct 31 08:04:36.971101 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:04:36.971067    4720 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-6bbcf98bf7-7sk52"
Oct 31 08:04:37.087808 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:04:37.087754    4720 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-6bbcf98bf7-7sk52" podStartSLOduration=4.327129558 podStartE2EDuration="8.087739129s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.760609571s" isStatefulPod=false podCreationTimestamp="2026-10-31 08:04:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 08:04:37.0859443 +0000 UTC m=+2131.217012371" watchObservedRunningTime="2026-10-31 08:04:37.087739129 +0000 UTC m=+2131.218807246"
Oct 31 08:04:37.692552 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:04:37.692495    4720 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-679758fddf-hh2nc" podUID="94456f54-4f33-4f9c-9e5a-1125d3955ef7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/health\": dial tcp 10.132.0.51:8000: connect: connection refused"
Oct 31 08:04:37.772694 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:04:37.772655    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 08:04:38.982363 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:04:38.981276    4720 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-79d7476f4b-9dq2q"
Oct 31 08:04:39.007134 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:04:39.007100    4720 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-79d7476f4b-9dq2q"
Oct 31 08:04:39.993387 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:04:39.993356    4720 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-79d7476f4b-9dq2q"
Oct 31 08:04:42.272895 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:04:42.272859    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 08:04:43.461719 ip-10-0-139-83 kubenswrapper[4720]: E1031 08:04:43.461685    4720 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8m768" containerName="extractor"
Oct 31 08:04:47.692602 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:04:47.692556    4720 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-679758fddf-hh2nc" podUID="94456f54-4f33-4f9c-9e5a-1125d3955ef7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/health\": dial tcp 10.132.0.51:8000: connect: connection refused"
Oct 31 08:04:57.692911 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:04:57.692859    4720 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-679758fddf-hh2nc" podUID="94456f54-4f33-4f9c-9e5a-1125d3955ef7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/health\": dial tcp 10.132.0.51:8000: connect: connection refused"
Oct 31 08:05:07.692529 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:05:07.692488    4720 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-679758fddf-hh2nc" podUID="94456f54-4f33-4f9c-9e5a-1125d3955ef7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/health\": dial tcp 10.132.0.51:8000: connect: connection refused"
Oct 31 08:05:07.978407 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:05:07.978299    4720 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-6bbcf98bf7-7sk52" podUID="a2d24fa7-975c-4976-8420-0e861eb88c30" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/health\": dial tcp 10.132.0.53:8000: connect: connection refused"
Oct 31 08:05:12.558626 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:05:12.558586    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 08:05:13.461694 ip-10-0-139-83 kubenswrapper[4720]: E1031 08:05:13.461655    4720 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8m768" containerName="extractor"
Oct 31 08:05:17.124993 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:05:17.124947    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 08:05:17.692763 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:05:17.692719    4720 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-679758fddf-hh2nc" podUID="94456f54-4f33-4f9c-9e5a-1125d3955ef7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/health\": dial tcp 10.132.0.51:8000: connect: connection refused"
Oct 31 08:05:17.987198 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:05:17.987105    4720 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-6bbcf98bf7-7sk52"
Oct 31 08:05:22.679113 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:05:22.679078    4720 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-cf47448b_tokenizer-clean-path-test-tokenizer-6bbcf98bf7-7sk52_a2d24fa7-975c-4976-8420-0e861eb88c30/main/0.log"
Oct 31 08:05:25.671542 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:05:25.671506    4720 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-79d7476f4b-9dq2q"]
Oct 31 08:05:25.683733 ip-10-0-139-83 kubenswrapper[4720]: E1031 08:05:25.683677    4720 log.go:32] "ExecSync cmd from runtime service failed" err="rpc error: code = Unknown desc = command error: , stdout: , stderr: , exit code -1" containerID="0fa4febce910182c67a8a435c9ee7cf780562024e90c0ab984b0ffc2b8fc89f0" cmd=["/bin/sleep","15"]
Oct 31 08:05:25.683896 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:05:25.683760    4720 handlers.go:85] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-79d7476f4b-9dq2q" message="" err="rpc error: code = Unknown desc = command error: , stdout: , stderr: , exit code -1"
Oct 31 08:05:25.683896 ip-10-0-139-83 kubenswrapper[4720]: E1031 08:05:25.683779    4720 kuberuntime_container.go:780] "PreStop hook failed" err="rpc error: code = Unknown desc = command error: , stdout: , stderr: , exit code -1" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-79d7476f4b-9dq2q" podUID="26959502-cb50-4bc4-9bb0-56966ad5a7f6" containerName="main" containerID="cri-o://0fa4febce910182c67a8a435c9ee7cf780562024e90c0ab984b0ffc2b8fc89f0"
Oct 31 08:05:25.683896 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:05:25.683845    4720 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-79d7476f4b-9dq2q" podUID="26959502-cb50-4bc4-9bb0-56966ad5a7f6" containerName="main" containerID="cri-o://0fa4febce910182c67a8a435c9ee7cf780562024e90c0ab984b0ffc2b8fc89f0" gracePeriod=60
Oct 31 08:05:25.687027 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:05:25.686998    4720 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-6bbcf98bf7-7sk52"]
Oct 31 08:05:25.687432 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:05:25.687352    4720 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-6bbcf98bf7-7sk52" podUID="a2d24fa7-975c-4976-8420-0e861eb88c30" containerName="main" containerID="cri-o://3a51649f98752e97d4c8117ef86a5293e212df21f1baeea60ad5d857e03b1d81" gracePeriod=30
Oct 31 08:05:25.702245 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:05:25.702204    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 08:05:25.855608 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:05:25.855571    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 08:05:25.994352 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:05:25.994232    4720 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-79d7476f4b-9dq2q" podUID="26959502-cb50-4bc4-9bb0-56966ad5a7f6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/health\": dial tcp 10.132.0.52:8000: connect: connection refused"
Oct 31 08:05:26.256989 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:05:26.256903    4720 generic.go:356] "Generic (PLEG): container finished" podID="26959502-cb50-4bc4-9bb0-56966ad5a7f6" containerID="0fa4febce910182c67a8a435c9ee7cf780562024e90c0ab984b0ffc2b8fc89f0" exitCode=0
Oct 31 08:05:26.256989 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:05:26.256934    4720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-79d7476f4b-9dq2q" event={"ID":"26959502-cb50-4bc4-9bb0-56966ad5a7f6","Type":"ContainerDied","Data":"0fa4febce910182c67a8a435c9ee7cf780562024e90c0ab984b0ffc2b8fc89f0"}
Oct 31 08:05:26.259061 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:05:26.259032    4720 generic.go:356] "Generic (PLEG): container finished" podID="a2d24fa7-975c-4976-8420-0e861eb88c30" containerID="3a51649f98752e97d4c8117ef86a5293e212df21f1baeea60ad5d857e03b1d81" exitCode=0
Oct 31 08:05:26.259203 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:05:26.259078    4720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-6bbcf98bf7-7sk52" event={"ID":"a2d24fa7-975c-4976-8420-0e861eb88c30","Type":"ContainerDied","Data":"3a51649f98752e97d4c8117ef86a5293e212df21f1baeea60ad5d857e03b1d81"}
Oct 31 08:05:26.351141 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:05:26.351107    4720 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-6bbcf98bf7-7sk52"
Oct 31 08:05:26.399044 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:05:26.399003    4720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a2d24fa7-975c-4976-8420-0e861eb88c30-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a2d24fa7-975c-4976-8420-0e861eb88c30-kserve-provision-location\") pod \"a2d24fa7-975c-4976-8420-0e861eb88c30\" (UID: \"a2d24fa7-975c-4976-8420-0e861eb88c30\") "
Oct 31 08:05:26.399189 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:05:26.399084    4720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a2d24fa7-975c-4976-8420-0e861eb88c30-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a2d24fa7-975c-4976-8420-0e861eb88c30-tmp-dir\") pod \"a2d24fa7-975c-4976-8420-0e861eb88c30\" (UID: \"a2d24fa7-975c-4976-8420-0e861eb88c30\") "
Oct 31 08:05:26.399189 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:05:26.399124    4720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a2d24fa7-975c-4976-8420-0e861eb88c30-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a2d24fa7-975c-4976-8420-0e861eb88c30-cabundle-cert\") pod \"a2d24fa7-975c-4976-8420-0e861eb88c30\" (UID: \"a2d24fa7-975c-4976-8420-0e861eb88c30\") "
Oct 31 08:05:26.399303 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:05:26.399188    4720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a2d24fa7-975c-4976-8420-0e861eb88c30-tls-certs\" (UniqueName: \"kubernetes.io/secret/a2d24fa7-975c-4976-8420-0e861eb88c30-tls-certs\") pod \"a2d24fa7-975c-4976-8420-0e861eb88c30\" (UID: \"a2d24fa7-975c-4976-8420-0e861eb88c30\") "
Oct 31 08:05:26.399612 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:05:26.399580    4720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a2d24fa7-975c-4976-8420-0e861eb88c30-cabundle-cert" pod "a2d24fa7-975c-4976-8420-0e861eb88c30" (UID: "a2d24fa7-975c-4976-8420-0e861eb88c30"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 08:05:26.399713 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:05:26.399583    4720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2d24fa7-975c-4976-8420-0e861eb88c30-tmp-dir" pod "a2d24fa7-975c-4976-8420-0e861eb88c30" (UID: "a2d24fa7-975c-4976-8420-0e861eb88c30"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 08:05:26.401724 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:05:26.401693    4720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a2d24fa7-975c-4976-8420-0e861eb88c30-tls-certs" pod "a2d24fa7-975c-4976-8420-0e861eb88c30" (UID: "a2d24fa7-975c-4976-8420-0e861eb88c30"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 08:05:26.420260 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:05:26.420225    4720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2d24fa7-975c-4976-8420-0e861eb88c30-kserve-provision-location" pod "a2d24fa7-975c-4976-8420-0e861eb88c30" (UID: "a2d24fa7-975c-4976-8420-0e861eb88c30"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 08:05:26.459869 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:05:26.459842    4720 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-79d7476f4b-9dq2q"
Oct 31 08:05:26.500015 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:05:26.499980    4720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/26959502-cb50-4bc4-9bb0-56966ad5a7f6-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/26959502-cb50-4bc4-9bb0-56966ad5a7f6-tmp-dir\") pod \"26959502-cb50-4bc4-9bb0-56966ad5a7f6\" (UID: \"26959502-cb50-4bc4-9bb0-56966ad5a7f6\") "
Oct 31 08:05:26.500208 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:05:26.500061    4720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/26959502-cb50-4bc4-9bb0-56966ad5a7f6-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/26959502-cb50-4bc4-9bb0-56966ad5a7f6-cabundle-cert\") pod \"26959502-cb50-4bc4-9bb0-56966ad5a7f6\" (UID: \"26959502-cb50-4bc4-9bb0-56966ad5a7f6\") "
Oct 31 08:05:26.500208 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:05:26.500091    4720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/26959502-cb50-4bc4-9bb0-56966ad5a7f6-dshm\" (UniqueName: \"kubernetes.io/empty-dir/26959502-cb50-4bc4-9bb0-56966ad5a7f6-dshm\") pod \"26959502-cb50-4bc4-9bb0-56966ad5a7f6\" (UID: \"26959502-cb50-4bc4-9bb0-56966ad5a7f6\") "
Oct 31 08:05:26.500208 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:05:26.500146    4720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/26959502-cb50-4bc4-9bb0-56966ad5a7f6-kube-api-access-df6mm\" (UniqueName: \"kubernetes.io/projected/26959502-cb50-4bc4-9bb0-56966ad5a7f6-kube-api-access-df6mm\") pod \"26959502-cb50-4bc4-9bb0-56966ad5a7f6\" (UID: \"26959502-cb50-4bc4-9bb0-56966ad5a7f6\") "
Oct 31 08:05:26.500417 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:05:26.500256    4720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/26959502-cb50-4bc4-9bb0-56966ad5a7f6-tmp-dir" pod "26959502-cb50-4bc4-9bb0-56966ad5a7f6" (UID: "26959502-cb50-4bc4-9bb0-56966ad5a7f6"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 08:05:26.500417 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:05:26.500262    4720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/26959502-cb50-4bc4-9bb0-56966ad5a7f6-tls-certs\" (UniqueName: \"kubernetes.io/secret/26959502-cb50-4bc4-9bb0-56966ad5a7f6-tls-certs\") pod \"26959502-cb50-4bc4-9bb0-56966ad5a7f6\" (UID: \"26959502-cb50-4bc4-9bb0-56966ad5a7f6\") "
Oct 31 08:05:26.500417 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:05:26.500381    4720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/26959502-cb50-4bc4-9bb0-56966ad5a7f6-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/26959502-cb50-4bc4-9bb0-56966ad5a7f6-kserve-provision-location\") pod \"26959502-cb50-4bc4-9bb0-56966ad5a7f6\" (UID: \"26959502-cb50-4bc4-9bb0-56966ad5a7f6\") "
Oct 31 08:05:26.500564 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:05:26.500475    4720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/26959502-cb50-4bc4-9bb0-56966ad5a7f6-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/26959502-cb50-4bc4-9bb0-56966ad5a7f6-model-cache\") pod \"26959502-cb50-4bc4-9bb0-56966ad5a7f6\" (UID: \"26959502-cb50-4bc4-9bb0-56966ad5a7f6\") "
Oct 31 08:05:26.500564 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:05:26.500508    4720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/26959502-cb50-4bc4-9bb0-56966ad5a7f6-home\" (UniqueName: \"kubernetes.io/empty-dir/26959502-cb50-4bc4-9bb0-56966ad5a7f6-home\") pod \"26959502-cb50-4bc4-9bb0-56966ad5a7f6\" (UID: \"26959502-cb50-4bc4-9bb0-56966ad5a7f6\") "
Oct 31 08:05:26.500564 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:05:26.500525    4720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/26959502-cb50-4bc4-9bb0-56966ad5a7f6-cabundle-cert" pod "26959502-cb50-4bc4-9bb0-56966ad5a7f6" (UID: "26959502-cb50-4bc4-9bb0-56966ad5a7f6"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 08:05:26.500841 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:05:26.500817    4720 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a2d24fa7-975c-4976-8420-0e861eb88c30-kserve-provision-location\") on node \"ip-10-0-139-83.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 08:05:26.500926 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:05:26.500848    4720 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a2d24fa7-975c-4976-8420-0e861eb88c30-tmp-dir\") on node \"ip-10-0-139-83.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 08:05:26.500926 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:05:26.500862    4720 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a2d24fa7-975c-4976-8420-0e861eb88c30-cabundle-cert\") on node \"ip-10-0-139-83.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 08:05:26.500926 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:05:26.500875    4720 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/26959502-cb50-4bc4-9bb0-56966ad5a7f6-tmp-dir\") on node \"ip-10-0-139-83.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 08:05:26.500926 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:05:26.500888    4720 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a2d24fa7-975c-4976-8420-0e861eb88c30-tls-certs\") on node \"ip-10-0-139-83.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 08:05:26.500926 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:05:26.500903    4720 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/26959502-cb50-4bc4-9bb0-56966ad5a7f6-cabundle-cert\") on node \"ip-10-0-139-83.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 08:05:26.501175 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:05:26.500953    4720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/26959502-cb50-4bc4-9bb0-56966ad5a7f6-model-cache" pod "26959502-cb50-4bc4-9bb0-56966ad5a7f6" (UID: "26959502-cb50-4bc4-9bb0-56966ad5a7f6"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 08:05:26.501175 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:05:26.501022    4720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/26959502-cb50-4bc4-9bb0-56966ad5a7f6-home" pod "26959502-cb50-4bc4-9bb0-56966ad5a7f6" (UID: "26959502-cb50-4bc4-9bb0-56966ad5a7f6"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 08:05:26.502724 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:05:26.502686    4720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/26959502-cb50-4bc4-9bb0-56966ad5a7f6-tls-certs" pod "26959502-cb50-4bc4-9bb0-56966ad5a7f6" (UID: "26959502-cb50-4bc4-9bb0-56966ad5a7f6"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 08:05:26.502829 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:05:26.502790    4720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/26959502-cb50-4bc4-9bb0-56966ad5a7f6-dshm" pod "26959502-cb50-4bc4-9bb0-56966ad5a7f6" (UID: "26959502-cb50-4bc4-9bb0-56966ad5a7f6"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 08:05:26.502982 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:05:26.502966    4720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/26959502-cb50-4bc4-9bb0-56966ad5a7f6-kube-api-access-df6mm" pod "26959502-cb50-4bc4-9bb0-56966ad5a7f6" (UID: "26959502-cb50-4bc4-9bb0-56966ad5a7f6"). InnerVolumeSpecName "kube-api-access-df6mm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 08:05:26.531143 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:05:26.531074    4720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/26959502-cb50-4bc4-9bb0-56966ad5a7f6-kserve-provision-location" pod "26959502-cb50-4bc4-9bb0-56966ad5a7f6" (UID: "26959502-cb50-4bc4-9bb0-56966ad5a7f6"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 08:05:26.601912 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:05:26.601871    4720 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/26959502-cb50-4bc4-9bb0-56966ad5a7f6-dshm\") on node \"ip-10-0-139-83.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 08:05:26.601912 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:05:26.601913    4720 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-df6mm\" (UniqueName: \"kubernetes.io/projected/26959502-cb50-4bc4-9bb0-56966ad5a7f6-kube-api-access-df6mm\") on node \"ip-10-0-139-83.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 08:05:26.602114 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:05:26.601928    4720 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/26959502-cb50-4bc4-9bb0-56966ad5a7f6-tls-certs\") on node \"ip-10-0-139-83.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 08:05:26.602114 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:05:26.601942    4720 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/26959502-cb50-4bc4-9bb0-56966ad5a7f6-kserve-provision-location\") on node \"ip-10-0-139-83.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 08:05:26.602114 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:05:26.601955    4720 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/26959502-cb50-4bc4-9bb0-56966ad5a7f6-model-cache\") on node \"ip-10-0-139-83.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 08:05:26.602114 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:05:26.601970    4720 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/26959502-cb50-4bc4-9bb0-56966ad5a7f6-home\") on node \"ip-10-0-139-83.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 08:05:27.266734 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:05:27.266698    4720 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-6bbcf98bf7-7sk52"
Oct 31 08:05:27.266734 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:05:27.266727    4720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-6bbcf98bf7-7sk52" event={"ID":"a2d24fa7-975c-4976-8420-0e861eb88c30","Type":"ContainerDied","Data":"1dd088ad151c0747e102fc99933f47d94a5734721557000ca0a61970805cfe2a"}
Oct 31 08:05:27.267295 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:05:27.266794    4720 scope.go:122] "RemoveContainer" containerID="3a51649f98752e97d4c8117ef86a5293e212df21f1baeea60ad5d857e03b1d81"
Oct 31 08:05:27.268687 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:05:27.268649    4720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-79d7476f4b-9dq2q" event={"ID":"26959502-cb50-4bc4-9bb0-56966ad5a7f6","Type":"ContainerDied","Data":"ed6dc702b5e6920ea3ca6e6be5c41976f9e0f08aa70055b21c88f17c478931c4"}
Oct 31 08:05:27.268830 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:05:27.268687    4720 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-79d7476f4b-9dq2q"
Oct 31 08:05:27.277031 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:05:27.277011    4720 scope.go:122] "RemoveContainer" containerID="208848e7b8b84d3a1523d4587d16d480f6e0c2db6d4f44a28be6efc93d334f2f"
Oct 31 08:05:27.286113 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:05:27.286092    4720 scope.go:122] "RemoveContainer" containerID="0fa4febce910182c67a8a435c9ee7cf780562024e90c0ab984b0ffc2b8fc89f0"
Oct 31 08:05:27.295242 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:05:27.295222    4720 scope.go:122] "RemoveContainer" containerID="1337e45cec97ff6b1876d05cbbc47f3575638457ebc0166320c0cbe3994bbf7f"
Oct 31 08:05:27.428464 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:05:27.428425    4720 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-6bbcf98bf7-7sk52"]
Oct 31 08:05:27.477742 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:05:27.477716    4720 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-6bbcf98bf7-7sk52"]
Oct 31 08:05:27.586619 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:05:27.586585    4720 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-79d7476f4b-9dq2q"]
Oct 31 08:05:27.593490 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:05:27.590167    4720 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-79d7476f4b-9dq2q"]
Oct 31 08:05:27.693098 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:05:27.693045    4720 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-679758fddf-hh2nc" podUID="94456f54-4f33-4f9c-9e5a-1125d3955ef7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/health\": dial tcp 10.132.0.51:8000: connect: connection refused"
Oct 31 08:05:28.029361 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:05:28.029260    4720 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="26959502-cb50-4bc4-9bb0-56966ad5a7f6" path="/var/lib/kubelet/pods/26959502-cb50-4bc4-9bb0-56966ad5a7f6/volumes"
Oct 31 08:05:28.029752 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:05:28.029738    4720 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a2d24fa7-975c-4976-8420-0e861eb88c30" path="/var/lib/kubelet/pods/a2d24fa7-975c-4976-8420-0e861eb88c30/volumes"
Oct 31 08:05:37.693347 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:05:37.693273    4720 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-679758fddf-hh2nc" podUID="94456f54-4f33-4f9c-9e5a-1125d3955ef7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/health\": dial tcp 10.132.0.51:8000: connect: connection refused"
Oct 31 08:05:42.623584 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:05:42.623512    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 08:05:43.464367 ip-10-0-139-83 kubenswrapper[4720]: E1031 08:05:43.464308    4720 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8m768" containerName="extractor"
Oct 31 08:05:47.692570 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:05:47.692467    4720 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-679758fddf-hh2nc" podUID="94456f54-4f33-4f9c-9e5a-1125d3955ef7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/health\": dial tcp 10.132.0.51:8000: connect: connection refused"
Oct 31 08:05:57.692680 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:05:57.692631    4720 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-679758fddf-hh2nc" podUID="94456f54-4f33-4f9c-9e5a-1125d3955ef7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/health\": dial tcp 10.132.0.51:8000: connect: connection refused"
Oct 31 08:06:07.693485 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:06:07.693440    4720 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-679758fddf-hh2nc" podUID="94456f54-4f33-4f9c-9e5a-1125d3955ef7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/health\": dial tcp 10.132.0.51:8000: connect: connection refused"
Oct 31 08:06:12.904544 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:06:12.904500    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 08:06:13.462008 ip-10-0-139-83 kubenswrapper[4720]: E1031 08:06:13.461973    4720 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8m768" containerName="extractor"
Oct 31 08:06:13.502757 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:06:13.502717    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 08:06:13.677053 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:06:13.677010    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 08:06:17.446132 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:06:17.446093    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 08:06:17.702786 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:06:17.702700    4720 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-679758fddf-hh2nc"
Oct 31 08:06:17.710509 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:06:17.710483    4720 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-679758fddf-hh2nc"
Oct 31 08:06:19.506945 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:06:19.506912    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 08:06:20.547388 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:06:20.547353    4720 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-8b767c9df-b5fpz"]
Oct 31 08:06:20.547780 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:06:20.547731    4720 cpu_manager.go:414] "containerMap: removing container" podUID="26959502-cb50-4bc4-9bb0-56966ad5a7f6" containerName="storage-initializer"
Oct 31 08:06:20.547780 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:06:20.547742    4720 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="26959502-cb50-4bc4-9bb0-56966ad5a7f6" containerName="storage-initializer"
Oct 31 08:06:20.547780 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:06:20.547772    4720 cpu_manager.go:414] "containerMap: removing container" podUID="a2d24fa7-975c-4976-8420-0e861eb88c30" containerName="main"
Oct 31 08:06:20.547780 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:06:20.547778    4720 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a2d24fa7-975c-4976-8420-0e861eb88c30" containerName="main"
Oct 31 08:06:20.547913 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:06:20.547792    4720 cpu_manager.go:414] "containerMap: removing container" podUID="a2d24fa7-975c-4976-8420-0e861eb88c30" containerName="storage-initializer"
Oct 31 08:06:20.547913 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:06:20.547797    4720 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a2d24fa7-975c-4976-8420-0e861eb88c30" containerName="storage-initializer"
Oct 31 08:06:20.547913 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:06:20.547804    4720 cpu_manager.go:414] "containerMap: removing container" podUID="26959502-cb50-4bc4-9bb0-56966ad5a7f6" containerName="main"
Oct 31 08:06:20.547913 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:06:20.547809    4720 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="26959502-cb50-4bc4-9bb0-56966ad5a7f6" containerName="main"
Oct 31 08:06:20.547913 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:06:20.547864    4720 memory_manager.go:365] "RemoveStaleState removing state" podUID="a2d24fa7-975c-4976-8420-0e861eb88c30" containerName="main"
Oct 31 08:06:20.547913 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:06:20.547873    4720 memory_manager.go:365] "RemoveStaleState removing state" podUID="26959502-cb50-4bc4-9bb0-56966ad5a7f6" containerName="main"
Oct 31 08:06:20.551941 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:06:20.551920    4720 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-8b767c9df-b5fpz"
Oct 31 08:06:20.603552 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:06:20.603514    4720 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"tokenizer-migration-path-test-kserve-self-signed-certs\""
Oct 31 08:06:20.603753 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:06:20.603598    4720 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"seaweedfs-s3-creds\""
Oct 31 08:06:20.604060 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:06:20.604036    4720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/45fe1515-14e2-457d-afa2-92eb268580a6-home\") pod \"tokenizer-migration-path-test-kserve-8b767c9df-b5fpz\" (UID: \"45fe1515-14e2-457d-afa2-92eb268580a6\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-8b767c9df-b5fpz"
Oct 31 08:06:20.604107 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:06:20.604091    4720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/45fe1515-14e2-457d-afa2-92eb268580a6-kserve-provision-location\") pod \"tokenizer-migration-path-test-kserve-8b767c9df-b5fpz\" (UID: \"45fe1515-14e2-457d-afa2-92eb268580a6\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-8b767c9df-b5fpz"
Oct 31 08:06:20.604240 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:06:20.604219    4720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w7gzh\" (UniqueName: \"kubernetes.io/projected/45fe1515-14e2-457d-afa2-92eb268580a6-kube-api-access-w7gzh\") pod \"tokenizer-migration-path-test-kserve-8b767c9df-b5fpz\" (UID: \"45fe1515-14e2-457d-afa2-92eb268580a6\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-8b767c9df-b5fpz"
Oct 31 08:06:20.604308 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:06:20.604272    4720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/45fe1515-14e2-457d-afa2-92eb268580a6-tmp-dir\") pod \"tokenizer-migration-path-test-kserve-8b767c9df-b5fpz\" (UID: \"45fe1515-14e2-457d-afa2-92eb268580a6\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-8b767c9df-b5fpz"
Oct 31 08:06:20.604421 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:06:20.604315    4720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/45fe1515-14e2-457d-afa2-92eb268580a6-dshm\") pod \"tokenizer-migration-path-test-kserve-8b767c9df-b5fpz\" (UID: \"45fe1515-14e2-457d-afa2-92eb268580a6\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-8b767c9df-b5fpz"
Oct 31 08:06:20.604479 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:06:20.604416    4720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/45fe1515-14e2-457d-afa2-92eb268580a6-model-cache\") pod \"tokenizer-migration-path-test-kserve-8b767c9df-b5fpz\" (UID: \"45fe1515-14e2-457d-afa2-92eb268580a6\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-8b767c9df-b5fpz"
Oct 31 08:06:20.604521 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:06:20.604494    4720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/45fe1515-14e2-457d-afa2-92eb268580a6-tls-certs\") pod \"tokenizer-migration-path-test-kserve-8b767c9df-b5fpz\" (UID: \"45fe1515-14e2-457d-afa2-92eb268580a6\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-8b767c9df-b5fpz"
Oct 31 08:06:20.604564 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:06:20.604529    4720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/45fe1515-14e2-457d-afa2-92eb268580a6-cabundle-cert\") pod \"tokenizer-migration-path-test-kserve-8b767c9df-b5fpz\" (UID: \"45fe1515-14e2-457d-afa2-92eb268580a6\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-8b767c9df-b5fpz"
Oct 31 08:06:20.652587 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:06:20.652553    4720 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"odh-kserve-custom-ca-bundle\""
Oct 31 08:06:20.652807 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:06:20.652662    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 08:06:20.652807 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:06:20.652765    4720 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"openshift-service-ca.crt\""
Oct 31 08:06:20.652807 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:06:20.652804    4720 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"kube-root-ca.crt\""
Oct 31 08:06:20.653007 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:06:20.652836    4720 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"default-dockercfg-s4gtz\""
Oct 31 08:06:20.654777 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:06:20.654753    4720 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-8b767c9df-b5fpz"]
Oct 31 08:06:20.704947 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:06:20.704906    4720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/45fe1515-14e2-457d-afa2-92eb268580a6-dshm\") pod \"tokenizer-migration-path-test-kserve-8b767c9df-b5fpz\" (UID: \"45fe1515-14e2-457d-afa2-92eb268580a6\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-8b767c9df-b5fpz"
Oct 31 08:06:20.704947 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:06:20.704949    4720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/45fe1515-14e2-457d-afa2-92eb268580a6-model-cache\") pod \"tokenizer-migration-path-test-kserve-8b767c9df-b5fpz\" (UID: \"45fe1515-14e2-457d-afa2-92eb268580a6\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-8b767c9df-b5fpz"
Oct 31 08:06:20.705203 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:06:20.704994    4720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/45fe1515-14e2-457d-afa2-92eb268580a6-tls-certs\") pod \"tokenizer-migration-path-test-kserve-8b767c9df-b5fpz\" (UID: \"45fe1515-14e2-457d-afa2-92eb268580a6\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-8b767c9df-b5fpz"
Oct 31 08:06:20.705203 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:06:20.705020    4720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/45fe1515-14e2-457d-afa2-92eb268580a6-cabundle-cert\") pod \"tokenizer-migration-path-test-kserve-8b767c9df-b5fpz\" (UID: \"45fe1515-14e2-457d-afa2-92eb268580a6\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-8b767c9df-b5fpz"
Oct 31 08:06:20.705203 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:06:20.705062    4720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/45fe1515-14e2-457d-afa2-92eb268580a6-home\") pod \"tokenizer-migration-path-test-kserve-8b767c9df-b5fpz\" (UID: \"45fe1515-14e2-457d-afa2-92eb268580a6\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-8b767c9df-b5fpz"
Oct 31 08:06:20.705203 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:06:20.705102    4720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/45fe1515-14e2-457d-afa2-92eb268580a6-kserve-provision-location\") pod \"tokenizer-migration-path-test-kserve-8b767c9df-b5fpz\" (UID: \"45fe1515-14e2-457d-afa2-92eb268580a6\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-8b767c9df-b5fpz"
Oct 31 08:06:20.705203 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:06:20.705141    4720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w7gzh\" (UniqueName: \"kubernetes.io/projected/45fe1515-14e2-457d-afa2-92eb268580a6-kube-api-access-w7gzh\") pod \"tokenizer-migration-path-test-kserve-8b767c9df-b5fpz\" (UID: \"45fe1515-14e2-457d-afa2-92eb268580a6\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-8b767c9df-b5fpz"
Oct 31 08:06:20.705203 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:06:20.705177    4720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/45fe1515-14e2-457d-afa2-92eb268580a6-tmp-dir\") pod \"tokenizer-migration-path-test-kserve-8b767c9df-b5fpz\" (UID: \"45fe1515-14e2-457d-afa2-92eb268580a6\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-8b767c9df-b5fpz"
Oct 31 08:06:20.705542 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:06:20.705481    4720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/45fe1515-14e2-457d-afa2-92eb268580a6-model-cache\") pod \"tokenizer-migration-path-test-kserve-8b767c9df-b5fpz\" (UID: \"45fe1515-14e2-457d-afa2-92eb268580a6\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-8b767c9df-b5fpz"
Oct 31 08:06:20.705911 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:06:20.705792    4720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/45fe1515-14e2-457d-afa2-92eb268580a6-kserve-provision-location\") pod \"tokenizer-migration-path-test-kserve-8b767c9df-b5fpz\" (UID: \"45fe1515-14e2-457d-afa2-92eb268580a6\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-8b767c9df-b5fpz"
Oct 31 08:06:20.705911 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:06:20.705868    4720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/45fe1515-14e2-457d-afa2-92eb268580a6-home\") pod \"tokenizer-migration-path-test-kserve-8b767c9df-b5fpz\" (UID: \"45fe1515-14e2-457d-afa2-92eb268580a6\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-8b767c9df-b5fpz"
Oct 31 08:06:20.706066 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:06:20.705918    4720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/45fe1515-14e2-457d-afa2-92eb268580a6-tmp-dir\") pod \"tokenizer-migration-path-test-kserve-8b767c9df-b5fpz\" (UID: \"45fe1515-14e2-457d-afa2-92eb268580a6\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-8b767c9df-b5fpz"
Oct 31 08:06:20.706066 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:06:20.706037    4720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/45fe1515-14e2-457d-afa2-92eb268580a6-cabundle-cert\") pod \"tokenizer-migration-path-test-kserve-8b767c9df-b5fpz\" (UID: \"45fe1515-14e2-457d-afa2-92eb268580a6\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-8b767c9df-b5fpz"
Oct 31 08:06:20.707550 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:06:20.707522    4720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/45fe1515-14e2-457d-afa2-92eb268580a6-dshm\") pod \"tokenizer-migration-path-test-kserve-8b767c9df-b5fpz\" (UID: \"45fe1515-14e2-457d-afa2-92eb268580a6\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-8b767c9df-b5fpz"
Oct 31 08:06:20.707821 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:06:20.707802    4720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/45fe1515-14e2-457d-afa2-92eb268580a6-tls-certs\") pod \"tokenizer-migration-path-test-kserve-8b767c9df-b5fpz\" (UID: \"45fe1515-14e2-457d-afa2-92eb268580a6\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-8b767c9df-b5fpz"
Oct 31 08:06:20.761182 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:06:20.761141    4720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w7gzh\" (UniqueName: \"kubernetes.io/projected/45fe1515-14e2-457d-afa2-92eb268580a6-kube-api-access-w7gzh\") pod \"tokenizer-migration-path-test-kserve-8b767c9df-b5fpz\" (UID: \"45fe1515-14e2-457d-afa2-92eb268580a6\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-8b767c9df-b5fpz"
Oct 31 08:06:20.868125 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:06:20.868082    4720 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-8b767c9df-b5fpz"
Oct 31 08:06:21.129942 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:06:21.129871    4720 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-5cd7c9f67f-plhh9"]
Oct 31 08:06:21.134385 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:06:21.134364    4720 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-5cd7c9f67f-plhh9"
Oct 31 08:06:21.191458 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:06:21.191427    4720 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-5cd7c9f67f-plhh9"]
Oct 31 08:06:21.210073 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:06:21.210045    4720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5947348e-e436-4e5e-8a0d-a14da00383fb-tls-certs\") pod \"tokenizer-migration-path-test-tokenizer-5cd7c9f67f-plhh9\" (UID: \"5947348e-e436-4e5e-8a0d-a14da00383fb\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-5cd7c9f67f-plhh9"
Oct 31 08:06:21.210257 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:06:21.210115    4720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5947348e-e436-4e5e-8a0d-a14da00383fb-kserve-provision-location\") pod \"tokenizer-migration-path-test-tokenizer-5cd7c9f67f-plhh9\" (UID: \"5947348e-e436-4e5e-8a0d-a14da00383fb\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-5cd7c9f67f-plhh9"
Oct 31 08:06:21.210257 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:06:21.210220    4720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5947348e-e436-4e5e-8a0d-a14da00383fb-cabundle-cert\") pod \"tokenizer-migration-path-test-tokenizer-5cd7c9f67f-plhh9\" (UID: \"5947348e-e436-4e5e-8a0d-a14da00383fb\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-5cd7c9f67f-plhh9"
Oct 31 08:06:21.210417 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:06:21.210255    4720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5947348e-e436-4e5e-8a0d-a14da00383fb-tmp-dir\") pod \"tokenizer-migration-path-test-tokenizer-5cd7c9f67f-plhh9\" (UID: \"5947348e-e436-4e5e-8a0d-a14da00383fb\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-5cd7c9f67f-plhh9"
Oct 31 08:06:21.243509 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:06:21.243478    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 08:06:21.298136 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:06:21.298093    4720 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-8b767c9df-b5fpz"]
Oct 31 08:06:21.302313 ip-10-0-139-83 kubenswrapper[4720]: W1031 08:06:21.302284    4720 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod45fe1515_14e2_457d_afa2_92eb268580a6.slice/crio-d4993c0978698195fa101d7d5a7338daa8c32ec916173ed286717b2a4a803c8f WatchSource:0}: Error finding container d4993c0978698195fa101d7d5a7338daa8c32ec916173ed286717b2a4a803c8f: Status 404 returned error can't find the container with id d4993c0978698195fa101d7d5a7338daa8c32ec916173ed286717b2a4a803c8f
Oct 31 08:06:21.310897 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:06:21.310871    4720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5947348e-e436-4e5e-8a0d-a14da00383fb-cabundle-cert\") pod \"tokenizer-migration-path-test-tokenizer-5cd7c9f67f-plhh9\" (UID: \"5947348e-e436-4e5e-8a0d-a14da00383fb\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-5cd7c9f67f-plhh9"
Oct 31 08:06:21.311000 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:06:21.310908    4720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5947348e-e436-4e5e-8a0d-a14da00383fb-tmp-dir\") pod \"tokenizer-migration-path-test-tokenizer-5cd7c9f67f-plhh9\" (UID: \"5947348e-e436-4e5e-8a0d-a14da00383fb\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-5cd7c9f67f-plhh9"
Oct 31 08:06:21.311000 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:06:21.310980    4720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5947348e-e436-4e5e-8a0d-a14da00383fb-tls-certs\") pod \"tokenizer-migration-path-test-tokenizer-5cd7c9f67f-plhh9\" (UID: \"5947348e-e436-4e5e-8a0d-a14da00383fb\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-5cd7c9f67f-plhh9"
Oct 31 08:06:21.311097 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:06:21.311042    4720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5947348e-e436-4e5e-8a0d-a14da00383fb-kserve-provision-location\") pod \"tokenizer-migration-path-test-tokenizer-5cd7c9f67f-plhh9\" (UID: \"5947348e-e436-4e5e-8a0d-a14da00383fb\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-5cd7c9f67f-plhh9"
Oct 31 08:06:21.311412 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:06:21.311390    4720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5947348e-e436-4e5e-8a0d-a14da00383fb-tmp-dir\") pod \"tokenizer-migration-path-test-tokenizer-5cd7c9f67f-plhh9\" (UID: \"5947348e-e436-4e5e-8a0d-a14da00383fb\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-5cd7c9f67f-plhh9"
Oct 31 08:06:21.311468 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:06:21.311435    4720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5947348e-e436-4e5e-8a0d-a14da00383fb-kserve-provision-location\") pod \"tokenizer-migration-path-test-tokenizer-5cd7c9f67f-plhh9\" (UID: \"5947348e-e436-4e5e-8a0d-a14da00383fb\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-5cd7c9f67f-plhh9"
Oct 31 08:06:21.311623 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:06:21.311602    4720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5947348e-e436-4e5e-8a0d-a14da00383fb-cabundle-cert\") pod \"tokenizer-migration-path-test-tokenizer-5cd7c9f67f-plhh9\" (UID: \"5947348e-e436-4e5e-8a0d-a14da00383fb\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-5cd7c9f67f-plhh9"
Oct 31 08:06:21.313864 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:06:21.313840    4720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5947348e-e436-4e5e-8a0d-a14da00383fb-tls-certs\") pod \"tokenizer-migration-path-test-tokenizer-5cd7c9f67f-plhh9\" (UID: \"5947348e-e436-4e5e-8a0d-a14da00383fb\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-5cd7c9f67f-plhh9"
Oct 31 08:06:21.400950 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:06:21.400699    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 08:06:21.451940 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:06:21.451906    4720 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-5cd7c9f67f-plhh9"
Oct 31 08:06:21.537521 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:06:21.537469    4720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-8b767c9df-b5fpz" event={"ID":"45fe1515-14e2-457d-afa2-92eb268580a6","Type":"ContainerStarted","Data":"ca192d087bb88c6a137e17558df67f0455116863f58826318ff221d70dbc2608"}
Oct 31 08:06:21.537715 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:06:21.537543    4720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-8b767c9df-b5fpz" event={"ID":"45fe1515-14e2-457d-afa2-92eb268580a6","Type":"ContainerStarted","Data":"d4993c0978698195fa101d7d5a7338daa8c32ec916173ed286717b2a4a803c8f"}
Oct 31 08:06:21.828051 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:06:21.827948    4720 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-5cd7c9f67f-plhh9"]
Oct 31 08:06:21.831289 ip-10-0-139-83 kubenswrapper[4720]: W1031 08:06:21.831255    4720 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5947348e_e436_4e5e_8a0d_a14da00383fb.slice/crio-81ccf2a49442f1c6bb52e862e2b55064df9c286f78b5b396326cd9f0c40ffb4c WatchSource:0}: Error finding container 81ccf2a49442f1c6bb52e862e2b55064df9c286f78b5b396326cd9f0c40ffb4c: Status 404 returned error can't find the container with id 81ccf2a49442f1c6bb52e862e2b55064df9c286f78b5b396326cd9f0c40ffb4c
Oct 31 08:06:22.545524 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:06:22.545483    4720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-5cd7c9f67f-plhh9" event={"ID":"5947348e-e436-4e5e-8a0d-a14da00383fb","Type":"ContainerStarted","Data":"acffef2c58e30befffd1046682a07abb9095ae5736d78089874d947d10dd6921"}
Oct 31 08:06:22.546431 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:06:22.545550    4720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-5cd7c9f67f-plhh9" event={"ID":"5947348e-e436-4e5e-8a0d-a14da00383fb","Type":"ContainerStarted","Data":"81ccf2a49442f1c6bb52e862e2b55064df9c286f78b5b396326cd9f0c40ffb4c"}
Oct 31 08:06:24.557206 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:06:24.557170    4720 generic.go:356] "Generic (PLEG): container finished" podID="45fe1515-14e2-457d-afa2-92eb268580a6" containerID="ca192d087bb88c6a137e17558df67f0455116863f58826318ff221d70dbc2608" exitCode=0
Oct 31 08:06:24.557691 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:06:24.557250    4720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-8b767c9df-b5fpz" event={"ID":"45fe1515-14e2-457d-afa2-92eb268580a6","Type":"ContainerDied","Data":"ca192d087bb88c6a137e17558df67f0455116863f58826318ff221d70dbc2608"}
Oct 31 08:06:25.481713 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:06:25.481671    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 08:06:25.566701 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:06:25.566651    4720 generic.go:356] "Generic (PLEG): container finished" podID="5947348e-e436-4e5e-8a0d-a14da00383fb" containerID="acffef2c58e30befffd1046682a07abb9095ae5736d78089874d947d10dd6921" exitCode=0
Oct 31 08:06:25.567218 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:06:25.566732    4720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-5cd7c9f67f-plhh9" event={"ID":"5947348e-e436-4e5e-8a0d-a14da00383fb","Type":"ContainerDied","Data":"acffef2c58e30befffd1046682a07abb9095ae5736d78089874d947d10dd6921"}
Oct 31 08:06:25.569322 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:06:25.569268    4720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-8b767c9df-b5fpz" event={"ID":"45fe1515-14e2-457d-afa2-92eb268580a6","Type":"ContainerStarted","Data":"9a9b6f36279510d18003f9dccbaf514f5d8c3df1e23352272081a978ed74cec1"}
Oct 31 08:06:25.583637 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:06:25.583397    4720 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-679758fddf-hh2nc"]
Oct 31 08:06:25.644877 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:06:25.644848    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 08:06:25.801522 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:06:25.801409    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 08:06:25.869208 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:06:25.869160    4720 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-8b767c9df-b5fpz"
Oct 31 08:06:25.921572 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:06:25.921522    4720 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-8b767c9df-b5fpz" podStartSLOduration=3.276772616 podStartE2EDuration="5.921508106s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.64473549s" isStatefulPod=false podCreationTimestamp="2026-10-31 08:06:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 08:06:25.92075562 +0000 UTC m=+2240.051823691" watchObservedRunningTime="2026-10-31 08:06:25.921508106 +0000 UTC m=+2240.052576177"
Oct 31 08:06:26.576236 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:06:26.576196    4720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-5cd7c9f67f-plhh9" event={"ID":"5947348e-e436-4e5e-8a0d-a14da00383fb","Type":"ContainerStarted","Data":"9c8df5144c2c1cfb394891177ac626723ba6ed9f3826d3c77a3c6d2c3e0f06d6"}
Oct 31 08:06:26.693079 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:06:26.693012    4720 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-5cd7c9f67f-plhh9" podStartSLOduration=3.041293133 podStartE2EDuration="5.692992685s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.651699552s" isStatefulPod=false podCreationTimestamp="2026-10-31 08:06:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 08:06:26.692106863 +0000 UTC m=+2240.823174919" watchObservedRunningTime="2026-10-31 08:06:26.692992685 +0000 UTC m=+2240.824060757"
Oct 31 08:06:27.371513 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:06:27.371456    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 08:06:30.869058 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:06:30.869019    4720 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-8b767c9df-b5fpz"
Oct 31 08:06:30.882063 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:06:30.882032    4720 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-8b767c9df-b5fpz"
Oct 31 08:06:31.619658 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:06:31.619625    4720 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-8b767c9df-b5fpz"
Oct 31 08:06:36.056231 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:06:36.056179    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 08:06:36.577441 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:06:36.577404    4720 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-5cd7c9f67f-plhh9"
Oct 31 08:06:40.612371 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:06:40.612280    4720 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-679758fddf-hh2nc" podUID="94456f54-4f33-4f9c-9e5a-1125d3955ef7" containerName="main" containerID="cri-o://b1955df30e6d77c634fa4bb113921a52d10d3464a89d19c9db3ac0a1af0b9b5e" gracePeriod=45
Oct 31 08:06:40.704195 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:06:40.704142    4720 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-679758fddf-hh2nc" podUID="94456f54-4f33-4f9c-9e5a-1125d3955ef7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/health\": dial tcp 10.132.0.51:8000: connect: connection refused"
Oct 31 08:06:41.383879 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:06:41.383848    4720 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-679758fddf-hh2nc"
Oct 31 08:06:41.519240 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:06:41.519149    4720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/94456f54-4f33-4f9c-9e5a-1125d3955ef7-kube-api-access-2cwgs\" (UniqueName: \"kubernetes.io/projected/94456f54-4f33-4f9c-9e5a-1125d3955ef7-kube-api-access-2cwgs\") pod \"94456f54-4f33-4f9c-9e5a-1125d3955ef7\" (UID: \"94456f54-4f33-4f9c-9e5a-1125d3955ef7\") "
Oct 31 08:06:41.519240 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:06:41.519194    4720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/94456f54-4f33-4f9c-9e5a-1125d3955ef7-tls-certs\" (UniqueName: \"kubernetes.io/secret/94456f54-4f33-4f9c-9e5a-1125d3955ef7-tls-certs\") pod \"94456f54-4f33-4f9c-9e5a-1125d3955ef7\" (UID: \"94456f54-4f33-4f9c-9e5a-1125d3955ef7\") "
Oct 31 08:06:41.519500 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:06:41.519244    4720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/94456f54-4f33-4f9c-9e5a-1125d3955ef7-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/94456f54-4f33-4f9c-9e5a-1125d3955ef7-kserve-provision-location\") pod \"94456f54-4f33-4f9c-9e5a-1125d3955ef7\" (UID: \"94456f54-4f33-4f9c-9e5a-1125d3955ef7\") "
Oct 31 08:06:41.519500 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:06:41.519263    4720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/94456f54-4f33-4f9c-9e5a-1125d3955ef7-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/94456f54-4f33-4f9c-9e5a-1125d3955ef7-model-cache\") pod \"94456f54-4f33-4f9c-9e5a-1125d3955ef7\" (UID: \"94456f54-4f33-4f9c-9e5a-1125d3955ef7\") "
Oct 31 08:06:41.519500 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:06:41.519301    4720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/94456f54-4f33-4f9c-9e5a-1125d3955ef7-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/94456f54-4f33-4f9c-9e5a-1125d3955ef7-tmp-dir\") pod \"94456f54-4f33-4f9c-9e5a-1125d3955ef7\" (UID: \"94456f54-4f33-4f9c-9e5a-1125d3955ef7\") "
Oct 31 08:06:41.519500 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:06:41.519377    4720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/94456f54-4f33-4f9c-9e5a-1125d3955ef7-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/94456f54-4f33-4f9c-9e5a-1125d3955ef7-cabundle-cert\") pod \"94456f54-4f33-4f9c-9e5a-1125d3955ef7\" (UID: \"94456f54-4f33-4f9c-9e5a-1125d3955ef7\") "
Oct 31 08:06:41.519500 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:06:41.519403    4720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/94456f54-4f33-4f9c-9e5a-1125d3955ef7-dshm\" (UniqueName: \"kubernetes.io/empty-dir/94456f54-4f33-4f9c-9e5a-1125d3955ef7-dshm\") pod \"94456f54-4f33-4f9c-9e5a-1125d3955ef7\" (UID: \"94456f54-4f33-4f9c-9e5a-1125d3955ef7\") "
Oct 31 08:06:41.519500 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:06:41.519474    4720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/94456f54-4f33-4f9c-9e5a-1125d3955ef7-home\" (UniqueName: \"kubernetes.io/empty-dir/94456f54-4f33-4f9c-9e5a-1125d3955ef7-home\") pod \"94456f54-4f33-4f9c-9e5a-1125d3955ef7\" (UID: \"94456f54-4f33-4f9c-9e5a-1125d3955ef7\") "
Oct 31 08:06:41.519797 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:06:41.519627    4720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/94456f54-4f33-4f9c-9e5a-1125d3955ef7-model-cache" pod "94456f54-4f33-4f9c-9e5a-1125d3955ef7" (UID: "94456f54-4f33-4f9c-9e5a-1125d3955ef7"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 08:06:41.519797 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:06:41.519780    4720 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/94456f54-4f33-4f9c-9e5a-1125d3955ef7-model-cache\") on node \"ip-10-0-139-83.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 08:06:41.519989 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:06:41.519942    4720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/94456f54-4f33-4f9c-9e5a-1125d3955ef7-cabundle-cert" pod "94456f54-4f33-4f9c-9e5a-1125d3955ef7" (UID: "94456f54-4f33-4f9c-9e5a-1125d3955ef7"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 08:06:41.520234 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:06:41.520207    4720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/94456f54-4f33-4f9c-9e5a-1125d3955ef7-home" pod "94456f54-4f33-4f9c-9e5a-1125d3955ef7" (UID: "94456f54-4f33-4f9c-9e5a-1125d3955ef7"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 08:06:41.522248 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:06:41.522215    4720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/94456f54-4f33-4f9c-9e5a-1125d3955ef7-kube-api-access-2cwgs" pod "94456f54-4f33-4f9c-9e5a-1125d3955ef7" (UID: "94456f54-4f33-4f9c-9e5a-1125d3955ef7"). InnerVolumeSpecName "kube-api-access-2cwgs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 08:06:41.522440 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:06:41.522421    4720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/94456f54-4f33-4f9c-9e5a-1125d3955ef7-dshm" pod "94456f54-4f33-4f9c-9e5a-1125d3955ef7" (UID: "94456f54-4f33-4f9c-9e5a-1125d3955ef7"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 08:06:41.522515 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:06:41.522501    4720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/94456f54-4f33-4f9c-9e5a-1125d3955ef7-tls-certs" pod "94456f54-4f33-4f9c-9e5a-1125d3955ef7" (UID: "94456f54-4f33-4f9c-9e5a-1125d3955ef7"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 08:06:41.533276 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:06:41.533243    4720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/94456f54-4f33-4f9c-9e5a-1125d3955ef7-tmp-dir" pod "94456f54-4f33-4f9c-9e5a-1125d3955ef7" (UID: "94456f54-4f33-4f9c-9e5a-1125d3955ef7"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 08:06:41.557925 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:06:41.557878    4720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/94456f54-4f33-4f9c-9e5a-1125d3955ef7-kserve-provision-location" pod "94456f54-4f33-4f9c-9e5a-1125d3955ef7" (UID: "94456f54-4f33-4f9c-9e5a-1125d3955ef7"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 08:06:41.620789 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:06:41.620752    4720 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/94456f54-4f33-4f9c-9e5a-1125d3955ef7-tmp-dir\") on node \"ip-10-0-139-83.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 08:06:41.620789 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:06:41.620792    4720 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/94456f54-4f33-4f9c-9e5a-1125d3955ef7-cabundle-cert\") on node \"ip-10-0-139-83.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 08:06:41.621226 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:06:41.620809    4720 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/94456f54-4f33-4f9c-9e5a-1125d3955ef7-dshm\") on node \"ip-10-0-139-83.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 08:06:41.621226 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:06:41.620822    4720 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/94456f54-4f33-4f9c-9e5a-1125d3955ef7-home\") on node \"ip-10-0-139-83.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 08:06:41.621226 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:06:41.620837    4720 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2cwgs\" (UniqueName: \"kubernetes.io/projected/94456f54-4f33-4f9c-9e5a-1125d3955ef7-kube-api-access-2cwgs\") on node \"ip-10-0-139-83.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 08:06:41.621226 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:06:41.620851    4720 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/94456f54-4f33-4f9c-9e5a-1125d3955ef7-tls-certs\") on node \"ip-10-0-139-83.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 08:06:41.621226 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:06:41.620865    4720 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/94456f54-4f33-4f9c-9e5a-1125d3955ef7-kserve-provision-location\") on node \"ip-10-0-139-83.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 08:06:41.661395 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:06:41.661356    4720 generic.go:356] "Generic (PLEG): container finished" podID="94456f54-4f33-4f9c-9e5a-1125d3955ef7" containerID="b1955df30e6d77c634fa4bb113921a52d10d3464a89d19c9db3ac0a1af0b9b5e" exitCode=0
Oct 31 08:06:41.661599 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:06:41.661408    4720 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-679758fddf-hh2nc"
Oct 31 08:06:41.661599 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:06:41.661428    4720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-679758fddf-hh2nc" event={"ID":"94456f54-4f33-4f9c-9e5a-1125d3955ef7","Type":"ContainerDied","Data":"b1955df30e6d77c634fa4bb113921a52d10d3464a89d19c9db3ac0a1af0b9b5e"}
Oct 31 08:06:41.661599 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:06:41.661486    4720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-679758fddf-hh2nc" event={"ID":"94456f54-4f33-4f9c-9e5a-1125d3955ef7","Type":"ContainerDied","Data":"039efbfe82d52682deb101be2be3548c884148fe2dbd529aa47cef80146c137e"}
Oct 31 08:06:41.661599 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:06:41.661510    4720 scope.go:122] "RemoveContainer" containerID="b1955df30e6d77c634fa4bb113921a52d10d3464a89d19c9db3ac0a1af0b9b5e"
Oct 31 08:06:41.671389 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:06:41.671361    4720 scope.go:122] "RemoveContainer" containerID="766fd28b174a193dfc46e254f315f56339365167195e90da7fe24726b6c85c8e"
Oct 31 08:06:41.679773 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:06:41.679753    4720 scope.go:122] "RemoveContainer" containerID="b1955df30e6d77c634fa4bb113921a52d10d3464a89d19c9db3ac0a1af0b9b5e"
Oct 31 08:06:41.680035 ip-10-0-139-83 kubenswrapper[4720]: E1031 08:06:41.680018    4720 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b1955df30e6d77c634fa4bb113921a52d10d3464a89d19c9db3ac0a1af0b9b5e\": container with ID starting with b1955df30e6d77c634fa4bb113921a52d10d3464a89d19c9db3ac0a1af0b9b5e not found: ID does not exist" containerID="b1955df30e6d77c634fa4bb113921a52d10d3464a89d19c9db3ac0a1af0b9b5e"
Oct 31 08:06:41.680082 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:06:41.680044    4720 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b1955df30e6d77c634fa4bb113921a52d10d3464a89d19c9db3ac0a1af0b9b5e"} err="failed to get container status \"b1955df30e6d77c634fa4bb113921a52d10d3464a89d19c9db3ac0a1af0b9b5e\": rpc error: code = NotFound desc = could not find container \"b1955df30e6d77c634fa4bb113921a52d10d3464a89d19c9db3ac0a1af0b9b5e\": container with ID starting with b1955df30e6d77c634fa4bb113921a52d10d3464a89d19c9db3ac0a1af0b9b5e not found: ID does not exist"
Oct 31 08:06:41.680082 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:06:41.680064    4720 scope.go:122] "RemoveContainer" containerID="766fd28b174a193dfc46e254f315f56339365167195e90da7fe24726b6c85c8e"
Oct 31 08:06:41.680375 ip-10-0-139-83 kubenswrapper[4720]: E1031 08:06:41.680354    4720 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"766fd28b174a193dfc46e254f315f56339365167195e90da7fe24726b6c85c8e\": container with ID starting with 766fd28b174a193dfc46e254f315f56339365167195e90da7fe24726b6c85c8e not found: ID does not exist" containerID="766fd28b174a193dfc46e254f315f56339365167195e90da7fe24726b6c85c8e"
Oct 31 08:06:41.680440 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:06:41.680383    4720 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"766fd28b174a193dfc46e254f315f56339365167195e90da7fe24726b6c85c8e"} err="failed to get container status \"766fd28b174a193dfc46e254f315f56339365167195e90da7fe24726b6c85c8e\": rpc error: code = NotFound desc = could not find container \"766fd28b174a193dfc46e254f315f56339365167195e90da7fe24726b6c85c8e\": container with ID starting with 766fd28b174a193dfc46e254f315f56339365167195e90da7fe24726b6c85c8e not found: ID does not exist"
Oct 31 08:06:41.824634 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:06:41.824551    4720 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-679758fddf-hh2nc"]
Oct 31 08:06:41.873850 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:06:41.873813    4720 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-679758fddf-hh2nc"]
Oct 31 08:06:42.026193 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:06:42.026159    4720 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="94456f54-4f33-4f9c-9e5a-1125d3955ef7" path="/var/lib/kubelet/pods/94456f54-4f33-4f9c-9e5a-1125d3955ef7/volumes"
Oct 31 08:06:43.461852 ip-10-0-139-83 kubenswrapper[4720]: E1031 08:06:43.461810    4720 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8m768" containerName="extractor"
Oct 31 08:06:56.580666 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:06:56.580612    4720 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-5cd7c9f67f-plhh9" podUID="5947348e-e436-4e5e-8a0d-a14da00383fb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/health\": dial tcp 10.132.0.55:8000: connect: connection refused"
Oct 31 08:07:06.588537 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:07:06.588506    4720 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-5cd7c9f67f-plhh9"
Oct 31 08:07:10.642915 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:07:10.642884    4720 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-312caf2d_tokenizer-migration-path-test-tokenizer-5cd7c9f67f-plhh9_5947348e-e436-4e5e-8a0d-a14da00383fb/main/0.log"
Oct 31 08:07:13.276712 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:07:13.276679    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 08:07:13.461441 ip-10-0-139-83 kubenswrapper[4720]: E1031 08:07:13.461399    4720 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8m768" containerName="extractor"
Oct 31 08:07:13.686636 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:07:13.686599    4720 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-8b767c9df-b5fpz"]
Oct 31 08:07:13.697683 ip-10-0-139-83 kubenswrapper[4720]: E1031 08:07:13.697633    4720 log.go:32] "ExecSync cmd from runtime service failed" err="rpc error: code = Unknown desc = command error: , stdout: , stderr: , exit code -1" containerID="9a9b6f36279510d18003f9dccbaf514f5d8c3df1e23352272081a978ed74cec1" cmd=["/bin/sleep","15"]
Oct 31 08:07:13.697876 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:07:13.697702    4720 handlers.go:85] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-8b767c9df-b5fpz" message="" err="rpc error: code = Unknown desc = command error: , stdout: , stderr: , exit code -1"
Oct 31 08:07:13.697876 ip-10-0-139-83 kubenswrapper[4720]: E1031 08:07:13.697717    4720 kuberuntime_container.go:780] "PreStop hook failed" err="rpc error: code = Unknown desc = command error: , stdout: , stderr: , exit code -1" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-8b767c9df-b5fpz" podUID="45fe1515-14e2-457d-afa2-92eb268580a6" containerName="main" containerID="cri-o://9a9b6f36279510d18003f9dccbaf514f5d8c3df1e23352272081a978ed74cec1"
Oct 31 08:07:13.697876 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:07:13.697767    4720 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-8b767c9df-b5fpz" podUID="45fe1515-14e2-457d-afa2-92eb268580a6" containerName="main" containerID="cri-o://9a9b6f36279510d18003f9dccbaf514f5d8c3df1e23352272081a978ed74cec1" gracePeriod=60
Oct 31 08:07:13.700573 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:07:13.700545    4720 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-5cd7c9f67f-plhh9"]
Oct 31 08:07:13.700797 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:07:13.700769    4720 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-5cd7c9f67f-plhh9" podUID="5947348e-e436-4e5e-8a0d-a14da00383fb" containerName="main" containerID="cri-o://9c8df5144c2c1cfb394891177ac626723ba6ed9f3826d3c77a3c6d2c3e0f06d6" gracePeriod=30
Oct 31 08:07:13.742524 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:07:13.742492    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 08:07:13.915603 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:07:13.915543    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 08:07:14.073030 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:07:14.073002    4720 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-8b767c9df-b5fpz"
Oct 31 08:07:14.133750 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:07:14.133726    4720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/45fe1515-14e2-457d-afa2-92eb268580a6-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/45fe1515-14e2-457d-afa2-92eb268580a6-cabundle-cert\") pod \"45fe1515-14e2-457d-afa2-92eb268580a6\" (UID: \"45fe1515-14e2-457d-afa2-92eb268580a6\") "
Oct 31 08:07:14.133871 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:07:14.133771    4720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/45fe1515-14e2-457d-afa2-92eb268580a6-dshm\" (UniqueName: \"kubernetes.io/empty-dir/45fe1515-14e2-457d-afa2-92eb268580a6-dshm\") pod \"45fe1515-14e2-457d-afa2-92eb268580a6\" (UID: \"45fe1515-14e2-457d-afa2-92eb268580a6\") "
Oct 31 08:07:14.133871 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:07:14.133853    4720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/45fe1515-14e2-457d-afa2-92eb268580a6-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/45fe1515-14e2-457d-afa2-92eb268580a6-tmp-dir\") pod \"45fe1515-14e2-457d-afa2-92eb268580a6\" (UID: \"45fe1515-14e2-457d-afa2-92eb268580a6\") "
Oct 31 08:07:14.133982 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:07:14.133872    4720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/45fe1515-14e2-457d-afa2-92eb268580a6-home\" (UniqueName: \"kubernetes.io/empty-dir/45fe1515-14e2-457d-afa2-92eb268580a6-home\") pod \"45fe1515-14e2-457d-afa2-92eb268580a6\" (UID: \"45fe1515-14e2-457d-afa2-92eb268580a6\") "
Oct 31 08:07:14.133982 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:07:14.133893    4720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/45fe1515-14e2-457d-afa2-92eb268580a6-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/45fe1515-14e2-457d-afa2-92eb268580a6-kserve-provision-location\") pod \"45fe1515-14e2-457d-afa2-92eb268580a6\" (UID: \"45fe1515-14e2-457d-afa2-92eb268580a6\") "
Oct 31 08:07:14.133982 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:07:14.133941    4720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/45fe1515-14e2-457d-afa2-92eb268580a6-kube-api-access-w7gzh\" (UniqueName: \"kubernetes.io/projected/45fe1515-14e2-457d-afa2-92eb268580a6-kube-api-access-w7gzh\") pod \"45fe1515-14e2-457d-afa2-92eb268580a6\" (UID: \"45fe1515-14e2-457d-afa2-92eb268580a6\") "
Oct 31 08:07:14.133982 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:07:14.133976    4720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/45fe1515-14e2-457d-afa2-92eb268580a6-tls-certs\" (UniqueName: \"kubernetes.io/secret/45fe1515-14e2-457d-afa2-92eb268580a6-tls-certs\") pod \"45fe1515-14e2-457d-afa2-92eb268580a6\" (UID: \"45fe1515-14e2-457d-afa2-92eb268580a6\") "
Oct 31 08:07:14.134187 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:07:14.134042    4720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/45fe1515-14e2-457d-afa2-92eb268580a6-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/45fe1515-14e2-457d-afa2-92eb268580a6-model-cache\") pod \"45fe1515-14e2-457d-afa2-92eb268580a6\" (UID: \"45fe1515-14e2-457d-afa2-92eb268580a6\") "
Oct 31 08:07:14.134187 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:07:14.134089    4720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/45fe1515-14e2-457d-afa2-92eb268580a6-home" pod "45fe1515-14e2-457d-afa2-92eb268580a6" (UID: "45fe1515-14e2-457d-afa2-92eb268580a6"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 08:07:14.134187 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:07:14.134147    4720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/45fe1515-14e2-457d-afa2-92eb268580a6-tmp-dir" pod "45fe1515-14e2-457d-afa2-92eb268580a6" (UID: "45fe1515-14e2-457d-afa2-92eb268580a6"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 08:07:14.134372 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:07:14.134207    4720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/45fe1515-14e2-457d-afa2-92eb268580a6-cabundle-cert" pod "45fe1515-14e2-457d-afa2-92eb268580a6" (UID: "45fe1515-14e2-457d-afa2-92eb268580a6"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 08:07:14.134431 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:07:14.134390    4720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/45fe1515-14e2-457d-afa2-92eb268580a6-model-cache" pod "45fe1515-14e2-457d-afa2-92eb268580a6" (UID: "45fe1515-14e2-457d-afa2-92eb268580a6"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 08:07:14.134527 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:07:14.134506    4720 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/45fe1515-14e2-457d-afa2-92eb268580a6-cabundle-cert\") on node \"ip-10-0-139-83.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 08:07:14.134746 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:07:14.134532    4720 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/45fe1515-14e2-457d-afa2-92eb268580a6-tmp-dir\") on node \"ip-10-0-139-83.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 08:07:14.134746 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:07:14.134548    4720 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/45fe1515-14e2-457d-afa2-92eb268580a6-home\") on node \"ip-10-0-139-83.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 08:07:14.134746 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:07:14.134561    4720 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/45fe1515-14e2-457d-afa2-92eb268580a6-model-cache\") on node \"ip-10-0-139-83.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 08:07:14.149776 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:07:14.149740    4720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/45fe1515-14e2-457d-afa2-92eb268580a6-dshm" pod "45fe1515-14e2-457d-afa2-92eb268580a6" (UID: "45fe1515-14e2-457d-afa2-92eb268580a6"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 08:07:14.153833 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:07:14.153808    4720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/45fe1515-14e2-457d-afa2-92eb268580a6-kserve-provision-location" pod "45fe1515-14e2-457d-afa2-92eb268580a6" (UID: "45fe1515-14e2-457d-afa2-92eb268580a6"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 08:07:14.159761 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:07:14.159732    4720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/45fe1515-14e2-457d-afa2-92eb268580a6-kube-api-access-w7gzh" pod "45fe1515-14e2-457d-afa2-92eb268580a6" (UID: "45fe1515-14e2-457d-afa2-92eb268580a6"). InnerVolumeSpecName "kube-api-access-w7gzh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 08:07:14.160948 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:07:14.160925    4720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/45fe1515-14e2-457d-afa2-92eb268580a6-tls-certs" pod "45fe1515-14e2-457d-afa2-92eb268580a6" (UID: "45fe1515-14e2-457d-afa2-92eb268580a6"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 08:07:14.235354 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:07:14.235259    4720 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/45fe1515-14e2-457d-afa2-92eb268580a6-dshm\") on node \"ip-10-0-139-83.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 08:07:14.235354 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:07:14.235288    4720 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/45fe1515-14e2-457d-afa2-92eb268580a6-kserve-provision-location\") on node \"ip-10-0-139-83.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 08:07:14.235354 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:07:14.235305    4720 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w7gzh\" (UniqueName: \"kubernetes.io/projected/45fe1515-14e2-457d-afa2-92eb268580a6-kube-api-access-w7gzh\") on node \"ip-10-0-139-83.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 08:07:14.235354 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:07:14.235314    4720 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/45fe1515-14e2-457d-afa2-92eb268580a6-tls-certs\") on node \"ip-10-0-139-83.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 08:07:14.349912 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:07:14.349882    4720 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-5cd7c9f67f-plhh9"
Oct 31 08:07:14.436841 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:07:14.436805    4720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5947348e-e436-4e5e-8a0d-a14da00383fb-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5947348e-e436-4e5e-8a0d-a14da00383fb-cabundle-cert\") pod \"5947348e-e436-4e5e-8a0d-a14da00383fb\" (UID: \"5947348e-e436-4e5e-8a0d-a14da00383fb\") "
Oct 31 08:07:14.437020 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:07:14.436950    4720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5947348e-e436-4e5e-8a0d-a14da00383fb-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5947348e-e436-4e5e-8a0d-a14da00383fb-kserve-provision-location\") pod \"5947348e-e436-4e5e-8a0d-a14da00383fb\" (UID: \"5947348e-e436-4e5e-8a0d-a14da00383fb\") "
Oct 31 08:07:14.437020 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:07:14.436977    4720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5947348e-e436-4e5e-8a0d-a14da00383fb-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5947348e-e436-4e5e-8a0d-a14da00383fb-tmp-dir\") pod \"5947348e-e436-4e5e-8a0d-a14da00383fb\" (UID: \"5947348e-e436-4e5e-8a0d-a14da00383fb\") "
Oct 31 08:07:14.437124 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:07:14.437027    4720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5947348e-e436-4e5e-8a0d-a14da00383fb-tls-certs\" (UniqueName: \"kubernetes.io/secret/5947348e-e436-4e5e-8a0d-a14da00383fb-tls-certs\") pod \"5947348e-e436-4e5e-8a0d-a14da00383fb\" (UID: \"5947348e-e436-4e5e-8a0d-a14da00383fb\") "
Oct 31 08:07:14.437291 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:07:14.437188    4720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5947348e-e436-4e5e-8a0d-a14da00383fb-cabundle-cert" pod "5947348e-e436-4e5e-8a0d-a14da00383fb" (UID: "5947348e-e436-4e5e-8a0d-a14da00383fb"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 08:07:14.437431 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:07:14.437408    4720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5947348e-e436-4e5e-8a0d-a14da00383fb-tmp-dir" pod "5947348e-e436-4e5e-8a0d-a14da00383fb" (UID: "5947348e-e436-4e5e-8a0d-a14da00383fb"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 08:07:14.439558 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:07:14.439534    4720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5947348e-e436-4e5e-8a0d-a14da00383fb-tls-certs" pod "5947348e-e436-4e5e-8a0d-a14da00383fb" (UID: "5947348e-e436-4e5e-8a0d-a14da00383fb"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 08:07:14.459666 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:07:14.459629    4720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5947348e-e436-4e5e-8a0d-a14da00383fb-kserve-provision-location" pod "5947348e-e436-4e5e-8a0d-a14da00383fb" (UID: "5947348e-e436-4e5e-8a0d-a14da00383fb"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 08:07:14.538574 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:07:14.538495    4720 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5947348e-e436-4e5e-8a0d-a14da00383fb-tls-certs\") on node \"ip-10-0-139-83.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 08:07:14.538574 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:07:14.538523    4720 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5947348e-e436-4e5e-8a0d-a14da00383fb-cabundle-cert\") on node \"ip-10-0-139-83.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 08:07:14.538574 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:07:14.538533    4720 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5947348e-e436-4e5e-8a0d-a14da00383fb-kserve-provision-location\") on node \"ip-10-0-139-83.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 08:07:14.538574 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:07:14.538542    4720 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5947348e-e436-4e5e-8a0d-a14da00383fb-tmp-dir\") on node \"ip-10-0-139-83.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 08:07:14.832965 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:07:14.832931    4720 generic.go:356] "Generic (PLEG): container finished" podID="5947348e-e436-4e5e-8a0d-a14da00383fb" containerID="9c8df5144c2c1cfb394891177ac626723ba6ed9f3826d3c77a3c6d2c3e0f06d6" exitCode=0
Oct 31 08:07:14.833158 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:07:14.832994    4720 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-5cd7c9f67f-plhh9"
Oct 31 08:07:14.833158 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:07:14.833017    4720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-5cd7c9f67f-plhh9" event={"ID":"5947348e-e436-4e5e-8a0d-a14da00383fb","Type":"ContainerDied","Data":"9c8df5144c2c1cfb394891177ac626723ba6ed9f3826d3c77a3c6d2c3e0f06d6"}
Oct 31 08:07:14.833158 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:07:14.833124    4720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-5cd7c9f67f-plhh9" event={"ID":"5947348e-e436-4e5e-8a0d-a14da00383fb","Type":"ContainerDied","Data":"81ccf2a49442f1c6bb52e862e2b55064df9c286f78b5b396326cd9f0c40ffb4c"}
Oct 31 08:07:14.833158 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:07:14.833154    4720 scope.go:122] "RemoveContainer" containerID="9c8df5144c2c1cfb394891177ac626723ba6ed9f3826d3c77a3c6d2c3e0f06d6"
Oct 31 08:07:14.834854 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:07:14.834820    4720 generic.go:356] "Generic (PLEG): container finished" podID="45fe1515-14e2-457d-afa2-92eb268580a6" containerID="9a9b6f36279510d18003f9dccbaf514f5d8c3df1e23352272081a978ed74cec1" exitCode=0
Oct 31 08:07:14.834962 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:07:14.834901    4720 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-8b767c9df-b5fpz"
Oct 31 08:07:14.834962 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:07:14.834915    4720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-8b767c9df-b5fpz" event={"ID":"45fe1515-14e2-457d-afa2-92eb268580a6","Type":"ContainerDied","Data":"9a9b6f36279510d18003f9dccbaf514f5d8c3df1e23352272081a978ed74cec1"}
Oct 31 08:07:14.835057 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:07:14.834961    4720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-8b767c9df-b5fpz" event={"ID":"45fe1515-14e2-457d-afa2-92eb268580a6","Type":"ContainerDied","Data":"d4993c0978698195fa101d7d5a7338daa8c32ec916173ed286717b2a4a803c8f"}
Oct 31 08:07:14.843309 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:07:14.843288    4720 scope.go:122] "RemoveContainer" containerID="acffef2c58e30befffd1046682a07abb9095ae5736d78089874d947d10dd6921"
Oct 31 08:07:14.852815 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:07:14.852790    4720 scope.go:122] "RemoveContainer" containerID="9c8df5144c2c1cfb394891177ac626723ba6ed9f3826d3c77a3c6d2c3e0f06d6"
Oct 31 08:07:14.853130 ip-10-0-139-83 kubenswrapper[4720]: E1031 08:07:14.853109    4720 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9c8df5144c2c1cfb394891177ac626723ba6ed9f3826d3c77a3c6d2c3e0f06d6\": container with ID starting with 9c8df5144c2c1cfb394891177ac626723ba6ed9f3826d3c77a3c6d2c3e0f06d6 not found: ID does not exist" containerID="9c8df5144c2c1cfb394891177ac626723ba6ed9f3826d3c77a3c6d2c3e0f06d6"
Oct 31 08:07:14.853181 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:07:14.853140    4720 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9c8df5144c2c1cfb394891177ac626723ba6ed9f3826d3c77a3c6d2c3e0f06d6"} err="failed to get container status \"9c8df5144c2c1cfb394891177ac626723ba6ed9f3826d3c77a3c6d2c3e0f06d6\": rpc error: code = NotFound desc = could not find container \"9c8df5144c2c1cfb394891177ac626723ba6ed9f3826d3c77a3c6d2c3e0f06d6\": container with ID starting with 9c8df5144c2c1cfb394891177ac626723ba6ed9f3826d3c77a3c6d2c3e0f06d6 not found: ID does not exist"
Oct 31 08:07:14.853181 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:07:14.853161    4720 scope.go:122] "RemoveContainer" containerID="acffef2c58e30befffd1046682a07abb9095ae5736d78089874d947d10dd6921"
Oct 31 08:07:14.853483 ip-10-0-139-83 kubenswrapper[4720]: E1031 08:07:14.853463    4720 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"acffef2c58e30befffd1046682a07abb9095ae5736d78089874d947d10dd6921\": container with ID starting with acffef2c58e30befffd1046682a07abb9095ae5736d78089874d947d10dd6921 not found: ID does not exist" containerID="acffef2c58e30befffd1046682a07abb9095ae5736d78089874d947d10dd6921"
Oct 31 08:07:14.853549 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:07:14.853489    4720 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"acffef2c58e30befffd1046682a07abb9095ae5736d78089874d947d10dd6921"} err="failed to get container status \"acffef2c58e30befffd1046682a07abb9095ae5736d78089874d947d10dd6921\": rpc error: code = NotFound desc = could not find container \"acffef2c58e30befffd1046682a07abb9095ae5736d78089874d947d10dd6921\": container with ID starting with acffef2c58e30befffd1046682a07abb9095ae5736d78089874d947d10dd6921 not found: ID does not exist"
Oct 31 08:07:14.853549 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:07:14.853506    4720 scope.go:122] "RemoveContainer" containerID="9a9b6f36279510d18003f9dccbaf514f5d8c3df1e23352272081a978ed74cec1"
Oct 31 08:07:14.862491 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:07:14.862470    4720 scope.go:122] "RemoveContainer" containerID="ca192d087bb88c6a137e17558df67f0455116863f58826318ff221d70dbc2608"
Oct 31 08:07:14.870725 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:07:14.870703    4720 scope.go:122] "RemoveContainer" containerID="9a9b6f36279510d18003f9dccbaf514f5d8c3df1e23352272081a978ed74cec1"
Oct 31 08:07:14.871026 ip-10-0-139-83 kubenswrapper[4720]: E1031 08:07:14.871004    4720 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9a9b6f36279510d18003f9dccbaf514f5d8c3df1e23352272081a978ed74cec1\": container with ID starting with 9a9b6f36279510d18003f9dccbaf514f5d8c3df1e23352272081a978ed74cec1 not found: ID does not exist" containerID="9a9b6f36279510d18003f9dccbaf514f5d8c3df1e23352272081a978ed74cec1"
Oct 31 08:07:14.871107 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:07:14.871034    4720 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9a9b6f36279510d18003f9dccbaf514f5d8c3df1e23352272081a978ed74cec1"} err="failed to get container status \"9a9b6f36279510d18003f9dccbaf514f5d8c3df1e23352272081a978ed74cec1\": rpc error: code = NotFound desc = could not find container \"9a9b6f36279510d18003f9dccbaf514f5d8c3df1e23352272081a978ed74cec1\": container with ID starting with 9a9b6f36279510d18003f9dccbaf514f5d8c3df1e23352272081a978ed74cec1 not found: ID does not exist"
Oct 31 08:07:14.871107 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:07:14.871053    4720 scope.go:122] "RemoveContainer" containerID="ca192d087bb88c6a137e17558df67f0455116863f58826318ff221d70dbc2608"
Oct 31 08:07:14.871279 ip-10-0-139-83 kubenswrapper[4720]: E1031 08:07:14.871262    4720 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ca192d087bb88c6a137e17558df67f0455116863f58826318ff221d70dbc2608\": container with ID starting with ca192d087bb88c6a137e17558df67f0455116863f58826318ff221d70dbc2608 not found: ID does not exist" containerID="ca192d087bb88c6a137e17558df67f0455116863f58826318ff221d70dbc2608"
Oct 31 08:07:14.871358 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:07:14.871285    4720 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ca192d087bb88c6a137e17558df67f0455116863f58826318ff221d70dbc2608"} err="failed to get container status \"ca192d087bb88c6a137e17558df67f0455116863f58826318ff221d70dbc2608\": rpc error: code = NotFound desc = could not find container \"ca192d087bb88c6a137e17558df67f0455116863f58826318ff221d70dbc2608\": container with ID starting with ca192d087bb88c6a137e17558df67f0455116863f58826318ff221d70dbc2608 not found: ID does not exist"
Oct 31 08:07:14.994835 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:07:14.994800    4720 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-8b767c9df-b5fpz"]
Oct 31 08:07:14.998032 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:07:14.998010    4720 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-8b767c9df-b5fpz"]
Oct 31 08:07:15.107987 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:07:15.107949    4720 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-5cd7c9f67f-plhh9"]
Oct 31 08:07:15.111929 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:07:15.111899    4720 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-5cd7c9f67f-plhh9"]
Oct 31 08:07:16.027042 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:07:16.027010    4720 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="45fe1515-14e2-457d-afa2-92eb268580a6" path="/var/lib/kubelet/pods/45fe1515-14e2-457d-afa2-92eb268580a6/volumes"
Oct 31 08:07:16.027471 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:07:16.027458    4720 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5947348e-e436-4e5e-8a0d-a14da00383fb" path="/var/lib/kubelet/pods/5947348e-e436-4e5e-8a0d-a14da00383fb/volumes"
Oct 31 08:07:19.909672 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:07:19.909637    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 08:07:25.363548 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:07:25.363511    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 08:07:25.514409 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:07:25.514375    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 08:07:26.506043 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:07:26.506008    4720 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-597d55d8b8-vhlt6"]
Oct 31 08:07:26.506458 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:07:26.506408    4720 cpu_manager.go:414] "containerMap: removing container" podUID="94456f54-4f33-4f9c-9e5a-1125d3955ef7" containerName="storage-initializer"
Oct 31 08:07:26.506458 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:07:26.506424    4720 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="94456f54-4f33-4f9c-9e5a-1125d3955ef7" containerName="storage-initializer"
Oct 31 08:07:26.506458 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:07:26.506434    4720 cpu_manager.go:414] "containerMap: removing container" podUID="45fe1515-14e2-457d-afa2-92eb268580a6" containerName="storage-initializer"
Oct 31 08:07:26.506458 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:07:26.506439    4720 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="45fe1515-14e2-457d-afa2-92eb268580a6" containerName="storage-initializer"
Oct 31 08:07:26.506458 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:07:26.506452    4720 cpu_manager.go:414] "containerMap: removing container" podUID="5947348e-e436-4e5e-8a0d-a14da00383fb" containerName="storage-initializer"
Oct 31 08:07:26.506458 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:07:26.506456    4720 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5947348e-e436-4e5e-8a0d-a14da00383fb" containerName="storage-initializer"
Oct 31 08:07:26.506653 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:07:26.506469    4720 cpu_manager.go:414] "containerMap: removing container" podUID="5947348e-e436-4e5e-8a0d-a14da00383fb" containerName="main"
Oct 31 08:07:26.506653 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:07:26.506474    4720 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5947348e-e436-4e5e-8a0d-a14da00383fb" containerName="main"
Oct 31 08:07:26.506653 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:07:26.506482    4720 cpu_manager.go:414] "containerMap: removing container" podUID="94456f54-4f33-4f9c-9e5a-1125d3955ef7" containerName="main"
Oct 31 08:07:26.506653 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:07:26.506488    4720 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="94456f54-4f33-4f9c-9e5a-1125d3955ef7" containerName="main"
Oct 31 08:07:26.506653 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:07:26.506499    4720 cpu_manager.go:414] "containerMap: removing container" podUID="45fe1515-14e2-457d-afa2-92eb268580a6" containerName="main"
Oct 31 08:07:26.506653 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:07:26.506504    4720 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="45fe1515-14e2-457d-afa2-92eb268580a6" containerName="main"
Oct 31 08:07:26.506653 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:07:26.506577    4720 memory_manager.go:365] "RemoveStaleState removing state" podUID="5947348e-e436-4e5e-8a0d-a14da00383fb" containerName="main"
Oct 31 08:07:26.506653 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:07:26.506586    4720 memory_manager.go:365] "RemoveStaleState removing state" podUID="94456f54-4f33-4f9c-9e5a-1125d3955ef7" containerName="main"
Oct 31 08:07:26.506653 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:07:26.506592    4720 memory_manager.go:365] "RemoveStaleState removing state" podUID="45fe1515-14e2-457d-afa2-92eb268580a6" containerName="main"
Oct 31 08:07:26.509464 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:07:26.509446    4720 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-597d55d8b8-vhlt6"
Oct 31 08:07:26.562967 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:07:26.562932    4720 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-597d55d8b8-vhlt6"]
Oct 31 08:07:26.567632 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:07:26.567604    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 08:07:26.610359 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:07:26.610294    4720 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-disabled-no-token-required-396e4821\"/\"kube-root-ca.crt\""
Oct 31 08:07:26.610359 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:07:26.610305    4720 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-disabled-no-token-required-396e4821\"/\"odh-kserve-custom-ca-bundle\""
Oct 31 08:07:26.610626 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:07:26.610314    4720 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-disabled-no-token-required-396e4821\"/\"seaweedfs-s3-creds\""
Oct 31 08:07:26.610626 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:07:26.610360    4720 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-disabled-no-token-required-396e4821\"/\"openshift-service-ca.crt\""
Oct 31 08:07:26.610626 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:07:26.610446    4720 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-disabled-no-token-required-396e4821\"/\"default-dockercfg-hzqmp\""
Oct 31 08:07:26.610626 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:07:26.610531    4720 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-disabled-no-token-required-396e4821\"/\"auth-disabled-test-kserve-self-signed-certs\""
Oct 31 08:07:26.648861 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:07:26.648819    4720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8gwzh\" (UniqueName: \"kubernetes.io/projected/6c76be18-e721-4f19-8378-5dd0b1c0d265-kube-api-access-8gwzh\") pod \"auth-disabled-test-kserve-597d55d8b8-vhlt6\" (UID: \"6c76be18-e721-4f19-8378-5dd0b1c0d265\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-597d55d8b8-vhlt6"
Oct 31 08:07:26.648861 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:07:26.648867    4720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6c76be18-e721-4f19-8378-5dd0b1c0d265-dshm\") pod \"auth-disabled-test-kserve-597d55d8b8-vhlt6\" (UID: \"6c76be18-e721-4f19-8378-5dd0b1c0d265\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-597d55d8b8-vhlt6"
Oct 31 08:07:26.649083 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:07:26.648945    4720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6c76be18-e721-4f19-8378-5dd0b1c0d265-tmp-dir\") pod \"auth-disabled-test-kserve-597d55d8b8-vhlt6\" (UID: \"6c76be18-e721-4f19-8378-5dd0b1c0d265\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-597d55d8b8-vhlt6"
Oct 31 08:07:26.649083 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:07:26.648997    4720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6c76be18-e721-4f19-8378-5dd0b1c0d265-kserve-provision-location\") pod \"auth-disabled-test-kserve-597d55d8b8-vhlt6\" (UID: \"6c76be18-e721-4f19-8378-5dd0b1c0d265\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-597d55d8b8-vhlt6"
Oct 31 08:07:26.649083 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:07:26.649022    4720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6c76be18-e721-4f19-8378-5dd0b1c0d265-tls-certs\") pod \"auth-disabled-test-kserve-597d55d8b8-vhlt6\" (UID: \"6c76be18-e721-4f19-8378-5dd0b1c0d265\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-597d55d8b8-vhlt6"
Oct 31 08:07:26.649083 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:07:26.649044    4720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6c76be18-e721-4f19-8378-5dd0b1c0d265-home\") pod \"auth-disabled-test-kserve-597d55d8b8-vhlt6\" (UID: \"6c76be18-e721-4f19-8378-5dd0b1c0d265\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-597d55d8b8-vhlt6"
Oct 31 08:07:26.649207 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:07:26.649136    4720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6c76be18-e721-4f19-8378-5dd0b1c0d265-model-cache\") pod \"auth-disabled-test-kserve-597d55d8b8-vhlt6\" (UID: \"6c76be18-e721-4f19-8378-5dd0b1c0d265\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-597d55d8b8-vhlt6"
Oct 31 08:07:26.649207 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:07:26.649164    4720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6c76be18-e721-4f19-8378-5dd0b1c0d265-cabundle-cert\") pod \"auth-disabled-test-kserve-597d55d8b8-vhlt6\" (UID: \"6c76be18-e721-4f19-8378-5dd0b1c0d265\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-597d55d8b8-vhlt6"
Oct 31 08:07:26.750008 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:07:26.749974    4720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6c76be18-e721-4f19-8378-5dd0b1c0d265-dshm\") pod \"auth-disabled-test-kserve-597d55d8b8-vhlt6\" (UID: \"6c76be18-e721-4f19-8378-5dd0b1c0d265\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-597d55d8b8-vhlt6"
Oct 31 08:07:26.750008 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:07:26.750011    4720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6c76be18-e721-4f19-8378-5dd0b1c0d265-tmp-dir\") pod \"auth-disabled-test-kserve-597d55d8b8-vhlt6\" (UID: \"6c76be18-e721-4f19-8378-5dd0b1c0d265\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-597d55d8b8-vhlt6"
Oct 31 08:07:26.750292 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:07:26.750029    4720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6c76be18-e721-4f19-8378-5dd0b1c0d265-kserve-provision-location\") pod \"auth-disabled-test-kserve-597d55d8b8-vhlt6\" (UID: \"6c76be18-e721-4f19-8378-5dd0b1c0d265\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-597d55d8b8-vhlt6"
Oct 31 08:07:26.750292 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:07:26.750046    4720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6c76be18-e721-4f19-8378-5dd0b1c0d265-tls-certs\") pod \"auth-disabled-test-kserve-597d55d8b8-vhlt6\" (UID: \"6c76be18-e721-4f19-8378-5dd0b1c0d265\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-597d55d8b8-vhlt6"
Oct 31 08:07:26.750292 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:07:26.750062    4720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6c76be18-e721-4f19-8378-5dd0b1c0d265-home\") pod \"auth-disabled-test-kserve-597d55d8b8-vhlt6\" (UID: \"6c76be18-e721-4f19-8378-5dd0b1c0d265\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-597d55d8b8-vhlt6"
Oct 31 08:07:26.750292 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:07:26.750142    4720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6c76be18-e721-4f19-8378-5dd0b1c0d265-model-cache\") pod \"auth-disabled-test-kserve-597d55d8b8-vhlt6\" (UID: \"6c76be18-e721-4f19-8378-5dd0b1c0d265\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-597d55d8b8-vhlt6"
Oct 31 08:07:26.750292 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:07:26.750194    4720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6c76be18-e721-4f19-8378-5dd0b1c0d265-cabundle-cert\") pod \"auth-disabled-test-kserve-597d55d8b8-vhlt6\" (UID: \"6c76be18-e721-4f19-8378-5dd0b1c0d265\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-597d55d8b8-vhlt6"
Oct 31 08:07:26.750292 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:07:26.750276    4720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8gwzh\" (UniqueName: \"kubernetes.io/projected/6c76be18-e721-4f19-8378-5dd0b1c0d265-kube-api-access-8gwzh\") pod \"auth-disabled-test-kserve-597d55d8b8-vhlt6\" (UID: \"6c76be18-e721-4f19-8378-5dd0b1c0d265\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-597d55d8b8-vhlt6"
Oct 31 08:07:26.750647 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:07:26.750563    4720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6c76be18-e721-4f19-8378-5dd0b1c0d265-kserve-provision-location\") pod \"auth-disabled-test-kserve-597d55d8b8-vhlt6\" (UID: \"6c76be18-e721-4f19-8378-5dd0b1c0d265\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-597d55d8b8-vhlt6"
Oct 31 08:07:26.750647 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:07:26.750588    4720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6c76be18-e721-4f19-8378-5dd0b1c0d265-home\") pod \"auth-disabled-test-kserve-597d55d8b8-vhlt6\" (UID: \"6c76be18-e721-4f19-8378-5dd0b1c0d265\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-597d55d8b8-vhlt6"
Oct 31 08:07:26.750647 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:07:26.750642    4720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6c76be18-e721-4f19-8378-5dd0b1c0d265-tmp-dir\") pod \"auth-disabled-test-kserve-597d55d8b8-vhlt6\" (UID: \"6c76be18-e721-4f19-8378-5dd0b1c0d265\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-597d55d8b8-vhlt6"
Oct 31 08:07:26.750944 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:07:26.750787    4720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6c76be18-e721-4f19-8378-5dd0b1c0d265-model-cache\") pod \"auth-disabled-test-kserve-597d55d8b8-vhlt6\" (UID: \"6c76be18-e721-4f19-8378-5dd0b1c0d265\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-597d55d8b8-vhlt6"
Oct 31 08:07:26.751137 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:07:26.751117    4720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6c76be18-e721-4f19-8378-5dd0b1c0d265-cabundle-cert\") pod \"auth-disabled-test-kserve-597d55d8b8-vhlt6\" (UID: \"6c76be18-e721-4f19-8378-5dd0b1c0d265\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-597d55d8b8-vhlt6"
Oct 31 08:07:26.752596 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:07:26.752565    4720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6c76be18-e721-4f19-8378-5dd0b1c0d265-dshm\") pod \"auth-disabled-test-kserve-597d55d8b8-vhlt6\" (UID: \"6c76be18-e721-4f19-8378-5dd0b1c0d265\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-597d55d8b8-vhlt6"
Oct 31 08:07:26.752839 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:07:26.752824    4720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6c76be18-e721-4f19-8378-5dd0b1c0d265-tls-certs\") pod \"auth-disabled-test-kserve-597d55d8b8-vhlt6\" (UID: \"6c76be18-e721-4f19-8378-5dd0b1c0d265\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-597d55d8b8-vhlt6"
Oct 31 08:07:26.806596 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:07:26.806521    4720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8gwzh\" (UniqueName: \"kubernetes.io/projected/6c76be18-e721-4f19-8378-5dd0b1c0d265-kube-api-access-8gwzh\") pod \"auth-disabled-test-kserve-597d55d8b8-vhlt6\" (UID: \"6c76be18-e721-4f19-8378-5dd0b1c0d265\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-597d55d8b8-vhlt6"
Oct 31 08:07:26.826348 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:07:26.826303    4720 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-597d55d8b8-vhlt6"
Oct 31 08:07:26.928278 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:07:26.928238    4720 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-5d987cbbfd-b2zlm"]
Oct 31 08:07:26.930923 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:07:26.930903    4720 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-5d987cbbfd-b2zlm"
Oct 31 08:07:26.982593 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:07:26.982551    4720 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-disabled-no-token-required-396e4821\"/\"auth-disabled-test-epp-sa-dockercfg-578bk\""
Oct 31 08:07:26.987908 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:07:26.987878    4720 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-5d987cbbfd-b2zlm"]
Oct 31 08:07:27.029915 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:07:27.029856    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 08:07:27.052666 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:07:27.052625    4720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/50e53f44-4485-40c5-bcb2-10ccca40c7c4-tls-certs\") pod \"auth-disabled-test-kserve-router-scheduler-5d987cbbfd-b2zlm\" (UID: \"50e53f44-4485-40c5-bcb2-10ccca40c7c4\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-5d987cbbfd-b2zlm"
Oct 31 08:07:27.052835 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:07:27.052671    4720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-csxfl\" (UniqueName: \"kubernetes.io/projected/50e53f44-4485-40c5-bcb2-10ccca40c7c4-kube-api-access-csxfl\") pod \"auth-disabled-test-kserve-router-scheduler-5d987cbbfd-b2zlm\" (UID: \"50e53f44-4485-40c5-bcb2-10ccca40c7c4\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-5d987cbbfd-b2zlm"
Oct 31 08:07:27.153892 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:07:27.153843    4720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/50e53f44-4485-40c5-bcb2-10ccca40c7c4-tls-certs\") pod \"auth-disabled-test-kserve-router-scheduler-5d987cbbfd-b2zlm\" (UID: \"50e53f44-4485-40c5-bcb2-10ccca40c7c4\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-5d987cbbfd-b2zlm"
Oct 31 08:07:27.154088 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:07:27.153905    4720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-csxfl\" (UniqueName: \"kubernetes.io/projected/50e53f44-4485-40c5-bcb2-10ccca40c7c4-kube-api-access-csxfl\") pod \"auth-disabled-test-kserve-router-scheduler-5d987cbbfd-b2zlm\" (UID: \"50e53f44-4485-40c5-bcb2-10ccca40c7c4\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-5d987cbbfd-b2zlm"
Oct 31 08:07:27.156710 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:07:27.156687    4720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/50e53f44-4485-40c5-bcb2-10ccca40c7c4-tls-certs\") pod \"auth-disabled-test-kserve-router-scheduler-5d987cbbfd-b2zlm\" (UID: \"50e53f44-4485-40c5-bcb2-10ccca40c7c4\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-5d987cbbfd-b2zlm"
Oct 31 08:07:27.210149 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:07:27.210115    4720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-csxfl\" (UniqueName: \"kubernetes.io/projected/50e53f44-4485-40c5-bcb2-10ccca40c7c4-kube-api-access-csxfl\") pod \"auth-disabled-test-kserve-router-scheduler-5d987cbbfd-b2zlm\" (UID: \"50e53f44-4485-40c5-bcb2-10ccca40c7c4\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-5d987cbbfd-b2zlm"
Oct 31 08:07:27.242982 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:07:27.242942    4720 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-5d987cbbfd-b2zlm"
Oct 31 08:07:27.255432 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:07:27.255400    4720 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-597d55d8b8-vhlt6"]
Oct 31 08:07:27.258971 ip-10-0-139-83 kubenswrapper[4720]: W1031 08:07:27.258940    4720 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6c76be18_e721_4f19_8378_5dd0b1c0d265.slice/crio-a2c28bcd0762beb31d93f544429dd27d09f22f02c282aa5d71ece9b884237152 WatchSource:0}: Error finding container a2c28bcd0762beb31d93f544429dd27d09f22f02c282aa5d71ece9b884237152: Status 404 returned error can't find the container with id a2c28bcd0762beb31d93f544429dd27d09f22f02c282aa5d71ece9b884237152
Oct 31 08:07:27.424093 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:07:27.423997    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 08:07:27.622901 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:07:27.622865    4720 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-5d987cbbfd-b2zlm"]
Oct 31 08:07:27.626407 ip-10-0-139-83 kubenswrapper[4720]: W1031 08:07:27.626375    4720 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod50e53f44_4485_40c5_bcb2_10ccca40c7c4.slice/crio-b1fd32aa4c24857469f13acff3d41330b2649a1d355dc3678c79c32f66565f13 WatchSource:0}: Error finding container b1fd32aa4c24857469f13acff3d41330b2649a1d355dc3678c79c32f66565f13: Status 404 returned error can't find the container with id b1fd32aa4c24857469f13acff3d41330b2649a1d355dc3678c79c32f66565f13
Oct 31 08:07:27.780937 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:07:27.780901    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 08:07:27.906665 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:07:27.906631    4720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-597d55d8b8-vhlt6" event={"ID":"6c76be18-e721-4f19-8378-5dd0b1c0d265","Type":"ContainerStarted","Data":"025b6b81db4529d4a9fff01ddeeb5ad1889a18a537ed67ad611c64c6bd7922d5"}
Oct 31 08:07:27.906859 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:07:27.906688    4720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-597d55d8b8-vhlt6" event={"ID":"6c76be18-e721-4f19-8378-5dd0b1c0d265","Type":"ContainerStarted","Data":"a2c28bcd0762beb31d93f544429dd27d09f22f02c282aa5d71ece9b884237152"}
Oct 31 08:07:27.908551 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:07:27.908524    4720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-5d987cbbfd-b2zlm" event={"ID":"50e53f44-4485-40c5-bcb2-10ccca40c7c4","Type":"ContainerStarted","Data":"4979c2390176b26b9bbe03c13e2d2ace5cdd863adfd2ffc220a8306015582819"}
Oct 31 08:07:27.908707 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:07:27.908569    4720 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-5d987cbbfd-b2zlm"
Oct 31 08:07:27.908707 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:07:27.908591    4720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-5d987cbbfd-b2zlm" event={"ID":"50e53f44-4485-40c5-bcb2-10ccca40c7c4","Type":"ContainerStarted","Data":"b1fd32aa4c24857469f13acff3d41330b2649a1d355dc3678c79c32f66565f13"}
Oct 31 08:07:28.135509 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:07:28.135456    4720 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-5d987cbbfd-b2zlm" podStartSLOduration=2.135440632 podStartE2EDuration="2.135440632s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 08:07:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 08:07:28.133547917 +0000 UTC m=+2302.264615999" watchObservedRunningTime="2026-10-31 08:07:28.135440632 +0000 UTC m=+2302.266508702"
Oct 31 08:07:29.919185 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:07:29.919151    4720 generic.go:356] "Generic (PLEG): container finished" podID="6c76be18-e721-4f19-8378-5dd0b1c0d265" containerID="025b6b81db4529d4a9fff01ddeeb5ad1889a18a537ed67ad611c64c6bd7922d5" exitCode=0
Oct 31 08:07:29.919630 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:07:29.919233    4720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-597d55d8b8-vhlt6" event={"ID":"6c76be18-e721-4f19-8378-5dd0b1c0d265","Type":"ContainerDied","Data":"025b6b81db4529d4a9fff01ddeeb5ad1889a18a537ed67ad611c64c6bd7922d5"}
Oct 31 08:07:30.926591 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:07:30.926548    4720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-597d55d8b8-vhlt6" event={"ID":"6c76be18-e721-4f19-8378-5dd0b1c0d265","Type":"ContainerStarted","Data":"e4da5962e17169d72706fb0eb5d9d0e02261f274b74328339ff704186e5084e4"}
Oct 31 08:07:31.142398 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:07:31.142320    4720 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-597d55d8b8-vhlt6" podStartSLOduration=2.79702868 podStartE2EDuration="5.142303679s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.345274999s" isStatefulPod=false podCreationTimestamp="2026-10-31 08:07:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 08:07:31.140971652 +0000 UTC m=+2305.272039725" watchObservedRunningTime="2026-10-31 08:07:31.142303679 +0000 UTC m=+2305.273371749"
Oct 31 08:07:31.478779 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:07:31.478697    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 08:07:31.827491 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:07:31.827397    4720 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-597d55d8b8-vhlt6"
Oct 31 08:07:36.827344 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:07:36.827294    4720 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-597d55d8b8-vhlt6"
Oct 31 08:07:36.829168 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:07:36.829141    4720 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-597d55d8b8-vhlt6" podUID="6c76be18-e721-4f19-8378-5dd0b1c0d265" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/health\": dial tcp 10.132.0.56:8000: connect: connection refused"
Oct 31 08:07:43.462162 ip-10-0-139-83 kubenswrapper[4720]: E1031 08:07:43.462111    4720 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8m768" containerName="extractor"
Oct 31 08:07:46.827581 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:07:46.827526    4720 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-597d55d8b8-vhlt6" podUID="6c76be18-e721-4f19-8378-5dd0b1c0d265" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/health\": dial tcp 10.132.0.56:8000: connect: connection refused"
Oct 31 08:07:56.826873 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:07:56.826810    4720 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-597d55d8b8-vhlt6" podUID="6c76be18-e721-4f19-8378-5dd0b1c0d265" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/health\": dial tcp 10.132.0.56:8000: connect: connection refused"
Oct 31 08:07:58.914900 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:07:58.914861    4720 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-5d987cbbfd-b2zlm"
Oct 31 08:08:01.444936 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:08:01.444899    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 08:08:06.827802 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:08:06.827758    4720 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-597d55d8b8-vhlt6" podUID="6c76be18-e721-4f19-8378-5dd0b1c0d265" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/health\": dial tcp 10.132.0.56:8000: connect: connection refused"
Oct 31 08:08:07.466319 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:08:07.466277    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 08:08:12.839029 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:08:12.838995    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 08:08:12.991884 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:08:12.991843    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 08:08:13.462123 ip-10-0-139-83 kubenswrapper[4720]: E1031 08:08:13.462078    4720 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8m768" containerName="extractor"
Oct 31 08:08:13.610146 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:08:13.610111    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 08:08:16.826888 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:08:16.826845    4720 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-597d55d8b8-vhlt6" podUID="6c76be18-e721-4f19-8378-5dd0b1c0d265" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/health\": dial tcp 10.132.0.56:8000: connect: connection refused"
Oct 31 08:08:26.827174 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:08:26.827115    4720 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-597d55d8b8-vhlt6" podUID="6c76be18-e721-4f19-8378-5dd0b1c0d265" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/health\": dial tcp 10.132.0.56:8000: connect: connection refused"
Oct 31 08:08:28.134493 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:08:28.134457    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 08:08:36.827443 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:08:36.827400    4720 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-597d55d8b8-vhlt6" podUID="6c76be18-e721-4f19-8378-5dd0b1c0d265" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/health\": dial tcp 10.132.0.56:8000: connect: connection refused"
Oct 31 08:08:43.463119 ip-10-0-139-83 kubenswrapper[4720]: E1031 08:08:43.463073    4720 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8m768" containerName="extractor"
Oct 31 08:08:46.827530 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:08:46.827437    4720 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-597d55d8b8-vhlt6" podUID="6c76be18-e721-4f19-8378-5dd0b1c0d265" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/health\": dial tcp 10.132.0.56:8000: connect: connection refused"
Oct 31 08:08:52.975456 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:08:52.975415    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 08:08:56.827342 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:08:56.827295    4720 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-597d55d8b8-vhlt6" podUID="6c76be18-e721-4f19-8378-5dd0b1c0d265" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/health\": dial tcp 10.132.0.56:8000: connect: connection refused"
Oct 31 08:09:06.237027 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:09:06.236915    4720 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-4jszp_b1db0e3b-e85d-4769-8012-cf4b6f740d9d/console-operator/2.log"
Oct 31 08:09:06.245121 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:09:06.245098    4720 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-4jszp_b1db0e3b-e85d-4769-8012-cf4b6f740d9d/console-operator/2.log"
Oct 31 08:09:06.827445 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:09:06.827394    4720 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-597d55d8b8-vhlt6" podUID="6c76be18-e721-4f19-8378-5dd0b1c0d265" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/health\": dial tcp 10.132.0.56:8000: connect: connection refused"
Oct 31 08:09:13.465137 ip-10-0-139-83 kubenswrapper[4720]: E1031 08:09:13.465081    4720 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8m768" containerName="extractor"
Oct 31 08:09:13.957128 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:09:13.957087    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 08:09:16.827538 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:09:16.827491    4720 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-597d55d8b8-vhlt6" podUID="6c76be18-e721-4f19-8378-5dd0b1c0d265" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/health\": dial tcp 10.132.0.56:8000: connect: connection refused"
Oct 31 08:09:26.837138 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:09:26.837099    4720 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-597d55d8b8-vhlt6"
Oct 31 08:09:26.845196 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:09:26.845158    4720 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-597d55d8b8-vhlt6"
Oct 31 08:09:28.467421 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:09:28.467383    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 08:09:35.384385 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:09:35.384342    4720 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-597d55d8b8-vhlt6"]
Oct 31 08:09:35.385960 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:09:35.385927    4720 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-5d987cbbfd-b2zlm"]
Oct 31 08:09:35.417970 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:09:35.417935    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 08:09:35.567080 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:09:35.567040    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 08:09:38.914372 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:09:38.914309    4720 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-5d987cbbfd-b2zlm" podUID="50e53f44-4485-40c5-bcb2-10ccca40c7c4" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 08:09:43.463174 ip-10-0-139-83 kubenswrapper[4720]: E1031 08:09:43.463133    4720 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8m768" containerName="extractor"
Oct 31 08:09:48.914581 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:09:48.914538    4720 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-5d987cbbfd-b2zlm" podUID="50e53f44-4485-40c5-bcb2-10ccca40c7c4" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 08:09:50.406190 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:09:50.406148    4720 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-5d987cbbfd-b2zlm" podUID="50e53f44-4485-40c5-bcb2-10ccca40c7c4" containerName="main" containerID="cri-o://4979c2390176b26b9bbe03c13e2d2ace5cdd863adfd2ffc220a8306015582819" gracePeriod=45
Oct 31 08:09:50.409132 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:09:50.409094    4720 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-597d55d8b8-vhlt6" podUID="6c76be18-e721-4f19-8378-5dd0b1c0d265" containerName="main" containerID="cri-o://e4da5962e17169d72706fb0eb5d9d0e02261f274b74328339ff704186e5084e4" gracePeriod=45
Oct 31 08:09:50.838428 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:09:50.838388    4720 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-597d55d8b8-vhlt6" podUID="6c76be18-e721-4f19-8378-5dd0b1c0d265" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/health\": dial tcp 10.132.0.56:8000: connect: connection refused"
Oct 31 08:09:51.165047 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:09:51.165018    4720 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-597d55d8b8-vhlt6"
Oct 31 08:09:51.301323 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:09:51.301280    4720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6c76be18-e721-4f19-8378-5dd0b1c0d265-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6c76be18-e721-4f19-8378-5dd0b1c0d265-model-cache\") pod \"6c76be18-e721-4f19-8378-5dd0b1c0d265\" (UID: \"6c76be18-e721-4f19-8378-5dd0b1c0d265\") "
Oct 31 08:09:51.301323 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:09:51.301380    4720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6c76be18-e721-4f19-8378-5dd0b1c0d265-tls-certs\" (UniqueName: \"kubernetes.io/secret/6c76be18-e721-4f19-8378-5dd0b1c0d265-tls-certs\") pod \"6c76be18-e721-4f19-8378-5dd0b1c0d265\" (UID: \"6c76be18-e721-4f19-8378-5dd0b1c0d265\") "
Oct 31 08:09:51.301663 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:09:51.301404    4720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6c76be18-e721-4f19-8378-5dd0b1c0d265-kube-api-access-8gwzh\" (UniqueName: \"kubernetes.io/projected/6c76be18-e721-4f19-8378-5dd0b1c0d265-kube-api-access-8gwzh\") pod \"6c76be18-e721-4f19-8378-5dd0b1c0d265\" (UID: \"6c76be18-e721-4f19-8378-5dd0b1c0d265\") "
Oct 31 08:09:51.301663 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:09:51.301477    4720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6c76be18-e721-4f19-8378-5dd0b1c0d265-dshm\" (UniqueName: \"kubernetes.io/empty-dir/6c76be18-e721-4f19-8378-5dd0b1c0d265-dshm\") pod \"6c76be18-e721-4f19-8378-5dd0b1c0d265\" (UID: \"6c76be18-e721-4f19-8378-5dd0b1c0d265\") "
Oct 31 08:09:51.301663 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:09:51.301493    4720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6c76be18-e721-4f19-8378-5dd0b1c0d265-home\" (UniqueName: \"kubernetes.io/empty-dir/6c76be18-e721-4f19-8378-5dd0b1c0d265-home\") pod \"6c76be18-e721-4f19-8378-5dd0b1c0d265\" (UID: \"6c76be18-e721-4f19-8378-5dd0b1c0d265\") "
Oct 31 08:09:51.301663 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:09:51.301552    4720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6c76be18-e721-4f19-8378-5dd0b1c0d265-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6c76be18-e721-4f19-8378-5dd0b1c0d265-kserve-provision-location\") pod \"6c76be18-e721-4f19-8378-5dd0b1c0d265\" (UID: \"6c76be18-e721-4f19-8378-5dd0b1c0d265\") "
Oct 31 08:09:51.301663 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:09:51.301585    4720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6c76be18-e721-4f19-8378-5dd0b1c0d265-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6c76be18-e721-4f19-8378-5dd0b1c0d265-cabundle-cert\") pod \"6c76be18-e721-4f19-8378-5dd0b1c0d265\" (UID: \"6c76be18-e721-4f19-8378-5dd0b1c0d265\") "
Oct 31 08:09:51.301663 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:09:51.301589    4720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c76be18-e721-4f19-8378-5dd0b1c0d265-model-cache" pod "6c76be18-e721-4f19-8378-5dd0b1c0d265" (UID: "6c76be18-e721-4f19-8378-5dd0b1c0d265"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 08:09:51.301663 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:09:51.301619    4720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6c76be18-e721-4f19-8378-5dd0b1c0d265-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6c76be18-e721-4f19-8378-5dd0b1c0d265-tmp-dir\") pod \"6c76be18-e721-4f19-8378-5dd0b1c0d265\" (UID: \"6c76be18-e721-4f19-8378-5dd0b1c0d265\") "
Oct 31 08:09:51.302062 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:09:51.301992    4720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6c76be18-e721-4f19-8378-5dd0b1c0d265-cabundle-cert" pod "6c76be18-e721-4f19-8378-5dd0b1c0d265" (UID: "6c76be18-e721-4f19-8378-5dd0b1c0d265"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 08:09:51.302062 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:09:51.302001    4720 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6c76be18-e721-4f19-8378-5dd0b1c0d265-model-cache\") on node \"ip-10-0-139-83.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 08:09:51.302644 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:09:51.302612    4720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c76be18-e721-4f19-8378-5dd0b1c0d265-home" pod "6c76be18-e721-4f19-8378-5dd0b1c0d265" (UID: "6c76be18-e721-4f19-8378-5dd0b1c0d265"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 08:09:51.304071 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:09:51.304035    4720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c76be18-e721-4f19-8378-5dd0b1c0d265-dshm" pod "6c76be18-e721-4f19-8378-5dd0b1c0d265" (UID: "6c76be18-e721-4f19-8378-5dd0b1c0d265"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 08:09:51.304177 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:09:51.304137    4720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6c76be18-e721-4f19-8378-5dd0b1c0d265-tls-certs" pod "6c76be18-e721-4f19-8378-5dd0b1c0d265" (UID: "6c76be18-e721-4f19-8378-5dd0b1c0d265"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 08:09:51.304418 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:09:51.304400    4720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6c76be18-e721-4f19-8378-5dd0b1c0d265-kube-api-access-8gwzh" pod "6c76be18-e721-4f19-8378-5dd0b1c0d265" (UID: "6c76be18-e721-4f19-8378-5dd0b1c0d265"). InnerVolumeSpecName "kube-api-access-8gwzh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 08:09:51.315424 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:09:51.315394    4720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c76be18-e721-4f19-8378-5dd0b1c0d265-tmp-dir" pod "6c76be18-e721-4f19-8378-5dd0b1c0d265" (UID: "6c76be18-e721-4f19-8378-5dd0b1c0d265"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 08:09:51.326108 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:09:51.326079    4720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c76be18-e721-4f19-8378-5dd0b1c0d265-kserve-provision-location" pod "6c76be18-e721-4f19-8378-5dd0b1c0d265" (UID: "6c76be18-e721-4f19-8378-5dd0b1c0d265"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 08:09:51.403210 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:09:51.403121    4720 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6c76be18-e721-4f19-8378-5dd0b1c0d265-dshm\") on node \"ip-10-0-139-83.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 08:09:51.403210 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:09:51.403158    4720 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6c76be18-e721-4f19-8378-5dd0b1c0d265-home\") on node \"ip-10-0-139-83.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 08:09:51.403210 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:09:51.403168    4720 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6c76be18-e721-4f19-8378-5dd0b1c0d265-kserve-provision-location\") on node \"ip-10-0-139-83.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 08:09:51.403210 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:09:51.403177    4720 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6c76be18-e721-4f19-8378-5dd0b1c0d265-cabundle-cert\") on node \"ip-10-0-139-83.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 08:09:51.403210 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:09:51.403190    4720 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6c76be18-e721-4f19-8378-5dd0b1c0d265-tmp-dir\") on node \"ip-10-0-139-83.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 08:09:51.403210 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:09:51.403198    4720 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6c76be18-e721-4f19-8378-5dd0b1c0d265-tls-certs\") on node \"ip-10-0-139-83.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 08:09:51.403210 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:09:51.403211    4720 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8gwzh\" (UniqueName: \"kubernetes.io/projected/6c76be18-e721-4f19-8378-5dd0b1c0d265-kube-api-access-8gwzh\") on node \"ip-10-0-139-83.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 08:09:51.641441 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:09:51.641405    4720 generic.go:356] "Generic (PLEG): container finished" podID="6c76be18-e721-4f19-8378-5dd0b1c0d265" containerID="e4da5962e17169d72706fb0eb5d9d0e02261f274b74328339ff704186e5084e4" exitCode=0
Oct 31 08:09:51.641879 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:09:51.641461    4720 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-597d55d8b8-vhlt6"
Oct 31 08:09:51.641879 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:09:51.641490    4720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-597d55d8b8-vhlt6" event={"ID":"6c76be18-e721-4f19-8378-5dd0b1c0d265","Type":"ContainerDied","Data":"e4da5962e17169d72706fb0eb5d9d0e02261f274b74328339ff704186e5084e4"}
Oct 31 08:09:51.641879 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:09:51.641563    4720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-597d55d8b8-vhlt6" event={"ID":"6c76be18-e721-4f19-8378-5dd0b1c0d265","Type":"ContainerDied","Data":"a2c28bcd0762beb31d93f544429dd27d09f22f02c282aa5d71ece9b884237152"}
Oct 31 08:09:51.641879 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:09:51.641588    4720 scope.go:122] "RemoveContainer" containerID="e4da5962e17169d72706fb0eb5d9d0e02261f274b74328339ff704186e5084e4"
Oct 31 08:09:51.651578 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:09:51.651559    4720 scope.go:122] "RemoveContainer" containerID="025b6b81db4529d4a9fff01ddeeb5ad1889a18a537ed67ad611c64c6bd7922d5"
Oct 31 08:09:51.660282 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:09:51.660263    4720 scope.go:122] "RemoveContainer" containerID="e4da5962e17169d72706fb0eb5d9d0e02261f274b74328339ff704186e5084e4"
Oct 31 08:09:51.660591 ip-10-0-139-83 kubenswrapper[4720]: E1031 08:09:51.660569    4720 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e4da5962e17169d72706fb0eb5d9d0e02261f274b74328339ff704186e5084e4\": container with ID starting with e4da5962e17169d72706fb0eb5d9d0e02261f274b74328339ff704186e5084e4 not found: ID does not exist" containerID="e4da5962e17169d72706fb0eb5d9d0e02261f274b74328339ff704186e5084e4"
Oct 31 08:09:51.660655 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:09:51.660602    4720 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e4da5962e17169d72706fb0eb5d9d0e02261f274b74328339ff704186e5084e4"} err="failed to get container status \"e4da5962e17169d72706fb0eb5d9d0e02261f274b74328339ff704186e5084e4\": rpc error: code = NotFound desc = could not find container \"e4da5962e17169d72706fb0eb5d9d0e02261f274b74328339ff704186e5084e4\": container with ID starting with e4da5962e17169d72706fb0eb5d9d0e02261f274b74328339ff704186e5084e4 not found: ID does not exist"
Oct 31 08:09:51.660655 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:09:51.660622    4720 scope.go:122] "RemoveContainer" containerID="025b6b81db4529d4a9fff01ddeeb5ad1889a18a537ed67ad611c64c6bd7922d5"
Oct 31 08:09:51.660855 ip-10-0-139-83 kubenswrapper[4720]: E1031 08:09:51.660836    4720 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"025b6b81db4529d4a9fff01ddeeb5ad1889a18a537ed67ad611c64c6bd7922d5\": container with ID starting with 025b6b81db4529d4a9fff01ddeeb5ad1889a18a537ed67ad611c64c6bd7922d5 not found: ID does not exist" containerID="025b6b81db4529d4a9fff01ddeeb5ad1889a18a537ed67ad611c64c6bd7922d5"
Oct 31 08:09:51.660890 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:09:51.660863    4720 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"025b6b81db4529d4a9fff01ddeeb5ad1889a18a537ed67ad611c64c6bd7922d5"} err="failed to get container status \"025b6b81db4529d4a9fff01ddeeb5ad1889a18a537ed67ad611c64c6bd7922d5\": rpc error: code = NotFound desc = could not find container \"025b6b81db4529d4a9fff01ddeeb5ad1889a18a537ed67ad611c64c6bd7922d5\": container with ID starting with 025b6b81db4529d4a9fff01ddeeb5ad1889a18a537ed67ad611c64c6bd7922d5 not found: ID does not exist"
Oct 31 08:09:51.804763 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:09:51.804731    4720 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-597d55d8b8-vhlt6"]
Oct 31 08:09:51.809706 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:09:51.809679    4720 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-597d55d8b8-vhlt6"]
Oct 31 08:09:52.029616 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:09:52.027713    4720 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6c76be18-e721-4f19-8378-5dd0b1c0d265" path="/var/lib/kubelet/pods/6c76be18-e721-4f19-8378-5dd0b1c0d265/volumes"
Oct 31 08:09:53.318546 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:09:53.318510    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 08:09:58.914567 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:09:58.914518    4720 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-5d987cbbfd-b2zlm" podUID="50e53f44-4485-40c5-bcb2-10ccca40c7c4" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 08:09:58.914989 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:09:58.914637    4720 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-5d987cbbfd-b2zlm"
Oct 31 08:10:08.914644 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:10:08.914598    4720 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-5d987cbbfd-b2zlm" podUID="50e53f44-4485-40c5-bcb2-10ccca40c7c4" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 08:10:11.493029 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:10:11.492997    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 08:10:11.653780 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:10:11.653742    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 08:10:13.462998 ip-10-0-139-83 kubenswrapper[4720]: E1031 08:10:13.462959    4720 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8m768" containerName="extractor"
Oct 31 08:10:14.290365 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:10:14.290302    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 08:10:18.914829 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:10:18.914746    4720 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-5d987cbbfd-b2zlm" podUID="50e53f44-4485-40c5-bcb2-10ccca40c7c4" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 08:10:20.667982 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:10:20.667917    4720 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-5d987cbbfd-b2zlm"
Oct 31 08:10:20.722801 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:10:20.722767    4720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/50e53f44-4485-40c5-bcb2-10ccca40c7c4-tls-certs\" (UniqueName: \"kubernetes.io/secret/50e53f44-4485-40c5-bcb2-10ccca40c7c4-tls-certs\") pod \"50e53f44-4485-40c5-bcb2-10ccca40c7c4\" (UID: \"50e53f44-4485-40c5-bcb2-10ccca40c7c4\") "
Oct 31 08:10:20.723000 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:10:20.722827    4720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/50e53f44-4485-40c5-bcb2-10ccca40c7c4-kube-api-access-csxfl\" (UniqueName: \"kubernetes.io/projected/50e53f44-4485-40c5-bcb2-10ccca40c7c4-kube-api-access-csxfl\") pod \"50e53f44-4485-40c5-bcb2-10ccca40c7c4\" (UID: \"50e53f44-4485-40c5-bcb2-10ccca40c7c4\") "
Oct 31 08:10:20.725105 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:10:20.725070    4720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/50e53f44-4485-40c5-bcb2-10ccca40c7c4-tls-certs" pod "50e53f44-4485-40c5-bcb2-10ccca40c7c4" (UID: "50e53f44-4485-40c5-bcb2-10ccca40c7c4"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 08:10:20.725213 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:10:20.725108    4720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/50e53f44-4485-40c5-bcb2-10ccca40c7c4-kube-api-access-csxfl" pod "50e53f44-4485-40c5-bcb2-10ccca40c7c4" (UID: "50e53f44-4485-40c5-bcb2-10ccca40c7c4"). InnerVolumeSpecName "kube-api-access-csxfl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 08:10:20.773983 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:10:20.773945    4720 generic.go:356] "Generic (PLEG): container finished" podID="50e53f44-4485-40c5-bcb2-10ccca40c7c4" containerID="4979c2390176b26b9bbe03c13e2d2ace5cdd863adfd2ffc220a8306015582819" exitCode=0
Oct 31 08:10:20.774173 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:10:20.773997    4720 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-5d987cbbfd-b2zlm"
Oct 31 08:10:20.774173 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:10:20.773997    4720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-5d987cbbfd-b2zlm" event={"ID":"50e53f44-4485-40c5-bcb2-10ccca40c7c4","Type":"ContainerDied","Data":"4979c2390176b26b9bbe03c13e2d2ace5cdd863adfd2ffc220a8306015582819"}
Oct 31 08:10:20.774309 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:10:20.774171    4720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-5d987cbbfd-b2zlm" event={"ID":"50e53f44-4485-40c5-bcb2-10ccca40c7c4","Type":"ContainerDied","Data":"b1fd32aa4c24857469f13acff3d41330b2649a1d355dc3678c79c32f66565f13"}
Oct 31 08:10:20.774309 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:10:20.774210    4720 scope.go:122] "RemoveContainer" containerID="4979c2390176b26b9bbe03c13e2d2ace5cdd863adfd2ffc220a8306015582819"
Oct 31 08:10:20.785090 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:10:20.785068    4720 scope.go:122] "RemoveContainer" containerID="4979c2390176b26b9bbe03c13e2d2ace5cdd863adfd2ffc220a8306015582819"
Oct 31 08:10:20.785437 ip-10-0-139-83 kubenswrapper[4720]: E1031 08:10:20.785418    4720 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4979c2390176b26b9bbe03c13e2d2ace5cdd863adfd2ffc220a8306015582819\": container with ID starting with 4979c2390176b26b9bbe03c13e2d2ace5cdd863adfd2ffc220a8306015582819 not found: ID does not exist" containerID="4979c2390176b26b9bbe03c13e2d2ace5cdd863adfd2ffc220a8306015582819"
Oct 31 08:10:20.785499 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:10:20.785449    4720 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4979c2390176b26b9bbe03c13e2d2ace5cdd863adfd2ffc220a8306015582819"} err="failed to get container status \"4979c2390176b26b9bbe03c13e2d2ace5cdd863adfd2ffc220a8306015582819\": rpc error: code = NotFound desc = could not find container \"4979c2390176b26b9bbe03c13e2d2ace5cdd863adfd2ffc220a8306015582819\": container with ID starting with 4979c2390176b26b9bbe03c13e2d2ace5cdd863adfd2ffc220a8306015582819 not found: ID does not exist"
Oct 31 08:10:20.824114 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:10:20.824077    4720 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-csxfl\" (UniqueName: \"kubernetes.io/projected/50e53f44-4485-40c5-bcb2-10ccca40c7c4-kube-api-access-csxfl\") on node \"ip-10-0-139-83.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 08:10:20.824114 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:10:20.824108    4720 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/50e53f44-4485-40c5-bcb2-10ccca40c7c4-tls-certs\") on node \"ip-10-0-139-83.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 08:10:20.940719 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:10:20.940631    4720 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-5d987cbbfd-b2zlm"]
Oct 31 08:10:20.944748 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:10:20.944717    4720 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-5d987cbbfd-b2zlm"]
Oct 31 08:10:21.916611 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:10:21.916573    4720 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6784ddf685-wsmjj"]
Oct 31 08:10:21.917017 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:10:21.916916    4720 cpu_manager.go:414] "containerMap: removing container" podUID="50e53f44-4485-40c5-bcb2-10ccca40c7c4" containerName="main"
Oct 31 08:10:21.917017 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:10:21.916927    4720 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="50e53f44-4485-40c5-bcb2-10ccca40c7c4" containerName="main"
Oct 31 08:10:21.917017 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:10:21.916940    4720 cpu_manager.go:414] "containerMap: removing container" podUID="6c76be18-e721-4f19-8378-5dd0b1c0d265" containerName="storage-initializer"
Oct 31 08:10:21.917017 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:10:21.916945    4720 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6c76be18-e721-4f19-8378-5dd0b1c0d265" containerName="storage-initializer"
Oct 31 08:10:21.917017 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:10:21.916954    4720 cpu_manager.go:414] "containerMap: removing container" podUID="6c76be18-e721-4f19-8378-5dd0b1c0d265" containerName="main"
Oct 31 08:10:21.917017 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:10:21.916959    4720 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6c76be18-e721-4f19-8378-5dd0b1c0d265" containerName="main"
Oct 31 08:10:21.917203 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:10:21.917026    4720 memory_manager.go:365] "RemoveStaleState removing state" podUID="50e53f44-4485-40c5-bcb2-10ccca40c7c4" containerName="main"
Oct 31 08:10:21.917203 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:10:21.917035    4720 memory_manager.go:365] "RemoveStaleState removing state" podUID="6c76be18-e721-4f19-8378-5dd0b1c0d265" containerName="main"
Oct 31 08:10:21.921587 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:10:21.921562    4720 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6784ddf685-wsmjj"
Oct 31 08:10:22.008287 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:10:22.008250    4720 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-0eb24714\"/\"seaweedfs-s3-creds\""
Oct 31 08:10:22.022944 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:10:22.022751    4720 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-0eb24714\"/\"default-dockercfg-dpjc7\""
Oct 31 08:10:22.023111 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:10:22.023002    4720 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-0eb24714\"/\"tokenizer-template-args-test-kserve-self-signed-certs\""
Oct 31 08:10:22.023975 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:10:22.023949    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 08:10:22.027235 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:10:22.027211    4720 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="50e53f44-4485-40c5-bcb2-10ccca40c7c4" path="/var/lib/kubelet/pods/50e53f44-4485-40c5-bcb2-10ccca40c7c4/volumes"
Oct 31 08:10:22.027519 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:10:22.027506    4720 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6784ddf685-wsmjj"]
Oct 31 08:10:22.033747 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:10:22.033728    4720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v872x\" (UniqueName: \"kubernetes.io/projected/9856d031-a5d9-4fa0-af75-1e2ede74e2c5-kube-api-access-v872x\") pod \"tokenizer-template-args-test-kserve-6784ddf685-wsmjj\" (UID: \"9856d031-a5d9-4fa0-af75-1e2ede74e2c5\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6784ddf685-wsmjj"
Oct 31 08:10:22.033822 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:10:22.033759    4720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9856d031-a5d9-4fa0-af75-1e2ede74e2c5-home\") pod \"tokenizer-template-args-test-kserve-6784ddf685-wsmjj\" (UID: \"9856d031-a5d9-4fa0-af75-1e2ede74e2c5\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6784ddf685-wsmjj"
Oct 31 08:10:22.033859 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:10:22.033842    4720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9856d031-a5d9-4fa0-af75-1e2ede74e2c5-cabundle-cert\") pod \"tokenizer-template-args-test-kserve-6784ddf685-wsmjj\" (UID: \"9856d031-a5d9-4fa0-af75-1e2ede74e2c5\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6784ddf685-wsmjj"
Oct 31 08:10:22.033910 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:10:22.033898    4720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9856d031-a5d9-4fa0-af75-1e2ede74e2c5-dshm\") pod \"tokenizer-template-args-test-kserve-6784ddf685-wsmjj\" (UID: \"9856d031-a5d9-4fa0-af75-1e2ede74e2c5\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6784ddf685-wsmjj"
Oct 31 08:10:22.034002 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:10:22.033958    4720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9856d031-a5d9-4fa0-af75-1e2ede74e2c5-model-cache\") pod \"tokenizer-template-args-test-kserve-6784ddf685-wsmjj\" (UID: \"9856d031-a5d9-4fa0-af75-1e2ede74e2c5\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6784ddf685-wsmjj"
Oct 31 08:10:22.034038 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:10:22.034020    4720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9856d031-a5d9-4fa0-af75-1e2ede74e2c5-tmp-dir\") pod \"tokenizer-template-args-test-kserve-6784ddf685-wsmjj\" (UID: \"9856d031-a5d9-4fa0-af75-1e2ede74e2c5\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6784ddf685-wsmjj"
Oct 31 08:10:22.034119 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:10:22.034073    4720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9856d031-a5d9-4fa0-af75-1e2ede74e2c5-tls-certs\") pod \"tokenizer-template-args-test-kserve-6784ddf685-wsmjj\" (UID: \"9856d031-a5d9-4fa0-af75-1e2ede74e2c5\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6784ddf685-wsmjj"
Oct 31 08:10:22.034119 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:10:22.034096    4720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9856d031-a5d9-4fa0-af75-1e2ede74e2c5-kserve-provision-location\") pod \"tokenizer-template-args-test-kserve-6784ddf685-wsmjj\" (UID: \"9856d031-a5d9-4fa0-af75-1e2ede74e2c5\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6784ddf685-wsmjj"
Oct 31 08:10:22.061738 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:10:22.061712    4720 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-0eb24714\"/\"kube-root-ca.crt\""
Oct 31 08:10:22.061909 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:10:22.061892    4720 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-0eb24714\"/\"odh-kserve-custom-ca-bundle\""
Oct 31 08:10:22.072437 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:10:22.072413    4720 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-0eb24714\"/\"openshift-service-ca.crt\""
Oct 31 08:10:22.135022 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:10:22.134989    4720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9856d031-a5d9-4fa0-af75-1e2ede74e2c5-dshm\") pod \"tokenizer-template-args-test-kserve-6784ddf685-wsmjj\" (UID: \"9856d031-a5d9-4fa0-af75-1e2ede74e2c5\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6784ddf685-wsmjj"
Oct 31 08:10:22.135213 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:10:22.135027    4720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9856d031-a5d9-4fa0-af75-1e2ede74e2c5-model-cache\") pod \"tokenizer-template-args-test-kserve-6784ddf685-wsmjj\" (UID: \"9856d031-a5d9-4fa0-af75-1e2ede74e2c5\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6784ddf685-wsmjj"
Oct 31 08:10:22.135213 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:10:22.135055    4720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9856d031-a5d9-4fa0-af75-1e2ede74e2c5-tmp-dir\") pod \"tokenizer-template-args-test-kserve-6784ddf685-wsmjj\" (UID: \"9856d031-a5d9-4fa0-af75-1e2ede74e2c5\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6784ddf685-wsmjj"
Oct 31 08:10:22.135213 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:10:22.135080    4720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9856d031-a5d9-4fa0-af75-1e2ede74e2c5-tls-certs\") pod \"tokenizer-template-args-test-kserve-6784ddf685-wsmjj\" (UID: \"9856d031-a5d9-4fa0-af75-1e2ede74e2c5\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6784ddf685-wsmjj"
Oct 31 08:10:22.135213 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:10:22.135109    4720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9856d031-a5d9-4fa0-af75-1e2ede74e2c5-kserve-provision-location\") pod \"tokenizer-template-args-test-kserve-6784ddf685-wsmjj\" (UID: \"9856d031-a5d9-4fa0-af75-1e2ede74e2c5\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6784ddf685-wsmjj"
Oct 31 08:10:22.135441 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:10:22.135220    4720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v872x\" (UniqueName: \"kubernetes.io/projected/9856d031-a5d9-4fa0-af75-1e2ede74e2c5-kube-api-access-v872x\") pod \"tokenizer-template-args-test-kserve-6784ddf685-wsmjj\" (UID: \"9856d031-a5d9-4fa0-af75-1e2ede74e2c5\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6784ddf685-wsmjj"
Oct 31 08:10:22.135441 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:10:22.135250    4720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9856d031-a5d9-4fa0-af75-1e2ede74e2c5-home\") pod \"tokenizer-template-args-test-kserve-6784ddf685-wsmjj\" (UID: \"9856d031-a5d9-4fa0-af75-1e2ede74e2c5\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6784ddf685-wsmjj"
Oct 31 08:10:22.135441 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:10:22.135289    4720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9856d031-a5d9-4fa0-af75-1e2ede74e2c5-cabundle-cert\") pod \"tokenizer-template-args-test-kserve-6784ddf685-wsmjj\" (UID: \"9856d031-a5d9-4fa0-af75-1e2ede74e2c5\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6784ddf685-wsmjj"
Oct 31 08:10:22.135664 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:10:22.135621    4720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9856d031-a5d9-4fa0-af75-1e2ede74e2c5-tmp-dir\") pod \"tokenizer-template-args-test-kserve-6784ddf685-wsmjj\" (UID: \"9856d031-a5d9-4fa0-af75-1e2ede74e2c5\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6784ddf685-wsmjj"
Oct 31 08:10:22.135725 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:10:22.135699    4720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9856d031-a5d9-4fa0-af75-1e2ede74e2c5-kserve-provision-location\") pod \"tokenizer-template-args-test-kserve-6784ddf685-wsmjj\" (UID: \"9856d031-a5d9-4fa0-af75-1e2ede74e2c5\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6784ddf685-wsmjj"
Oct 31 08:10:22.135841 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:10:22.135819    4720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9856d031-a5d9-4fa0-af75-1e2ede74e2c5-model-cache\") pod \"tokenizer-template-args-test-kserve-6784ddf685-wsmjj\" (UID: \"9856d031-a5d9-4fa0-af75-1e2ede74e2c5\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6784ddf685-wsmjj"
Oct 31 08:10:22.135901 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:10:22.135853    4720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9856d031-a5d9-4fa0-af75-1e2ede74e2c5-home\") pod \"tokenizer-template-args-test-kserve-6784ddf685-wsmjj\" (UID: \"9856d031-a5d9-4fa0-af75-1e2ede74e2c5\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6784ddf685-wsmjj"
Oct 31 08:10:22.136085 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:10:22.136037    4720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9856d031-a5d9-4fa0-af75-1e2ede74e2c5-cabundle-cert\") pod \"tokenizer-template-args-test-kserve-6784ddf685-wsmjj\" (UID: \"9856d031-a5d9-4fa0-af75-1e2ede74e2c5\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6784ddf685-wsmjj"
Oct 31 08:10:22.137815 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:10:22.137793    4720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9856d031-a5d9-4fa0-af75-1e2ede74e2c5-dshm\") pod \"tokenizer-template-args-test-kserve-6784ddf685-wsmjj\" (UID: \"9856d031-a5d9-4fa0-af75-1e2ede74e2c5\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6784ddf685-wsmjj"
Oct 31 08:10:22.137977 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:10:22.137961    4720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9856d031-a5d9-4fa0-af75-1e2ede74e2c5-tls-certs\") pod \"tokenizer-template-args-test-kserve-6784ddf685-wsmjj\" (UID: \"9856d031-a5d9-4fa0-af75-1e2ede74e2c5\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6784ddf685-wsmjj"
Oct 31 08:10:22.191528 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:10:22.191455    4720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v872x\" (UniqueName: \"kubernetes.io/projected/9856d031-a5d9-4fa0-af75-1e2ede74e2c5-kube-api-access-v872x\") pod \"tokenizer-template-args-test-kserve-6784ddf685-wsmjj\" (UID: \"9856d031-a5d9-4fa0-af75-1e2ede74e2c5\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6784ddf685-wsmjj"
Oct 31 08:10:22.236081 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:10:22.236043    4720 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6784ddf685-wsmjj"
Oct 31 08:10:22.469601 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:10:22.469512    4720 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-5b5cc5746b-n4htw"]
Oct 31 08:10:22.473188 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:10:22.473166    4720 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-5b5cc5746b-n4htw"
Oct 31 08:10:22.529783 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:10:22.529745    4720 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-5b5cc5746b-n4htw"]
Oct 31 08:10:22.539220 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:10:22.539190    4720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f7a7f584-5678-4004-90ad-01cd877da402-cabundle-cert\") pod \"tokenizer-template-args-test-tokenizer-5b5cc5746b-n4htw\" (UID: \"f7a7f584-5678-4004-90ad-01cd877da402\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-5b5cc5746b-n4htw"
Oct 31 08:10:22.539402 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:10:22.539229    4720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f7a7f584-5678-4004-90ad-01cd877da402-kserve-provision-location\") pod \"tokenizer-template-args-test-tokenizer-5b5cc5746b-n4htw\" (UID: \"f7a7f584-5678-4004-90ad-01cd877da402\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-5b5cc5746b-n4htw"
Oct 31 08:10:22.539402 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:10:22.539256    4720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f7a7f584-5678-4004-90ad-01cd877da402-tmp-dir\") pod \"tokenizer-template-args-test-tokenizer-5b5cc5746b-n4htw\" (UID: \"f7a7f584-5678-4004-90ad-01cd877da402\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-5b5cc5746b-n4htw"
Oct 31 08:10:22.539402 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:10:22.539352    4720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f7a7f584-5678-4004-90ad-01cd877da402-tls-certs\") pod \"tokenizer-template-args-test-tokenizer-5b5cc5746b-n4htw\" (UID: \"f7a7f584-5678-4004-90ad-01cd877da402\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-5b5cc5746b-n4htw"
Oct 31 08:10:22.602074 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:10:22.602042    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 08:10:22.640143 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:10:22.640107    4720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f7a7f584-5678-4004-90ad-01cd877da402-cabundle-cert\") pod \"tokenizer-template-args-test-tokenizer-5b5cc5746b-n4htw\" (UID: \"f7a7f584-5678-4004-90ad-01cd877da402\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-5b5cc5746b-n4htw"
Oct 31 08:10:22.640143 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:10:22.640152    4720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f7a7f584-5678-4004-90ad-01cd877da402-kserve-provision-location\") pod \"tokenizer-template-args-test-tokenizer-5b5cc5746b-n4htw\" (UID: \"f7a7f584-5678-4004-90ad-01cd877da402\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-5b5cc5746b-n4htw"
Oct 31 08:10:22.640441 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:10:22.640188    4720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f7a7f584-5678-4004-90ad-01cd877da402-tmp-dir\") pod \"tokenizer-template-args-test-tokenizer-5b5cc5746b-n4htw\" (UID: \"f7a7f584-5678-4004-90ad-01cd877da402\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-5b5cc5746b-n4htw"
Oct 31 08:10:22.640441 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:10:22.640212    4720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f7a7f584-5678-4004-90ad-01cd877da402-tls-certs\") pod \"tokenizer-template-args-test-tokenizer-5b5cc5746b-n4htw\" (UID: \"f7a7f584-5678-4004-90ad-01cd877da402\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-5b5cc5746b-n4htw"
Oct 31 08:10:22.640735 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:10:22.640708    4720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f7a7f584-5678-4004-90ad-01cd877da402-kserve-provision-location\") pod \"tokenizer-template-args-test-tokenizer-5b5cc5746b-n4htw\" (UID: \"f7a7f584-5678-4004-90ad-01cd877da402\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-5b5cc5746b-n4htw"
Oct 31 08:10:22.640828 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:10:22.640754    4720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f7a7f584-5678-4004-90ad-01cd877da402-tmp-dir\") pod \"tokenizer-template-args-test-tokenizer-5b5cc5746b-n4htw\" (UID: \"f7a7f584-5678-4004-90ad-01cd877da402\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-5b5cc5746b-n4htw"
Oct 31 08:10:22.640998 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:10:22.640975    4720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f7a7f584-5678-4004-90ad-01cd877da402-cabundle-cert\") pod \"tokenizer-template-args-test-tokenizer-5b5cc5746b-n4htw\" (UID: \"f7a7f584-5678-4004-90ad-01cd877da402\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-5b5cc5746b-n4htw"
Oct 31 08:10:22.643181 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:10:22.643156    4720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f7a7f584-5678-4004-90ad-01cd877da402-tls-certs\") pod \"tokenizer-template-args-test-tokenizer-5b5cc5746b-n4htw\" (UID: \"f7a7f584-5678-4004-90ad-01cd877da402\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-5b5cc5746b-n4htw"
Oct 31 08:10:22.658294 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:10:22.658266    4720 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6784ddf685-wsmjj"]
Oct 31 08:10:22.658830 ip-10-0-139-83 kubenswrapper[4720]: W1031 08:10:22.658806    4720 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9856d031_a5d9_4fa0_af75_1e2ede74e2c5.slice/crio-221815af95a251594d9c101f16f874b76f4d2ee1259df92463731844e5d7d46e WatchSource:0}: Error finding container 221815af95a251594d9c101f16f874b76f4d2ee1259df92463731844e5d7d46e: Status 404 returned error can't find the container with id 221815af95a251594d9c101f16f874b76f4d2ee1259df92463731844e5d7d46e
Oct 31 08:10:22.751274 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:10:22.751184    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 08:10:22.786938 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:10:22.786900    4720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6784ddf685-wsmjj" event={"ID":"9856d031-a5d9-4fa0-af75-1e2ede74e2c5","Type":"ContainerStarted","Data":"923917f3c26be86f79a56dca8120ac5e984daf1c8df79fa7412bbc2f4584c4d1"}
Oct 31 08:10:22.787165 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:10:22.786957    4720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6784ddf685-wsmjj" event={"ID":"9856d031-a5d9-4fa0-af75-1e2ede74e2c5","Type":"ContainerStarted","Data":"221815af95a251594d9c101f16f874b76f4d2ee1259df92463731844e5d7d46e"}
Oct 31 08:10:22.788262 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:10:22.788235    4720 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-5b5cc5746b-n4htw"
Oct 31 08:10:23.166185 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:10:23.166161    4720 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-5b5cc5746b-n4htw"]
Oct 31 08:10:23.169288 ip-10-0-139-83 kubenswrapper[4720]: W1031 08:10:23.169239    4720 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf7a7f584_5678_4004_90ad_01cd877da402.slice/crio-c49d4f8f71ae496ff0c7e42386dd404212acfc6cfc2b047d99ba85a30fbe86de WatchSource:0}: Error finding container c49d4f8f71ae496ff0c7e42386dd404212acfc6cfc2b047d99ba85a30fbe86de: Status 404 returned error can't find the container with id c49d4f8f71ae496ff0c7e42386dd404212acfc6cfc2b047d99ba85a30fbe86de
Oct 31 08:10:23.795577 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:10:23.795539    4720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-5b5cc5746b-n4htw" event={"ID":"f7a7f584-5678-4004-90ad-01cd877da402","Type":"ContainerStarted","Data":"b01fa78eb325de154becbde47b4d1db19294b571d1bc0cb8d8d825bccb7d5e71"}
Oct 31 08:10:23.795775 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:10:23.795607    4720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-5b5cc5746b-n4htw" event={"ID":"f7a7f584-5678-4004-90ad-01cd877da402","Type":"ContainerStarted","Data":"c49d4f8f71ae496ff0c7e42386dd404212acfc6cfc2b047d99ba85a30fbe86de"}
Oct 31 08:10:24.821938 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:10:24.821900    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 08:10:25.000355 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:10:24.994394    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 08:10:25.615601 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:10:25.590889    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 08:10:25.777360 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:10:25.777300    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 08:10:26.851531 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:10:26.851416    4720 generic.go:356] "Generic (PLEG): container finished" podID="9856d031-a5d9-4fa0-af75-1e2ede74e2c5" containerID="923917f3c26be86f79a56dca8120ac5e984daf1c8df79fa7412bbc2f4584c4d1" exitCode=0
Oct 31 08:10:26.851946 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:10:26.851601    4720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6784ddf685-wsmjj" event={"ID":"9856d031-a5d9-4fa0-af75-1e2ede74e2c5","Type":"ContainerDied","Data":"923917f3c26be86f79a56dca8120ac5e984daf1c8df79fa7412bbc2f4584c4d1"}
Oct 31 08:10:26.853401 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:10:26.853377    4720 generic.go:356] "Generic (PLEG): container finished" podID="f7a7f584-5678-4004-90ad-01cd877da402" containerID="b01fa78eb325de154becbde47b4d1db19294b571d1bc0cb8d8d825bccb7d5e71" exitCode=0
Oct 31 08:10:26.853520 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:10:26.853416    4720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-5b5cc5746b-n4htw" event={"ID":"f7a7f584-5678-4004-90ad-01cd877da402","Type":"ContainerDied","Data":"b01fa78eb325de154becbde47b4d1db19294b571d1bc0cb8d8d825bccb7d5e71"}
Oct 31 08:10:27.290417 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:10:27.290290    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 08:10:27.873003 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:10:27.872958    4720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6784ddf685-wsmjj" event={"ID":"9856d031-a5d9-4fa0-af75-1e2ede74e2c5","Type":"ContainerStarted","Data":"0a908fb23991bb85069124fdb485850595af4723ffc3b05bf0e3b1ce52d537a6"}
Oct 31 08:10:27.881029 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:10:27.880986    4720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-5b5cc5746b-n4htw" event={"ID":"f7a7f584-5678-4004-90ad-01cd877da402","Type":"ContainerStarted","Data":"371dc468b642a73853983b315188871144a47a1c3e3f6da18877bc1422267134"}
Oct 31 08:10:27.882198 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:10:27.882176    4720 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-5b5cc5746b-n4htw"
Oct 31 08:10:28.111729 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:10:28.111642    4720 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6784ddf685-wsmjj" podStartSLOduration=3.8118027100000003 podStartE2EDuration="7.111613775s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.299811065s" isStatefulPod=false podCreationTimestamp="2026-10-31 08:10:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 08:10:28.044834372 +0000 UTC m=+2482.175902447" watchObservedRunningTime="2026-10-31 08:10:28.111613775 +0000 UTC m=+2482.242681847"
Oct 31 08:10:28.166252 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:10:28.166090    4720 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-5b5cc5746b-n4htw" podStartSLOduration=2.798414238 podStartE2EDuration="6.166070224s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.367655986s" isStatefulPod=false podCreationTimestamp="2026-10-31 08:10:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 08:10:28.164833826 +0000 UTC m=+2482.295902109" watchObservedRunningTime="2026-10-31 08:10:28.166070224 +0000 UTC m=+2482.297138297"
Oct 31 08:10:28.236528 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:10:28.236476    4720 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6784ddf685-wsmjj"
Oct 31 08:10:28.900078 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:10:28.900043    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 08:10:29.351742 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:10:29.351675    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 08:10:30.798421 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:10:30.798387    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 08:10:32.237035 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:10:32.236987    4720 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6784ddf685-wsmjj"
Oct 31 08:10:32.250368 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:10:32.250314    4720 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6784ddf685-wsmjj"
Oct 31 08:10:32.929954 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:10:32.929919    4720 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6784ddf685-wsmjj"
Oct 31 08:10:36.271659 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:10:36.271619    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 08:10:36.414473 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:10:36.414441    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 08:10:38.613547 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:10:38.613501    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 08:10:43.462765 ip-10-0-139-83 kubenswrapper[4720]: E1031 08:10:43.462716    4720 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8m768" containerName="extractor"
Oct 31 08:10:53.658396 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:10:53.658356    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 08:10:58.895884 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:10:58.895824    4720 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-5b5cc5746b-n4htw" podUID="f7a7f584-5678-4004-90ad-01cd877da402" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/health\": dial tcp 10.132.0.59:8000: connect: connection refused"
Oct 31 08:11:08.904301 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:11:08.904254    4720 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-5b5cc5746b-n4htw"
Oct 31 08:11:12.931555 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:11:12.931519    4720 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-0eb24714_tokenizer-template-args-test-tokenizer-5b5cc5746b-n4htw_f7a7f584-5678-4004-90ad-01cd877da402/main/0.log"
Oct 31 08:11:13.462465 ip-10-0-139-83 kubenswrapper[4720]: E1031 08:11:13.462427    4720 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8m768" containerName="extractor"
Oct 31 08:11:14.620057 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:11:14.620027    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 08:11:16.244265 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:11:16.244226    4720 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-5b5cc5746b-n4htw"]
Oct 31 08:11:16.244730 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:11:16.244520    4720 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-5b5cc5746b-n4htw" podUID="f7a7f584-5678-4004-90ad-01cd877da402" containerName="main" containerID="cri-o://371dc468b642a73853983b315188871144a47a1c3e3f6da18877bc1422267134" gracePeriod=30
Oct 31 08:11:16.250475 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:11:16.250446    4720 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6784ddf685-wsmjj"]
Oct 31 08:11:16.260276 ip-10-0-139-83 kubenswrapper[4720]: E1031 08:11:16.260225    4720 log.go:32] "ExecSync cmd from runtime service failed" err="rpc error: code = Unknown desc = command error: , stdout: , stderr: , exit code -1" containerID="0a908fb23991bb85069124fdb485850595af4723ffc3b05bf0e3b1ce52d537a6" cmd=["/bin/sleep","15"]
Oct 31 08:11:16.260445 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:11:16.260297    4720 handlers.go:85] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6784ddf685-wsmjj" message="" err="rpc error: code = Unknown desc = command error: , stdout: , stderr: , exit code -1"
Oct 31 08:11:16.260445 ip-10-0-139-83 kubenswrapper[4720]: E1031 08:11:16.260316    4720 kuberuntime_container.go:780] "PreStop hook failed" err="rpc error: code = Unknown desc = command error: , stdout: , stderr: , exit code -1" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6784ddf685-wsmjj" podUID="9856d031-a5d9-4fa0-af75-1e2ede74e2c5" containerName="main" containerID="cri-o://0a908fb23991bb85069124fdb485850595af4723ffc3b05bf0e3b1ce52d537a6"
Oct 31 08:11:16.260445 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:11:16.260387    4720 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6784ddf685-wsmjj" podUID="9856d031-a5d9-4fa0-af75-1e2ede74e2c5" containerName="main" containerID="cri-o://0a908fb23991bb85069124fdb485850595af4723ffc3b05bf0e3b1ce52d537a6" gracePeriod=60
Oct 31 08:11:16.277557 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:11:16.277527    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 08:11:16.423517 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:11:16.423476    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 08:11:16.631116 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:11:16.631083    4720 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6784ddf685-wsmjj"
Oct 31 08:11:16.732100 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:11:16.732059    4720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9856d031-a5d9-4fa0-af75-1e2ede74e2c5-home\" (UniqueName: \"kubernetes.io/empty-dir/9856d031-a5d9-4fa0-af75-1e2ede74e2c5-home\") pod \"9856d031-a5d9-4fa0-af75-1e2ede74e2c5\" (UID: \"9856d031-a5d9-4fa0-af75-1e2ede74e2c5\") "
Oct 31 08:11:16.732368 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:11:16.732121    4720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9856d031-a5d9-4fa0-af75-1e2ede74e2c5-tls-certs\" (UniqueName: \"kubernetes.io/secret/9856d031-a5d9-4fa0-af75-1e2ede74e2c5-tls-certs\") pod \"9856d031-a5d9-4fa0-af75-1e2ede74e2c5\" (UID: \"9856d031-a5d9-4fa0-af75-1e2ede74e2c5\") "
Oct 31 08:11:16.732368 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:11:16.732180    4720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9856d031-a5d9-4fa0-af75-1e2ede74e2c5-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9856d031-a5d9-4fa0-af75-1e2ede74e2c5-tmp-dir\") pod \"9856d031-a5d9-4fa0-af75-1e2ede74e2c5\" (UID: \"9856d031-a5d9-4fa0-af75-1e2ede74e2c5\") "
Oct 31 08:11:16.732368 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:11:16.732208    4720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9856d031-a5d9-4fa0-af75-1e2ede74e2c5-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9856d031-a5d9-4fa0-af75-1e2ede74e2c5-kserve-provision-location\") pod \"9856d031-a5d9-4fa0-af75-1e2ede74e2c5\" (UID: \"9856d031-a5d9-4fa0-af75-1e2ede74e2c5\") "
Oct 31 08:11:16.732368 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:11:16.732283    4720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9856d031-a5d9-4fa0-af75-1e2ede74e2c5-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9856d031-a5d9-4fa0-af75-1e2ede74e2c5-cabundle-cert\") pod \"9856d031-a5d9-4fa0-af75-1e2ede74e2c5\" (UID: \"9856d031-a5d9-4fa0-af75-1e2ede74e2c5\") "
Oct 31 08:11:16.732606 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:11:16.732386    4720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9856d031-a5d9-4fa0-af75-1e2ede74e2c5-dshm\" (UniqueName: \"kubernetes.io/empty-dir/9856d031-a5d9-4fa0-af75-1e2ede74e2c5-dshm\") pod \"9856d031-a5d9-4fa0-af75-1e2ede74e2c5\" (UID: \"9856d031-a5d9-4fa0-af75-1e2ede74e2c5\") "
Oct 31 08:11:16.732606 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:11:16.732432    4720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9856d031-a5d9-4fa0-af75-1e2ede74e2c5-kube-api-access-v872x\" (UniqueName: \"kubernetes.io/projected/9856d031-a5d9-4fa0-af75-1e2ede74e2c5-kube-api-access-v872x\") pod \"9856d031-a5d9-4fa0-af75-1e2ede74e2c5\" (UID: \"9856d031-a5d9-4fa0-af75-1e2ede74e2c5\") "
Oct 31 08:11:16.732606 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:11:16.732463    4720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9856d031-a5d9-4fa0-af75-1e2ede74e2c5-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9856d031-a5d9-4fa0-af75-1e2ede74e2c5-model-cache\") pod \"9856d031-a5d9-4fa0-af75-1e2ede74e2c5\" (UID: \"9856d031-a5d9-4fa0-af75-1e2ede74e2c5\") "
Oct 31 08:11:16.732869 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:11:16.732817    4720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9856d031-a5d9-4fa0-af75-1e2ede74e2c5-tmp-dir" pod "9856d031-a5d9-4fa0-af75-1e2ede74e2c5" (UID: "9856d031-a5d9-4fa0-af75-1e2ede74e2c5"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 08:11:16.733131 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:11:16.732873    4720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9856d031-a5d9-4fa0-af75-1e2ede74e2c5-home" pod "9856d031-a5d9-4fa0-af75-1e2ede74e2c5" (UID: "9856d031-a5d9-4fa0-af75-1e2ede74e2c5"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 08:11:16.733220 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:11:16.733083    4720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9856d031-a5d9-4fa0-af75-1e2ede74e2c5-model-cache" pod "9856d031-a5d9-4fa0-af75-1e2ede74e2c5" (UID: "9856d031-a5d9-4fa0-af75-1e2ede74e2c5"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 08:11:16.733490 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:11:16.733314    4720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9856d031-a5d9-4fa0-af75-1e2ede74e2c5-cabundle-cert" pod "9856d031-a5d9-4fa0-af75-1e2ede74e2c5" (UID: "9856d031-a5d9-4fa0-af75-1e2ede74e2c5"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 08:11:16.735061 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:11:16.735025    4720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9856d031-a5d9-4fa0-af75-1e2ede74e2c5-tls-certs" pod "9856d031-a5d9-4fa0-af75-1e2ede74e2c5" (UID: "9856d031-a5d9-4fa0-af75-1e2ede74e2c5"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 08:11:16.735177 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:11:16.735076    4720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9856d031-a5d9-4fa0-af75-1e2ede74e2c5-kube-api-access-v872x" pod "9856d031-a5d9-4fa0-af75-1e2ede74e2c5" (UID: "9856d031-a5d9-4fa0-af75-1e2ede74e2c5"). InnerVolumeSpecName "kube-api-access-v872x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 08:11:16.735177 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:11:16.735118    4720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9856d031-a5d9-4fa0-af75-1e2ede74e2c5-dshm" pod "9856d031-a5d9-4fa0-af75-1e2ede74e2c5" (UID: "9856d031-a5d9-4fa0-af75-1e2ede74e2c5"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 08:11:16.760539 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:11:16.760489    4720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9856d031-a5d9-4fa0-af75-1e2ede74e2c5-kserve-provision-location" pod "9856d031-a5d9-4fa0-af75-1e2ede74e2c5" (UID: "9856d031-a5d9-4fa0-af75-1e2ede74e2c5"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 08:11:16.834212 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:11:16.834157    4720 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9856d031-a5d9-4fa0-af75-1e2ede74e2c5-cabundle-cert\") on node \"ip-10-0-139-83.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 08:11:16.834212 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:11:16.834195    4720 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9856d031-a5d9-4fa0-af75-1e2ede74e2c5-dshm\") on node \"ip-10-0-139-83.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 08:11:16.834212 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:11:16.834211    4720 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v872x\" (UniqueName: \"kubernetes.io/projected/9856d031-a5d9-4fa0-af75-1e2ede74e2c5-kube-api-access-v872x\") on node \"ip-10-0-139-83.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 08:11:16.834589 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:11:16.834227    4720 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9856d031-a5d9-4fa0-af75-1e2ede74e2c5-model-cache\") on node \"ip-10-0-139-83.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 08:11:16.834589 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:11:16.834437    4720 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9856d031-a5d9-4fa0-af75-1e2ede74e2c5-home\") on node \"ip-10-0-139-83.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 08:11:16.834589 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:11:16.834455    4720 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9856d031-a5d9-4fa0-af75-1e2ede74e2c5-tls-certs\") on node \"ip-10-0-139-83.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 08:11:16.834589 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:11:16.834469    4720 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9856d031-a5d9-4fa0-af75-1e2ede74e2c5-tmp-dir\") on node \"ip-10-0-139-83.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 08:11:16.834589 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:11:16.834480    4720 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9856d031-a5d9-4fa0-af75-1e2ede74e2c5-kserve-provision-location\") on node \"ip-10-0-139-83.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 08:11:16.890297 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:11:16.890269    4720 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-5b5cc5746b-n4htw"
Oct 31 08:11:16.934978 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:11:16.934944    4720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f7a7f584-5678-4004-90ad-01cd877da402-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f7a7f584-5678-4004-90ad-01cd877da402-kserve-provision-location\") pod \"f7a7f584-5678-4004-90ad-01cd877da402\" (UID: \"f7a7f584-5678-4004-90ad-01cd877da402\") "
Oct 31 08:11:16.935154 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:11:16.934987    4720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f7a7f584-5678-4004-90ad-01cd877da402-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f7a7f584-5678-4004-90ad-01cd877da402-tmp-dir\") pod \"f7a7f584-5678-4004-90ad-01cd877da402\" (UID: \"f7a7f584-5678-4004-90ad-01cd877da402\") "
Oct 31 08:11:16.935154 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:11:16.935009    4720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f7a7f584-5678-4004-90ad-01cd877da402-tls-certs\" (UniqueName: \"kubernetes.io/secret/f7a7f584-5678-4004-90ad-01cd877da402-tls-certs\") pod \"f7a7f584-5678-4004-90ad-01cd877da402\" (UID: \"f7a7f584-5678-4004-90ad-01cd877da402\") "
Oct 31 08:11:16.935154 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:11:16.935066    4720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f7a7f584-5678-4004-90ad-01cd877da402-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f7a7f584-5678-4004-90ad-01cd877da402-cabundle-cert\") pod \"f7a7f584-5678-4004-90ad-01cd877da402\" (UID: \"f7a7f584-5678-4004-90ad-01cd877da402\") "
Oct 31 08:11:16.935505 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:11:16.935476    4720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f7a7f584-5678-4004-90ad-01cd877da402-tmp-dir" pod "f7a7f584-5678-4004-90ad-01cd877da402" (UID: "f7a7f584-5678-4004-90ad-01cd877da402"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 08:11:16.935555 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:11:16.935500    4720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f7a7f584-5678-4004-90ad-01cd877da402-cabundle-cert" pod "f7a7f584-5678-4004-90ad-01cd877da402" (UID: "f7a7f584-5678-4004-90ad-01cd877da402"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 08:11:16.937381 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:11:16.937354    4720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f7a7f584-5678-4004-90ad-01cd877da402-tls-certs" pod "f7a7f584-5678-4004-90ad-01cd877da402" (UID: "f7a7f584-5678-4004-90ad-01cd877da402"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 08:11:16.959109 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:11:16.959071    4720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f7a7f584-5678-4004-90ad-01cd877da402-kserve-provision-location" pod "f7a7f584-5678-4004-90ad-01cd877da402" (UID: "f7a7f584-5678-4004-90ad-01cd877da402"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 08:11:17.036439 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:11:17.036392    4720 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f7a7f584-5678-4004-90ad-01cd877da402-kserve-provision-location\") on node \"ip-10-0-139-83.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 08:11:17.036439 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:11:17.036438    4720 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f7a7f584-5678-4004-90ad-01cd877da402-tmp-dir\") on node \"ip-10-0-139-83.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 08:11:17.036439 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:11:17.036453    4720 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f7a7f584-5678-4004-90ad-01cd877da402-tls-certs\") on node \"ip-10-0-139-83.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 08:11:17.036697 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:11:17.036467    4720 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f7a7f584-5678-4004-90ad-01cd877da402-cabundle-cert\") on node \"ip-10-0-139-83.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 08:11:17.133219 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:11:17.133177    4720 generic.go:356] "Generic (PLEG): container finished" podID="9856d031-a5d9-4fa0-af75-1e2ede74e2c5" containerID="0a908fb23991bb85069124fdb485850595af4723ffc3b05bf0e3b1ce52d537a6" exitCode=0
Oct 31 08:11:17.133435 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:11:17.133240    4720 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6784ddf685-wsmjj"
Oct 31 08:11:17.133435 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:11:17.133264    4720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6784ddf685-wsmjj" event={"ID":"9856d031-a5d9-4fa0-af75-1e2ede74e2c5","Type":"ContainerDied","Data":"0a908fb23991bb85069124fdb485850595af4723ffc3b05bf0e3b1ce52d537a6"}
Oct 31 08:11:17.133435 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:11:17.133344    4720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6784ddf685-wsmjj" event={"ID":"9856d031-a5d9-4fa0-af75-1e2ede74e2c5","Type":"ContainerDied","Data":"221815af95a251594d9c101f16f874b76f4d2ee1259df92463731844e5d7d46e"}
Oct 31 08:11:17.133435 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:11:17.133375    4720 scope.go:122] "RemoveContainer" containerID="0a908fb23991bb85069124fdb485850595af4723ffc3b05bf0e3b1ce52d537a6"
Oct 31 08:11:17.135177 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:11:17.135150    4720 generic.go:356] "Generic (PLEG): container finished" podID="f7a7f584-5678-4004-90ad-01cd877da402" containerID="371dc468b642a73853983b315188871144a47a1c3e3f6da18877bc1422267134" exitCode=0
Oct 31 08:11:17.135300 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:11:17.135208    4720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-5b5cc5746b-n4htw" event={"ID":"f7a7f584-5678-4004-90ad-01cd877da402","Type":"ContainerDied","Data":"371dc468b642a73853983b315188871144a47a1c3e3f6da18877bc1422267134"}
Oct 31 08:11:17.135300 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:11:17.135214    4720 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-5b5cc5746b-n4htw"
Oct 31 08:11:17.135300 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:11:17.135246    4720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-5b5cc5746b-n4htw" event={"ID":"f7a7f584-5678-4004-90ad-01cd877da402","Type":"ContainerDied","Data":"c49d4f8f71ae496ff0c7e42386dd404212acfc6cfc2b047d99ba85a30fbe86de"}
Oct 31 08:11:17.143703 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:11:17.143675    4720 scope.go:122] "RemoveContainer" containerID="923917f3c26be86f79a56dca8120ac5e984daf1c8df79fa7412bbc2f4584c4d1"
Oct 31 08:11:17.152027 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:11:17.152010    4720 scope.go:122] "RemoveContainer" containerID="0a908fb23991bb85069124fdb485850595af4723ffc3b05bf0e3b1ce52d537a6"
Oct 31 08:11:17.152313 ip-10-0-139-83 kubenswrapper[4720]: E1031 08:11:17.152293    4720 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0a908fb23991bb85069124fdb485850595af4723ffc3b05bf0e3b1ce52d537a6\": container with ID starting with 0a908fb23991bb85069124fdb485850595af4723ffc3b05bf0e3b1ce52d537a6 not found: ID does not exist" containerID="0a908fb23991bb85069124fdb485850595af4723ffc3b05bf0e3b1ce52d537a6"
Oct 31 08:11:17.152392 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:11:17.152321    4720 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0a908fb23991bb85069124fdb485850595af4723ffc3b05bf0e3b1ce52d537a6"} err="failed to get container status \"0a908fb23991bb85069124fdb485850595af4723ffc3b05bf0e3b1ce52d537a6\": rpc error: code = NotFound desc = could not find container \"0a908fb23991bb85069124fdb485850595af4723ffc3b05bf0e3b1ce52d537a6\": container with ID starting with 0a908fb23991bb85069124fdb485850595af4723ffc3b05bf0e3b1ce52d537a6 not found: ID does not exist"
Oct 31 08:11:17.152392 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:11:17.152361    4720 scope.go:122] "RemoveContainer" containerID="923917f3c26be86f79a56dca8120ac5e984daf1c8df79fa7412bbc2f4584c4d1"
Oct 31 08:11:17.152591 ip-10-0-139-83 kubenswrapper[4720]: E1031 08:11:17.152577    4720 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"923917f3c26be86f79a56dca8120ac5e984daf1c8df79fa7412bbc2f4584c4d1\": container with ID starting with 923917f3c26be86f79a56dca8120ac5e984daf1c8df79fa7412bbc2f4584c4d1 not found: ID does not exist" containerID="923917f3c26be86f79a56dca8120ac5e984daf1c8df79fa7412bbc2f4584c4d1"
Oct 31 08:11:17.152624 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:11:17.152595    4720 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"923917f3c26be86f79a56dca8120ac5e984daf1c8df79fa7412bbc2f4584c4d1"} err="failed to get container status \"923917f3c26be86f79a56dca8120ac5e984daf1c8df79fa7412bbc2f4584c4d1\": rpc error: code = NotFound desc = could not find container \"923917f3c26be86f79a56dca8120ac5e984daf1c8df79fa7412bbc2f4584c4d1\": container with ID starting with 923917f3c26be86f79a56dca8120ac5e984daf1c8df79fa7412bbc2f4584c4d1 not found: ID does not exist"
Oct 31 08:11:17.152624 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:11:17.152607    4720 scope.go:122] "RemoveContainer" containerID="371dc468b642a73853983b315188871144a47a1c3e3f6da18877bc1422267134"
Oct 31 08:11:17.160483 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:11:17.160462    4720 scope.go:122] "RemoveContainer" containerID="b01fa78eb325de154becbde47b4d1db19294b571d1bc0cb8d8d825bccb7d5e71"
Oct 31 08:11:17.168425 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:11:17.168399    4720 scope.go:122] "RemoveContainer" containerID="371dc468b642a73853983b315188871144a47a1c3e3f6da18877bc1422267134"
Oct 31 08:11:17.168699 ip-10-0-139-83 kubenswrapper[4720]: E1031 08:11:17.168683    4720 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"371dc468b642a73853983b315188871144a47a1c3e3f6da18877bc1422267134\": container with ID starting with 371dc468b642a73853983b315188871144a47a1c3e3f6da18877bc1422267134 not found: ID does not exist" containerID="371dc468b642a73853983b315188871144a47a1c3e3f6da18877bc1422267134"
Oct 31 08:11:17.168750 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:11:17.168708    4720 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"371dc468b642a73853983b315188871144a47a1c3e3f6da18877bc1422267134"} err="failed to get container status \"371dc468b642a73853983b315188871144a47a1c3e3f6da18877bc1422267134\": rpc error: code = NotFound desc = could not find container \"371dc468b642a73853983b315188871144a47a1c3e3f6da18877bc1422267134\": container with ID starting with 371dc468b642a73853983b315188871144a47a1c3e3f6da18877bc1422267134 not found: ID does not exist"
Oct 31 08:11:17.168750 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:11:17.168727    4720 scope.go:122] "RemoveContainer" containerID="b01fa78eb325de154becbde47b4d1db19294b571d1bc0cb8d8d825bccb7d5e71"
Oct 31 08:11:17.169017 ip-10-0-139-83 kubenswrapper[4720]: E1031 08:11:17.168982    4720 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b01fa78eb325de154becbde47b4d1db19294b571d1bc0cb8d8d825bccb7d5e71\": container with ID starting with b01fa78eb325de154becbde47b4d1db19294b571d1bc0cb8d8d825bccb7d5e71 not found: ID does not exist" containerID="b01fa78eb325de154becbde47b4d1db19294b571d1bc0cb8d8d825bccb7d5e71"
Oct 31 08:11:17.169078 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:11:17.169027    4720 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b01fa78eb325de154becbde47b4d1db19294b571d1bc0cb8d8d825bccb7d5e71"} err="failed to get container status \"b01fa78eb325de154becbde47b4d1db19294b571d1bc0cb8d8d825bccb7d5e71\": rpc error: code = NotFound desc = could not find container \"b01fa78eb325de154becbde47b4d1db19294b571d1bc0cb8d8d825bccb7d5e71\": container with ID starting with b01fa78eb325de154becbde47b4d1db19294b571d1bc0cb8d8d825bccb7d5e71 not found: ID does not exist"
Oct 31 08:11:17.296611 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:11:17.296577    4720 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-5b5cc5746b-n4htw"]
Oct 31 08:11:17.301823 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:11:17.301790    4720 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-5b5cc5746b-n4htw"]
Oct 31 08:11:17.409681 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:11:17.409592    4720 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6784ddf685-wsmjj"]
Oct 31 08:11:17.413411 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:11:17.413384    4720 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6784ddf685-wsmjj"]
Oct 31 08:11:18.025545 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:11:18.025510    4720 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9856d031-a5d9-4fa0-af75-1e2ede74e2c5" path="/var/lib/kubelet/pods/9856d031-a5d9-4fa0-af75-1e2ede74e2c5/volumes"
Oct 31 08:11:18.025957 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:11:18.025944    4720 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f7a7f584-5678-4004-90ad-01cd877da402" path="/var/lib/kubelet/pods/f7a7f584-5678-4004-90ad-01cd877da402/volumes"
Oct 31 08:11:27.736067 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:11:27.736029    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 08:11:29.246113 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:11:29.246074    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 08:11:32.039251 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:11:32.039218    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 08:11:32.198748 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:11:32.198712    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 08:11:33.736211 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:11:33.736171    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 08:11:37.074525 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:11:37.074483    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 08:11:38.214281 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:11:38.214243    4720 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5bc8cb7b8fwcv45"]
Oct 31 08:11:38.214822 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:11:38.214622    4720 cpu_manager.go:414] "containerMap: removing container" podUID="f7a7f584-5678-4004-90ad-01cd877da402" containerName="main"
Oct 31 08:11:38.214822 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:11:38.214634    4720 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f7a7f584-5678-4004-90ad-01cd877da402" containerName="main"
Oct 31 08:11:38.214822 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:11:38.214650    4720 cpu_manager.go:414] "containerMap: removing container" podUID="f7a7f584-5678-4004-90ad-01cd877da402" containerName="storage-initializer"
Oct 31 08:11:38.214822 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:11:38.214655    4720 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f7a7f584-5678-4004-90ad-01cd877da402" containerName="storage-initializer"
Oct 31 08:11:38.214822 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:11:38.214665    4720 cpu_manager.go:414] "containerMap: removing container" podUID="9856d031-a5d9-4fa0-af75-1e2ede74e2c5" containerName="storage-initializer"
Oct 31 08:11:38.214822 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:11:38.214670    4720 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9856d031-a5d9-4fa0-af75-1e2ede74e2c5" containerName="storage-initializer"
Oct 31 08:11:38.214822 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:11:38.214683    4720 cpu_manager.go:414] "containerMap: removing container" podUID="9856d031-a5d9-4fa0-af75-1e2ede74e2c5" containerName="main"
Oct 31 08:11:38.214822 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:11:38.214688    4720 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9856d031-a5d9-4fa0-af75-1e2ede74e2c5" containerName="main"
Oct 31 08:11:38.214822 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:11:38.214750    4720 memory_manager.go:365] "RemoveStaleState removing state" podUID="f7a7f584-5678-4004-90ad-01cd877da402" containerName="main"
Oct 31 08:11:38.214822 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:11:38.214760    4720 memory_manager.go:365] "RemoveStaleState removing state" podUID="9856d031-a5d9-4fa0-af75-1e2ede74e2c5" containerName="main"
Oct 31 08:11:38.216648 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:11:38.216631    4720 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5bc8cb7b8fwcv45"
Oct 31 08:11:38.269004 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:11:38.268951    4720 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-079cb970\"/\"seaweedfs-s3-creds\""
Oct 31 08:11:38.269196 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:11:38.269041    4720 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-079cb970\"/\"default-dockercfg-qgxnt\""
Oct 31 08:11:38.269196 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:11:38.269049    4720 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-079cb970\"/\"openshift-service-ca.crt\""
Oct 31 08:11:38.318351 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:11:38.318288    4720 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-079cb970\"/\"odh-kserve-custom-ca-bundle\""
Oct 31 08:11:38.318351 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:11:38.318305    4720 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-079cb970\"/\"llmisva690bbc929faec8bc98c767f16c003c1-kserve-self-signed-certs\""
Oct 31 08:11:38.318587 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:11:38.318299    4720 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-079cb970\"/\"kube-root-ca.crt\""
Oct 31 08:11:38.320084 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:11:38.320064    4720 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5bc8cb7b8fwcv45"]
Oct 31 08:11:38.328794 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:11:38.328765    4720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jqfqg\" (UniqueName: \"kubernetes.io/projected/223c3cb7-bbab-4b01-ac94-0edd36507cd0-kube-api-access-jqfqg\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5bc8cb7b8fwcv45\" (UID: \"223c3cb7-bbab-4b01-ac94-0edd36507cd0\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5bc8cb7b8fwcv45"
Oct 31 08:11:38.328922 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:11:38.328809    4720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/223c3cb7-bbab-4b01-ac94-0edd36507cd0-dshm\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5bc8cb7b8fwcv45\" (UID: \"223c3cb7-bbab-4b01-ac94-0edd36507cd0\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5bc8cb7b8fwcv45"
Oct 31 08:11:38.328922 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:11:38.328901    4720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/223c3cb7-bbab-4b01-ac94-0edd36507cd0-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5bc8cb7b8fwcv45\" (UID: \"223c3cb7-bbab-4b01-ac94-0edd36507cd0\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5bc8cb7b8fwcv45"
Oct 31 08:11:38.328997 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:11:38.328945    4720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/223c3cb7-bbab-4b01-ac94-0edd36507cd0-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5bc8cb7b8fwcv45\" (UID: \"223c3cb7-bbab-4b01-ac94-0edd36507cd0\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5bc8cb7b8fwcv45"
Oct 31 08:11:38.328997 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:11:38.328978    4720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/223c3cb7-bbab-4b01-ac94-0edd36507cd0-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5bc8cb7b8fwcv45\" (UID: \"223c3cb7-bbab-4b01-ac94-0edd36507cd0\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5bc8cb7b8fwcv45"
Oct 31 08:11:38.329093 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:11:38.329003    4720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/223c3cb7-bbab-4b01-ac94-0edd36507cd0-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5bc8cb7b8fwcv45\" (UID: \"223c3cb7-bbab-4b01-ac94-0edd36507cd0\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5bc8cb7b8fwcv45"
Oct 31 08:11:38.329152 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:11:38.329122    4720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/223c3cb7-bbab-4b01-ac94-0edd36507cd0-home\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5bc8cb7b8fwcv45\" (UID: \"223c3cb7-bbab-4b01-ac94-0edd36507cd0\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5bc8cb7b8fwcv45"
Oct 31 08:11:38.329209 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:11:38.329162    4720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/223c3cb7-bbab-4b01-ac94-0edd36507cd0-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5bc8cb7b8fwcv45\" (UID: \"223c3cb7-bbab-4b01-ac94-0edd36507cd0\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5bc8cb7b8fwcv45"
Oct 31 08:11:38.330566 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:11:38.330545    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 08:11:38.430475 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:11:38.430426    4720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jqfqg\" (UniqueName: \"kubernetes.io/projected/223c3cb7-bbab-4b01-ac94-0edd36507cd0-kube-api-access-jqfqg\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5bc8cb7b8fwcv45\" (UID: \"223c3cb7-bbab-4b01-ac94-0edd36507cd0\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5bc8cb7b8fwcv45"
Oct 31 08:11:38.430700 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:11:38.430484    4720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/223c3cb7-bbab-4b01-ac94-0edd36507cd0-dshm\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5bc8cb7b8fwcv45\" (UID: \"223c3cb7-bbab-4b01-ac94-0edd36507cd0\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5bc8cb7b8fwcv45"
Oct 31 08:11:38.430700 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:11:38.430536    4720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/223c3cb7-bbab-4b01-ac94-0edd36507cd0-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5bc8cb7b8fwcv45\" (UID: \"223c3cb7-bbab-4b01-ac94-0edd36507cd0\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5bc8cb7b8fwcv45"
Oct 31 08:11:38.430700 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:11:38.430575    4720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/223c3cb7-bbab-4b01-ac94-0edd36507cd0-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5bc8cb7b8fwcv45\" (UID: \"223c3cb7-bbab-4b01-ac94-0edd36507cd0\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5bc8cb7b8fwcv45"
Oct 31 08:11:38.430700 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:11:38.430618    4720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/223c3cb7-bbab-4b01-ac94-0edd36507cd0-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5bc8cb7b8fwcv45\" (UID: \"223c3cb7-bbab-4b01-ac94-0edd36507cd0\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5bc8cb7b8fwcv45"
Oct 31 08:11:38.430700 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:11:38.430649    4720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/223c3cb7-bbab-4b01-ac94-0edd36507cd0-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5bc8cb7b8fwcv45\" (UID: \"223c3cb7-bbab-4b01-ac94-0edd36507cd0\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5bc8cb7b8fwcv45"
Oct 31 08:11:38.430700 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:11:38.430683    4720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/223c3cb7-bbab-4b01-ac94-0edd36507cd0-home\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5bc8cb7b8fwcv45\" (UID: \"223c3cb7-bbab-4b01-ac94-0edd36507cd0\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5bc8cb7b8fwcv45"
Oct 31 08:11:38.431045 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:11:38.430708    4720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/223c3cb7-bbab-4b01-ac94-0edd36507cd0-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5bc8cb7b8fwcv45\" (UID: \"223c3cb7-bbab-4b01-ac94-0edd36507cd0\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5bc8cb7b8fwcv45"
Oct 31 08:11:38.431343 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:11:38.431286    4720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/223c3cb7-bbab-4b01-ac94-0edd36507cd0-home\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5bc8cb7b8fwcv45\" (UID: \"223c3cb7-bbab-4b01-ac94-0edd36507cd0\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5bc8cb7b8fwcv45"
Oct 31 08:11:38.431528 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:11:38.431504    4720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/223c3cb7-bbab-4b01-ac94-0edd36507cd0-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5bc8cb7b8fwcv45\" (UID: \"223c3cb7-bbab-4b01-ac94-0edd36507cd0\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5bc8cb7b8fwcv45"
Oct 31 08:11:38.431630 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:11:38.431601    4720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/223c3cb7-bbab-4b01-ac94-0edd36507cd0-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5bc8cb7b8fwcv45\" (UID: \"223c3cb7-bbab-4b01-ac94-0edd36507cd0\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5bc8cb7b8fwcv45"
Oct 31 08:11:38.431783 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:11:38.431765    4720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/223c3cb7-bbab-4b01-ac94-0edd36507cd0-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5bc8cb7b8fwcv45\" (UID: \"223c3cb7-bbab-4b01-ac94-0edd36507cd0\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5bc8cb7b8fwcv45"
Oct 31 08:11:38.431998 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:11:38.431976    4720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/223c3cb7-bbab-4b01-ac94-0edd36507cd0-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5bc8cb7b8fwcv45\" (UID: \"223c3cb7-bbab-4b01-ac94-0edd36507cd0\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5bc8cb7b8fwcv45"
Oct 31 08:11:38.434628 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:11:38.434598    4720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/223c3cb7-bbab-4b01-ac94-0edd36507cd0-dshm\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5bc8cb7b8fwcv45\" (UID: \"223c3cb7-bbab-4b01-ac94-0edd36507cd0\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5bc8cb7b8fwcv45"
Oct 31 08:11:38.435169 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:11:38.435145    4720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/223c3cb7-bbab-4b01-ac94-0edd36507cd0-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5bc8cb7b8fwcv45\" (UID: \"223c3cb7-bbab-4b01-ac94-0edd36507cd0\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5bc8cb7b8fwcv45"
Oct 31 08:11:38.487597 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:11:38.487516    4720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jqfqg\" (UniqueName: \"kubernetes.io/projected/223c3cb7-bbab-4b01-ac94-0edd36507cd0-kube-api-access-jqfqg\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5bc8cb7b8fwcv45\" (UID: \"223c3cb7-bbab-4b01-ac94-0edd36507cd0\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5bc8cb7b8fwcv45"
Oct 31 08:11:38.530451 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:11:38.530411    4720 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5bc8cb7b8fwcv45"
Oct 31 08:11:38.660882 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:11:38.660845    4720 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-scherx4cr"]
Oct 31 08:11:38.663141 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:11:38.663121    4720 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-scherx4cr"
Oct 31 08:11:38.714772 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:11:38.714741    4720 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-079cb970\"/\"llmisvc-model-fb-opt-125m-route-87882a8e-epp-sa-dockercfg-k579b\""
Oct 31 08:11:38.720405 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:11:38.720376    4720 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-scherx4cr"]
Oct 31 08:11:38.733955 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:11:38.733918    4720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1b98a7f9-d254-4019-9234-5927e1215f54-tls-certs\") pod \"llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-scherx4cr\" (UID: \"1b98a7f9-d254-4019-9234-5927e1215f54\") " pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-scherx4cr"
Oct 31 08:11:38.733955 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:11:38.733953    4720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-krb9b\" (UniqueName: \"kubernetes.io/projected/1b98a7f9-d254-4019-9234-5927e1215f54-kube-api-access-krb9b\") pod \"llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-scherx4cr\" (UID: \"1b98a7f9-d254-4019-9234-5927e1215f54\") " pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-scherx4cr"
Oct 31 08:11:38.779497 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:11:38.779406    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 08:11:38.834624 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:11:38.834583    4720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1b98a7f9-d254-4019-9234-5927e1215f54-tls-certs\") pod \"llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-scherx4cr\" (UID: \"1b98a7f9-d254-4019-9234-5927e1215f54\") " pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-scherx4cr"
Oct 31 08:11:38.834624 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:11:38.834624    4720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-krb9b\" (UniqueName: \"kubernetes.io/projected/1b98a7f9-d254-4019-9234-5927e1215f54-kube-api-access-krb9b\") pod \"llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-scherx4cr\" (UID: \"1b98a7f9-d254-4019-9234-5927e1215f54\") " pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-scherx4cr"
Oct 31 08:11:38.837294 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:11:38.837265    4720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1b98a7f9-d254-4019-9234-5927e1215f54-tls-certs\") pod \"llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-scherx4cr\" (UID: \"1b98a7f9-d254-4019-9234-5927e1215f54\") " pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-scherx4cr"
Oct 31 08:11:38.892879 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:11:38.892846    4720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-krb9b\" (UniqueName: \"kubernetes.io/projected/1b98a7f9-d254-4019-9234-5927e1215f54-kube-api-access-krb9b\") pod \"llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-scherx4cr\" (UID: \"1b98a7f9-d254-4019-9234-5927e1215f54\") " pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-scherx4cr"
Oct 31 08:11:38.958430 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:11:38.958398    4720 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5bc8cb7b8fwcv45"]
Oct 31 08:11:38.961599 ip-10-0-139-83 kubenswrapper[4720]: W1031 08:11:38.961569    4720 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod223c3cb7_bbab_4b01_ac94_0edd36507cd0.slice/crio-7430e80c650c436814bcf4d2f1abc193b987ef6447b7bcef28f049a38fed0dc5 WatchSource:0}: Error finding container 7430e80c650c436814bcf4d2f1abc193b987ef6447b7bcef28f049a38fed0dc5: Status 404 returned error can't find the container with id 7430e80c650c436814bcf4d2f1abc193b987ef6447b7bcef28f049a38fed0dc5
Oct 31 08:11:38.976028 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:11:38.976003    4720 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-scherx4cr"
Oct 31 08:11:39.244306 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:11:39.244271    4720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5bc8cb7b8fwcv45" event={"ID":"223c3cb7-bbab-4b01-ac94-0edd36507cd0","Type":"ContainerStarted","Data":"a8d1ca72eeec800c8cd00adf54d4f7517adbde1b517593bf8bb1afc509f56db6"}
Oct 31 08:11:39.244733 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:11:39.244358    4720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5bc8cb7b8fwcv45" event={"ID":"223c3cb7-bbab-4b01-ac94-0edd36507cd0","Type":"ContainerStarted","Data":"7430e80c650c436814bcf4d2f1abc193b987ef6447b7bcef28f049a38fed0dc5"}
Oct 31 08:11:39.349307 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:11:39.349272    4720 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-scherx4cr"]
Oct 31 08:11:39.352677 ip-10-0-139-83 kubenswrapper[4720]: W1031 08:11:39.352646    4720 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b98a7f9_d254_4019_9234_5927e1215f54.slice/crio-d6c6d45cb2880279e1ef27f31846e4a139f1146296b2efc2a82b904121b9f766 WatchSource:0}: Error finding container d6c6d45cb2880279e1ef27f31846e4a139f1146296b2efc2a82b904121b9f766: Status 404 returned error can't find the container with id d6c6d45cb2880279e1ef27f31846e4a139f1146296b2efc2a82b904121b9f766
Oct 31 08:11:40.250764 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:11:40.250729    4720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-scherx4cr" event={"ID":"1b98a7f9-d254-4019-9234-5927e1215f54","Type":"ContainerStarted","Data":"4974787593c223a7d572f95ac07c84e4c18c96ca1e696c7e3a28f482c5a19d41"}
Oct 31 08:11:40.251199 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:11:40.250780    4720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-scherx4cr" event={"ID":"1b98a7f9-d254-4019-9234-5927e1215f54","Type":"ContainerStarted","Data":"d6c6d45cb2880279e1ef27f31846e4a139f1146296b2efc2a82b904121b9f766"}
Oct 31 08:11:40.363361 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:11:40.363276    4720 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-scherx4cr" podStartSLOduration=2.363260879 podStartE2EDuration="2.363260879s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 08:11:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 08:11:40.362828555 +0000 UTC m=+2554.493896629" watchObservedRunningTime="2026-10-31 08:11:40.363260879 +0000 UTC m=+2554.494328950"
Oct 31 08:11:41.256408 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:11:41.256362    4720 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-scherx4cr"
Oct 31 08:11:42.263843 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:11:42.263806    4720 generic.go:356] "Generic (PLEG): container finished" podID="223c3cb7-bbab-4b01-ac94-0edd36507cd0" containerID="a8d1ca72eeec800c8cd00adf54d4f7517adbde1b517593bf8bb1afc509f56db6" exitCode=0
Oct 31 08:11:42.264387 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:11:42.263882    4720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5bc8cb7b8fwcv45" event={"ID":"223c3cb7-bbab-4b01-ac94-0edd36507cd0","Type":"ContainerDied","Data":"a8d1ca72eeec800c8cd00adf54d4f7517adbde1b517593bf8bb1afc509f56db6"}
Oct 31 08:11:43.272140 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:11:43.272102    4720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5bc8cb7b8fwcv45" event={"ID":"223c3cb7-bbab-4b01-ac94-0edd36507cd0","Type":"ContainerStarted","Data":"306e552625fe06f70c0abb811bca9f697fce6af6b1dfe18b78308e705dd9332b"}
Oct 31 08:11:43.438674 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:11:43.438616    4720 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5bc8cb7b8fwcv45" podStartSLOduration=3.043828581 podStartE2EDuration="5.438601511s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.39477293s" isStatefulPod=false podCreationTimestamp="2026-10-31 08:11:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 08:11:43.437206603 +0000 UTC m=+2557.568274710" watchObservedRunningTime="2026-10-31 08:11:43.438601511 +0000 UTC m=+2557.569669582"
Oct 31 08:11:43.462427 ip-10-0-139-83 kubenswrapper[4720]: E1031 08:11:43.462390    4720 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8m768" containerName="extractor"
Oct 31 08:11:43.531134 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:11:43.531051    4720 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5bc8cb7b8fwcv45"
Oct 31 08:11:45.112547 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:11:45.112516    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 08:11:48.531108 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:11:48.531060    4720 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5bc8cb7b8fwcv45"
Oct 31 08:11:48.532713 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:11:48.532686    4720 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5bc8cb7b8fwcv45" podUID="223c3cb7-bbab-4b01-ac94-0edd36507cd0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/health\": dial tcp 10.132.0.60:8000: connect: connection refused"
Oct 31 08:11:52.714479 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:11:52.714446    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 08:11:52.854091 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:11:52.854049    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 08:11:53.997984 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:11:53.997947    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 08:11:58.111801 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:11:58.111751    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 08:11:58.531183 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:11:58.531132    4720 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5bc8cb7b8fwcv45" podUID="223c3cb7-bbab-4b01-ac94-0edd36507cd0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/health\": dial tcp 10.132.0.60:8000: connect: connection refused"
Oct 31 08:11:59.180427 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:11:59.180385    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 08:12:03.715852 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:12:03.715819    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 08:12:03.876397 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:12:03.876359    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 08:12:04.807055 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:12:04.807013    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 08:12:08.531198 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:12:08.531148    4720 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5bc8cb7b8fwcv45" podUID="223c3cb7-bbab-4b01-ac94-0edd36507cd0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/health\": dial tcp 10.132.0.60:8000: connect: connection refused"
Oct 31 08:12:09.592150 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:12:09.592115    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 08:12:09.735036 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:12:09.735003    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 08:12:10.084547 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:12:10.084508    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 08:12:12.266476 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:12:12.266441    4720 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-scherx4cr"
Oct 31 08:12:13.549109 ip-10-0-139-83 kubenswrapper[4720]: E1031 08:12:13.549071    4720 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8m768" containerName="extractor"
Oct 31 08:12:14.960456 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:12:14.960417    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 08:12:15.109223 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:12:15.109183    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 08:12:15.311040 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:12:15.310954    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 08:12:15.455355 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:12:15.455300    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 08:12:18.531793 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:12:18.531750    4720 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5bc8cb7b8fwcv45" podUID="223c3cb7-bbab-4b01-ac94-0edd36507cd0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/health\": dial tcp 10.132.0.60:8000: connect: connection refused"
Oct 31 08:12:18.888168 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:12:18.888135    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 08:12:19.046376 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:12:19.046342    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 08:12:20.654107 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:12:20.654068    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 08:12:28.530872 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:12:28.530824    4720 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5bc8cb7b8fwcv45" podUID="223c3cb7-bbab-4b01-ac94-0edd36507cd0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/health\": dial tcp 10.132.0.60:8000: connect: connection refused"
Oct 31 08:12:29.587571 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:12:29.587533    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 08:12:38.531058 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:12:38.531015    4720 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5bc8cb7b8fwcv45" podUID="223c3cb7-bbab-4b01-ac94-0edd36507cd0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/health\": dial tcp 10.132.0.60:8000: connect: connection refused"
Oct 31 08:12:43.462938 ip-10-0-139-83 kubenswrapper[4720]: E1031 08:12:43.462897    4720 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8m768" containerName="extractor"
Oct 31 08:12:48.531745 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:12:48.531695    4720 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5bc8cb7b8fwcv45" podUID="223c3cb7-bbab-4b01-ac94-0edd36507cd0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/health\": dial tcp 10.132.0.60:8000: connect: connection refused"
Oct 31 08:12:49.289894 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:12:49.289857    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 08:12:49.437421 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:12:49.437382    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 08:12:54.333948 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:12:54.333912    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 08:12:58.531511 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:12:58.531466    4720 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5bc8cb7b8fwcv45" podUID="223c3cb7-bbab-4b01-ac94-0edd36507cd0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/health\": dial tcp 10.132.0.60:8000: connect: connection refused"
Oct 31 08:13:08.531970 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:13:08.531914    4720 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5bc8cb7b8fwcv45" podUID="223c3cb7-bbab-4b01-ac94-0edd36507cd0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/health\": dial tcp 10.132.0.60:8000: connect: connection refused"
Oct 31 08:13:13.463458 ip-10-0-139-83 kubenswrapper[4720]: E1031 08:13:13.463419    4720 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8m768" containerName="extractor"
Oct 31 08:13:15.354297 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:13:15.354195    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 08:13:18.531348 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:13:18.531269    4720 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5bc8cb7b8fwcv45" podUID="223c3cb7-bbab-4b01-ac94-0edd36507cd0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/health\": dial tcp 10.132.0.60:8000: connect: connection refused"
Oct 31 08:13:28.530881 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:13:28.530818    4720 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5bc8cb7b8fwcv45" podUID="223c3cb7-bbab-4b01-ac94-0edd36507cd0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/health\": dial tcp 10.132.0.60:8000: connect: connection refused"
Oct 31 08:13:29.923599 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:13:29.923543    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 08:13:36.755799 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:13:36.755755    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 08:13:36.917882 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:13:36.917849    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 08:13:38.541683 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:13:38.541644    4720 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5bc8cb7b8fwcv45"
Oct 31 08:13:38.549679 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:13:38.549651    4720 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5bc8cb7b8fwcv45"
Oct 31 08:13:42.644513 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:13:42.644466    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 08:13:43.462177 ip-10-0-139-83 kubenswrapper[4720]: E1031 08:13:43.462127    4720 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8m768" containerName="extractor"
Oct 31 08:13:44.557248 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:13:44.557209    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 08:13:44.993758 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:13:44.993715    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 08:13:48.278177 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:13:48.278030    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 08:13:48.437172 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:13:48.437138    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 08:13:49.212968 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:13:49.212930    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 08:13:49.214208 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:13:49.214181    4720 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5bc8cb7b8fwcv45"]
Oct 31 08:13:49.236065 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:13:49.236028    4720 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-scherx4cr"]
Oct 31 08:13:49.368076 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:13:49.368039    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 08:13:51.853172 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:13:51.853117    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 08:13:52.266591 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:13:52.266506    4720 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-scherx4cr" podUID="1b98a7f9-d254-4019-9234-5927e1215f54" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 08:13:54.677627 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:13:54.677575    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 08:14:02.265507 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:14:02.265466    4720 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-scherx4cr" podUID="1b98a7f9-d254-4019-9234-5927e1215f54" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 08:14:04.243946 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:14:04.243900    4720 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5bc8cb7b8fwcv45" podUID="223c3cb7-bbab-4b01-ac94-0edd36507cd0" containerName="main" containerID="cri-o://306e552625fe06f70c0abb811bca9f697fce6af6b1dfe18b78308e705dd9332b" gracePeriod=45
Oct 31 08:14:04.256685 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:14:04.256646    4720 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-scherx4cr" podUID="1b98a7f9-d254-4019-9234-5927e1215f54" containerName="main" containerID="cri-o://4974787593c223a7d572f95ac07c84e4c18c96ca1e696c7e3a28f482c5a19d41" gracePeriod=45
Oct 31 08:14:04.543200 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:14:04.543089    4720 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5bc8cb7b8fwcv45" podUID="223c3cb7-bbab-4b01-ac94-0edd36507cd0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/health\": dial tcp 10.132.0.60:8000: connect: connection refused"
Oct 31 08:14:04.980645 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:14:04.980605    4720 generic.go:356] "Generic (PLEG): container finished" podID="223c3cb7-bbab-4b01-ac94-0edd36507cd0" containerID="306e552625fe06f70c0abb811bca9f697fce6af6b1dfe18b78308e705dd9332b" exitCode=0
Oct 31 08:14:04.980806 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:14:04.980667    4720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5bc8cb7b8fwcv45" event={"ID":"223c3cb7-bbab-4b01-ac94-0edd36507cd0","Type":"ContainerDied","Data":"306e552625fe06f70c0abb811bca9f697fce6af6b1dfe18b78308e705dd9332b"}
Oct 31 08:14:04.997370 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:14:04.997345    4720 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5bc8cb7b8fwcv45"
Oct 31 08:14:05.048063 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:14:05.047964    4720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/223c3cb7-bbab-4b01-ac94-0edd36507cd0-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/223c3cb7-bbab-4b01-ac94-0edd36507cd0-kserve-provision-location\") pod \"223c3cb7-bbab-4b01-ac94-0edd36507cd0\" (UID: \"223c3cb7-bbab-4b01-ac94-0edd36507cd0\") "
Oct 31 08:14:05.048243 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:14:05.048069    4720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/223c3cb7-bbab-4b01-ac94-0edd36507cd0-home\" (UniqueName: \"kubernetes.io/empty-dir/223c3cb7-bbab-4b01-ac94-0edd36507cd0-home\") pod \"223c3cb7-bbab-4b01-ac94-0edd36507cd0\" (UID: \"223c3cb7-bbab-4b01-ac94-0edd36507cd0\") "
Oct 31 08:14:05.048243 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:14:05.048099    4720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/223c3cb7-bbab-4b01-ac94-0edd36507cd0-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/223c3cb7-bbab-4b01-ac94-0edd36507cd0-cabundle-cert\") pod \"223c3cb7-bbab-4b01-ac94-0edd36507cd0\" (UID: \"223c3cb7-bbab-4b01-ac94-0edd36507cd0\") "
Oct 31 08:14:05.048243 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:14:05.048134    4720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/223c3cb7-bbab-4b01-ac94-0edd36507cd0-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/223c3cb7-bbab-4b01-ac94-0edd36507cd0-tmp-dir\") pod \"223c3cb7-bbab-4b01-ac94-0edd36507cd0\" (UID: \"223c3cb7-bbab-4b01-ac94-0edd36507cd0\") "
Oct 31 08:14:05.048243 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:14:05.048172    4720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/223c3cb7-bbab-4b01-ac94-0edd36507cd0-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/223c3cb7-bbab-4b01-ac94-0edd36507cd0-model-cache\") pod \"223c3cb7-bbab-4b01-ac94-0edd36507cd0\" (UID: \"223c3cb7-bbab-4b01-ac94-0edd36507cd0\") "
Oct 31 08:14:05.048243 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:14:05.048201    4720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/223c3cb7-bbab-4b01-ac94-0edd36507cd0-tls-certs\" (UniqueName: \"kubernetes.io/secret/223c3cb7-bbab-4b01-ac94-0edd36507cd0-tls-certs\") pod \"223c3cb7-bbab-4b01-ac94-0edd36507cd0\" (UID: \"223c3cb7-bbab-4b01-ac94-0edd36507cd0\") "
Oct 31 08:14:05.048243 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:14:05.048228    4720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/223c3cb7-bbab-4b01-ac94-0edd36507cd0-kube-api-access-jqfqg\" (UniqueName: \"kubernetes.io/projected/223c3cb7-bbab-4b01-ac94-0edd36507cd0-kube-api-access-jqfqg\") pod \"223c3cb7-bbab-4b01-ac94-0edd36507cd0\" (UID: \"223c3cb7-bbab-4b01-ac94-0edd36507cd0\") "
Oct 31 08:14:05.048584 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:14:05.048255    4720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/223c3cb7-bbab-4b01-ac94-0edd36507cd0-dshm\" (UniqueName: \"kubernetes.io/empty-dir/223c3cb7-bbab-4b01-ac94-0edd36507cd0-dshm\") pod \"223c3cb7-bbab-4b01-ac94-0edd36507cd0\" (UID: \"223c3cb7-bbab-4b01-ac94-0edd36507cd0\") "
Oct 31 08:14:05.048584 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:14:05.048526    4720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/223c3cb7-bbab-4b01-ac94-0edd36507cd0-model-cache" pod "223c3cb7-bbab-4b01-ac94-0edd36507cd0" (UID: "223c3cb7-bbab-4b01-ac94-0edd36507cd0"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 08:14:05.048695 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:14:05.048661    4720 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/223c3cb7-bbab-4b01-ac94-0edd36507cd0-model-cache\") on node \"ip-10-0-139-83.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 08:14:05.048757 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:14:05.048696    4720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/223c3cb7-bbab-4b01-ac94-0edd36507cd0-cabundle-cert" pod "223c3cb7-bbab-4b01-ac94-0edd36507cd0" (UID: "223c3cb7-bbab-4b01-ac94-0edd36507cd0"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 08:14:05.048958 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:14:05.048921    4720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/223c3cb7-bbab-4b01-ac94-0edd36507cd0-home" pod "223c3cb7-bbab-4b01-ac94-0edd36507cd0" (UID: "223c3cb7-bbab-4b01-ac94-0edd36507cd0"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 08:14:05.051025 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:14:05.050991    4720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/223c3cb7-bbab-4b01-ac94-0edd36507cd0-tls-certs" pod "223c3cb7-bbab-4b01-ac94-0edd36507cd0" (UID: "223c3cb7-bbab-4b01-ac94-0edd36507cd0"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 08:14:05.051025 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:14:05.051001    4720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/223c3cb7-bbab-4b01-ac94-0edd36507cd0-kube-api-access-jqfqg" pod "223c3cb7-bbab-4b01-ac94-0edd36507cd0" (UID: "223c3cb7-bbab-4b01-ac94-0edd36507cd0"). InnerVolumeSpecName "kube-api-access-jqfqg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 08:14:05.051169 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:14:05.051065    4720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/223c3cb7-bbab-4b01-ac94-0edd36507cd0-dshm" pod "223c3cb7-bbab-4b01-ac94-0edd36507cd0" (UID: "223c3cb7-bbab-4b01-ac94-0edd36507cd0"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 08:14:05.068185 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:14:05.068115    4720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/223c3cb7-bbab-4b01-ac94-0edd36507cd0-tmp-dir" pod "223c3cb7-bbab-4b01-ac94-0edd36507cd0" (UID: "223c3cb7-bbab-4b01-ac94-0edd36507cd0"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 08:14:05.078417 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:14:05.078388    4720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/223c3cb7-bbab-4b01-ac94-0edd36507cd0-kserve-provision-location" pod "223c3cb7-bbab-4b01-ac94-0edd36507cd0" (UID: "223c3cb7-bbab-4b01-ac94-0edd36507cd0"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 08:14:05.149154 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:14:05.149111    4720 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/223c3cb7-bbab-4b01-ac94-0edd36507cd0-home\") on node \"ip-10-0-139-83.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 08:14:05.149154 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:14:05.149152    4720 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/223c3cb7-bbab-4b01-ac94-0edd36507cd0-cabundle-cert\") on node \"ip-10-0-139-83.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 08:14:05.149431 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:14:05.149186    4720 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/223c3cb7-bbab-4b01-ac94-0edd36507cd0-tmp-dir\") on node \"ip-10-0-139-83.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 08:14:05.149431 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:14:05.149195    4720 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/223c3cb7-bbab-4b01-ac94-0edd36507cd0-tls-certs\") on node \"ip-10-0-139-83.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 08:14:05.149431 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:14:05.149206    4720 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jqfqg\" (UniqueName: \"kubernetes.io/projected/223c3cb7-bbab-4b01-ac94-0edd36507cd0-kube-api-access-jqfqg\") on node \"ip-10-0-139-83.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 08:14:05.149431 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:14:05.149219    4720 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/223c3cb7-bbab-4b01-ac94-0edd36507cd0-dshm\") on node \"ip-10-0-139-83.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 08:14:05.149431 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:14:05.149232    4720 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/223c3cb7-bbab-4b01-ac94-0edd36507cd0-kserve-provision-location\") on node \"ip-10-0-139-83.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 08:14:05.986832 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:14:05.986795    4720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5bc8cb7b8fwcv45" event={"ID":"223c3cb7-bbab-4b01-ac94-0edd36507cd0","Type":"ContainerDied","Data":"7430e80c650c436814bcf4d2f1abc193b987ef6447b7bcef28f049a38fed0dc5"}
Oct 31 08:14:05.986832 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:14:05.986813    4720 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5bc8cb7b8fwcv45"
Oct 31 08:14:05.987369 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:14:05.986859    4720 scope.go:122] "RemoveContainer" containerID="306e552625fe06f70c0abb811bca9f697fce6af6b1dfe18b78308e705dd9332b"
Oct 31 08:14:05.997012 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:14:05.996994    4720 scope.go:122] "RemoveContainer" containerID="a8d1ca72eeec800c8cd00adf54d4f7517adbde1b517593bf8bb1afc509f56db6"
Oct 31 08:14:06.198832 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:14:06.198800    4720 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5bc8cb7b8fwcv45"]
Oct 31 08:14:06.249029 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:14:06.248875    4720 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5bc8cb7b8fwcv45"]
Oct 31 08:14:06.267707 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:14:06.267683    4720 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-4jszp_b1db0e3b-e85d-4769-8012-cf4b6f740d9d/console-operator/2.log"
Oct 31 08:14:06.275263 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:14:06.275237    4720 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-4jszp_b1db0e3b-e85d-4769-8012-cf4b6f740d9d/console-operator/2.log"
Oct 31 08:14:08.025383 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:14:08.025348    4720 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="223c3cb7-bbab-4b01-ac94-0edd36507cd0" path="/var/lib/kubelet/pods/223c3cb7-bbab-4b01-ac94-0edd36507cd0/volumes"
Oct 31 08:14:12.266152 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:14:12.266110    4720 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-scherx4cr" podUID="1b98a7f9-d254-4019-9234-5927e1215f54" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 08:14:12.266629 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:14:12.266257    4720 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-scherx4cr"
Oct 31 08:14:13.462490 ip-10-0-139-83 kubenswrapper[4720]: E1031 08:14:13.462444    4720 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8m768" containerName="extractor"
Oct 31 08:14:15.636170 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:14:15.636134    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 08:14:22.265668 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:14:22.265629    4720 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-scherx4cr" podUID="1b98a7f9-d254-4019-9234-5927e1215f54" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 08:14:24.223023 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:14:24.222980    4720 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d7985584b8p9lp"]
Oct 31 08:14:24.223527 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:14:24.223357    4720 cpu_manager.go:414] "containerMap: removing container" podUID="223c3cb7-bbab-4b01-ac94-0edd36507cd0" containerName="storage-initializer"
Oct 31 08:14:24.223527 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:14:24.223370    4720 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="223c3cb7-bbab-4b01-ac94-0edd36507cd0" containerName="storage-initializer"
Oct 31 08:14:24.223527 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:14:24.223386    4720 cpu_manager.go:414] "containerMap: removing container" podUID="223c3cb7-bbab-4b01-ac94-0edd36507cd0" containerName="main"
Oct 31 08:14:24.223527 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:14:24.223392    4720 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="223c3cb7-bbab-4b01-ac94-0edd36507cd0" containerName="main"
Oct 31 08:14:24.223527 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:14:24.223473    4720 memory_manager.go:365] "RemoveStaleState removing state" podUID="223c3cb7-bbab-4b01-ac94-0edd36507cd0" containerName="main"
Oct 31 08:14:24.227744 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:14:24.227720    4720 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d7985584b8p9lp"
Oct 31 08:14:24.282866 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:14:24.282832    4720 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d7985584b8p9lp"]
Oct 31 08:14:24.329533 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:14:24.329500    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 08:14:24.329533 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:14:24.329504    4720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/04ce9a21-6083-462c-9806-4b5fa8fd42ba-tls-certs\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d7985584b8p9lp\" (UID: \"04ce9a21-6083-462c-9806-4b5fa8fd42ba\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d7985584b8p9lp"
Oct 31 08:14:24.329817 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:14:24.329583    4720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/04ce9a21-6083-462c-9806-4b5fa8fd42ba-dshm\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d7985584b8p9lp\" (UID: \"04ce9a21-6083-462c-9806-4b5fa8fd42ba\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d7985584b8p9lp"
Oct 31 08:14:24.329817 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:14:24.329608    4720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t9ksd\" (UniqueName: \"kubernetes.io/projected/04ce9a21-6083-462c-9806-4b5fa8fd42ba-kube-api-access-t9ksd\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d7985584b8p9lp\" (UID: \"04ce9a21-6083-462c-9806-4b5fa8fd42ba\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d7985584b8p9lp"
Oct 31 08:14:24.329817 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:14:24.329647    4720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/04ce9a21-6083-462c-9806-4b5fa8fd42ba-home\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d7985584b8p9lp\" (UID: \"04ce9a21-6083-462c-9806-4b5fa8fd42ba\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d7985584b8p9lp"
Oct 31 08:14:24.329817 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:14:24.329709    4720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/04ce9a21-6083-462c-9806-4b5fa8fd42ba-tmp-dir\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d7985584b8p9lp\" (UID: \"04ce9a21-6083-462c-9806-4b5fa8fd42ba\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d7985584b8p9lp"
Oct 31 08:14:24.329817 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:14:24.329785    4720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/04ce9a21-6083-462c-9806-4b5fa8fd42ba-model-cache\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d7985584b8p9lp\" (UID: \"04ce9a21-6083-462c-9806-4b5fa8fd42ba\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d7985584b8p9lp"
Oct 31 08:14:24.330080 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:14:24.330057    4720 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"kube-root-ca.crt\""
Oct 31 08:14:24.330162 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:14:24.330082    4720 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"llmisvb11a9c9832b99b016bc8f8e0ea095712-kserve-self-signed-certs\""
Oct 31 08:14:24.330162 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:14:24.330064    4720 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"openshift-service-ca.crt\""
Oct 31 08:14:24.330162 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:14:24.330109    4720 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"default-dockercfg-p52xl\""
Oct 31 08:14:24.430985 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:14:24.430948    4720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/04ce9a21-6083-462c-9806-4b5fa8fd42ba-model-cache\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d7985584b8p9lp\" (UID: \"04ce9a21-6083-462c-9806-4b5fa8fd42ba\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d7985584b8p9lp"
Oct 31 08:14:24.431319 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:14:24.431289    4720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/04ce9a21-6083-462c-9806-4b5fa8fd42ba-tls-certs\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d7985584b8p9lp\" (UID: \"04ce9a21-6083-462c-9806-4b5fa8fd42ba\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d7985584b8p9lp"
Oct 31 08:14:24.431502 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:14:24.431366    4720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/04ce9a21-6083-462c-9806-4b5fa8fd42ba-dshm\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d7985584b8p9lp\" (UID: \"04ce9a21-6083-462c-9806-4b5fa8fd42ba\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d7985584b8p9lp"
Oct 31 08:14:24.431502 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:14:24.431477    4720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/04ce9a21-6083-462c-9806-4b5fa8fd42ba-model-cache\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d7985584b8p9lp\" (UID: \"04ce9a21-6083-462c-9806-4b5fa8fd42ba\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d7985584b8p9lp"
Oct 31 08:14:24.431502 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:14:24.431408    4720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t9ksd\" (UniqueName: \"kubernetes.io/projected/04ce9a21-6083-462c-9806-4b5fa8fd42ba-kube-api-access-t9ksd\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d7985584b8p9lp\" (UID: \"04ce9a21-6083-462c-9806-4b5fa8fd42ba\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d7985584b8p9lp"
Oct 31 08:14:24.431703 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:14:24.431564    4720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/04ce9a21-6083-462c-9806-4b5fa8fd42ba-home\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d7985584b8p9lp\" (UID: \"04ce9a21-6083-462c-9806-4b5fa8fd42ba\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d7985584b8p9lp"
Oct 31 08:14:24.431703 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:14:24.431595    4720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/04ce9a21-6083-462c-9806-4b5fa8fd42ba-tmp-dir\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d7985584b8p9lp\" (UID: \"04ce9a21-6083-462c-9806-4b5fa8fd42ba\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d7985584b8p9lp"
Oct 31 08:14:24.431952 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:14:24.431927    4720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/04ce9a21-6083-462c-9806-4b5fa8fd42ba-tmp-dir\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d7985584b8p9lp\" (UID: \"04ce9a21-6083-462c-9806-4b5fa8fd42ba\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d7985584b8p9lp"
Oct 31 08:14:24.432232 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:14:24.432206    4720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/04ce9a21-6083-462c-9806-4b5fa8fd42ba-home\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d7985584b8p9lp\" (UID: \"04ce9a21-6083-462c-9806-4b5fa8fd42ba\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d7985584b8p9lp"
Oct 31 08:14:24.435027 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:14:24.434992    4720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/04ce9a21-6083-462c-9806-4b5fa8fd42ba-tls-certs\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d7985584b8p9lp\" (UID: \"04ce9a21-6083-462c-9806-4b5fa8fd42ba\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d7985584b8p9lp"
Oct 31 08:14:24.435137 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:14:24.435067    4720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/04ce9a21-6083-462c-9806-4b5fa8fd42ba-dshm\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d7985584b8p9lp\" (UID: \"04ce9a21-6083-462c-9806-4b5fa8fd42ba\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d7985584b8p9lp"
Oct 31 08:14:24.488037 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:14:24.487959    4720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t9ksd\" (UniqueName: \"kubernetes.io/projected/04ce9a21-6083-462c-9806-4b5fa8fd42ba-kube-api-access-t9ksd\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d7985584b8p9lp\" (UID: \"04ce9a21-6083-462c-9806-4b5fa8fd42ba\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d7985584b8p9lp"
Oct 31 08:14:24.541243 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:14:24.541207    4720 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d7985584b8p9lp"
Oct 31 08:14:24.771956 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:14:24.771874    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 08:14:24.915981 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:14:24.915947    4720 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d7985584b8p9lp"]
Oct 31 08:14:24.919472 ip-10-0-139-83 kubenswrapper[4720]: W1031 08:14:24.919443    4720 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod04ce9a21_6083_462c_9806_4b5fa8fd42ba.slice/crio-31d397fc4df020d094547d3ad2de259aed1a0c3dcd7377c70215e908effb7210 WatchSource:0}: Error finding container 31d397fc4df020d094547d3ad2de259aed1a0c3dcd7377c70215e908effb7210: Status 404 returned error can't find the container with id 31d397fc4df020d094547d3ad2de259aed1a0c3dcd7377c70215e908effb7210
Oct 31 08:14:25.072769 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:14:25.072689    4720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d7985584b8p9lp" event={"ID":"04ce9a21-6083-462c-9806-4b5fa8fd42ba","Type":"ContainerStarted","Data":"b9f892eefc7e980f1ab25b76ef9e8960045ee2f8e7bcfdb76e53c8a413be9354"}
Oct 31 08:14:25.072769 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:14:25.072746    4720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d7985584b8p9lp" event={"ID":"04ce9a21-6083-462c-9806-4b5fa8fd42ba","Type":"ContainerStarted","Data":"31d397fc4df020d094547d3ad2de259aed1a0c3dcd7377c70215e908effb7210"}
Oct 31 08:14:25.186871 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:14:25.186817    4720 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d7985584b8p9lp" podStartSLOduration=1.1867994 podStartE2EDuration="1.1867994s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 08:14:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 08:14:25.1854783 +0000 UTC m=+2719.316546374" watchObservedRunningTime="2026-10-31 08:14:25.1867994 +0000 UTC m=+2719.317867496"
Oct 31 08:14:25.542395 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:14:25.542354    4720 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d7985584b8p9lp"
Oct 31 08:14:30.267321 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:14:30.267275    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 08:14:32.265960 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:14:32.265920    4720 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-scherx4cr" podUID="1b98a7f9-d254-4019-9234-5927e1215f54" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 08:14:34.517761 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:14:34.517697    4720 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-scherx4cr"
Oct 31 08:14:34.525543 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:14:34.525516    4720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1b98a7f9-d254-4019-9234-5927e1215f54-kube-api-access-krb9b\" (UniqueName: \"kubernetes.io/projected/1b98a7f9-d254-4019-9234-5927e1215f54-kube-api-access-krb9b\") pod \"1b98a7f9-d254-4019-9234-5927e1215f54\" (UID: \"1b98a7f9-d254-4019-9234-5927e1215f54\") "
Oct 31 08:14:34.525703 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:14:34.525555    4720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1b98a7f9-d254-4019-9234-5927e1215f54-tls-certs\" (UniqueName: \"kubernetes.io/secret/1b98a7f9-d254-4019-9234-5927e1215f54-tls-certs\") pod \"1b98a7f9-d254-4019-9234-5927e1215f54\" (UID: \"1b98a7f9-d254-4019-9234-5927e1215f54\") "
Oct 31 08:14:34.527778 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:14:34.527750    4720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1b98a7f9-d254-4019-9234-5927e1215f54-tls-certs" pod "1b98a7f9-d254-4019-9234-5927e1215f54" (UID: "1b98a7f9-d254-4019-9234-5927e1215f54"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 08:14:34.527778 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:14:34.527754    4720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1b98a7f9-d254-4019-9234-5927e1215f54-kube-api-access-krb9b" pod "1b98a7f9-d254-4019-9234-5927e1215f54" (UID: "1b98a7f9-d254-4019-9234-5927e1215f54"). InnerVolumeSpecName "kube-api-access-krb9b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 08:14:34.542434 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:14:34.542391    4720 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d7985584b8p9lp"
Oct 31 08:14:34.555608 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:14:34.555579    4720 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d7985584b8p9lp"
Oct 31 08:14:34.626318 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:14:34.626275    4720 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-krb9b\" (UniqueName: \"kubernetes.io/projected/1b98a7f9-d254-4019-9234-5927e1215f54-kube-api-access-krb9b\") on node \"ip-10-0-139-83.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 08:14:34.626318 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:14:34.626308    4720 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1b98a7f9-d254-4019-9234-5927e1215f54-tls-certs\") on node \"ip-10-0-139-83.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 08:14:35.117987 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:14:35.117945    4720 generic.go:356] "Generic (PLEG): container finished" podID="1b98a7f9-d254-4019-9234-5927e1215f54" containerID="4974787593c223a7d572f95ac07c84e4c18c96ca1e696c7e3a28f482c5a19d41" exitCode=0
Oct 31 08:14:35.118181 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:14:35.118005    4720 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-scherx4cr"
Oct 31 08:14:35.118181 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:14:35.118028    4720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-scherx4cr" event={"ID":"1b98a7f9-d254-4019-9234-5927e1215f54","Type":"ContainerDied","Data":"4974787593c223a7d572f95ac07c84e4c18c96ca1e696c7e3a28f482c5a19d41"}
Oct 31 08:14:35.118181 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:14:35.118076    4720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-scherx4cr" event={"ID":"1b98a7f9-d254-4019-9234-5927e1215f54","Type":"ContainerDied","Data":"d6c6d45cb2880279e1ef27f31846e4a139f1146296b2efc2a82b904121b9f766"}
Oct 31 08:14:35.118181 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:14:35.118097    4720 scope.go:122] "RemoveContainer" containerID="4974787593c223a7d572f95ac07c84e4c18c96ca1e696c7e3a28f482c5a19d41"
Oct 31 08:14:35.128075 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:14:35.128060    4720 scope.go:122] "RemoveContainer" containerID="4974787593c223a7d572f95ac07c84e4c18c96ca1e696c7e3a28f482c5a19d41"
Oct 31 08:14:35.128371 ip-10-0-139-83 kubenswrapper[4720]: E1031 08:14:35.128322    4720 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4974787593c223a7d572f95ac07c84e4c18c96ca1e696c7e3a28f482c5a19d41\": container with ID starting with 4974787593c223a7d572f95ac07c84e4c18c96ca1e696c7e3a28f482c5a19d41 not found: ID does not exist" containerID="4974787593c223a7d572f95ac07c84e4c18c96ca1e696c7e3a28f482c5a19d41"
Oct 31 08:14:35.128433 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:14:35.128380    4720 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4974787593c223a7d572f95ac07c84e4c18c96ca1e696c7e3a28f482c5a19d41"} err="failed to get container status \"4974787593c223a7d572f95ac07c84e4c18c96ca1e696c7e3a28f482c5a19d41\": rpc error: code = NotFound desc = could not find container \"4974787593c223a7d572f95ac07c84e4c18c96ca1e696c7e3a28f482c5a19d41\": container with ID starting with 4974787593c223a7d572f95ac07c84e4c18c96ca1e696c7e3a28f482c5a19d41 not found: ID does not exist"
Oct 31 08:14:35.130258 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:14:35.130236    4720 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d7985584b8p9lp"
Oct 31 08:14:43.462216 ip-10-0-139-83 kubenswrapper[4720]: E1031 08:14:43.462166    4720 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8m768" containerName="extractor"
Oct 31 08:14:44.085350 ip-10-0-139-83 kubenswrapper[4720]: E1031 08:14:44.085279    4720 controller.go:251] "Failed to update lease" err="Put \"https://adf9bd3603f5d4feb94c1d7340fbfba2-7a76273f21d70fcd.elb.us-east-2.amazonaws.com:6443/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/ip-10-0-139-83.us-west-1.compute.internal?timeout=10s\": context deadline exceeded"
Oct 31 08:14:54.085979 ip-10-0-139-83 kubenswrapper[4720]: E1031 08:14:54.085889    4720 controller.go:251] "Failed to update lease" err="Put \"https://adf9bd3603f5d4feb94c1d7340fbfba2-7a76273f21d70fcd.elb.us-east-2.amazonaws.com:6443/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/ip-10-0-139-83.us-west-1.compute.internal?timeout=10s\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 31 08:15:04.087163 ip-10-0-139-83 kubenswrapper[4720]: E1031 08:15:04.087103    4720 controller.go:251] "Failed to update lease" err="Put \"https://adf9bd3603f5d4feb94c1d7340fbfba2-7a76273f21d70fcd.elb.us-east-2.amazonaws.com:6443/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/ip-10-0-139-83.us-west-1.compute.internal?timeout=10s\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 31 08:15:13.462429 ip-10-0-139-83 kubenswrapper[4720]: E1031 08:15:13.462388    4720 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8m768" containerName="extractor"
Oct 31 08:15:14.087719 ip-10-0-139-83 kubenswrapper[4720]: E1031 08:15:14.087674    4720 controller.go:251] "Failed to update lease" err="Put \"https://adf9bd3603f5d4feb94c1d7340fbfba2-7a76273f21d70fcd.elb.us-east-2.amazonaws.com:6443/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/ip-10-0-139-83.us-west-1.compute.internal?timeout=10s\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 31 08:15:15.317183 ip-10-0-139-83 kubenswrapper[4720]: E1031 08:15:15.317145    4720 kubelet_node_status.go:524] "Error updating node status, will retry" err="failed to patch status \"{\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"MemoryPressure\\\"},{\\\"type\\\":\\\"DiskPressure\\\"},{\\\"type\\\":\\\"PIDPressure\\\"},{\\\"type\\\":\\\"Ready\\\"}],\\\"conditions\\\":[{\\\"lastHeartbeatTime\\\":\\\"2026-10-31T08:15:05Z\\\",\\\"type\\\":\\\"MemoryPressure\\\"},{\\\"lastHeartbeatTime\\\":\\\"2026-10-31T08:15:05Z\\\",\\\"type\\\":\\\"DiskPressure\\\"},{\\\"lastHeartbeatTime\\\":\\\"2026-10-31T08:15:05Z\\\",\\\"type\\\":\\\"PIDPressure\\\"},{\\\"lastHeartbeatTime\\\":\\\"2026-10-31T08:15:05Z\\\",\\\"type\\\":\\\"Ready\\\"}],\\\"runtimeHandlers\\\":[{\\\"features\\\":{\\\"recursiveReadOnlyMounts\\\":true,\\\"userNamespaces\\\":true},\\\"name\\\":\\\"runc\\\"},{\\\"features\\\":{\\\"recursiveReadOnlyMounts\\\":true,\\\"userNamespaces\\\":true},\\\"name\\\":\\\"crun\\\"},{\\\"features\\\":{\\\"recursiveReadOnlyMounts\\\":true,\\\"userNamespaces\\\":true},\\\"name\\\":\\\"\\\"}]}}\" for node \"ip-10-0-139-83.us-west-1.compute.internal\": Patch \"https://adf9bd3603f5d4feb94c1d7340fbfba2-7a76273f21d70fcd.elb.us-east-2.amazonaws.com:6443/api/v1/nodes/ip-10-0-139-83.us-west-1.compute.internal/status?timeout=10s\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 08:15:15.423212 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:15:15.423174    4720 reflector.go:578] "Warning: watch ended with error" reflector="object-\"kube-system\"/\"original-pull-secret\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Oct 31 08:15:15.423431 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:15:15.423224    4720 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-console-operator\"/\"console-operator-config\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Oct 31 08:15:15.423431 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:15:15.423242    4720 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-console-operator\"/\"trusted-ca\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Oct 31 08:15:15.423431 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:15:15.423223    4720 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Oct 31 08:15:15.423431 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:15:15.423264    4720 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Oct 31 08:15:15.423431 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:15:15.423232    4720 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Oct 31 08:15:15.423431 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:15:15.423174    4720 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-ldrdc\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Oct 31 08:15:15.423431 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:15:15.423198    4720 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-6g5bx\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Oct 31 08:15:15.423431 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:15:15.423232    4720 reflector.go:578] "Warning: watch ended with error" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Oct 31 08:15:15.423431 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:15:15.423322    4720 reflector.go:578] "Warning: watch ended with error" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-gcj9k\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Oct 31 08:15:15.423431 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:15:15.423377    4720 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Oct 31 08:15:15.423431 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:15:15.423376    4720 reflector.go:578] "Warning: watch ended with error" reflector="object-\"kserve\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Oct 31 08:15:15.423431 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:15:15.423398    4720 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-pvmcp\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Oct 31 08:15:15.423431 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:15:15.423379    4720 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Oct 31 08:15:15.423431 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:15:15.423420    4720 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Oct 31 08:15:15.423431 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:15:15.423436    4720 reflector.go:578] "Warning: watch ended with error" reflector="object-\"kuadrant-system\"/\"default-dockercfg-2mh85\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Oct 31 08:15:15.424178 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:15:15.423440    4720 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Oct 31 08:15:15.424178 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:15:15.423446    4720 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Oct 31 08:15:15.424178 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:15:15.423464    4720 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-798mv\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Oct 31 08:15:15.424178 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:15:15.423468    4720 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-6jm8r\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Oct 31 08:15:15.424178 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:15:15.423501    4720 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Oct 31 08:15:15.424178 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:15:15.423505    4720 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Oct 31 08:15:15.424178 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:15:15.423525    4720 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-dns\"/\"dns-default\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Oct 31 08:15:15.424178 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:15:15.423560    4720 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Oct 31 08:15:15.424178 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:15:15.423576    4720 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Oct 31 08:15:15.424178 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:15:15.423576    4720 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Oct 31 08:15:15.424178 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:15:15.423593    4720 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Oct 31 08:15:15.424178 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:15:15.423594    4720 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Oct 31 08:15:15.424178 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:15:15.423602    4720 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Oct 31 08:15:15.424178 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:15:15.423604    4720 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Oct 31 08:15:15.424178 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:15:15.423622    4720 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Oct 31 08:15:15.424178 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:15:15.423631    4720 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Oct 31 08:15:15.424178 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:15:15.423631    4720 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Node" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Oct 31 08:15:15.424178 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:15:15.423654    4720 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-g75wk\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Oct 31 08:15:15.424178 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:15:15.423655    4720 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Oct 31 08:15:15.424178 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:15:15.423670    4720 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Oct 31 08:15:15.424178 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:15:15.423663    4720 reflector.go:578] "Warning: watch ended with error" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-ffr4f\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Oct 31 08:15:15.424178 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:15:15.423688    4720 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-console-operator\"/\"serving-cert\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Oct 31 08:15:15.424178 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:15:15.423695    4720 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-crxcw\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Oct 31 08:15:15.424178 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:15:15.423683    4720 reflector.go:578] "Warning: watch ended with error" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Oct 31 08:15:15.424178 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:15:15.423709    4720 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Oct 31 08:15:15.424178 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:15:15.423717    4720 reflector.go:578] "Warning: watch ended with error" reflector="object-\"kserve\"/\"odh-model-controller-dockercfg-s8mnc\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Oct 31 08:15:15.424178 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:15:15.423726    4720 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Oct 31 08:15:15.424178 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:15:15.423733    4720 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Oct 31 08:15:15.424178 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:15:15.423742    4720 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-dns\"/\"dns-dockercfg-xb4h7\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Oct 31 08:15:15.424178 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:15:15.423752    4720 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Oct 31 08:15:15.424178 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:15:15.423764    4720 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Oct 31 08:15:15.424178 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:15:15.423757    4720 reflector.go:578] "Warning: watch ended with error" reflector="object-\"kserve\"/\"odh-model-controller-webhook-cert\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Oct 31 08:15:15.424178 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:15:15.423781    4720 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Oct 31 08:15:15.424178 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:15:15.423797    4720 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-m5kd5\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Oct 31 08:15:15.424178 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:15:15.423801    4720 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Oct 31 08:15:15.424178 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:15:15.423834    4720 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Oct 31 08:15:15.424178 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:15:15.423851    4720 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"telemetry-config\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Oct 31 08:15:15.424178 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:15:15.423851    4720 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Oct 31 08:15:15.424178 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:15:15.423859    4720 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Oct 31 08:15:15.424178 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:15:15.423870    4720 reflector.go:578] "Warning: watch ended with error" reflector="object-\"kube-system\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Oct 31 08:15:15.424178 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:15:15.423885    4720 reflector.go:578] "Warning: watch ended with error" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Oct 31 08:15:15.424178 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:15:15.423886    4720 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Oct 31 08:15:15.424178 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:15:15.423906    4720 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Oct 31 08:15:15.424178 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:15:15.423906    4720 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Oct 31 08:15:15.424178 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:15:15.423917    4720 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Oct 31 08:15:15.424178 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:15:15.423928    4720 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-4d4mq\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Oct 31 08:15:15.424178 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:15:15.423930    4720 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Oct 31 08:15:15.424178 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:15:15.423938    4720 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Oct 31 08:15:15.424178 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:15:15.423940    4720 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Oct 31 08:15:15.424178 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:15:15.423959    4720 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Oct 31 08:15:15.424178 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:15:15.423955    4720 reflector.go:578] "Warning: watch ended with error" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-fzkxj\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Oct 31 08:15:15.424178 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:15:15.423984    4720 reflector.go:578] "Warning: watch ended with error" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Oct 31 08:15:15.424178 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:15:15.423988    4720 reflector.go:578] "Warning: watch ended with error" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"llmisvb11a9c9832b99b016bc8f8e0ea095712-kserve-self-signed-certs\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Oct 31 08:15:15.424178 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:15:15.423995    4720 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-gttml\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Oct 31 08:15:15.424178 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:15:15.424010    4720 reflector.go:578] "Warning: watch ended with error" reflector="object-\"kube-system\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Oct 31 08:15:15.424178 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:15:15.424014    4720 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Oct 31 08:15:15.424178 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:15:15.424026    4720 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-bpwql\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Oct 31 08:15:15.424178 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:15:15.424026    4720 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Oct 31 08:15:15.424178 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:15:15.424040    4720 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Oct 31 08:15:15.424178 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:15:15.424056    4720 reflector.go:578] "Warning: watch ended with error" reflector="object-\"kserve\"/\"odh-model-controller-metrics-tls\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Oct 31 08:15:15.424178 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:15:15.424057    4720 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Oct 31 08:15:15.424178 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:15:15.424070    4720 reflector.go:578] "Warning: watch ended with error" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Oct 31 08:15:15.424178 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:15:15.424072    4720 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Oct 31 08:15:15.424178 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:15:15.424086    4720 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"router-stats-default\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Oct 31 08:15:15.424178 ip-10-0-139-83 kubenswrapper[4720]: E1031 08:15:15.423981    4720 event.go:368] "Unable to write event (may retry after sleeping)" err="Patch \"https://adf9bd3603f5d4feb94c1d7340fbfba2-7a76273f21d70fcd.elb.us-east-2.amazonaws.com:6443/api/v1/namespaces/e2e-test-llm-inference-service-079cb970/events/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-scherx4cr.18e38f1d3733ae7f\": http2: client connection lost" event="&Event{ObjectMeta:{llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-scherx4cr.18e38f1d3733ae7f  e2e-test-llm-inference-service-079cb970   45287 0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Pod,Namespace:e2e-test-llm-inference-service-079cb970,Name:llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-scherx4cr,UID:1b98a7f9-d254-4019-9234-5927e1215f54,APIVersion:v1,ResourceVersion:42833,FieldPath:spec.containers{main},},Reason:Unhealthy,Message:Readiness probe failed: service unhealthy (responded with \"NOT_SERVING\"),Source:EventSource{Component:kubelet,Host:ip-10-0-139-83.us-west-1.compute.internal,},FirstTimestamp:2026-10-31 08:13:52 +0000 UTC,LastTimestamp:2026-10-31 08:14:32.265960276 +0000 UTC m=+2726.397028350,Count:5,Type:Warning,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-139-83.us-west-1.compute.internal,}"
Oct 31 08:15:15.424178 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:15:15.424104    4720 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Oct 31 08:15:15.424178 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:15:15.424113    4720 reflector.go:578] "Warning: watch ended with error" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-5qqnh\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Oct 31 08:15:15.424178 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:15:15.424123    4720 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Oct 31 08:15:15.424178 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:15:15.424132    4720 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Oct 31 08:15:15.424178 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:15:15.424138    4720 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-x96s4\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Oct 31 08:15:15.424178 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:15:15.424142    4720 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"openshift-ai-inference-openshift-default-dockercfg-z29g8\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Oct 31 08:15:15.424178 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:15:15.424117    4720 reflector.go:578] "Warning: watch ended with error" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Oct 31 08:15:15.424178 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:15:15.424155    4720 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Oct 31 08:15:15.424178 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:15:15.424127    4720 reflector.go:578] "Warning: watch ended with error" reflector="object-\"kserve\"/\"default-dockercfg-zv87w\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Oct 31 08:15:15.424178 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:15:15.424167    4720 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Oct 31 08:15:15.424178 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:15:15.424149    4720 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-dockercfg-d69tc\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Oct 31 08:15:15.424178 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:15:15.424173    4720 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Oct 31 08:15:15.424178 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:15:15.424168    4720 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Oct 31 08:15:15.424178 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:15:15.424195    4720 reflector.go:578] "Warning: watch ended with error" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-jtbjn\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Oct 31 08:15:15.424178 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:15:15.424174    4720 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-w8pvl\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Oct 31 08:15:15.424178 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:15:15.424203    4720 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-7fk4f\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Oct 31 08:15:15.424178 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:15:15.424203    4720 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Oct 31 08:15:15.424178 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:15:15.424203    4720 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Oct 31 08:15:15.424178 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:15:15.424226    4720 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Oct 31 08:15:15.424178 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:15:15.424219    4720 reflector.go:578] "Warning: watch ended with error" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-xhv4l\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Oct 31 08:15:15.424178 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:15:15.424240    4720 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Oct 31 08:15:15.424178 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:15:15.424244    4720 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Oct 31 08:15:15.424178 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:15:15.424226    4720 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Oct 31 08:15:15.424178 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:15:15.424228    4720 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Oct 31 08:15:15.428308 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:15:15.424240    4720 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Oct 31 08:15:15.428308 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:15:15.424249    4720 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-config\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Oct 31 08:15:15.428308 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:15:15.424268    4720 reflector.go:578] "Warning: watch ended with error" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"default-dockercfg-p52xl\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Oct 31 08:15:15.428308 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:15:15.424276    4720 reflector.go:578] "Warning: watch ended with error" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Oct 31 08:15:15.428308 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:15:15.424238    4720 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Oct 31 08:15:15.428308 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:15:15.424276    4720 reflector.go:578] "Warning: watch ended with error" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Oct 31 08:15:15.428308 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:15:15.424196    4720 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Oct 31 08:15:15.428308 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:15:15.424281    4720 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Oct 31 08:15:15.428308 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:15:15.424276    4720 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Oct 31 08:15:15.428308 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:15:15.424304    4720 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Oct 31 08:15:15.428308 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:15:15.424286    4720 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Oct 31 08:15:15.428308 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:15:15.424343    4720 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Oct 31 08:15:15.428308 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:15:15.424353    4720 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Oct 31 08:15:15.428308 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:15:15.424356    4720 reflector.go:578] "Warning: watch ended with error" reflector="pkg/kubelet/config/apiserver.go:66" type="*v1.Pod" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Oct 31 08:15:15.428308 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:15:15.424315    4720 reflector.go:578] "Warning: watch ended with error" reflector="object-\"kserve\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Oct 31 08:15:15.428308 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:15:15.424356    4720 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Oct 31 08:15:15.428308 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:15:15.424364    4720 reflector.go:578] "Warning: watch ended with error" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Oct 31 08:15:15.428308 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:15:15.424382    4720 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Oct 31 08:15:15.428308 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:15:15.424389    4720 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Oct 31 08:15:15.428308 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:15:15.424343    4720 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Oct 31 08:15:15.428308 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:15:15.424393    4720 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Oct 31 08:15:15.428308 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:15:15.424401    4720 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-46dh7\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Oct 31 08:15:15.428308 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:15:15.424418    4720 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Oct 31 08:15:15.428308 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:15:15.424418    4720 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Oct 31 08:15:15.428308 ip-10-0-139-83 kubenswrapper[4720]: E1031 08:15:15.424360    4720 controller.go:251] "Failed to update lease" err="Put \"https://adf9bd3603f5d4feb94c1d7340fbfba2-7a76273f21d70fcd.elb.us-east-2.amazonaws.com:6443/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/ip-10-0-139-83.us-west-1.compute.internal?timeout=10s\": http2: client connection lost"
Oct 31 08:15:15.428308 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:15:15.424369    4720 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-c7ffn\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Oct 31 08:15:15.428308 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:15:15.424831    4720 controller.go:171] "failed to update lease using latest lease, fallback to ensure lease" err="failed 5 attempts to update lease"
Oct 31 08:15:15.428308 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:15:15.424955    4720 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Oct 31 08:15:15.428308 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:15:15.424401    4720 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Oct 31 08:15:15.428308 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:15:15.424404    4720 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Oct 31 08:15:15.428308 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:15:15.425043    4720 reflector.go:578] "Warning: watch ended with error" reflector="object-\"kube-system\"/\"konnectivity-agent\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Oct 31 08:15:15.428308 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:15:15.425046    4720 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-kwsqb\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Oct 31 08:15:15.428308 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:15:15.425072    4720 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Oct 31 08:15:16.060868 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:15:16.060828    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 08:15:16.285242 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:15:16.285209    4720 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\""
Oct 31 08:15:16.290214 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:15:16.290194    4720 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-c7ffn\""
Oct 31 08:15:16.339427 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:15:16.339405    4720 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-xb4h7\""
Oct 31 08:15:16.339427 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:15:16.339420    4720 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Oct 31 08:15:16.339893 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:15:16.339495    4720 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\""
Oct 31 08:15:16.339893 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:15:16.339494    4720 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\""
Oct 31 08:15:16.339893 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:15:16.339575    4720 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-4d4mq\""
Oct 31 08:15:16.343617 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:15:16.343600    4720 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-7fk4f\""
Oct 31 08:15:16.383810 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:15:16.383785    4720 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Oct 31 08:15:16.388691 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:15:16.388674    4720 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-m5kd5\""
Oct 31 08:15:16.388790 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:15:16.388694    4720 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\""
Oct 31 08:15:16.438301 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:15:16.438268    4720 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\""
Oct 31 08:15:16.438499 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:15:16.438485    4720 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-pvmcp\""
Oct 31 08:15:16.438573 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:15:16.438556    4720 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-fzkxj\""
Oct 31 08:15:16.438618 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:15:16.438562    4720 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\""
Oct 31 08:15:16.438830 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:15:16.438817    4720 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\""
Oct 31 08:15:16.487670 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:15:16.487641    4720 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Oct 31 08:15:16.491857 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:15:16.491826    4720 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-config\""
Oct 31 08:15:16.491857 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:15:16.491842    4720 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 31 08:15:16.491857 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:15:16.491838    4720 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Oct 31 08:15:16.492065 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:15:16.491827    4720 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Oct 31 08:15:16.531546 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:15:16.531506    4720 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Oct 31 08:15:16.531843 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:15:16.531802    4720 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\""
Oct 31 08:15:16.536832 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:15:16.536811    4720 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-metrics-tls\""
Oct 31 08:15:16.536955 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:15:16.536935    4720 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"openshift-ai-inference-openshift-default-dockercfg-z29g8\""
Oct 31 08:15:16.536999 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:15:16.536977    4720 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\""
Oct 31 08:15:16.536999 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:15:16.536984    4720 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Oct 31 08:15:16.537193 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:15:16.537180    4720 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 31 08:15:16.537234 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:15:16.537211    4720 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Oct 31 08:15:16.537357 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:15:16.537318    4720 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\""
Oct 31 08:15:16.537838 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:15:16.537821    4720 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Oct 31 08:15:16.538103 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:15:16.538081    4720 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"kube-root-ca.crt\""
Oct 31 08:15:16.538103 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:15:16.538091    4720 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\""
Oct 31 08:15:16.538270 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:15:16.538092    4720 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Oct 31 08:15:16.540786 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:15:16.540768    4720 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-kwsqb\""
Oct 31 08:15:16.540786 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:15:16.540773    4720 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\""
Oct 31 08:15:16.540916 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:15:16.540899    4720 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\""
Oct 31 08:15:16.541093 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:15:16.541078    4720 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\""
Oct 31 08:15:16.541288 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:15:16.541270    4720 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\""
Oct 31 08:15:16.541407 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:15:16.541304    4720 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Oct 31 08:15:16.541407 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:15:16.541388    4720 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Oct 31 08:15:16.541407 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:15:16.541405    4720 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\""
Oct 31 08:15:16.541566 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:15:16.541425    4720 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Oct 31 08:15:16.541566 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:15:16.541452    4720 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Oct 31 08:15:16.541566 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:15:16.541461    4720 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Oct 31 08:15:16.541566 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:15:16.541504    4720 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-2mh85\""
Oct 31 08:15:16.541566 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:15:16.541551    4720 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Oct 31 08:15:16.541906 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:15:16.541772    4720 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\""
Oct 31 08:15:16.541906 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:15:16.541851    4720 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Oct 31 08:15:16.543051 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:15:16.543034    4720 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\""
Oct 31 08:15:16.545230 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:15:16.545216    4720 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Oct 31 08:15:16.546427 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:15:16.546411    4720 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-zv87w\""
Oct 31 08:15:16.557200 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:15:16.557183    4720 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Oct 31 08:15:16.565526 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:15:16.565511    4720 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-gcj9k\""
Oct 31 08:15:16.580825 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:15:16.580807    4720 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"openshift-service-ca.crt\""
Oct 31 08:15:16.584617 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:15:16.584603    4720 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Oct 31 08:15:16.606624 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:15:16.606564    4720 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Oct 31 08:15:16.608592 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:15:16.608571    4720 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\""
Oct 31 08:15:16.613778 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:15:16.613756    4720 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"kube-root-ca.crt\""
Oct 31 08:15:16.614611 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:15:16.614594    4720 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Oct 31 08:15:16.616647 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:15:16.616623    4720 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\""
Oct 31 08:15:16.630807 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:15:16.630786    4720 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 31 08:15:16.634704 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:15:16.634683    4720 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Oct 31 08:15:16.646054 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:15:16.646037    4720 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Oct 31 08:15:16.657873 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:15:16.657850    4720 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-w8pvl\""
Oct 31 08:15:16.658794 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:15:16.658775    4720 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Oct 31 08:15:16.661834 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:15:16.661819    4720 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-crxcw\""
Oct 31 08:15:16.665771 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:15:16.665756    4720 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Oct 31 08:15:16.666573 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:15:16.666561    4720 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Oct 31 08:15:16.667791 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:15:16.667775    4720 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\""
Oct 31 08:15:16.669851 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:15:16.669836    4720 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-x96s4\""
Oct 31 08:15:16.678981 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:15:16.678966    4720 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Oct 31 08:15:16.687075 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:15:16.687054    4720 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\""
Oct 31 08:15:16.690056 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:15:16.690040    4720 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Oct 31 08:15:16.693138 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:15:16.693123    4720 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Oct 31 08:15:16.715547 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:15:16.715528    4720 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\""
Oct 31 08:15:16.731363 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:15:16.731312    4720 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Oct 31 08:15:16.732261 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:15:16.732244    4720 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\""
Oct 31 08:15:16.735467 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:15:16.735451    4720 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Oct 31 08:15:16.739222 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:15:16.739199    4720 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Oct 31 08:15:16.739861 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:15:16.739844    4720 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\""
Oct 31 08:15:16.741116 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:15:16.741097    4720 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Oct 31 08:15:16.751361 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:15:16.751340    4720 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Oct 31 08:15:16.752477 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:15:16.752464    4720 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\""
Oct 31 08:15:16.755681 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:15:16.755666    4720 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\""
Oct 31 08:15:16.766695 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:15:16.766678    4720 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"openshift-service-ca.crt\""
Oct 31 08:15:16.766989 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:15:16.766966    4720 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
Oct 31 08:15:16.770842 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:15:16.770826    4720 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Oct 31 08:15:16.773247 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:15:16.773231    4720 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\""
Oct 31 08:15:16.779562 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:15:16.779546    4720 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Oct 31 08:15:16.796892 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:15:16.796871    4720 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"llmisvb11a9c9832b99b016bc8f8e0ea095712-kserve-self-signed-certs\""
Oct 31 08:15:16.799698 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:15:16.799681    4720 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-6g5bx\""
Oct 31 08:15:16.822860 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:15:16.822834    4720 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-scherx4cr"]
Oct 31 08:15:16.827509 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:15:16.827477    4720 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-scherx4cr"]
Oct 31 08:15:16.837124 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:15:16.837092    4720 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-ldrdc\""
Oct 31 08:15:16.842041 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:15:16.842013    4720 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\""
Oct 31 08:15:16.845257 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:15:16.845232    4720 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Oct 31 08:15:16.845257 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:15:16.845241    4720 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-jtbjn\""
Oct 31 08:15:16.849335 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:15:16.849296    4720 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-xhv4l\""
Oct 31 08:15:16.857864 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:15:16.857798    4720 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Oct 31 08:15:16.869117 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:15:16.869034    4720 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Oct 31 08:15:16.870099 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:15:16.870074    4720 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Oct 31 08:15:16.871702 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:15:16.871677    4720 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 31 08:15:16.879314 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:15:16.879287    4720 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-6jm8r\""
Oct 31 08:15:16.880901 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:15:16.880877    4720 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Oct 31 08:15:16.894312 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:15:16.894282    4720 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Oct 31 08:15:16.895096 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:15:16.895083    4720 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\""
Oct 31 08:15:16.897763 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:15:16.897742    4720 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-dockercfg-s8mnc\""
Oct 31 08:15:16.897883 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:15:16.897819    4720 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\""
Oct 31 08:15:16.903685 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:15:16.903661    4720 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Oct 31 08:15:16.910258 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:15:16.910230    4720 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-gttml\""
Oct 31 08:15:16.923292 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:15:16.923262    4720 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\""
Oct 31 08:15:16.924724 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:15:16.924704    4720 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\""
Oct 31 08:15:16.926458 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:15:16.926441    4720 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"default-dockercfg-p52xl\""
Oct 31 08:15:16.929764 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:15:16.929745    4720 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Oct 31 08:15:16.942490 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:15:16.942464    4720 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\""
Oct 31 08:15:16.956877 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:15:16.956845    4720 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\""
Oct 31 08:15:16.959097 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:15:16.959073    4720 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-46dh7\""
Oct 31 08:15:16.979768 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:15:16.979738    4720 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Oct 31 08:15:16.980021 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:15:16.980004    4720 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-g75wk\""
Oct 31 08:15:16.981023 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:15:16.981008    4720 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\""
Oct 31 08:15:16.982949 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:15:16.982932    4720 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Oct 31 08:15:16.990711 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:15:16.990694    4720 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-5qqnh\""
Oct 31 08:15:16.991225 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:15:16.991209    4720 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\""
Oct 31 08:15:16.995804 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:15:16.995790    4720 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-dockercfg-d69tc\""
Oct 31 08:15:17.006907 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:15:17.006890    4720 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Oct 31 08:15:17.008832 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:15:17.008815    4720 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Oct 31 08:15:17.011344 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:15:17.011314    4720 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\""
Oct 31 08:15:17.038520 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:15:17.038494    4720 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-webhook-cert\""
Oct 31 08:15:17.045569 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:15:17.045552    4720 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Oct 31 08:15:17.050230 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:15:17.050217    4720 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Oct 31 08:15:17.064898 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:15:17.064874    4720 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-798mv\""
Oct 31 08:15:17.066826 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:15:17.066812    4720 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
Oct 31 08:15:17.070077 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:15:17.070060    4720 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-ffr4f\""
Oct 31 08:15:17.070138 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:15:17.070094    4720 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Oct 31 08:15:17.070174 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:15:17.070155    4720 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-bpwql\""
Oct 31 08:15:17.072143 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:15:17.072126    4720 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Oct 31 08:15:17.844849 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:15:17.844810    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 08:15:18.025056 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:15:18.025026    4720 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1b98a7f9-d254-4019-9234-5927e1215f54" path="/var/lib/kubelet/pods/1b98a7f9-d254-4019-9234-5927e1215f54/volumes"
Oct 31 08:15:21.574998 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:15:21.574968    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 08:15:21.733246 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:15:21.733215    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 08:15:22.471722 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:15:22.471681    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 08:15:22.477357 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:15:22.477312    4720 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d7985584b8p9lp"]
Oct 31 08:15:22.488024 ip-10-0-139-83 kubenswrapper[4720]: E1031 08:15:22.487984    4720 log.go:32] "ExecSync cmd from runtime service failed" err="rpc error: code = Unknown desc = command error: , stdout: , stderr: , exit code -1" containerID="b9f892eefc7e980f1ab25b76ef9e8960045ee2f8e7bcfdb76e53c8a413be9354" cmd=["/bin/sleep","15"]
Oct 31 08:15:22.488185 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:15:22.488050    4720 handlers.go:85] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d7985584b8p9lp" message="" err="rpc error: code = Unknown desc = command error: , stdout: , stderr: , exit code -1"
Oct 31 08:15:22.488185 ip-10-0-139-83 kubenswrapper[4720]: E1031 08:15:22.488072    4720 kuberuntime_container.go:780] "PreStop hook failed" err="rpc error: code = Unknown desc = command error: , stdout: , stderr: , exit code -1" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d7985584b8p9lp" podUID="04ce9a21-6083-462c-9806-4b5fa8fd42ba" containerName="main" containerID="cri-o://b9f892eefc7e980f1ab25b76ef9e8960045ee2f8e7bcfdb76e53c8a413be9354"
Oct 31 08:15:22.488185 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:15:22.488133    4720 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d7985584b8p9lp" podUID="04ce9a21-6083-462c-9806-4b5fa8fd42ba" containerName="main" containerID="cri-o://b9f892eefc7e980f1ab25b76ef9e8960045ee2f8e7bcfdb76e53c8a413be9354" gracePeriod=60
Oct 31 08:15:22.638571 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:15:22.638537    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 08:15:22.842983 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:15:22.842949    4720 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d7985584b8p9lp"
Oct 31 08:15:22.865116 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:15:22.865083    4720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/04ce9a21-6083-462c-9806-4b5fa8fd42ba-dshm\" (UniqueName: \"kubernetes.io/empty-dir/04ce9a21-6083-462c-9806-4b5fa8fd42ba-dshm\") pod \"04ce9a21-6083-462c-9806-4b5fa8fd42ba\" (UID: \"04ce9a21-6083-462c-9806-4b5fa8fd42ba\") "
Oct 31 08:15:22.865116 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:15:22.865119    4720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/04ce9a21-6083-462c-9806-4b5fa8fd42ba-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/04ce9a21-6083-462c-9806-4b5fa8fd42ba-model-cache\") pod \"04ce9a21-6083-462c-9806-4b5fa8fd42ba\" (UID: \"04ce9a21-6083-462c-9806-4b5fa8fd42ba\") "
Oct 31 08:15:22.865466 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:15:22.865141    4720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/04ce9a21-6083-462c-9806-4b5fa8fd42ba-home\" (UniqueName: \"kubernetes.io/empty-dir/04ce9a21-6083-462c-9806-4b5fa8fd42ba-home\") pod \"04ce9a21-6083-462c-9806-4b5fa8fd42ba\" (UID: \"04ce9a21-6083-462c-9806-4b5fa8fd42ba\") "
Oct 31 08:15:22.865466 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:15:22.865174    4720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/04ce9a21-6083-462c-9806-4b5fa8fd42ba-kube-api-access-t9ksd\" (UniqueName: \"kubernetes.io/projected/04ce9a21-6083-462c-9806-4b5fa8fd42ba-kube-api-access-t9ksd\") pod \"04ce9a21-6083-462c-9806-4b5fa8fd42ba\" (UID: \"04ce9a21-6083-462c-9806-4b5fa8fd42ba\") "
Oct 31 08:15:22.865466 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:15:22.865208    4720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/04ce9a21-6083-462c-9806-4b5fa8fd42ba-tls-certs\" (UniqueName: \"kubernetes.io/secret/04ce9a21-6083-462c-9806-4b5fa8fd42ba-tls-certs\") pod \"04ce9a21-6083-462c-9806-4b5fa8fd42ba\" (UID: \"04ce9a21-6083-462c-9806-4b5fa8fd42ba\") "
Oct 31 08:15:22.865466 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:15:22.865230    4720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/04ce9a21-6083-462c-9806-4b5fa8fd42ba-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/04ce9a21-6083-462c-9806-4b5fa8fd42ba-tmp-dir\") pod \"04ce9a21-6083-462c-9806-4b5fa8fd42ba\" (UID: \"04ce9a21-6083-462c-9806-4b5fa8fd42ba\") "
Oct 31 08:15:22.865654 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:15:22.865474    4720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04ce9a21-6083-462c-9806-4b5fa8fd42ba-model-cache" pod "04ce9a21-6083-462c-9806-4b5fa8fd42ba" (UID: "04ce9a21-6083-462c-9806-4b5fa8fd42ba"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 08:15:22.865654 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:15:22.865576    4720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04ce9a21-6083-462c-9806-4b5fa8fd42ba-home" pod "04ce9a21-6083-462c-9806-4b5fa8fd42ba" (UID: "04ce9a21-6083-462c-9806-4b5fa8fd42ba"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 08:15:22.865654 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:15:22.865603    4720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04ce9a21-6083-462c-9806-4b5fa8fd42ba-tmp-dir" pod "04ce9a21-6083-462c-9806-4b5fa8fd42ba" (UID: "04ce9a21-6083-462c-9806-4b5fa8fd42ba"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 08:15:22.867645 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:15:22.867623    4720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04ce9a21-6083-462c-9806-4b5fa8fd42ba-dshm" pod "04ce9a21-6083-462c-9806-4b5fa8fd42ba" (UID: "04ce9a21-6083-462c-9806-4b5fa8fd42ba"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 08:15:22.867739 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:15:22.867641    4720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/04ce9a21-6083-462c-9806-4b5fa8fd42ba-kube-api-access-t9ksd" pod "04ce9a21-6083-462c-9806-4b5fa8fd42ba" (UID: "04ce9a21-6083-462c-9806-4b5fa8fd42ba"). InnerVolumeSpecName "kube-api-access-t9ksd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 08:15:22.867739 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:15:22.867655    4720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/04ce9a21-6083-462c-9806-4b5fa8fd42ba-tls-certs" pod "04ce9a21-6083-462c-9806-4b5fa8fd42ba" (UID: "04ce9a21-6083-462c-9806-4b5fa8fd42ba"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 08:15:22.966468 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:15:22.966432    4720 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/04ce9a21-6083-462c-9806-4b5fa8fd42ba-dshm\") on node \"ip-10-0-139-83.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 08:15:22.966468 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:15:22.966462    4720 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/04ce9a21-6083-462c-9806-4b5fa8fd42ba-model-cache\") on node \"ip-10-0-139-83.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 08:15:22.966468 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:15:22.966472    4720 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/04ce9a21-6083-462c-9806-4b5fa8fd42ba-home\") on node \"ip-10-0-139-83.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 08:15:22.966777 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:15:22.966483    4720 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t9ksd\" (UniqueName: \"kubernetes.io/projected/04ce9a21-6083-462c-9806-4b5fa8fd42ba-kube-api-access-t9ksd\") on node \"ip-10-0-139-83.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 08:15:22.966777 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:15:22.966493    4720 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/04ce9a21-6083-462c-9806-4b5fa8fd42ba-tls-certs\") on node \"ip-10-0-139-83.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 08:15:22.966777 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:15:22.966502    4720 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/04ce9a21-6083-462c-9806-4b5fa8fd42ba-tmp-dir\") on node \"ip-10-0-139-83.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 08:15:23.345869 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:15:23.345827    4720 generic.go:356] "Generic (PLEG): container finished" podID="04ce9a21-6083-462c-9806-4b5fa8fd42ba" containerID="b9f892eefc7e980f1ab25b76ef9e8960045ee2f8e7bcfdb76e53c8a413be9354" exitCode=0
Oct 31 08:15:23.346074 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:15:23.345887    4720 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d7985584b8p9lp"
Oct 31 08:15:23.346074 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:15:23.345924    4720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d7985584b8p9lp" event={"ID":"04ce9a21-6083-462c-9806-4b5fa8fd42ba","Type":"ContainerDied","Data":"b9f892eefc7e980f1ab25b76ef9e8960045ee2f8e7bcfdb76e53c8a413be9354"}
Oct 31 08:15:23.346074 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:15:23.345992    4720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d7985584b8p9lp" event={"ID":"04ce9a21-6083-462c-9806-4b5fa8fd42ba","Type":"ContainerDied","Data":"31d397fc4df020d094547d3ad2de259aed1a0c3dcd7377c70215e908effb7210"}
Oct 31 08:15:23.346074 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:15:23.346025    4720 scope.go:122] "RemoveContainer" containerID="b9f892eefc7e980f1ab25b76ef9e8960045ee2f8e7bcfdb76e53c8a413be9354"
Oct 31 08:15:23.355965 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:15:23.355945    4720 scope.go:122] "RemoveContainer" containerID="b9f892eefc7e980f1ab25b76ef9e8960045ee2f8e7bcfdb76e53c8a413be9354"
Oct 31 08:15:23.356216 ip-10-0-139-83 kubenswrapper[4720]: E1031 08:15:23.356194    4720 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b9f892eefc7e980f1ab25b76ef9e8960045ee2f8e7bcfdb76e53c8a413be9354\": container with ID starting with b9f892eefc7e980f1ab25b76ef9e8960045ee2f8e7bcfdb76e53c8a413be9354 not found: ID does not exist" containerID="b9f892eefc7e980f1ab25b76ef9e8960045ee2f8e7bcfdb76e53c8a413be9354"
Oct 31 08:15:23.356264 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:15:23.356225    4720 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b9f892eefc7e980f1ab25b76ef9e8960045ee2f8e7bcfdb76e53c8a413be9354"} err="failed to get container status \"b9f892eefc7e980f1ab25b76ef9e8960045ee2f8e7bcfdb76e53c8a413be9354\": rpc error: code = NotFound desc = could not find container \"b9f892eefc7e980f1ab25b76ef9e8960045ee2f8e7bcfdb76e53c8a413be9354\": container with ID starting with b9f892eefc7e980f1ab25b76ef9e8960045ee2f8e7bcfdb76e53c8a413be9354 not found: ID does not exist"
Oct 31 08:15:23.511439 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:15:23.511409    4720 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d7985584b8p9lp"]
Oct 31 08:15:23.560062 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:15:23.560023    4720 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d7985584b8p9lp"]
Oct 31 08:15:24.025730 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:15:24.025696    4720 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="04ce9a21-6083-462c-9806-4b5fa8fd42ba" path="/var/lib/kubelet/pods/04ce9a21-6083-462c-9806-4b5fa8fd42ba/volumes"
Oct 31 08:15:27.702391 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:15:27.702355    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 08:15:28.281142 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:15:28.281111    4720 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-7687c48f9bmgqtl"]
Oct 31 08:15:28.281578 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:15:28.281549    4720 cpu_manager.go:414] "containerMap: removing container" podUID="04ce9a21-6083-462c-9806-4b5fa8fd42ba" containerName="main"
Oct 31 08:15:28.281578 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:15:28.281569    4720 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="04ce9a21-6083-462c-9806-4b5fa8fd42ba" containerName="main"
Oct 31 08:15:28.281733 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:15:28.281606    4720 cpu_manager.go:414] "containerMap: removing container" podUID="1b98a7f9-d254-4019-9234-5927e1215f54" containerName="main"
Oct 31 08:15:28.281733 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:15:28.281614    4720 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1b98a7f9-d254-4019-9234-5927e1215f54" containerName="main"
Oct 31 08:15:28.281733 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:15:28.281704    4720 memory_manager.go:365] "RemoveStaleState removing state" podUID="04ce9a21-6083-462c-9806-4b5fa8fd42ba" containerName="main"
Oct 31 08:15:28.281733 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:15:28.281722    4720 memory_manager.go:365] "RemoveStaleState removing state" podUID="1b98a7f9-d254-4019-9234-5927e1215f54" containerName="main"
Oct 31 08:15:28.286298 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:15:28.286267    4720 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-7687c48f9bmgqtl"
Oct 31 08:15:28.314318 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:15:28.314288    4720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/16065498-6407-448c-8ca7-24763e3710c8-dshm\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-7687c48f9bmgqtl\" (UID: \"16065498-6407-448c-8ca7-24763e3710c8\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-7687c48f9bmgqtl"
Oct 31 08:15:28.314468 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:15:28.314347    4720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/16065498-6407-448c-8ca7-24763e3710c8-tls-certs\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-7687c48f9bmgqtl\" (UID: \"16065498-6407-448c-8ca7-24763e3710c8\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-7687c48f9bmgqtl"
Oct 31 08:15:28.314468 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:15:28.314430    4720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/16065498-6407-448c-8ca7-24763e3710c8-model-cache\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-7687c48f9bmgqtl\" (UID: \"16065498-6407-448c-8ca7-24763e3710c8\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-7687c48f9bmgqtl"
Oct 31 08:15:28.314553 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:15:28.314477    4720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/16065498-6407-448c-8ca7-24763e3710c8-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-7687c48f9bmgqtl\" (UID: \"16065498-6407-448c-8ca7-24763e3710c8\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-7687c48f9bmgqtl"
Oct 31 08:15:28.314553 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:15:28.314532    4720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/16065498-6407-448c-8ca7-24763e3710c8-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-7687c48f9bmgqtl\" (UID: \"16065498-6407-448c-8ca7-24763e3710c8\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-7687c48f9bmgqtl"
Oct 31 08:15:28.314627 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:15:28.314556    4720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nmcft\" (UniqueName: \"kubernetes.io/projected/16065498-6407-448c-8ca7-24763e3710c8-kube-api-access-nmcft\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-7687c48f9bmgqtl\" (UID: \"16065498-6407-448c-8ca7-24763e3710c8\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-7687c48f9bmgqtl"
Oct 31 08:15:28.314627 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:15:28.314585    4720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/16065498-6407-448c-8ca7-24763e3710c8-home\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-7687c48f9bmgqtl\" (UID: \"16065498-6407-448c-8ca7-24763e3710c8\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-7687c48f9bmgqtl"
Oct 31 08:15:28.345753 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:15:28.345725    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 08:15:28.388589 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:15:28.388556    4720 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-7bc4bd10\"/\"llmisv20d1e8ef181cac7effb0fab7b8590b15-kserve-self-signed-certs\""
Oct 31 08:15:28.388776 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:15:28.388740    4720 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-7bc4bd10\"/\"default-dockercfg-nb7dk\""
Oct 31 08:15:28.388776 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:15:28.388744    4720 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-7bc4bd10\"/\"openshift-service-ca.crt\""
Oct 31 08:15:28.388886 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:15:28.388780    4720 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-7bc4bd10\"/\"kube-root-ca.crt\""
Oct 31 08:15:28.391254 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:15:28.391223    4720 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-7687c48f9bmgqtl"]
Oct 31 08:15:28.415148 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:15:28.415128    4720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/16065498-6407-448c-8ca7-24763e3710c8-model-cache\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-7687c48f9bmgqtl\" (UID: \"16065498-6407-448c-8ca7-24763e3710c8\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-7687c48f9bmgqtl"
Oct 31 08:15:28.415227 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:15:28.415164    4720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/16065498-6407-448c-8ca7-24763e3710c8-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-7687c48f9bmgqtl\" (UID: \"16065498-6407-448c-8ca7-24763e3710c8\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-7687c48f9bmgqtl"
Oct 31 08:15:28.415227 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:15:28.415184    4720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/16065498-6407-448c-8ca7-24763e3710c8-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-7687c48f9bmgqtl\" (UID: \"16065498-6407-448c-8ca7-24763e3710c8\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-7687c48f9bmgqtl"
Oct 31 08:15:28.415464 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:15:28.415446    4720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nmcft\" (UniqueName: \"kubernetes.io/projected/16065498-6407-448c-8ca7-24763e3710c8-kube-api-access-nmcft\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-7687c48f9bmgqtl\" (UID: \"16065498-6407-448c-8ca7-24763e3710c8\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-7687c48f9bmgqtl"
Oct 31 08:15:28.415548 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:15:28.415495    4720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/16065498-6407-448c-8ca7-24763e3710c8-home\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-7687c48f9bmgqtl\" (UID: \"16065498-6407-448c-8ca7-24763e3710c8\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-7687c48f9bmgqtl"
Oct 31 08:15:28.415609 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:15:28.415553    4720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/16065498-6407-448c-8ca7-24763e3710c8-model-cache\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-7687c48f9bmgqtl\" (UID: \"16065498-6407-448c-8ca7-24763e3710c8\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-7687c48f9bmgqtl"
Oct 31 08:15:28.415665 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:15:28.415605    4720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/16065498-6407-448c-8ca7-24763e3710c8-dshm\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-7687c48f9bmgqtl\" (UID: \"16065498-6407-448c-8ca7-24763e3710c8\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-7687c48f9bmgqtl"
Oct 31 08:15:28.415665 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:15:28.415614    4720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/16065498-6407-448c-8ca7-24763e3710c8-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-7687c48f9bmgqtl\" (UID: \"16065498-6407-448c-8ca7-24763e3710c8\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-7687c48f9bmgqtl"
Oct 31 08:15:28.415665 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:15:28.415636    4720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/16065498-6407-448c-8ca7-24763e3710c8-tls-certs\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-7687c48f9bmgqtl\" (UID: \"16065498-6407-448c-8ca7-24763e3710c8\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-7687c48f9bmgqtl"
Oct 31 08:15:28.415782 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:15:28.415688    4720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/16065498-6407-448c-8ca7-24763e3710c8-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-7687c48f9bmgqtl\" (UID: \"16065498-6407-448c-8ca7-24763e3710c8\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-7687c48f9bmgqtl"
Oct 31 08:15:28.415823 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:15:28.415813    4720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/16065498-6407-448c-8ca7-24763e3710c8-home\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-7687c48f9bmgqtl\" (UID: \"16065498-6407-448c-8ca7-24763e3710c8\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-7687c48f9bmgqtl"
Oct 31 08:15:28.417995 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:15:28.417968    4720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/16065498-6407-448c-8ca7-24763e3710c8-dshm\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-7687c48f9bmgqtl\" (UID: \"16065498-6407-448c-8ca7-24763e3710c8\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-7687c48f9bmgqtl"
Oct 31 08:15:28.418228 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:15:28.418208    4720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/16065498-6407-448c-8ca7-24763e3710c8-tls-certs\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-7687c48f9bmgqtl\" (UID: \"16065498-6407-448c-8ca7-24763e3710c8\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-7687c48f9bmgqtl"
Oct 31 08:15:28.471344 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:15:28.471294    4720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nmcft\" (UniqueName: \"kubernetes.io/projected/16065498-6407-448c-8ca7-24763e3710c8-kube-api-access-nmcft\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-7687c48f9bmgqtl\" (UID: \"16065498-6407-448c-8ca7-24763e3710c8\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-7687c48f9bmgqtl"
Oct 31 08:15:28.604981 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:15:28.604947    4720 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-7687c48f9bmgqtl"
Oct 31 08:15:28.647028 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:15:28.646995    4720 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-schenpfzd"]
Oct 31 08:15:28.650659 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:15:28.650635    4720 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-schenpfzd"
Oct 31 08:15:28.703816 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:15:28.703412    4720 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-7bc4bd10\"/\"llmisvc-model-fb-opt-125m-oci-r-0fba19de-epp-sa-dockercfg-bvslj\""
Oct 31 08:15:28.706521 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:15:28.706494    4720 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-schenpfzd"]
Oct 31 08:15:28.717778 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:15:28.717748    4720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5jgkq\" (UniqueName: \"kubernetes.io/projected/772d92f9-2bd0-419f-86b6-b7877d5b4bc2-kube-api-access-5jgkq\") pod \"llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-schenpfzd\" (UID: \"772d92f9-2bd0-419f-86b6-b7877d5b4bc2\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-schenpfzd"
Oct 31 08:15:28.717934 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:15:28.717819    4720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/772d92f9-2bd0-419f-86b6-b7877d5b4bc2-tls-certs\") pod \"llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-schenpfzd\" (UID: \"772d92f9-2bd0-419f-86b6-b7877d5b4bc2\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-schenpfzd"
Oct 31 08:15:28.748680 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:15:28.748643    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 08:15:28.819002 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:15:28.818967    4720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/772d92f9-2bd0-419f-86b6-b7877d5b4bc2-tls-certs\") pod \"llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-schenpfzd\" (UID: \"772d92f9-2bd0-419f-86b6-b7877d5b4bc2\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-schenpfzd"
Oct 31 08:15:28.819180 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:15:28.819039    4720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5jgkq\" (UniqueName: \"kubernetes.io/projected/772d92f9-2bd0-419f-86b6-b7877d5b4bc2-kube-api-access-5jgkq\") pod \"llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-schenpfzd\" (UID: \"772d92f9-2bd0-419f-86b6-b7877d5b4bc2\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-schenpfzd"
Oct 31 08:15:28.821715 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:15:28.821686    4720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/772d92f9-2bd0-419f-86b6-b7877d5b4bc2-tls-certs\") pod \"llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-schenpfzd\" (UID: \"772d92f9-2bd0-419f-86b6-b7877d5b4bc2\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-schenpfzd"
Oct 31 08:15:28.874772 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:15:28.874700    4720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5jgkq\" (UniqueName: \"kubernetes.io/projected/772d92f9-2bd0-419f-86b6-b7877d5b4bc2-kube-api-access-5jgkq\") pod \"llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-schenpfzd\" (UID: \"772d92f9-2bd0-419f-86b6-b7877d5b4bc2\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-schenpfzd"
Oct 31 08:15:28.976301 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:15:28.976264    4720 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-schenpfzd"
Oct 31 08:15:29.032568 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:15:29.032532    4720 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-7687c48f9bmgqtl"]
Oct 31 08:15:29.097226 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:15:29.097199    4720 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 31 08:15:29.349237 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:15:29.349207    4720 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-schenpfzd"]
Oct 31 08:15:29.351810 ip-10-0-139-83 kubenswrapper[4720]: W1031 08:15:29.351782    4720 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod772d92f9_2bd0_419f_86b6_b7877d5b4bc2.slice/crio-87d36dc9b1e61edafaa15ae934eecebeca950a742d66725e2f7f75543b38167f WatchSource:0}: Error finding container 87d36dc9b1e61edafaa15ae934eecebeca950a742d66725e2f7f75543b38167f: Status 404 returned error can't find the container with id 87d36dc9b1e61edafaa15ae934eecebeca950a742d66725e2f7f75543b38167f
Oct 31 08:15:29.379060 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:15:29.379027    4720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-7687c48f9bmgqtl" event={"ID":"16065498-6407-448c-8ca7-24763e3710c8","Type":"ContainerStarted","Data":"8627783f80d4d8bdb9433b8d8ba4fa6e322438876b6dd3cd9b7d9cc5096a454b"}
Oct 31 08:15:29.380222 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:15:29.380198    4720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-schenpfzd" event={"ID":"772d92f9-2bd0-419f-86b6-b7877d5b4bc2","Type":"ContainerStarted","Data":"87d36dc9b1e61edafaa15ae934eecebeca950a742d66725e2f7f75543b38167f"}
Oct 31 08:15:30.386856 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:15:30.386815    4720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-schenpfzd" event={"ID":"772d92f9-2bd0-419f-86b6-b7877d5b4bc2","Type":"ContainerStarted","Data":"7bf814f0104f326f881ce59f6fc8173a0ad16e343679093c2ccf830a19bc8ab9"}
Oct 31 08:15:30.387296 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:15:30.387038    4720 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-schenpfzd"
Oct 31 08:15:30.498887 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:15:30.498832    4720 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-schenpfzd" podStartSLOduration=2.498816363 podStartE2EDuration="2.498816363s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 08:15:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 08:15:30.497794915 +0000 UTC m=+2784.628862985" watchObservedRunningTime="2026-10-31 08:15:30.498816363 +0000 UTC m=+2784.629884431"
Oct 31 08:15:30.598260 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:15:30.598221    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 08:15:32.663069 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:15:32.663037    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 08:15:33.167844 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:15:33.167806    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 08:15:33.316125 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:15:33.316066    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 08:15:43.463383 ip-10-0-139-83 kubenswrapper[4720]: E1031 08:15:43.463339    4720 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8m768" containerName="extractor"
Oct 31 08:15:44.459342 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:15:44.459280    4720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-7687c48f9bmgqtl" event={"ID":"16065498-6407-448c-8ca7-24763e3710c8","Type":"ContainerStarted","Data":"1d8673799b09771d7a4715e34a613e4445ca4819cb2c635f330e3ef341bc86cd"}
Oct 31 08:15:45.465103 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:15:45.465066    4720 generic.go:356] "Generic (PLEG): container finished" podID="16065498-6407-448c-8ca7-24763e3710c8" containerID="1d8673799b09771d7a4715e34a613e4445ca4819cb2c635f330e3ef341bc86cd" exitCode=0
Oct 31 08:15:45.465563 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:15:45.465154    4720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-7687c48f9bmgqtl" event={"ID":"16065498-6407-448c-8ca7-24763e3710c8","Type":"ContainerDied","Data":"1d8673799b09771d7a4715e34a613e4445ca4819cb2c635f330e3ef341bc86cd"}
Oct 31 08:15:46.471312 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:15:46.471270    4720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-7687c48f9bmgqtl" event={"ID":"16065498-6407-448c-8ca7-24763e3710c8","Type":"ContainerStarted","Data":"e2476d4267148b473e8a067fcecd6f83a7e1a7896c785901e40e370bb9d72859"}
Oct 31 08:15:47.478257 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:15:47.478219    4720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-7687c48f9bmgqtl" event={"ID":"16065498-6407-448c-8ca7-24763e3710c8","Type":"ContainerStarted","Data":"278312a43cc017359a0b106596719d566fe22e1f7c284c2c006e1ebf76a1ac8f"}
Oct 31 08:15:47.605987 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:15:47.605951    4720 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-7687c48f9bmgqtl"
Oct 31 08:15:47.645482 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:15:47.645428    4720 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-7687c48f9bmgqtl" podStartSLOduration=3.4529377820000002 podStartE2EDuration="19.645412211s" totalImagesPullingTime="15.107428575s" totalInitContainerRuntime="1.085045854s" isStatefulPod=false podCreationTimestamp="2026-10-31 08:15:28 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 08:15:47.642742717 +0000 UTC m=+2801.773810791" watchObservedRunningTime="2026-10-31 08:15:47.645412211 +0000 UTC m=+2801.776480282"
Oct 31 08:15:48.605893 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:15:48.605850    4720 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-7687c48f9bmgqtl"
Oct 31 08:15:48.607368 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:15:48.607309    4720 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-7687c48f9bmgqtl" podUID="16065498-6407-448c-8ca7-24763e3710c8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/health\": dial tcp 10.132.0.63:8000: connect: connection refused"
Oct 31 08:15:55.357619 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:15:55.357571    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 08:15:58.605727 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:15:58.605685    4720 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-7687c48f9bmgqtl" podUID="16065498-6407-448c-8ca7-24763e3710c8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/health\": dial tcp 10.132.0.63:8000: connect: connection refused"
Oct 31 08:16:01.393578 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:16:01.393546    4720 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-schenpfzd"
Oct 31 08:16:08.605437 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:16:08.605389    4720 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-7687c48f9bmgqtl" podUID="16065498-6407-448c-8ca7-24763e3710c8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/health\": dial tcp 10.132.0.63:8000: connect: connection refused"
Oct 31 08:16:11.875210 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:16:11.875168    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 08:16:12.030487 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:16:12.030448    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 08:16:13.461710 ip-10-0-139-83 kubenswrapper[4720]: E1031 08:16:13.461673    4720 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8m768" containerName="extractor"
Oct 31 08:16:16.304640 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:16:16.304554    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 08:16:18.606212 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:16:18.606158    4720 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-7687c48f9bmgqtl" podUID="16065498-6407-448c-8ca7-24763e3710c8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/health\": dial tcp 10.132.0.63:8000: connect: connection refused"
Oct 31 08:16:28.605888 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:16:28.605835    4720 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-7687c48f9bmgqtl" podUID="16065498-6407-448c-8ca7-24763e3710c8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/health\": dial tcp 10.132.0.63:8000: connect: connection refused"
Oct 31 08:16:28.947535 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:16:28.947445    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 08:16:30.946980 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:16:30.946949    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 08:16:38.605567 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:16:38.605514    4720 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-7687c48f9bmgqtl" podUID="16065498-6407-448c-8ca7-24763e3710c8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/health\": dial tcp 10.132.0.63:8000: connect: connection refused"
Oct 31 08:16:43.462463 ip-10-0-139-83 kubenswrapper[4720]: E1031 08:16:43.462427    4720 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8m768" containerName="extractor"
Oct 31 08:16:48.605622 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:16:48.605570    4720 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-7687c48f9bmgqtl" podUID="16065498-6407-448c-8ca7-24763e3710c8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/health\": dial tcp 10.132.0.63:8000: connect: connection refused"
Oct 31 08:16:55.708006 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:16:55.707971    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 08:16:58.606231 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:16:58.606187    4720 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-7687c48f9bmgqtl" podUID="16065498-6407-448c-8ca7-24763e3710c8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/health\": dial tcp 10.132.0.63:8000: connect: connection refused"
Oct 31 08:17:01.292745 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:17:01.292711    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 08:17:01.448415 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:17:01.448376    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 08:17:06.652237 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:17:06.652187    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 08:17:07.199624 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:17:07.199582    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 08:17:07.952949 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:17:07.952919    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 08:17:08.605515 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:17:08.605469    4720 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-7687c48f9bmgqtl" podUID="16065498-6407-448c-8ca7-24763e3710c8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/health\": dial tcp 10.132.0.63:8000: connect: connection refused"
Oct 31 08:17:13.445887 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:17:13.445839    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 08:17:13.463405 ip-10-0-139-83 kubenswrapper[4720]: E1031 08:17:13.463367    4720 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8m768" containerName="extractor"
Oct 31 08:17:13.587599 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:17:13.587559    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 08:17:14.262422 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:17:14.262387    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 08:17:16.639477 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:17:16.639410    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 08:17:18.606211 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:17:18.606147    4720 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-7687c48f9bmgqtl" podUID="16065498-6407-448c-8ca7-24763e3710c8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/health\": dial tcp 10.132.0.63:8000: connect: connection refused"
Oct 31 08:17:28.606299 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:17:28.606247    4720 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-7687c48f9bmgqtl" podUID="16065498-6407-448c-8ca7-24763e3710c8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/health\": dial tcp 10.132.0.63:8000: connect: connection refused"
Oct 31 08:17:31.278102 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:17:31.278062    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 08:17:38.616045 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:17:38.616006    4720 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-7687c48f9bmgqtl"
Oct 31 08:17:38.624030 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:17:38.624005    4720 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-7687c48f9bmgqtl"
Oct 31 08:17:42.314175 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:17:42.314141    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 08:17:42.506460 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:17:42.506421    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 08:17:43.462954 ip-10-0-139-83 kubenswrapper[4720]: E1031 08:17:43.462919    4720 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8m768" containerName="extractor"
Oct 31 08:17:48.312928 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:17:48.312847    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 08:17:48.329983 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:17:48.329942    4720 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-7687c48f9bmgqtl"]
Oct 31 08:17:48.330344 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:17:48.330294    4720 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-7687c48f9bmgqtl" podUID="16065498-6407-448c-8ca7-24763e3710c8" containerName="modelcar" containerID="cri-o://278312a43cc017359a0b106596719d566fe22e1f7c284c2c006e1ebf76a1ac8f" gracePeriod=60
Oct 31 08:17:48.332795 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:17:48.332767    4720 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-schenpfzd"]
Oct 31 08:17:48.467530 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:17:48.467488    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 08:17:49.084453 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:17:49.084420    4720 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-7bc4bd10_llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-7687c48f9bmgqtl_16065498-6407-448c-8ca7-24763e3710c8/modelcar/0.log"
Oct 31 08:17:49.085117 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:17:49.085090    4720 generic.go:356] "Generic (PLEG): container finished" podID="16065498-6407-448c-8ca7-24763e3710c8" containerID="278312a43cc017359a0b106596719d566fe22e1f7c284c2c006e1ebf76a1ac8f" exitCode=143
Oct 31 08:17:49.085254 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:17:49.085169    4720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-7687c48f9bmgqtl" event={"ID":"16065498-6407-448c-8ca7-24763e3710c8","Type":"ContainerDied","Data":"278312a43cc017359a0b106596719d566fe22e1f7c284c2c006e1ebf76a1ac8f"}
Oct 31 08:17:51.394148 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:17:51.394097    4720 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-schenpfzd" podUID="772d92f9-2bd0-419f-86b6-b7877d5b4bc2" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 08:17:53.172204 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:17:53.172160    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 08:17:53.567574 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:17:53.567489    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 08:17:56.038535 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:17:56.038487    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 08:18:01.393898 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:18:01.393857    4720 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-schenpfzd" podUID="772d92f9-2bd0-419f-86b6-b7877d5b4bc2" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 08:18:01.985622 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:18:01.985587    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 08:18:03.352985 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:18:03.352912    4720 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-schenpfzd" podUID="772d92f9-2bd0-419f-86b6-b7877d5b4bc2" containerName="main" containerID="cri-o://7bf814f0104f326f881ce59f6fc8173a0ad16e343679093c2ccf830a19bc8ab9" gracePeriod=45
Oct 31 08:18:03.354490 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:18:03.354361    4720 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-7687c48f9bmgqtl" podUID="16065498-6407-448c-8ca7-24763e3710c8" containerName="main" containerID="cri-o://e2476d4267148b473e8a067fcecd6f83a7e1a7896c785901e40e370bb9d72859" gracePeriod=45
Oct 31 08:18:03.617694 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:18:03.617601    4720 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-7687c48f9bmgqtl" podUID="16065498-6407-448c-8ca7-24763e3710c8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/health\": dial tcp 10.132.0.63:8000: connect: connection refused"
Oct 31 08:18:04.154563 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:18:04.154494    4720 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-7bc4bd10_llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-7687c48f9bmgqtl_16065498-6407-448c-8ca7-24763e3710c8/modelcar/0.log"
Oct 31 08:18:04.157527 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:18:04.157503    4720 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-7687c48f9bmgqtl"
Oct 31 08:18:04.158052 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:18:04.158035    4720 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-7bc4bd10_llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-7687c48f9bmgqtl_16065498-6407-448c-8ca7-24763e3710c8/modelcar/0.log"
Oct 31 08:18:04.158730 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:18:04.158708    4720 generic.go:356] "Generic (PLEG): container finished" podID="16065498-6407-448c-8ca7-24763e3710c8" containerID="e2476d4267148b473e8a067fcecd6f83a7e1a7896c785901e40e370bb9d72859" exitCode=0
Oct 31 08:18:04.158816 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:18:04.158746    4720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-7687c48f9bmgqtl" event={"ID":"16065498-6407-448c-8ca7-24763e3710c8","Type":"ContainerDied","Data":"e2476d4267148b473e8a067fcecd6f83a7e1a7896c785901e40e370bb9d72859"}
Oct 31 08:18:04.158816 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:18:04.158793    4720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-7687c48f9bmgqtl" event={"ID":"16065498-6407-448c-8ca7-24763e3710c8","Type":"ContainerDied","Data":"8627783f80d4d8bdb9433b8d8ba4fa6e322438876b6dd3cd9b7d9cc5096a454b"}
Oct 31 08:18:04.158884 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:18:04.158819    4720 scope.go:122] "RemoveContainer" containerID="278312a43cc017359a0b106596719d566fe22e1f7c284c2c006e1ebf76a1ac8f"
Oct 31 08:18:04.167629 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:18:04.167611    4720 scope.go:122] "RemoveContainer" containerID="e2476d4267148b473e8a067fcecd6f83a7e1a7896c785901e40e370bb9d72859"
Oct 31 08:18:04.175837 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:18:04.175815    4720 scope.go:122] "RemoveContainer" containerID="1d8673799b09771d7a4715e34a613e4445ca4819cb2c635f330e3ef341bc86cd"
Oct 31 08:18:04.183454 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:18:04.183435    4720 scope.go:122] "RemoveContainer" containerID="278312a43cc017359a0b106596719d566fe22e1f7c284c2c006e1ebf76a1ac8f"
Oct 31 08:18:04.183731 ip-10-0-139-83 kubenswrapper[4720]: E1031 08:18:04.183711    4720 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"278312a43cc017359a0b106596719d566fe22e1f7c284c2c006e1ebf76a1ac8f\": container with ID starting with 278312a43cc017359a0b106596719d566fe22e1f7c284c2c006e1ebf76a1ac8f not found: ID does not exist" containerID="278312a43cc017359a0b106596719d566fe22e1f7c284c2c006e1ebf76a1ac8f"
Oct 31 08:18:04.183806 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:18:04.183747    4720 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"278312a43cc017359a0b106596719d566fe22e1f7c284c2c006e1ebf76a1ac8f"} err="failed to get container status \"278312a43cc017359a0b106596719d566fe22e1f7c284c2c006e1ebf76a1ac8f\": rpc error: code = NotFound desc = could not find container \"278312a43cc017359a0b106596719d566fe22e1f7c284c2c006e1ebf76a1ac8f\": container with ID starting with 278312a43cc017359a0b106596719d566fe22e1f7c284c2c006e1ebf76a1ac8f not found: ID does not exist"
Oct 31 08:18:04.183806 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:18:04.183774    4720 scope.go:122] "RemoveContainer" containerID="e2476d4267148b473e8a067fcecd6f83a7e1a7896c785901e40e370bb9d72859"
Oct 31 08:18:04.184048 ip-10-0-139-83 kubenswrapper[4720]: E1031 08:18:04.184011    4720 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e2476d4267148b473e8a067fcecd6f83a7e1a7896c785901e40e370bb9d72859\": container with ID starting with e2476d4267148b473e8a067fcecd6f83a7e1a7896c785901e40e370bb9d72859 not found: ID does not exist" containerID="e2476d4267148b473e8a067fcecd6f83a7e1a7896c785901e40e370bb9d72859"
Oct 31 08:18:04.184094 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:18:04.184053    4720 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e2476d4267148b473e8a067fcecd6f83a7e1a7896c785901e40e370bb9d72859"} err="failed to get container status \"e2476d4267148b473e8a067fcecd6f83a7e1a7896c785901e40e370bb9d72859\": rpc error: code = NotFound desc = could not find container \"e2476d4267148b473e8a067fcecd6f83a7e1a7896c785901e40e370bb9d72859\": container with ID starting with e2476d4267148b473e8a067fcecd6f83a7e1a7896c785901e40e370bb9d72859 not found: ID does not exist"
Oct 31 08:18:04.184094 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:18:04.184068    4720 scope.go:122] "RemoveContainer" containerID="1d8673799b09771d7a4715e34a613e4445ca4819cb2c635f330e3ef341bc86cd"
Oct 31 08:18:04.184294 ip-10-0-139-83 kubenswrapper[4720]: E1031 08:18:04.184277    4720 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1d8673799b09771d7a4715e34a613e4445ca4819cb2c635f330e3ef341bc86cd\": container with ID starting with 1d8673799b09771d7a4715e34a613e4445ca4819cb2c635f330e3ef341bc86cd not found: ID does not exist" containerID="1d8673799b09771d7a4715e34a613e4445ca4819cb2c635f330e3ef341bc86cd"
Oct 31 08:18:04.184378 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:18:04.184299    4720 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1d8673799b09771d7a4715e34a613e4445ca4819cb2c635f330e3ef341bc86cd"} err="failed to get container status \"1d8673799b09771d7a4715e34a613e4445ca4819cb2c635f330e3ef341bc86cd\": rpc error: code = NotFound desc = could not find container \"1d8673799b09771d7a4715e34a613e4445ca4819cb2c635f330e3ef341bc86cd\": container with ID starting with 1d8673799b09771d7a4715e34a613e4445ca4819cb2c635f330e3ef341bc86cd not found: ID does not exist"
Oct 31 08:18:04.237816 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:18:04.237773    4720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/16065498-6407-448c-8ca7-24763e3710c8-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/16065498-6407-448c-8ca7-24763e3710c8-model-cache\") pod \"16065498-6407-448c-8ca7-24763e3710c8\" (UID: \"16065498-6407-448c-8ca7-24763e3710c8\") "
Oct 31 08:18:04.238011 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:18:04.237834    4720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/16065498-6407-448c-8ca7-24763e3710c8-home\" (UniqueName: \"kubernetes.io/empty-dir/16065498-6407-448c-8ca7-24763e3710c8-home\") pod \"16065498-6407-448c-8ca7-24763e3710c8\" (UID: \"16065498-6407-448c-8ca7-24763e3710c8\") "
Oct 31 08:18:04.238011 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:18:04.237865    4720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/16065498-6407-448c-8ca7-24763e3710c8-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/16065498-6407-448c-8ca7-24763e3710c8-kserve-provision-location\") pod \"16065498-6407-448c-8ca7-24763e3710c8\" (UID: \"16065498-6407-448c-8ca7-24763e3710c8\") "
Oct 31 08:18:04.238011 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:18:04.237900    4720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/16065498-6407-448c-8ca7-24763e3710c8-dshm\" (UniqueName: \"kubernetes.io/empty-dir/16065498-6407-448c-8ca7-24763e3710c8-dshm\") pod \"16065498-6407-448c-8ca7-24763e3710c8\" (UID: \"16065498-6407-448c-8ca7-24763e3710c8\") "
Oct 31 08:18:04.238011 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:18:04.237945    4720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/16065498-6407-448c-8ca7-24763e3710c8-tls-certs\" (UniqueName: \"kubernetes.io/secret/16065498-6407-448c-8ca7-24763e3710c8-tls-certs\") pod \"16065498-6407-448c-8ca7-24763e3710c8\" (UID: \"16065498-6407-448c-8ca7-24763e3710c8\") "
Oct 31 08:18:04.238011 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:18:04.238006    4720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/16065498-6407-448c-8ca7-24763e3710c8-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/16065498-6407-448c-8ca7-24763e3710c8-tmp-dir\") pod \"16065498-6407-448c-8ca7-24763e3710c8\" (UID: \"16065498-6407-448c-8ca7-24763e3710c8\") "
Oct 31 08:18:04.238278 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:18:04.238085    4720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/16065498-6407-448c-8ca7-24763e3710c8-model-cache" pod "16065498-6407-448c-8ca7-24763e3710c8" (UID: "16065498-6407-448c-8ca7-24763e3710c8"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 08:18:04.238278 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:18:04.238122    4720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/16065498-6407-448c-8ca7-24763e3710c8-kube-api-access-nmcft\" (UniqueName: \"kubernetes.io/projected/16065498-6407-448c-8ca7-24763e3710c8-kube-api-access-nmcft\") pod \"16065498-6407-448c-8ca7-24763e3710c8\" (UID: \"16065498-6407-448c-8ca7-24763e3710c8\") "
Oct 31 08:18:04.238278 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:18:04.238182    4720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/16065498-6407-448c-8ca7-24763e3710c8-kserve-provision-location" pod "16065498-6407-448c-8ca7-24763e3710c8" (UID: "16065498-6407-448c-8ca7-24763e3710c8"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 08:18:04.238478 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:18:04.238459    4720 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/16065498-6407-448c-8ca7-24763e3710c8-model-cache\") on node \"ip-10-0-139-83.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 08:18:04.238537 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:18:04.238482    4720 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/16065498-6407-448c-8ca7-24763e3710c8-kserve-provision-location\") on node \"ip-10-0-139-83.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 08:18:04.238606 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:18:04.238583    4720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/16065498-6407-448c-8ca7-24763e3710c8-home" pod "16065498-6407-448c-8ca7-24763e3710c8" (UID: "16065498-6407-448c-8ca7-24763e3710c8"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 08:18:04.240531 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:18:04.240503    4720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/16065498-6407-448c-8ca7-24763e3710c8-tls-certs" pod "16065498-6407-448c-8ca7-24763e3710c8" (UID: "16065498-6407-448c-8ca7-24763e3710c8"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 08:18:04.240531 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:18:04.240510    4720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/16065498-6407-448c-8ca7-24763e3710c8-dshm" pod "16065498-6407-448c-8ca7-24763e3710c8" (UID: "16065498-6407-448c-8ca7-24763e3710c8"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 08:18:04.240721 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:18:04.240555    4720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/16065498-6407-448c-8ca7-24763e3710c8-kube-api-access-nmcft" pod "16065498-6407-448c-8ca7-24763e3710c8" (UID: "16065498-6407-448c-8ca7-24763e3710c8"). InnerVolumeSpecName "kube-api-access-nmcft". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 08:18:04.254267 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:18:04.254241    4720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/16065498-6407-448c-8ca7-24763e3710c8-tmp-dir" pod "16065498-6407-448c-8ca7-24763e3710c8" (UID: "16065498-6407-448c-8ca7-24763e3710c8"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 08:18:04.339663 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:18:04.339622    4720 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/16065498-6407-448c-8ca7-24763e3710c8-home\") on node \"ip-10-0-139-83.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 08:18:04.339663 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:18:04.339654    4720 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/16065498-6407-448c-8ca7-24763e3710c8-dshm\") on node \"ip-10-0-139-83.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 08:18:04.339663 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:18:04.339665    4720 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/16065498-6407-448c-8ca7-24763e3710c8-tls-certs\") on node \"ip-10-0-139-83.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 08:18:04.339663 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:18:04.339673    4720 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/16065498-6407-448c-8ca7-24763e3710c8-tmp-dir\") on node \"ip-10-0-139-83.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 08:18:04.339946 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:18:04.339682    4720 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nmcft\" (UniqueName: \"kubernetes.io/projected/16065498-6407-448c-8ca7-24763e3710c8-kube-api-access-nmcft\") on node \"ip-10-0-139-83.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 08:18:05.163864 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:18:05.163829    4720 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-7687c48f9bmgqtl"
Oct 31 08:18:05.376698 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:18:05.376658    4720 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-7687c48f9bmgqtl"]
Oct 31 08:18:05.425948 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:18:05.425866    4720 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-7687c48f9bmgqtl"]
Oct 31 08:18:06.025153 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:18:06.025114    4720 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="16065498-6407-448c-8ca7-24763e3710c8" path="/var/lib/kubelet/pods/16065498-6407-448c-8ca7-24763e3710c8/volumes"
Oct 31 08:18:11.393024 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:18:11.392975    4720 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-schenpfzd" podUID="772d92f9-2bd0-419f-86b6-b7877d5b4bc2" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 08:18:11.393653 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:18:11.393093    4720 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-schenpfzd"
Oct 31 08:18:13.462204 ip-10-0-139-83 kubenswrapper[4720]: E1031 08:18:13.462162    4720 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8m768" containerName="extractor"
Oct 31 08:18:16.981974 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:18:16.981940    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 08:18:21.393786 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:18:21.393746    4720 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-schenpfzd" podUID="772d92f9-2bd0-419f-86b6-b7877d5b4bc2" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 08:18:31.392852 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:18:31.392813    4720 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-schenpfzd" podUID="772d92f9-2bd0-419f-86b6-b7877d5b4bc2" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 08:18:31.625380 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:18:31.625312    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 08:18:32.946961 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:18:32.946747    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 08:18:33.110722 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:18:33.110688    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 08:18:33.611808 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:18:33.611743    4720 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-schenpfzd"
Oct 31 08:18:33.680005 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:18:33.679970    4720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/772d92f9-2bd0-419f-86b6-b7877d5b4bc2-kube-api-access-5jgkq\" (UniqueName: \"kubernetes.io/projected/772d92f9-2bd0-419f-86b6-b7877d5b4bc2-kube-api-access-5jgkq\") pod \"772d92f9-2bd0-419f-86b6-b7877d5b4bc2\" (UID: \"772d92f9-2bd0-419f-86b6-b7877d5b4bc2\") "
Oct 31 08:18:33.680187 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:18:33.680018    4720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/772d92f9-2bd0-419f-86b6-b7877d5b4bc2-tls-certs\" (UniqueName: \"kubernetes.io/secret/772d92f9-2bd0-419f-86b6-b7877d5b4bc2-tls-certs\") pod \"772d92f9-2bd0-419f-86b6-b7877d5b4bc2\" (UID: \"772d92f9-2bd0-419f-86b6-b7877d5b4bc2\") "
Oct 31 08:18:33.682263 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:18:33.682237    4720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/772d92f9-2bd0-419f-86b6-b7877d5b4bc2-tls-certs" pod "772d92f9-2bd0-419f-86b6-b7877d5b4bc2" (UID: "772d92f9-2bd0-419f-86b6-b7877d5b4bc2"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 08:18:33.682384 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:18:33.682237    4720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/772d92f9-2bd0-419f-86b6-b7877d5b4bc2-kube-api-access-5jgkq" pod "772d92f9-2bd0-419f-86b6-b7877d5b4bc2" (UID: "772d92f9-2bd0-419f-86b6-b7877d5b4bc2"). InnerVolumeSpecName "kube-api-access-5jgkq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 08:18:33.781229 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:18:33.781188    4720 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5jgkq\" (UniqueName: \"kubernetes.io/projected/772d92f9-2bd0-419f-86b6-b7877d5b4bc2-kube-api-access-5jgkq\") on node \"ip-10-0-139-83.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 08:18:33.781229 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:18:33.781224    4720 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/772d92f9-2bd0-419f-86b6-b7877d5b4bc2-tls-certs\") on node \"ip-10-0-139-83.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 08:18:34.301070 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:18:34.300980    4720 generic.go:356] "Generic (PLEG): container finished" podID="772d92f9-2bd0-419f-86b6-b7877d5b4bc2" containerID="7bf814f0104f326f881ce59f6fc8173a0ad16e343679093c2ccf830a19bc8ab9" exitCode=0
Oct 31 08:18:34.301070 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:18:34.301031    4720 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-schenpfzd"
Oct 31 08:18:34.301587 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:18:34.301066    4720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-schenpfzd" event={"ID":"772d92f9-2bd0-419f-86b6-b7877d5b4bc2","Type":"ContainerDied","Data":"7bf814f0104f326f881ce59f6fc8173a0ad16e343679093c2ccf830a19bc8ab9"}
Oct 31 08:18:34.301587 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:18:34.301121    4720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-schenpfzd" event={"ID":"772d92f9-2bd0-419f-86b6-b7877d5b4bc2","Type":"ContainerDied","Data":"87d36dc9b1e61edafaa15ae934eecebeca950a742d66725e2f7f75543b38167f"}
Oct 31 08:18:34.301587 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:18:34.301143    4720 scope.go:122] "RemoveContainer" containerID="7bf814f0104f326f881ce59f6fc8173a0ad16e343679093c2ccf830a19bc8ab9"
Oct 31 08:18:34.309624 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:18:34.309606    4720 scope.go:122] "RemoveContainer" containerID="7bf814f0104f326f881ce59f6fc8173a0ad16e343679093c2ccf830a19bc8ab9"
Oct 31 08:18:34.309916 ip-10-0-139-83 kubenswrapper[4720]: E1031 08:18:34.309896    4720 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7bf814f0104f326f881ce59f6fc8173a0ad16e343679093c2ccf830a19bc8ab9\": container with ID starting with 7bf814f0104f326f881ce59f6fc8173a0ad16e343679093c2ccf830a19bc8ab9 not found: ID does not exist" containerID="7bf814f0104f326f881ce59f6fc8173a0ad16e343679093c2ccf830a19bc8ab9"
Oct 31 08:18:34.309964 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:18:34.309926    4720 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7bf814f0104f326f881ce59f6fc8173a0ad16e343679093c2ccf830a19bc8ab9"} err="failed to get container status \"7bf814f0104f326f881ce59f6fc8173a0ad16e343679093c2ccf830a19bc8ab9\": rpc error: code = NotFound desc = could not find container \"7bf814f0104f326f881ce59f6fc8173a0ad16e343679093c2ccf830a19bc8ab9\": container with ID starting with 7bf814f0104f326f881ce59f6fc8173a0ad16e343679093c2ccf830a19bc8ab9 not found: ID does not exist"
Oct 31 08:18:34.463053 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:18:34.463015    4720 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-schenpfzd"]
Oct 31 08:18:34.512456 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:18:34.512422    4720 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-schenpfzd"]
Oct 31 08:18:36.025887 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:18:36.025845    4720 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="772d92f9-2bd0-419f-86b6-b7877d5b4bc2" path="/var/lib/kubelet/pods/772d92f9-2bd0-419f-86b6-b7877d5b4bc2/volumes"
Oct 31 08:18:37.302745 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:18:37.302712    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 08:18:37.465564 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:18:37.465534    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 08:18:41.406835 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:18:41.406799    4720 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-57d77d7f74-m88rz"]
Oct 31 08:18:41.407365 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:18:41.407343    4720 cpu_manager.go:414] "containerMap: removing container" podUID="16065498-6407-448c-8ca7-24763e3710c8" containerName="main"
Oct 31 08:18:41.407418 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:18:41.407363    4720 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="16065498-6407-448c-8ca7-24763e3710c8" containerName="main"
Oct 31 08:18:41.407418 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:18:41.407391    4720 cpu_manager.go:414] "containerMap: removing container" podUID="16065498-6407-448c-8ca7-24763e3710c8" containerName="modelcar-init"
Oct 31 08:18:41.407418 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:18:41.407401    4720 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="16065498-6407-448c-8ca7-24763e3710c8" containerName="modelcar-init"
Oct 31 08:18:41.407508 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:18:41.407431    4720 cpu_manager.go:414] "containerMap: removing container" podUID="16065498-6407-448c-8ca7-24763e3710c8" containerName="modelcar"
Oct 31 08:18:41.407508 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:18:41.407439    4720 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="16065498-6407-448c-8ca7-24763e3710c8" containerName="modelcar"
Oct 31 08:18:41.407508 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:18:41.407455    4720 cpu_manager.go:414] "containerMap: removing container" podUID="772d92f9-2bd0-419f-86b6-b7877d5b4bc2" containerName="main"
Oct 31 08:18:41.407508 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:18:41.407462    4720 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="772d92f9-2bd0-419f-86b6-b7877d5b4bc2" containerName="main"
Oct 31 08:18:41.407624 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:18:41.407549    4720 memory_manager.go:365] "RemoveStaleState removing state" podUID="16065498-6407-448c-8ca7-24763e3710c8" containerName="modelcar"
Oct 31 08:18:41.407624 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:18:41.407563    4720 memory_manager.go:365] "RemoveStaleState removing state" podUID="772d92f9-2bd0-419f-86b6-b7877d5b4bc2" containerName="main"
Oct 31 08:18:41.407624 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:18:41.407574    4720 memory_manager.go:365] "RemoveStaleState removing state" podUID="16065498-6407-448c-8ca7-24763e3710c8" containerName="main"
Oct 31 08:18:41.411212 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:18:41.411194    4720 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-57d77d7f74-m88rz"
Oct 31 08:18:41.461652 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:18:41.461613    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 08:18:41.463311 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:18:41.463286    4720 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-57d77d7f74-m88rz"]
Oct 31 08:18:41.509100 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:18:41.509066    4720 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-caba0b4a\"/\"openshift-service-ca.crt\""
Oct 31 08:18:41.509100 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:18:41.509084    4720 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-caba0b4a\"/\"kube-root-ca.crt\""
Oct 31 08:18:41.509315 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:18:41.509148    4720 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-caba0b4a\"/\"odh-kserve-custom-ca-bundle\""
Oct 31 08:18:41.509315 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:18:41.509222    4720 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-caba0b4a\"/\"default-dockercfg-hlsz2\""
Oct 31 08:18:41.509315 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:18:41.509254    4720 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-caba0b4a\"/\"seaweedfs-s3-creds\""
Oct 31 08:18:41.509315 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:18:41.509309    4720 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-caba0b4a\"/\"custom-route-timeout-test-kserve-self-signed-certs\""
Oct 31 08:18:41.553051 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:18:41.553013    4720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2kn6j\" (UniqueName: \"kubernetes.io/projected/46809f03-84a0-460b-9d64-4926110b24d7-kube-api-access-2kn6j\") pod \"custom-route-timeout-test-kserve-57d77d7f74-m88rz\" (UID: \"46809f03-84a0-460b-9d64-4926110b24d7\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-57d77d7f74-m88rz"
Oct 31 08:18:41.553163 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:18:41.553056    4720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/46809f03-84a0-460b-9d64-4926110b24d7-tmp-dir\") pod \"custom-route-timeout-test-kserve-57d77d7f74-m88rz\" (UID: \"46809f03-84a0-460b-9d64-4926110b24d7\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-57d77d7f74-m88rz"
Oct 31 08:18:41.553163 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:18:41.553147    4720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/46809f03-84a0-460b-9d64-4926110b24d7-dshm\") pod \"custom-route-timeout-test-kserve-57d77d7f74-m88rz\" (UID: \"46809f03-84a0-460b-9d64-4926110b24d7\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-57d77d7f74-m88rz"
Oct 31 08:18:41.553229 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:18:41.553200    4720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/46809f03-84a0-460b-9d64-4926110b24d7-kserve-provision-location\") pod \"custom-route-timeout-test-kserve-57d77d7f74-m88rz\" (UID: \"46809f03-84a0-460b-9d64-4926110b24d7\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-57d77d7f74-m88rz"
Oct 31 08:18:41.553361 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:18:41.553310    4720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/46809f03-84a0-460b-9d64-4926110b24d7-tls-certs\") pod \"custom-route-timeout-test-kserve-57d77d7f74-m88rz\" (UID: \"46809f03-84a0-460b-9d64-4926110b24d7\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-57d77d7f74-m88rz"
Oct 31 08:18:41.553407 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:18:41.553382    4720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/46809f03-84a0-460b-9d64-4926110b24d7-cabundle-cert\") pod \"custom-route-timeout-test-kserve-57d77d7f74-m88rz\" (UID: \"46809f03-84a0-460b-9d64-4926110b24d7\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-57d77d7f74-m88rz"
Oct 31 08:18:41.553457 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:18:41.553443    4720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/46809f03-84a0-460b-9d64-4926110b24d7-model-cache\") pod \"custom-route-timeout-test-kserve-57d77d7f74-m88rz\" (UID: \"46809f03-84a0-460b-9d64-4926110b24d7\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-57d77d7f74-m88rz"
Oct 31 08:18:41.553492 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:18:41.553470    4720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/46809f03-84a0-460b-9d64-4926110b24d7-home\") pod \"custom-route-timeout-test-kserve-57d77d7f74-m88rz\" (UID: \"46809f03-84a0-460b-9d64-4926110b24d7\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-57d77d7f74-m88rz"
Oct 31 08:18:41.654503 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:18:41.654460    4720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/46809f03-84a0-460b-9d64-4926110b24d7-model-cache\") pod \"custom-route-timeout-test-kserve-57d77d7f74-m88rz\" (UID: \"46809f03-84a0-460b-9d64-4926110b24d7\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-57d77d7f74-m88rz"
Oct 31 08:18:41.654503 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:18:41.654504    4720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/46809f03-84a0-460b-9d64-4926110b24d7-home\") pod \"custom-route-timeout-test-kserve-57d77d7f74-m88rz\" (UID: \"46809f03-84a0-460b-9d64-4926110b24d7\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-57d77d7f74-m88rz"
Oct 31 08:18:41.654785 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:18:41.654638    4720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2kn6j\" (UniqueName: \"kubernetes.io/projected/46809f03-84a0-460b-9d64-4926110b24d7-kube-api-access-2kn6j\") pod \"custom-route-timeout-test-kserve-57d77d7f74-m88rz\" (UID: \"46809f03-84a0-460b-9d64-4926110b24d7\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-57d77d7f74-m88rz"
Oct 31 08:18:41.654785 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:18:41.654689    4720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/46809f03-84a0-460b-9d64-4926110b24d7-tmp-dir\") pod \"custom-route-timeout-test-kserve-57d77d7f74-m88rz\" (UID: \"46809f03-84a0-460b-9d64-4926110b24d7\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-57d77d7f74-m88rz"
Oct 31 08:18:41.654785 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:18:41.654725    4720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/46809f03-84a0-460b-9d64-4926110b24d7-dshm\") pod \"custom-route-timeout-test-kserve-57d77d7f74-m88rz\" (UID: \"46809f03-84a0-460b-9d64-4926110b24d7\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-57d77d7f74-m88rz"
Oct 31 08:18:41.654785 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:18:41.654778    4720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/46809f03-84a0-460b-9d64-4926110b24d7-kserve-provision-location\") pod \"custom-route-timeout-test-kserve-57d77d7f74-m88rz\" (UID: \"46809f03-84a0-460b-9d64-4926110b24d7\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-57d77d7f74-m88rz"
Oct 31 08:18:41.654979 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:18:41.654869    4720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/46809f03-84a0-460b-9d64-4926110b24d7-tls-certs\") pod \"custom-route-timeout-test-kserve-57d77d7f74-m88rz\" (UID: \"46809f03-84a0-460b-9d64-4926110b24d7\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-57d77d7f74-m88rz"
Oct 31 08:18:41.654979 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:18:41.654904    4720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/46809f03-84a0-460b-9d64-4926110b24d7-home\") pod \"custom-route-timeout-test-kserve-57d77d7f74-m88rz\" (UID: \"46809f03-84a0-460b-9d64-4926110b24d7\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-57d77d7f74-m88rz"
Oct 31 08:18:41.654979 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:18:41.654949    4720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/46809f03-84a0-460b-9d64-4926110b24d7-model-cache\") pod \"custom-route-timeout-test-kserve-57d77d7f74-m88rz\" (UID: \"46809f03-84a0-460b-9d64-4926110b24d7\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-57d77d7f74-m88rz"
Oct 31 08:18:41.655090 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:18:41.654911    4720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/46809f03-84a0-460b-9d64-4926110b24d7-cabundle-cert\") pod \"custom-route-timeout-test-kserve-57d77d7f74-m88rz\" (UID: \"46809f03-84a0-460b-9d64-4926110b24d7\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-57d77d7f74-m88rz"
Oct 31 08:18:41.655128 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:18:41.655114    4720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/46809f03-84a0-460b-9d64-4926110b24d7-tmp-dir\") pod \"custom-route-timeout-test-kserve-57d77d7f74-m88rz\" (UID: \"46809f03-84a0-460b-9d64-4926110b24d7\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-57d77d7f74-m88rz"
Oct 31 08:18:41.655178 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:18:41.655159    4720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/46809f03-84a0-460b-9d64-4926110b24d7-kserve-provision-location\") pod \"custom-route-timeout-test-kserve-57d77d7f74-m88rz\" (UID: \"46809f03-84a0-460b-9d64-4926110b24d7\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-57d77d7f74-m88rz"
Oct 31 08:18:41.655521 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:18:41.655495    4720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/46809f03-84a0-460b-9d64-4926110b24d7-cabundle-cert\") pod \"custom-route-timeout-test-kserve-57d77d7f74-m88rz\" (UID: \"46809f03-84a0-460b-9d64-4926110b24d7\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-57d77d7f74-m88rz"
Oct 31 08:18:41.657137 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:18:41.657091    4720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/46809f03-84a0-460b-9d64-4926110b24d7-dshm\") pod \"custom-route-timeout-test-kserve-57d77d7f74-m88rz\" (UID: \"46809f03-84a0-460b-9d64-4926110b24d7\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-57d77d7f74-m88rz"
Oct 31 08:18:41.657479 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:18:41.657461    4720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/46809f03-84a0-460b-9d64-4926110b24d7-tls-certs\") pod \"custom-route-timeout-test-kserve-57d77d7f74-m88rz\" (UID: \"46809f03-84a0-460b-9d64-4926110b24d7\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-57d77d7f74-m88rz"
Oct 31 08:18:41.710898 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:18:41.710864    4720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2kn6j\" (UniqueName: \"kubernetes.io/projected/46809f03-84a0-460b-9d64-4926110b24d7-kube-api-access-2kn6j\") pod \"custom-route-timeout-test-kserve-57d77d7f74-m88rz\" (UID: \"46809f03-84a0-460b-9d64-4926110b24d7\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-57d77d7f74-m88rz"
Oct 31 08:18:41.728714 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:18:41.728676    4720 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-57d77d7f74-m88rz"
Oct 31 08:18:41.758086 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:18:41.758056    4720 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-5c6b9d54r9tqv"]
Oct 31 08:18:41.761743 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:18:41.761719    4720 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-5c6b9d54r9tqv"
Oct 31 08:18:41.815682 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:18:41.815628    4720 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-caba0b4a\"/\"custom-route-timeout-test-epp-sa-dockercfg-lg4qx\""
Oct 31 08:18:41.820199 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:18:41.820162    4720 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-5c6b9d54r9tqv"]
Oct 31 08:18:41.856549 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:18:41.856511    4720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6e859957-5568-4fcf-95ba-aca317743d82-tls-certs\") pod \"custom-route-timeout-test-kserve-router-scheduler-5c6b9d54r9tqv\" (UID: \"6e859957-5568-4fcf-95ba-aca317743d82\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-5c6b9d54r9tqv"
Oct 31 08:18:41.856737 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:18:41.856572    4720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pj6xg\" (UniqueName: \"kubernetes.io/projected/6e859957-5568-4fcf-95ba-aca317743d82-kube-api-access-pj6xg\") pod \"custom-route-timeout-test-kserve-router-scheduler-5c6b9d54r9tqv\" (UID: \"6e859957-5568-4fcf-95ba-aca317743d82\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-5c6b9d54r9tqv"
Oct 31 08:18:41.860778 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:18:41.860734    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 08:18:41.957350 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:18:41.957227    4720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pj6xg\" (UniqueName: \"kubernetes.io/projected/6e859957-5568-4fcf-95ba-aca317743d82-kube-api-access-pj6xg\") pod \"custom-route-timeout-test-kserve-router-scheduler-5c6b9d54r9tqv\" (UID: \"6e859957-5568-4fcf-95ba-aca317743d82\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-5c6b9d54r9tqv"
Oct 31 08:18:41.957544 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:18:41.957355    4720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6e859957-5568-4fcf-95ba-aca317743d82-tls-certs\") pod \"custom-route-timeout-test-kserve-router-scheduler-5c6b9d54r9tqv\" (UID: \"6e859957-5568-4fcf-95ba-aca317743d82\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-5c6b9d54r9tqv"
Oct 31 08:18:41.960084 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:18:41.960061    4720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6e859957-5568-4fcf-95ba-aca317743d82-tls-certs\") pod \"custom-route-timeout-test-kserve-router-scheduler-5c6b9d54r9tqv\" (UID: \"6e859957-5568-4fcf-95ba-aca317743d82\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-5c6b9d54r9tqv"
Oct 31 08:18:42.013975 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:18:42.013941    4720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pj6xg\" (UniqueName: \"kubernetes.io/projected/6e859957-5568-4fcf-95ba-aca317743d82-kube-api-access-pj6xg\") pod \"custom-route-timeout-test-kserve-router-scheduler-5c6b9d54r9tqv\" (UID: \"6e859957-5568-4fcf-95ba-aca317743d82\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-5c6b9d54r9tqv"
Oct 31 08:18:42.077532 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:18:42.077494    4720 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-5c6b9d54r9tqv"
Oct 31 08:18:42.162978 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:18:42.162949    4720 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-57d77d7f74-m88rz"]
Oct 31 08:18:42.165376 ip-10-0-139-83 kubenswrapper[4720]: W1031 08:18:42.165313    4720 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod46809f03_84a0_460b_9d64_4926110b24d7.slice/crio-ca6d6a8464c19d9110a396b67c97705b80dfe1ee7677151d471ba03f984b0433 WatchSource:0}: Error finding container ca6d6a8464c19d9110a396b67c97705b80dfe1ee7677151d471ba03f984b0433: Status 404 returned error can't find the container with id ca6d6a8464c19d9110a396b67c97705b80dfe1ee7677151d471ba03f984b0433
Oct 31 08:18:42.341278 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:18:42.341229    4720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-57d77d7f74-m88rz" event={"ID":"46809f03-84a0-460b-9d64-4926110b24d7","Type":"ContainerStarted","Data":"84d5c57ba38d623de9163b03f8339f84744c81517f86592549ef5227709286cb"}
Oct 31 08:18:42.341477 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:18:42.341300    4720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-57d77d7f74-m88rz" event={"ID":"46809f03-84a0-460b-9d64-4926110b24d7","Type":"ContainerStarted","Data":"ca6d6a8464c19d9110a396b67c97705b80dfe1ee7677151d471ba03f984b0433"}
Oct 31 08:18:42.456369 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:18:42.454960    4720 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-5c6b9d54r9tqv"]
Oct 31 08:18:43.346870 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:18:43.346829    4720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-5c6b9d54r9tqv" event={"ID":"6e859957-5568-4fcf-95ba-aca317743d82","Type":"ContainerStarted","Data":"94b9c856482a41b13f199c405739c8efc5f850c8cfefe931aed51bbc06298495"}
Oct 31 08:18:43.347082 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:18:43.346900    4720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-5c6b9d54r9tqv" event={"ID":"6e859957-5568-4fcf-95ba-aca317743d82","Type":"ContainerStarted","Data":"8c1c251db76034fc90680fd6cd75595fb53a4febef2d5da21cd84f97cf9897c7"}
Oct 31 08:18:43.347082 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:18:43.346929    4720 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-5c6b9d54r9tqv"
Oct 31 08:18:43.446790 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:18:43.446750    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 08:18:43.460365 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:18:43.460260    4720 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-5c6b9d54r9tqv" podStartSLOduration=2.460240772 podStartE2EDuration="2.460240772s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 08:18:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 08:18:43.459043937 +0000 UTC m=+2977.590112032" watchObservedRunningTime="2026-10-31 08:18:43.460240772 +0000 UTC m=+2977.591308845"
Oct 31 08:18:43.463255 ip-10-0-139-83 kubenswrapper[4720]: E1031 08:18:43.463219    4720 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8m768" containerName="extractor"
Oct 31 08:18:45.357065 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:18:45.357032    4720 generic.go:356] "Generic (PLEG): container finished" podID="46809f03-84a0-460b-9d64-4926110b24d7" containerID="84d5c57ba38d623de9163b03f8339f84744c81517f86592549ef5227709286cb" exitCode=0
Oct 31 08:18:45.357495 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:18:45.357087    4720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-57d77d7f74-m88rz" event={"ID":"46809f03-84a0-460b-9d64-4926110b24d7","Type":"ContainerDied","Data":"84d5c57ba38d623de9163b03f8339f84744c81517f86592549ef5227709286cb"}
Oct 31 08:18:45.929982 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:18:45.929952    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 08:18:46.364585 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:18:46.364549    4720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-57d77d7f74-m88rz" event={"ID":"46809f03-84a0-460b-9d64-4926110b24d7","Type":"ContainerStarted","Data":"499f7d7cc17be29aee7a2a2de207b8a30ba309a95e958184ccbb2064dfc62c96"}
Oct 31 08:18:46.579243 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:18:46.579177    4720 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-57d77d7f74-m88rz" podStartSLOduration=3.245661225 podStartE2EDuration="5.579162015s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.33350079s" isStatefulPod=false podCreationTimestamp="2026-10-31 08:18:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 08:18:46.528878833 +0000 UTC m=+2980.659946903" watchObservedRunningTime="2026-10-31 08:18:46.579162015 +0000 UTC m=+2980.710230086"
Oct 31 08:18:46.729739 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:18:46.729633    4720 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-57d77d7f74-m88rz"
Oct 31 08:18:48.867576 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:18:48.867534    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 08:18:49.017200 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:18:49.017165    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 08:18:51.729531 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:18:51.729490    4720 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-57d77d7f74-m88rz"
Oct 31 08:18:51.731252 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:18:51.731217    4720 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-57d77d7f74-m88rz" podUID="46809f03-84a0-460b-9d64-4926110b24d7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/health\": dial tcp 10.132.0.65:8000: connect: connection refused"
Oct 31 08:18:56.381206 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:18:56.381170    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 08:19:01.730052 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:19:01.730002    4720 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-57d77d7f74-m88rz" podUID="46809f03-84a0-460b-9d64-4926110b24d7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/health\": dial tcp 10.132.0.65:8000: connect: connection refused"
Oct 31 08:19:06.303069 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:19:06.302966    4720 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-4jszp_b1db0e3b-e85d-4769-8012-cf4b6f740d9d/console-operator/2.log"
Oct 31 08:19:06.316679 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:19:06.310819    4720 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-4jszp_b1db0e3b-e85d-4769-8012-cf4b6f740d9d/console-operator/2.log"
Oct 31 08:19:11.729480 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:19:11.729428    4720 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-57d77d7f74-m88rz" podUID="46809f03-84a0-460b-9d64-4926110b24d7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/health\": dial tcp 10.132.0.65:8000: connect: connection refused"
Oct 31 08:19:13.462885 ip-10-0-139-83 kubenswrapper[4720]: E1031 08:19:13.462837    4720 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8m768" containerName="extractor"
Oct 31 08:19:14.352861 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:19:14.352827    4720 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-5c6b9d54r9tqv"
Oct 31 08:19:17.317144 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:19:17.317066    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 08:19:21.459971 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:19:21.459938    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 08:19:21.613491 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:19:21.613457    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 08:19:21.729734 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:19:21.729645    4720 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-57d77d7f74-m88rz" podUID="46809f03-84a0-460b-9d64-4926110b24d7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/health\": dial tcp 10.132.0.65:8000: connect: connection refused"
Oct 31 08:19:31.729704 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:19:31.729648    4720 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-57d77d7f74-m88rz" podUID="46809f03-84a0-460b-9d64-4926110b24d7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/health\": dial tcp 10.132.0.65:8000: connect: connection refused"
Oct 31 08:19:32.009042 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:19:32.008956    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 08:19:41.729738 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:19:41.729687    4720 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-57d77d7f74-m88rz" podUID="46809f03-84a0-460b-9d64-4926110b24d7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/health\": dial tcp 10.132.0.65:8000: connect: connection refused"
Oct 31 08:19:43.030449 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:19:43.030409    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 08:19:43.462763 ip-10-0-139-83 kubenswrapper[4720]: E1031 08:19:43.462723    4720 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8m768" containerName="extractor"
Oct 31 08:19:51.729834 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:19:51.729782    4720 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-57d77d7f74-m88rz" podUID="46809f03-84a0-460b-9d64-4926110b24d7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/health\": dial tcp 10.132.0.65:8000: connect: connection refused"
Oct 31 08:19:56.716888 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:19:56.716845    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 08:20:01.730035 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:20:01.729983    4720 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-57d77d7f74-m88rz" podUID="46809f03-84a0-460b-9d64-4926110b24d7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/health\": dial tcp 10.132.0.65:8000: connect: connection refused"
Oct 31 08:20:10.873153 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:20:10.873113    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 08:20:11.032722 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:20:11.032687    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 08:20:11.729594 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:20:11.729548    4720 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-57d77d7f74-m88rz" podUID="46809f03-84a0-460b-9d64-4926110b24d7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/health\": dial tcp 10.132.0.65:8000: connect: connection refused"
Oct 31 08:20:13.462863 ip-10-0-139-83 kubenswrapper[4720]: E1031 08:20:13.462819    4720 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8m768" containerName="extractor"
Oct 31 08:20:17.248191 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:20:17.248152    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 08:20:17.643885 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:20:17.643845    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 08:20:19.478513 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:20:19.478476    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 08:20:21.500900 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:20:21.500858    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 08:20:21.729518 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:20:21.729465    4720 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-57d77d7f74-m88rz" podUID="46809f03-84a0-460b-9d64-4926110b24d7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/health\": dial tcp 10.132.0.65:8000: connect: connection refused"
Oct 31 08:20:22.595126 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:20:22.595091    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 08:20:22.747464 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:20:22.747431    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 08:20:29.306472 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:20:29.306435    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 08:20:31.729883 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:20:31.729842    4720 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-57d77d7f74-m88rz" podUID="46809f03-84a0-460b-9d64-4926110b24d7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/health\": dial tcp 10.132.0.65:8000: connect: connection refused"
Oct 31 08:20:32.306249 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:20:32.306213    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 08:20:41.739581 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:20:41.739550    4720 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-57d77d7f74-m88rz"
Oct 31 08:20:41.747439 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:20:41.747412    4720 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-57d77d7f74-m88rz"
Oct 31 08:20:43.461774 ip-10-0-139-83 kubenswrapper[4720]: E1031 08:20:43.461734    4720 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8m768" containerName="extractor"
Oct 31 08:20:50.591030 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:20:50.590954    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 08:20:50.596570 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:20:50.596538    4720 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-57d77d7f74-m88rz"]
Oct 31 08:20:50.599053 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:20:50.599027    4720 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-5c6b9d54r9tqv"]
Oct 31 08:20:50.743970 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:20:50.743939    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 08:20:54.353265 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:20:54.353221    4720 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-5c6b9d54r9tqv" podUID="6e859957-5568-4fcf-95ba-aca317743d82" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 08:20:55.986019 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:20:55.985984    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 08:20:56.130686 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:20:56.130655    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 08:20:57.052550 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:20:57.052518    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 08:21:04.353313 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:21:04.353265    4720 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-5c6b9d54r9tqv" podUID="6e859957-5568-4fcf-95ba-aca317743d82" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 08:21:05.618576 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:21:05.618500    4720 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-5c6b9d54r9tqv" podUID="6e859957-5568-4fcf-95ba-aca317743d82" containerName="main" containerID="cri-o://94b9c856482a41b13f199c405739c8efc5f850c8cfefe931aed51bbc06298495" gracePeriod=45
Oct 31 08:21:05.620115 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:21:05.620079    4720 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-57d77d7f74-m88rz" podUID="46809f03-84a0-460b-9d64-4926110b24d7" containerName="main" containerID="cri-o://499f7d7cc17be29aee7a2a2de207b8a30ba309a95e958184ccbb2064dfc62c96" gracePeriod=45
Oct 31 08:21:05.740890 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:21:05.740843    4720 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-57d77d7f74-m88rz" podUID="46809f03-84a0-460b-9d64-4926110b24d7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/health\": dial tcp 10.132.0.65:8000: connect: connection refused"
Oct 31 08:21:06.365596 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:21:06.365566    4720 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-57d77d7f74-m88rz"
Oct 31 08:21:06.477170 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:21:06.477073    4720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/46809f03-84a0-460b-9d64-4926110b24d7-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/46809f03-84a0-460b-9d64-4926110b24d7-tmp-dir\") pod \"46809f03-84a0-460b-9d64-4926110b24d7\" (UID: \"46809f03-84a0-460b-9d64-4926110b24d7\") "
Oct 31 08:21:06.477170 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:21:06.477131    4720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/46809f03-84a0-460b-9d64-4926110b24d7-kube-api-access-2kn6j\" (UniqueName: \"kubernetes.io/projected/46809f03-84a0-460b-9d64-4926110b24d7-kube-api-access-2kn6j\") pod \"46809f03-84a0-460b-9d64-4926110b24d7\" (UID: \"46809f03-84a0-460b-9d64-4926110b24d7\") "
Oct 31 08:21:06.477170 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:21:06.477150    4720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/46809f03-84a0-460b-9d64-4926110b24d7-home\" (UniqueName: \"kubernetes.io/empty-dir/46809f03-84a0-460b-9d64-4926110b24d7-home\") pod \"46809f03-84a0-460b-9d64-4926110b24d7\" (UID: \"46809f03-84a0-460b-9d64-4926110b24d7\") "
Oct 31 08:21:06.477486 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:21:06.477176    4720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/46809f03-84a0-460b-9d64-4926110b24d7-dshm\" (UniqueName: \"kubernetes.io/empty-dir/46809f03-84a0-460b-9d64-4926110b24d7-dshm\") pod \"46809f03-84a0-460b-9d64-4926110b24d7\" (UID: \"46809f03-84a0-460b-9d64-4926110b24d7\") "
Oct 31 08:21:06.477486 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:21:06.477248    4720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/46809f03-84a0-460b-9d64-4926110b24d7-tls-certs\" (UniqueName: \"kubernetes.io/secret/46809f03-84a0-460b-9d64-4926110b24d7-tls-certs\") pod \"46809f03-84a0-460b-9d64-4926110b24d7\" (UID: \"46809f03-84a0-460b-9d64-4926110b24d7\") "
Oct 31 08:21:06.477486 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:21:06.477278    4720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/46809f03-84a0-460b-9d64-4926110b24d7-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/46809f03-84a0-460b-9d64-4926110b24d7-kserve-provision-location\") pod \"46809f03-84a0-460b-9d64-4926110b24d7\" (UID: \"46809f03-84a0-460b-9d64-4926110b24d7\") "
Oct 31 08:21:06.477486 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:21:06.477314    4720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/46809f03-84a0-460b-9d64-4926110b24d7-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/46809f03-84a0-460b-9d64-4926110b24d7-cabundle-cert\") pod \"46809f03-84a0-460b-9d64-4926110b24d7\" (UID: \"46809f03-84a0-460b-9d64-4926110b24d7\") "
Oct 31 08:21:06.477486 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:21:06.477357    4720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/46809f03-84a0-460b-9d64-4926110b24d7-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/46809f03-84a0-460b-9d64-4926110b24d7-model-cache\") pod \"46809f03-84a0-460b-9d64-4926110b24d7\" (UID: \"46809f03-84a0-460b-9d64-4926110b24d7\") "
Oct 31 08:21:06.478105 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:21:06.478065    4720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46809f03-84a0-460b-9d64-4926110b24d7-home" pod "46809f03-84a0-460b-9d64-4926110b24d7" (UID: "46809f03-84a0-460b-9d64-4926110b24d7"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 08:21:06.478105 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:21:06.478077    4720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/46809f03-84a0-460b-9d64-4926110b24d7-cabundle-cert" pod "46809f03-84a0-460b-9d64-4926110b24d7" (UID: "46809f03-84a0-460b-9d64-4926110b24d7"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 08:21:06.478296 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:21:06.478201    4720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46809f03-84a0-460b-9d64-4926110b24d7-model-cache" pod "46809f03-84a0-460b-9d64-4926110b24d7" (UID: "46809f03-84a0-460b-9d64-4926110b24d7"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 08:21:06.480093 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:21:06.480051    4720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46809f03-84a0-460b-9d64-4926110b24d7-dshm" pod "46809f03-84a0-460b-9d64-4926110b24d7" (UID: "46809f03-84a0-460b-9d64-4926110b24d7"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 08:21:06.480231 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:21:06.480211    4720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/46809f03-84a0-460b-9d64-4926110b24d7-tls-certs" pod "46809f03-84a0-460b-9d64-4926110b24d7" (UID: "46809f03-84a0-460b-9d64-4926110b24d7"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 08:21:06.480290 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:21:06.480228    4720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/46809f03-84a0-460b-9d64-4926110b24d7-kube-api-access-2kn6j" pod "46809f03-84a0-460b-9d64-4926110b24d7" (UID: "46809f03-84a0-460b-9d64-4926110b24d7"). InnerVolumeSpecName "kube-api-access-2kn6j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 08:21:06.491581 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:21:06.491552    4720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46809f03-84a0-460b-9d64-4926110b24d7-tmp-dir" pod "46809f03-84a0-460b-9d64-4926110b24d7" (UID: "46809f03-84a0-460b-9d64-4926110b24d7"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 08:21:06.501558 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:21:06.501315    4720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46809f03-84a0-460b-9d64-4926110b24d7-kserve-provision-location" pod "46809f03-84a0-460b-9d64-4926110b24d7" (UID: "46809f03-84a0-460b-9d64-4926110b24d7"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 08:21:06.578409 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:21:06.578373    4720 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2kn6j\" (UniqueName: \"kubernetes.io/projected/46809f03-84a0-460b-9d64-4926110b24d7-kube-api-access-2kn6j\") on node \"ip-10-0-139-83.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 08:21:06.578409 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:21:06.578406    4720 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/46809f03-84a0-460b-9d64-4926110b24d7-home\") on node \"ip-10-0-139-83.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 08:21:06.578409 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:21:06.578415    4720 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/46809f03-84a0-460b-9d64-4926110b24d7-dshm\") on node \"ip-10-0-139-83.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 08:21:06.578409 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:21:06.578423    4720 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/46809f03-84a0-460b-9d64-4926110b24d7-tls-certs\") on node \"ip-10-0-139-83.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 08:21:06.578705 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:21:06.578434    4720 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/46809f03-84a0-460b-9d64-4926110b24d7-kserve-provision-location\") on node \"ip-10-0-139-83.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 08:21:06.578705 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:21:06.578443    4720 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/46809f03-84a0-460b-9d64-4926110b24d7-cabundle-cert\") on node \"ip-10-0-139-83.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 08:21:06.578705 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:21:06.578452    4720 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/46809f03-84a0-460b-9d64-4926110b24d7-model-cache\") on node \"ip-10-0-139-83.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 08:21:06.578705 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:21:06.578460    4720 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/46809f03-84a0-460b-9d64-4926110b24d7-tmp-dir\") on node \"ip-10-0-139-83.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 08:21:07.058062 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:21:07.058023    4720 generic.go:356] "Generic (PLEG): container finished" podID="46809f03-84a0-460b-9d64-4926110b24d7" containerID="499f7d7cc17be29aee7a2a2de207b8a30ba309a95e958184ccbb2064dfc62c96" exitCode=0
Oct 31 08:21:07.058584 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:21:07.058079    4720 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-57d77d7f74-m88rz"
Oct 31 08:21:07.058584 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:21:07.058094    4720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-57d77d7f74-m88rz" event={"ID":"46809f03-84a0-460b-9d64-4926110b24d7","Type":"ContainerDied","Data":"499f7d7cc17be29aee7a2a2de207b8a30ba309a95e958184ccbb2064dfc62c96"}
Oct 31 08:21:07.058584 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:21:07.058145    4720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-57d77d7f74-m88rz" event={"ID":"46809f03-84a0-460b-9d64-4926110b24d7","Type":"ContainerDied","Data":"ca6d6a8464c19d9110a396b67c97705b80dfe1ee7677151d471ba03f984b0433"}
Oct 31 08:21:07.058584 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:21:07.058165    4720 scope.go:122] "RemoveContainer" containerID="499f7d7cc17be29aee7a2a2de207b8a30ba309a95e958184ccbb2064dfc62c96"
Oct 31 08:21:07.067378 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:21:07.067360    4720 scope.go:122] "RemoveContainer" containerID="84d5c57ba38d623de9163b03f8339f84744c81517f86592549ef5227709286cb"
Oct 31 08:21:07.075220 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:21:07.075199    4720 scope.go:122] "RemoveContainer" containerID="499f7d7cc17be29aee7a2a2de207b8a30ba309a95e958184ccbb2064dfc62c96"
Oct 31 08:21:07.076122 ip-10-0-139-83 kubenswrapper[4720]: E1031 08:21:07.076103    4720 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"499f7d7cc17be29aee7a2a2de207b8a30ba309a95e958184ccbb2064dfc62c96\": container with ID starting with 499f7d7cc17be29aee7a2a2de207b8a30ba309a95e958184ccbb2064dfc62c96 not found: ID does not exist" containerID="499f7d7cc17be29aee7a2a2de207b8a30ba309a95e958184ccbb2064dfc62c96"
Oct 31 08:21:07.076178 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:21:07.076132    4720 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"499f7d7cc17be29aee7a2a2de207b8a30ba309a95e958184ccbb2064dfc62c96"} err="failed to get container status \"499f7d7cc17be29aee7a2a2de207b8a30ba309a95e958184ccbb2064dfc62c96\": rpc error: code = NotFound desc = could not find container \"499f7d7cc17be29aee7a2a2de207b8a30ba309a95e958184ccbb2064dfc62c96\": container with ID starting with 499f7d7cc17be29aee7a2a2de207b8a30ba309a95e958184ccbb2064dfc62c96 not found: ID does not exist"
Oct 31 08:21:07.076178 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:21:07.076153    4720 scope.go:122] "RemoveContainer" containerID="84d5c57ba38d623de9163b03f8339f84744c81517f86592549ef5227709286cb"
Oct 31 08:21:07.076448 ip-10-0-139-83 kubenswrapper[4720]: E1031 08:21:07.076428    4720 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"84d5c57ba38d623de9163b03f8339f84744c81517f86592549ef5227709286cb\": container with ID starting with 84d5c57ba38d623de9163b03f8339f84744c81517f86592549ef5227709286cb not found: ID does not exist" containerID="84d5c57ba38d623de9163b03f8339f84744c81517f86592549ef5227709286cb"
Oct 31 08:21:07.076494 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:21:07.076455    4720 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"84d5c57ba38d623de9163b03f8339f84744c81517f86592549ef5227709286cb"} err="failed to get container status \"84d5c57ba38d623de9163b03f8339f84744c81517f86592549ef5227709286cb\": rpc error: code = NotFound desc = could not find container \"84d5c57ba38d623de9163b03f8339f84744c81517f86592549ef5227709286cb\": container with ID starting with 84d5c57ba38d623de9163b03f8339f84744c81517f86592549ef5227709286cb not found: ID does not exist"
Oct 31 08:21:07.270572 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:21:07.270536    4720 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-57d77d7f74-m88rz"]
Oct 31 08:21:07.320135 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:21:07.320050    4720 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-57d77d7f74-m88rz"]
Oct 31 08:21:08.025616 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:21:08.025579    4720 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="46809f03-84a0-460b-9d64-4926110b24d7" path="/var/lib/kubelet/pods/46809f03-84a0-460b-9d64-4926110b24d7/volumes"
Oct 31 08:21:13.462041 ip-10-0-139-83 kubenswrapper[4720]: E1031 08:21:13.461998    4720 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8m768" containerName="extractor"
Oct 31 08:21:14.353005 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:21:14.352967    4720 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-5c6b9d54r9tqv" podUID="6e859957-5568-4fcf-95ba-aca317743d82" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 08:21:14.353189 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:21:14.353081    4720 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-5c6b9d54r9tqv"
Oct 31 08:21:17.991084 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:21:17.991044    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 08:21:24.353453 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:21:24.353415    4720 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-5c6b9d54r9tqv" podUID="6e859957-5568-4fcf-95ba-aca317743d82" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 08:21:32.627288 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:21:32.627242    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 08:21:34.353353 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:21:34.353295    4720 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-5c6b9d54r9tqv" podUID="6e859957-5568-4fcf-95ba-aca317743d82" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 08:21:35.880006 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:21:35.879936    4720 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-5c6b9d54r9tqv"
Oct 31 08:21:35.929008 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:21:35.928974    4720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6e859957-5568-4fcf-95ba-aca317743d82-kube-api-access-pj6xg\" (UniqueName: \"kubernetes.io/projected/6e859957-5568-4fcf-95ba-aca317743d82-kube-api-access-pj6xg\") pod \"6e859957-5568-4fcf-95ba-aca317743d82\" (UID: \"6e859957-5568-4fcf-95ba-aca317743d82\") "
Oct 31 08:21:35.929152 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:21:35.929040    4720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6e859957-5568-4fcf-95ba-aca317743d82-tls-certs\" (UniqueName: \"kubernetes.io/secret/6e859957-5568-4fcf-95ba-aca317743d82-tls-certs\") pod \"6e859957-5568-4fcf-95ba-aca317743d82\" (UID: \"6e859957-5568-4fcf-95ba-aca317743d82\") "
Oct 31 08:21:35.931410 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:21:35.931381    4720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6e859957-5568-4fcf-95ba-aca317743d82-tls-certs" pod "6e859957-5568-4fcf-95ba-aca317743d82" (UID: "6e859957-5568-4fcf-95ba-aca317743d82"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 08:21:35.931410 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:21:35.931381    4720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6e859957-5568-4fcf-95ba-aca317743d82-kube-api-access-pj6xg" pod "6e859957-5568-4fcf-95ba-aca317743d82" (UID: "6e859957-5568-4fcf-95ba-aca317743d82"). InnerVolumeSpecName "kube-api-access-pj6xg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 08:21:36.029926 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:21:36.029888    4720 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6e859957-5568-4fcf-95ba-aca317743d82-tls-certs\") on node \"ip-10-0-139-83.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 08:21:36.029926 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:21:36.029924    4720 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pj6xg\" (UniqueName: \"kubernetes.io/projected/6e859957-5568-4fcf-95ba-aca317743d82-kube-api-access-pj6xg\") on node \"ip-10-0-139-83.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 08:21:36.188212 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:21:36.188118    4720 generic.go:356] "Generic (PLEG): container finished" podID="6e859957-5568-4fcf-95ba-aca317743d82" containerID="94b9c856482a41b13f199c405739c8efc5f850c8cfefe931aed51bbc06298495" exitCode=0
Oct 31 08:21:36.188212 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:21:36.188168    4720 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-5c6b9d54r9tqv"
Oct 31 08:21:36.188425 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:21:36.188169    4720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-5c6b9d54r9tqv" event={"ID":"6e859957-5568-4fcf-95ba-aca317743d82","Type":"ContainerDied","Data":"94b9c856482a41b13f199c405739c8efc5f850c8cfefe931aed51bbc06298495"}
Oct 31 08:21:36.188425 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:21:36.188286    4720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-5c6b9d54r9tqv" event={"ID":"6e859957-5568-4fcf-95ba-aca317743d82","Type":"ContainerDied","Data":"8c1c251db76034fc90680fd6cd75595fb53a4febef2d5da21cd84f97cf9897c7"}
Oct 31 08:21:36.188425 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:21:36.188309    4720 scope.go:122] "RemoveContainer" containerID="94b9c856482a41b13f199c405739c8efc5f850c8cfefe931aed51bbc06298495"
Oct 31 08:21:36.199648 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:21:36.199629    4720 scope.go:122] "RemoveContainer" containerID="94b9c856482a41b13f199c405739c8efc5f850c8cfefe931aed51bbc06298495"
Oct 31 08:21:36.199946 ip-10-0-139-83 kubenswrapper[4720]: E1031 08:21:36.199921    4720 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"94b9c856482a41b13f199c405739c8efc5f850c8cfefe931aed51bbc06298495\": container with ID starting with 94b9c856482a41b13f199c405739c8efc5f850c8cfefe931aed51bbc06298495 not found: ID does not exist" containerID="94b9c856482a41b13f199c405739c8efc5f850c8cfefe931aed51bbc06298495"
Oct 31 08:21:36.200029 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:21:36.199956    4720 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"94b9c856482a41b13f199c405739c8efc5f850c8cfefe931aed51bbc06298495"} err="failed to get container status \"94b9c856482a41b13f199c405739c8efc5f850c8cfefe931aed51bbc06298495\": rpc error: code = NotFound desc = could not find container \"94b9c856482a41b13f199c405739c8efc5f850c8cfefe931aed51bbc06298495\": container with ID starting with 94b9c856482a41b13f199c405739c8efc5f850c8cfefe931aed51bbc06298495 not found: ID does not exist"
Oct 31 08:21:36.350389 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:21:36.350343    4720 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-5c6b9d54r9tqv"]
Oct 31 08:21:36.356171 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:21:36.356143    4720 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-5c6b9d54r9tqv"]
Oct 31 08:21:38.024860 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:21:38.024824    4720 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6e859957-5568-4fcf-95ba-aca317743d82" path="/var/lib/kubelet/pods/6e859957-5568-4fcf-95ba-aca317743d82/volumes"
Oct 31 08:21:40.011538 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:21:40.011504    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 08:21:40.177653 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:21:40.177616    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 08:21:40.615032 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:21:40.615001    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 08:21:40.764906 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:21:40.764872    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 08:21:43.462508 ip-10-0-139-83 kubenswrapper[4720]: E1031 08:21:43.462472    4720 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8m768" containerName="extractor"
Oct 31 08:21:44.671762 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:21:44.671731    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 08:21:46.208892 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:21:46.208847    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 08:21:47.495173 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:21:47.495132    4720 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-569c6bbc86-klpb4"]
Oct 31 08:21:47.495689 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:21:47.495642    4720 cpu_manager.go:414] "containerMap: removing container" podUID="6e859957-5568-4fcf-95ba-aca317743d82" containerName="main"
Oct 31 08:21:47.495689 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:21:47.495661    4720 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6e859957-5568-4fcf-95ba-aca317743d82" containerName="main"
Oct 31 08:21:47.495808 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:21:47.495706    4720 cpu_manager.go:414] "containerMap: removing container" podUID="46809f03-84a0-460b-9d64-4926110b24d7" containerName="main"
Oct 31 08:21:47.495808 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:21:47.495714    4720 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="46809f03-84a0-460b-9d64-4926110b24d7" containerName="main"
Oct 31 08:21:47.495808 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:21:47.495734    4720 cpu_manager.go:414] "containerMap: removing container" podUID="46809f03-84a0-460b-9d64-4926110b24d7" containerName="storage-initializer"
Oct 31 08:21:47.495808 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:21:47.495742    4720 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="46809f03-84a0-460b-9d64-4926110b24d7" containerName="storage-initializer"
Oct 31 08:21:47.496011 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:21:47.495828    4720 memory_manager.go:365] "RemoveStaleState removing state" podUID="6e859957-5568-4fcf-95ba-aca317743d82" containerName="main"
Oct 31 08:21:47.496011 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:21:47.495849    4720 memory_manager.go:365] "RemoveStaleState removing state" podUID="46809f03-84a0-460b-9d64-4926110b24d7" containerName="main"
Oct 31 08:21:47.499131 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:21:47.499107    4720 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-569c6bbc86-klpb4"
Oct 31 08:21:47.536887 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:21:47.536848    4720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c0febac0-957c-4001-8088-176900d4cfc6-tmp-dir\") pod \"router-with-refs-test-kserve-569c6bbc86-klpb4\" (UID: \"c0febac0-957c-4001-8088-176900d4cfc6\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-569c6bbc86-klpb4"
Oct 31 08:21:47.537092 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:21:47.536899    4720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c0febac0-957c-4001-8088-176900d4cfc6-dshm\") pod \"router-with-refs-test-kserve-569c6bbc86-klpb4\" (UID: \"c0febac0-957c-4001-8088-176900d4cfc6\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-569c6bbc86-klpb4"
Oct 31 08:21:47.537092 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:21:47.536989    4720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c0febac0-957c-4001-8088-176900d4cfc6-model-cache\") pod \"router-with-refs-test-kserve-569c6bbc86-klpb4\" (UID: \"c0febac0-957c-4001-8088-176900d4cfc6\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-569c6bbc86-klpb4"
Oct 31 08:21:47.537092 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:21:47.537041    4720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c0febac0-957c-4001-8088-176900d4cfc6-kserve-provision-location\") pod \"router-with-refs-test-kserve-569c6bbc86-klpb4\" (UID: \"c0febac0-957c-4001-8088-176900d4cfc6\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-569c6bbc86-klpb4"
Oct 31 08:21:47.537280 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:21:47.537137    4720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c0febac0-957c-4001-8088-176900d4cfc6-cabundle-cert\") pod \"router-with-refs-test-kserve-569c6bbc86-klpb4\" (UID: \"c0febac0-957c-4001-8088-176900d4cfc6\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-569c6bbc86-klpb4"
Oct 31 08:21:47.537280 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:21:47.537206    4720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c0febac0-957c-4001-8088-176900d4cfc6-home\") pod \"router-with-refs-test-kserve-569c6bbc86-klpb4\" (UID: \"c0febac0-957c-4001-8088-176900d4cfc6\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-569c6bbc86-klpb4"
Oct 31 08:21:47.537280 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:21:47.537242    4720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c0febac0-957c-4001-8088-176900d4cfc6-tls-certs\") pod \"router-with-refs-test-kserve-569c6bbc86-klpb4\" (UID: \"c0febac0-957c-4001-8088-176900d4cfc6\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-569c6bbc86-klpb4"
Oct 31 08:21:47.537280 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:21:47.537267    4720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9mqqh\" (UniqueName: \"kubernetes.io/projected/c0febac0-957c-4001-8088-176900d4cfc6-kube-api-access-9mqqh\") pod \"router-with-refs-test-kserve-569c6bbc86-klpb4\" (UID: \"c0febac0-957c-4001-8088-176900d4cfc6\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-569c6bbc86-klpb4"
Oct 31 08:21:47.549925 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:21:47.549887    4720 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-569c6bbc86-klpb4"]
Oct 31 08:21:47.565990 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:21:47.565937    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 08:21:47.596478 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:21:47.596419    4720 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-028f7809\"/\"openshift-service-ca.crt\""
Oct 31 08:21:47.596478 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:21:47.596419    4720 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-028f7809\"/\"odh-kserve-custom-ca-bundle\""
Oct 31 08:21:47.596478 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:21:47.596429    4720 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-028f7809\"/\"seaweedfs-s3-creds\""
Oct 31 08:21:47.596478 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:21:47.596466    4720 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-028f7809\"/\"router-with-refs-test-kserve-self-signed-certs\""
Oct 31 08:21:47.596798 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:21:47.596554    4720 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-028f7809\"/\"default-dockercfg-qv72z\""
Oct 31 08:21:47.596798 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:21:47.596702    4720 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-028f7809\"/\"kube-root-ca.crt\""
Oct 31 08:21:47.638528 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:21:47.638494    4720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c0febac0-957c-4001-8088-176900d4cfc6-tmp-dir\") pod \"router-with-refs-test-kserve-569c6bbc86-klpb4\" (UID: \"c0febac0-957c-4001-8088-176900d4cfc6\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-569c6bbc86-klpb4"
Oct 31 08:21:47.638528 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:21:47.638540    4720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c0febac0-957c-4001-8088-176900d4cfc6-dshm\") pod \"router-with-refs-test-kserve-569c6bbc86-klpb4\" (UID: \"c0febac0-957c-4001-8088-176900d4cfc6\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-569c6bbc86-klpb4"
Oct 31 08:21:47.638793 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:21:47.638564    4720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c0febac0-957c-4001-8088-176900d4cfc6-model-cache\") pod \"router-with-refs-test-kserve-569c6bbc86-klpb4\" (UID: \"c0febac0-957c-4001-8088-176900d4cfc6\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-569c6bbc86-klpb4"
Oct 31 08:21:47.638793 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:21:47.638587    4720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c0febac0-957c-4001-8088-176900d4cfc6-kserve-provision-location\") pod \"router-with-refs-test-kserve-569c6bbc86-klpb4\" (UID: \"c0febac0-957c-4001-8088-176900d4cfc6\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-569c6bbc86-klpb4"
Oct 31 08:21:47.638793 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:21:47.638629    4720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c0febac0-957c-4001-8088-176900d4cfc6-cabundle-cert\") pod \"router-with-refs-test-kserve-569c6bbc86-klpb4\" (UID: \"c0febac0-957c-4001-8088-176900d4cfc6\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-569c6bbc86-klpb4"
Oct 31 08:21:47.638793 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:21:47.638660    4720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c0febac0-957c-4001-8088-176900d4cfc6-home\") pod \"router-with-refs-test-kserve-569c6bbc86-klpb4\" (UID: \"c0febac0-957c-4001-8088-176900d4cfc6\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-569c6bbc86-klpb4"
Oct 31 08:21:47.638793 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:21:47.638686    4720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c0febac0-957c-4001-8088-176900d4cfc6-tls-certs\") pod \"router-with-refs-test-kserve-569c6bbc86-klpb4\" (UID: \"c0febac0-957c-4001-8088-176900d4cfc6\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-569c6bbc86-klpb4"
Oct 31 08:21:47.639054 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:21:47.639014    4720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c0febac0-957c-4001-8088-176900d4cfc6-tmp-dir\") pod \"router-with-refs-test-kserve-569c6bbc86-klpb4\" (UID: \"c0febac0-957c-4001-8088-176900d4cfc6\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-569c6bbc86-klpb4"
Oct 31 08:21:47.639112 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:21:47.639068    4720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9mqqh\" (UniqueName: \"kubernetes.io/projected/c0febac0-957c-4001-8088-176900d4cfc6-kube-api-access-9mqqh\") pod \"router-with-refs-test-kserve-569c6bbc86-klpb4\" (UID: \"c0febac0-957c-4001-8088-176900d4cfc6\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-569c6bbc86-klpb4"
Oct 31 08:21:47.639162 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:21:47.639105    4720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c0febac0-957c-4001-8088-176900d4cfc6-model-cache\") pod \"router-with-refs-test-kserve-569c6bbc86-klpb4\" (UID: \"c0febac0-957c-4001-8088-176900d4cfc6\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-569c6bbc86-klpb4"
Oct 31 08:21:47.639241 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:21:47.639219    4720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c0febac0-957c-4001-8088-176900d4cfc6-home\") pod \"router-with-refs-test-kserve-569c6bbc86-klpb4\" (UID: \"c0febac0-957c-4001-8088-176900d4cfc6\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-569c6bbc86-klpb4"
Oct 31 08:21:47.639611 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:21:47.639593    4720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c0febac0-957c-4001-8088-176900d4cfc6-kserve-provision-location\") pod \"router-with-refs-test-kserve-569c6bbc86-klpb4\" (UID: \"c0febac0-957c-4001-8088-176900d4cfc6\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-569c6bbc86-klpb4"
Oct 31 08:21:47.639731 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:21:47.639673    4720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c0febac0-957c-4001-8088-176900d4cfc6-cabundle-cert\") pod \"router-with-refs-test-kserve-569c6bbc86-klpb4\" (UID: \"c0febac0-957c-4001-8088-176900d4cfc6\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-569c6bbc86-klpb4"
Oct 31 08:21:47.641280 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:21:47.641257    4720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c0febac0-957c-4001-8088-176900d4cfc6-dshm\") pod \"router-with-refs-test-kserve-569c6bbc86-klpb4\" (UID: \"c0febac0-957c-4001-8088-176900d4cfc6\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-569c6bbc86-klpb4"
Oct 31 08:21:47.641466 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:21:47.641447    4720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c0febac0-957c-4001-8088-176900d4cfc6-tls-certs\") pod \"router-with-refs-test-kserve-569c6bbc86-klpb4\" (UID: \"c0febac0-957c-4001-8088-176900d4cfc6\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-569c6bbc86-klpb4"
Oct 31 08:21:47.695834 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:21:47.695800    4720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9mqqh\" (UniqueName: \"kubernetes.io/projected/c0febac0-957c-4001-8088-176900d4cfc6-kube-api-access-9mqqh\") pod \"router-with-refs-test-kserve-569c6bbc86-klpb4\" (UID: \"c0febac0-957c-4001-8088-176900d4cfc6\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-569c6bbc86-klpb4"
Oct 31 08:21:47.815198 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:21:47.815091    4720 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-569c6bbc86-klpb4"
Oct 31 08:21:47.893965 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:21:47.893600    4720 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-57f69774b-czqxj"]
Oct 31 08:21:47.897553 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:21:47.897532    4720 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-57f69774b-czqxj"
Oct 31 08:21:47.941647 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:21:47.941614    4720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0b8e1b06-9b4d-4147-8776-750797effaac-tls-certs\") pod \"router-with-refs-test-kserve-router-scheduler-57f69774b-czqxj\" (UID: \"0b8e1b06-9b4d-4147-8776-750797effaac\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-57f69774b-czqxj"
Oct 31 08:21:47.941802 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:21:47.941754    4720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lsjbl\" (UniqueName: \"kubernetes.io/projected/0b8e1b06-9b4d-4147-8776-750797effaac-kube-api-access-lsjbl\") pod \"router-with-refs-test-kserve-router-scheduler-57f69774b-czqxj\" (UID: \"0b8e1b06-9b4d-4147-8776-750797effaac\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-57f69774b-czqxj"
Oct 31 08:21:47.949438 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:21:47.949383    4720 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-028f7809\"/\"router-with-refs-test-epp-sa-dockercfg-kpprl\""
Oct 31 08:21:47.953658 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:21:47.953630    4720 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-57f69774b-czqxj"]
Oct 31 08:21:48.031094 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:21:48.031051    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 08:21:48.042319 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:21:48.042289    4720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lsjbl\" (UniqueName: \"kubernetes.io/projected/0b8e1b06-9b4d-4147-8776-750797effaac-kube-api-access-lsjbl\") pod \"router-with-refs-test-kserve-router-scheduler-57f69774b-czqxj\" (UID: \"0b8e1b06-9b4d-4147-8776-750797effaac\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-57f69774b-czqxj"
Oct 31 08:21:48.042445 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:21:48.042389    4720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0b8e1b06-9b4d-4147-8776-750797effaac-tls-certs\") pod \"router-with-refs-test-kserve-router-scheduler-57f69774b-czqxj\" (UID: \"0b8e1b06-9b4d-4147-8776-750797effaac\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-57f69774b-czqxj"
Oct 31 08:21:48.045242 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:21:48.045198    4720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0b8e1b06-9b4d-4147-8776-750797effaac-tls-certs\") pod \"router-with-refs-test-kserve-router-scheduler-57f69774b-czqxj\" (UID: \"0b8e1b06-9b4d-4147-8776-750797effaac\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-57f69774b-czqxj"
Oct 31 08:21:48.098692 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:21:48.098659    4720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lsjbl\" (UniqueName: \"kubernetes.io/projected/0b8e1b06-9b4d-4147-8776-750797effaac-kube-api-access-lsjbl\") pod \"router-with-refs-test-kserve-router-scheduler-57f69774b-czqxj\" (UID: \"0b8e1b06-9b4d-4147-8776-750797effaac\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-57f69774b-czqxj"
Oct 31 08:21:48.177922 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:21:48.177884    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 08:21:48.212277 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:21:48.212217    4720 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-57f69774b-czqxj"
Oct 31 08:21:48.243278 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:21:48.242742    4720 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-569c6bbc86-klpb4"]
Oct 31 08:21:48.247702 ip-10-0-139-83 kubenswrapper[4720]: W1031 08:21:48.247670    4720 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc0febac0_957c_4001_8088_176900d4cfc6.slice/crio-f2984302bce484e710df238fd9392b27ef1b6392da6a329b07f038e36a153508 WatchSource:0}: Error finding container f2984302bce484e710df238fd9392b27ef1b6392da6a329b07f038e36a153508: Status 404 returned error can't find the container with id f2984302bce484e710df238fd9392b27ef1b6392da6a329b07f038e36a153508
Oct 31 08:21:48.595772 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:21:48.595662    4720 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-57f69774b-czqxj"]
Oct 31 08:21:48.598019 ip-10-0-139-83 kubenswrapper[4720]: W1031 08:21:48.597988    4720 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b8e1b06_9b4d_4147_8776_750797effaac.slice/crio-35c885028011a89114405081f851c79717eed538606000120d0128b0a37e2fe4 WatchSource:0}: Error finding container 35c885028011a89114405081f851c79717eed538606000120d0128b0a37e2fe4: Status 404 returned error can't find the container with id 35c885028011a89114405081f851c79717eed538606000120d0128b0a37e2fe4
Oct 31 08:21:49.251070 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:21:49.251030    4720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-57f69774b-czqxj" event={"ID":"0b8e1b06-9b4d-4147-8776-750797effaac","Type":"ContainerStarted","Data":"84c7981e69db6f7337751e2dba6767f0bfef8e5d599f7a8dc951445dbc75b00a"}
Oct 31 08:21:49.251269 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:21:49.251095    4720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-57f69774b-czqxj" event={"ID":"0b8e1b06-9b4d-4147-8776-750797effaac","Type":"ContainerStarted","Data":"35c885028011a89114405081f851c79717eed538606000120d0128b0a37e2fe4"}
Oct 31 08:21:49.251269 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:21:49.251122    4720 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-57f69774b-czqxj"
Oct 31 08:21:49.252547 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:21:49.252523    4720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-569c6bbc86-klpb4" event={"ID":"c0febac0-957c-4001-8088-176900d4cfc6","Type":"ContainerStarted","Data":"a1c8f675d6c4a070e5770689f4ada68729c2d619e99d367648a185cb8288ff44"}
Oct 31 08:21:49.252650 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:21:49.252568    4720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-569c6bbc86-klpb4" event={"ID":"c0febac0-957c-4001-8088-176900d4cfc6","Type":"ContainerStarted","Data":"f2984302bce484e710df238fd9392b27ef1b6392da6a329b07f038e36a153508"}
Oct 31 08:21:49.363255 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:21:49.363127    4720 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-57f69774b-czqxj" podStartSLOduration=2.363113158 podStartE2EDuration="2.363113158s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 08:21:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 08:21:49.362737739 +0000 UTC m=+3163.493805820" watchObservedRunningTime="2026-10-31 08:21:49.363113158 +0000 UTC m=+3163.494181228"
Oct 31 08:21:51.262961 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:21:51.262924    4720 generic.go:356] "Generic (PLEG): container finished" podID="c0febac0-957c-4001-8088-176900d4cfc6" containerID="a1c8f675d6c4a070e5770689f4ada68729c2d619e99d367648a185cb8288ff44" exitCode=0
Oct 31 08:21:51.263408 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:21:51.262971    4720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-569c6bbc86-klpb4" event={"ID":"c0febac0-957c-4001-8088-176900d4cfc6","Type":"ContainerDied","Data":"a1c8f675d6c4a070e5770689f4ada68729c2d619e99d367648a185cb8288ff44"}
Oct 31 08:21:51.699378 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:21:51.699343    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 08:21:51.860043 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:21:51.860004    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 08:21:52.271026 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:21:52.270993    4720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-569c6bbc86-klpb4" event={"ID":"c0febac0-957c-4001-8088-176900d4cfc6","Type":"ContainerStarted","Data":"c1d026dbefe313402bfc06260165365f940ad6ff3264dc884ac62e839e82f3af"}
Oct 31 08:21:52.485837 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:21:52.485769    4720 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-569c6bbc86-klpb4" podStartSLOduration=3.18559617 podStartE2EDuration="5.485753035s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.300156865s" isStatefulPod=false podCreationTimestamp="2026-10-31 08:21:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 08:21:52.435932558 +0000 UTC m=+3166.567000629" watchObservedRunningTime="2026-10-31 08:21:52.485753035 +0000 UTC m=+3166.616821106"
Oct 31 08:21:52.815287 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:21:52.815252    4720 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-569c6bbc86-klpb4"
Oct 31 08:21:57.390976 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:21:57.390940    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 08:21:57.815938 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:21:57.815841    4720 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-569c6bbc86-klpb4"
Oct 31 08:21:57.817744 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:21:57.817716    4720 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-569c6bbc86-klpb4" podUID="c0febac0-957c-4001-8088-176900d4cfc6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/health\": dial tcp 10.132.0.67:8000: connect: connection refused"
Oct 31 08:22:07.815994 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:22:07.815943    4720 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-569c6bbc86-klpb4" podUID="c0febac0-957c-4001-8088-176900d4cfc6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/health\": dial tcp 10.132.0.67:8000: connect: connection refused"
Oct 31 08:22:13.463664 ip-10-0-139-83 kubenswrapper[4720]: E1031 08:22:13.463630    4720 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8m768" containerName="extractor"
Oct 31 08:22:17.815728 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:22:17.815640    4720 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-569c6bbc86-klpb4" podUID="c0febac0-957c-4001-8088-176900d4cfc6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/health\": dial tcp 10.132.0.67:8000: connect: connection refused"
Oct 31 08:22:20.259533 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:22:20.259499    4720 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-57f69774b-czqxj"
Oct 31 08:22:20.307008 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:22:20.306976    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 08:22:27.815780 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:22:27.815731    4720 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-569c6bbc86-klpb4" podUID="c0febac0-957c-4001-8088-176900d4cfc6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/health\": dial tcp 10.132.0.67:8000: connect: connection refused"
Oct 31 08:22:32.970410 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:22:32.970371    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 08:22:36.576013 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:22:36.575968    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 08:22:36.730689 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:22:36.730653    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 08:22:37.815931 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:22:37.815890    4720 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-569c6bbc86-klpb4" podUID="c0febac0-957c-4001-8088-176900d4cfc6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/health\": dial tcp 10.132.0.67:8000: connect: connection refused"
Oct 31 08:22:43.461836 ip-10-0-139-83 kubenswrapper[4720]: E1031 08:22:43.461788    4720 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8m768" containerName="extractor"
Oct 31 08:22:47.815660 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:22:47.815617    4720 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-569c6bbc86-klpb4" podUID="c0febac0-957c-4001-8088-176900d4cfc6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/health\": dial tcp 10.132.0.67:8000: connect: connection refused"
Oct 31 08:22:57.736415 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:22:57.736375    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 08:22:57.815996 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:22:57.815942    4720 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-569c6bbc86-klpb4" podUID="c0febac0-957c-4001-8088-176900d4cfc6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/health\": dial tcp 10.132.0.67:8000: connect: connection refused"
Oct 31 08:23:07.816128 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:23:07.816068    4720 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-569c6bbc86-klpb4" podUID="c0febac0-957c-4001-8088-176900d4cfc6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/health\": dial tcp 10.132.0.67:8000: connect: connection refused"
Oct 31 08:23:13.462213 ip-10-0-139-83 kubenswrapper[4720]: E1031 08:23:13.462174    4720 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8m768" containerName="extractor"
Oct 31 08:23:17.815951 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:23:17.815894    4720 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-569c6bbc86-klpb4" podUID="c0febac0-957c-4001-8088-176900d4cfc6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/health\": dial tcp 10.132.0.67:8000: connect: connection refused"
Oct 31 08:23:18.679020 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:23:18.678989    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 08:23:26.048945 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:23:26.048898    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 08:23:26.206986 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:23:26.206936    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 08:23:27.816507 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:23:27.816457    4720 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-569c6bbc86-klpb4" podUID="c0febac0-957c-4001-8088-176900d4cfc6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/health\": dial tcp 10.132.0.67:8000: connect: connection refused"
Oct 31 08:23:31.187514 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:23:31.187475    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 08:23:31.601576 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:23:31.601542    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 08:23:32.886512 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:23:32.886478    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 08:23:33.300726 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:23:33.300635    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 08:23:37.816073 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:23:37.816001    4720 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-569c6bbc86-klpb4" podUID="c0febac0-957c-4001-8088-176900d4cfc6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/health\": dial tcp 10.132.0.67:8000: connect: connection refused"
Oct 31 08:23:37.932051 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:23:37.932013    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 08:23:38.518355 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:23:38.518130    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 08:23:38.664887 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:23:38.664823    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 08:23:43.462984 ip-10-0-139-83 kubenswrapper[4720]: E1031 08:23:43.462937    4720 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8m768" containerName="extractor"
Oct 31 08:23:47.826119 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:23:47.826041    4720 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-569c6bbc86-klpb4"
Oct 31 08:23:47.834078 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:23:47.834050    4720 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-569c6bbc86-klpb4"
Oct 31 08:23:56.569506 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:23:56.569474    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 08:23:56.572094 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:23:56.572068    4720 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-569c6bbc86-klpb4"]
Oct 31 08:23:56.585785 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:23:56.585755    4720 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-57f69774b-czqxj"]
Oct 31 08:23:56.716647 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:23:56.716612    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 08:23:58.069265 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:23:58.069230    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 08:24:00.259608 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:24:00.259568    4720 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-57f69774b-czqxj" podUID="0b8e1b06-9b4d-4147-8776-750797effaac" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 08:24:06.333267 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:24:06.333237    4720 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-4jszp_b1db0e3b-e85d-4769-8012-cf4b6f740d9d/console-operator/2.log"
Oct 31 08:24:06.341625 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:24:06.341600    4720 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-4jszp_b1db0e3b-e85d-4769-8012-cf4b6f740d9d/console-operator/2.log"
Oct 31 08:24:10.258604 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:24:10.258559    4720 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-57f69774b-czqxj" podUID="0b8e1b06-9b4d-4147-8776-750797effaac" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 08:24:11.596968 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:24:11.596931    4720 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-569c6bbc86-klpb4" podUID="c0febac0-957c-4001-8088-176900d4cfc6" containerName="main" containerID="cri-o://c1d026dbefe313402bfc06260165365f940ad6ff3264dc884ac62e839e82f3af" gracePeriod=45
Oct 31 08:24:11.606102 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:24:11.606038    4720 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-57f69774b-czqxj" podUID="0b8e1b06-9b4d-4147-8776-750797effaac" containerName="main" containerID="cri-o://84c7981e69db6f7337751e2dba6767f0bfef8e5d599f7a8dc951445dbc75b00a" gracePeriod=45
Oct 31 08:24:11.827379 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:24:11.827322    4720 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-569c6bbc86-klpb4" podUID="c0febac0-957c-4001-8088-176900d4cfc6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/health\": dial tcp 10.132.0.67:8000: connect: connection refused"
Oct 31 08:24:12.346255 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:24:12.346228    4720 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-569c6bbc86-klpb4"
Oct 31 08:24:12.443282 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:24:12.443199    4720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c0febac0-957c-4001-8088-176900d4cfc6-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c0febac0-957c-4001-8088-176900d4cfc6-cabundle-cert\") pod \"c0febac0-957c-4001-8088-176900d4cfc6\" (UID: \"c0febac0-957c-4001-8088-176900d4cfc6\") "
Oct 31 08:24:12.443282 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:24:12.443260    4720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c0febac0-957c-4001-8088-176900d4cfc6-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c0febac0-957c-4001-8088-176900d4cfc6-tmp-dir\") pod \"c0febac0-957c-4001-8088-176900d4cfc6\" (UID: \"c0febac0-957c-4001-8088-176900d4cfc6\") "
Oct 31 08:24:12.443484 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:24:12.443304    4720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c0febac0-957c-4001-8088-176900d4cfc6-home\" (UniqueName: \"kubernetes.io/empty-dir/c0febac0-957c-4001-8088-176900d4cfc6-home\") pod \"c0febac0-957c-4001-8088-176900d4cfc6\" (UID: \"c0febac0-957c-4001-8088-176900d4cfc6\") "
Oct 31 08:24:12.443484 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:24:12.443367    4720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c0febac0-957c-4001-8088-176900d4cfc6-kube-api-access-9mqqh\" (UniqueName: \"kubernetes.io/projected/c0febac0-957c-4001-8088-176900d4cfc6-kube-api-access-9mqqh\") pod \"c0febac0-957c-4001-8088-176900d4cfc6\" (UID: \"c0febac0-957c-4001-8088-176900d4cfc6\") "
Oct 31 08:24:12.443484 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:24:12.443424    4720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c0febac0-957c-4001-8088-176900d4cfc6-tls-certs\" (UniqueName: \"kubernetes.io/secret/c0febac0-957c-4001-8088-176900d4cfc6-tls-certs\") pod \"c0febac0-957c-4001-8088-176900d4cfc6\" (UID: \"c0febac0-957c-4001-8088-176900d4cfc6\") "
Oct 31 08:24:12.443484 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:24:12.443474    4720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c0febac0-957c-4001-8088-176900d4cfc6-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c0febac0-957c-4001-8088-176900d4cfc6-model-cache\") pod \"c0febac0-957c-4001-8088-176900d4cfc6\" (UID: \"c0febac0-957c-4001-8088-176900d4cfc6\") "
Oct 31 08:24:12.443680 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:24:12.443501    4720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c0febac0-957c-4001-8088-176900d4cfc6-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c0febac0-957c-4001-8088-176900d4cfc6-kserve-provision-location\") pod \"c0febac0-957c-4001-8088-176900d4cfc6\" (UID: \"c0febac0-957c-4001-8088-176900d4cfc6\") "
Oct 31 08:24:12.443680 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:24:12.443548    4720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c0febac0-957c-4001-8088-176900d4cfc6-dshm\" (UniqueName: \"kubernetes.io/empty-dir/c0febac0-957c-4001-8088-176900d4cfc6-dshm\") pod \"c0febac0-957c-4001-8088-176900d4cfc6\" (UID: \"c0febac0-957c-4001-8088-176900d4cfc6\") "
Oct 31 08:24:12.443860 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:24:12.443711    4720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c0febac0-957c-4001-8088-176900d4cfc6-cabundle-cert" pod "c0febac0-957c-4001-8088-176900d4cfc6" (UID: "c0febac0-957c-4001-8088-176900d4cfc6"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 08:24:12.443860 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:24:12.443816    4720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0febac0-957c-4001-8088-176900d4cfc6-model-cache" pod "c0febac0-957c-4001-8088-176900d4cfc6" (UID: "c0febac0-957c-4001-8088-176900d4cfc6"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 08:24:12.444214 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:24:12.444162    4720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0febac0-957c-4001-8088-176900d4cfc6-home" pod "c0febac0-957c-4001-8088-176900d4cfc6" (UID: "c0febac0-957c-4001-8088-176900d4cfc6"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 08:24:12.445988 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:24:12.445957    4720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c0febac0-957c-4001-8088-176900d4cfc6-kube-api-access-9mqqh" pod "c0febac0-957c-4001-8088-176900d4cfc6" (UID: "c0febac0-957c-4001-8088-176900d4cfc6"). InnerVolumeSpecName "kube-api-access-9mqqh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 08:24:12.446083 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:24:12.445990    4720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0febac0-957c-4001-8088-176900d4cfc6-dshm" pod "c0febac0-957c-4001-8088-176900d4cfc6" (UID: "c0febac0-957c-4001-8088-176900d4cfc6"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 08:24:12.446143 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:24:12.446101    4720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c0febac0-957c-4001-8088-176900d4cfc6-tls-certs" pod "c0febac0-957c-4001-8088-176900d4cfc6" (UID: "c0febac0-957c-4001-8088-176900d4cfc6"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 08:24:12.459389 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:24:12.459318    4720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0febac0-957c-4001-8088-176900d4cfc6-tmp-dir" pod "c0febac0-957c-4001-8088-176900d4cfc6" (UID: "c0febac0-957c-4001-8088-176900d4cfc6"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 08:24:12.468956 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:24:12.468914    4720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0febac0-957c-4001-8088-176900d4cfc6-kserve-provision-location" pod "c0febac0-957c-4001-8088-176900d4cfc6" (UID: "c0febac0-957c-4001-8088-176900d4cfc6"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 08:24:12.545217 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:24:12.545176    4720 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c0febac0-957c-4001-8088-176900d4cfc6-tls-certs\") on node \"ip-10-0-139-83.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 08:24:12.545217 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:24:12.545215    4720 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c0febac0-957c-4001-8088-176900d4cfc6-model-cache\") on node \"ip-10-0-139-83.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 08:24:12.545481 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:24:12.545230    4720 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c0febac0-957c-4001-8088-176900d4cfc6-kserve-provision-location\") on node \"ip-10-0-139-83.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 08:24:12.545481 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:24:12.545244    4720 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c0febac0-957c-4001-8088-176900d4cfc6-dshm\") on node \"ip-10-0-139-83.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 08:24:12.545481 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:24:12.545257    4720 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c0febac0-957c-4001-8088-176900d4cfc6-cabundle-cert\") on node \"ip-10-0-139-83.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 08:24:12.545481 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:24:12.545293    4720 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c0febac0-957c-4001-8088-176900d4cfc6-tmp-dir\") on node \"ip-10-0-139-83.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 08:24:12.545481 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:24:12.545314    4720 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c0febac0-957c-4001-8088-176900d4cfc6-home\") on node \"ip-10-0-139-83.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 08:24:12.545481 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:24:12.545355    4720 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9mqqh\" (UniqueName: \"kubernetes.io/projected/c0febac0-957c-4001-8088-176900d4cfc6-kube-api-access-9mqqh\") on node \"ip-10-0-139-83.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 08:24:13.088729 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:24:13.088692    4720 generic.go:356] "Generic (PLEG): container finished" podID="c0febac0-957c-4001-8088-176900d4cfc6" containerID="c1d026dbefe313402bfc06260165365f940ad6ff3264dc884ac62e839e82f3af" exitCode=0
Oct 31 08:24:13.089210 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:24:13.088753    4720 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-569c6bbc86-klpb4"
Oct 31 08:24:13.089210 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:24:13.088763    4720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-569c6bbc86-klpb4" event={"ID":"c0febac0-957c-4001-8088-176900d4cfc6","Type":"ContainerDied","Data":"c1d026dbefe313402bfc06260165365f940ad6ff3264dc884ac62e839e82f3af"}
Oct 31 08:24:13.089210 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:24:13.088820    4720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-569c6bbc86-klpb4" event={"ID":"c0febac0-957c-4001-8088-176900d4cfc6","Type":"ContainerDied","Data":"f2984302bce484e710df238fd9392b27ef1b6392da6a329b07f038e36a153508"}
Oct 31 08:24:13.089210 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:24:13.088853    4720 scope.go:122] "RemoveContainer" containerID="c1d026dbefe313402bfc06260165365f940ad6ff3264dc884ac62e839e82f3af"
Oct 31 08:24:13.098521 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:24:13.098491    4720 scope.go:122] "RemoveContainer" containerID="a1c8f675d6c4a070e5770689f4ada68729c2d619e99d367648a185cb8288ff44"
Oct 31 08:24:13.106569 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:24:13.106543    4720 scope.go:122] "RemoveContainer" containerID="c1d026dbefe313402bfc06260165365f940ad6ff3264dc884ac62e839e82f3af"
Oct 31 08:24:13.106867 ip-10-0-139-83 kubenswrapper[4720]: E1031 08:24:13.106846    4720 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c1d026dbefe313402bfc06260165365f940ad6ff3264dc884ac62e839e82f3af\": container with ID starting with c1d026dbefe313402bfc06260165365f940ad6ff3264dc884ac62e839e82f3af not found: ID does not exist" containerID="c1d026dbefe313402bfc06260165365f940ad6ff3264dc884ac62e839e82f3af"
Oct 31 08:24:13.106938 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:24:13.106878    4720 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c1d026dbefe313402bfc06260165365f940ad6ff3264dc884ac62e839e82f3af"} err="failed to get container status \"c1d026dbefe313402bfc06260165365f940ad6ff3264dc884ac62e839e82f3af\": rpc error: code = NotFound desc = could not find container \"c1d026dbefe313402bfc06260165365f940ad6ff3264dc884ac62e839e82f3af\": container with ID starting with c1d026dbefe313402bfc06260165365f940ad6ff3264dc884ac62e839e82f3af not found: ID does not exist"
Oct 31 08:24:13.106938 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:24:13.106897    4720 scope.go:122] "RemoveContainer" containerID="a1c8f675d6c4a070e5770689f4ada68729c2d619e99d367648a185cb8288ff44"
Oct 31 08:24:13.107137 ip-10-0-139-83 kubenswrapper[4720]: E1031 08:24:13.107121    4720 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a1c8f675d6c4a070e5770689f4ada68729c2d619e99d367648a185cb8288ff44\": container with ID starting with a1c8f675d6c4a070e5770689f4ada68729c2d619e99d367648a185cb8288ff44 not found: ID does not exist" containerID="a1c8f675d6c4a070e5770689f4ada68729c2d619e99d367648a185cb8288ff44"
Oct 31 08:24:13.107182 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:24:13.107143    4720 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a1c8f675d6c4a070e5770689f4ada68729c2d619e99d367648a185cb8288ff44"} err="failed to get container status \"a1c8f675d6c4a070e5770689f4ada68729c2d619e99d367648a185cb8288ff44\": rpc error: code = NotFound desc = could not find container \"a1c8f675d6c4a070e5770689f4ada68729c2d619e99d367648a185cb8288ff44\": container with ID starting with a1c8f675d6c4a070e5770689f4ada68729c2d619e99d367648a185cb8288ff44 not found: ID does not exist"
Oct 31 08:24:13.301553 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:24:13.301516    4720 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-569c6bbc86-klpb4"]
Oct 31 08:24:13.351390 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:24:13.351288    4720 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-569c6bbc86-klpb4"]
Oct 31 08:24:13.461925 ip-10-0-139-83 kubenswrapper[4720]: E1031 08:24:13.461884    4720 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8m768" containerName="extractor"
Oct 31 08:24:14.025994 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:24:14.025962    4720 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c0febac0-957c-4001-8088-176900d4cfc6" path="/var/lib/kubelet/pods/c0febac0-957c-4001-8088-176900d4cfc6/volumes"
Oct 31 08:24:19.017544 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:24:19.017508    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 08:24:20.259020 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:24:20.258974    4720 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-57f69774b-czqxj" podUID="0b8e1b06-9b4d-4147-8776-750797effaac" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 08:24:20.259578 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:24:20.259557    4720 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-57f69774b-czqxj"
Oct 31 08:24:30.258911 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:24:30.258869    4720 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-57f69774b-czqxj" podUID="0b8e1b06-9b4d-4147-8776-750797effaac" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 08:24:33.637692 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:24:33.637656    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 08:24:40.259433 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:24:40.259397    4720 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-57f69774b-czqxj" podUID="0b8e1b06-9b4d-4147-8776-750797effaac" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 08:24:41.866114 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:24:41.866084    4720 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-57f69774b-czqxj"
Oct 31 08:24:42.010217 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:24:42.010111    4720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0b8e1b06-9b4d-4147-8776-750797effaac-tls-certs\" (UniqueName: \"kubernetes.io/secret/0b8e1b06-9b4d-4147-8776-750797effaac-tls-certs\") pod \"0b8e1b06-9b4d-4147-8776-750797effaac\" (UID: \"0b8e1b06-9b4d-4147-8776-750797effaac\") "
Oct 31 08:24:42.010437 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:24:42.010219    4720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0b8e1b06-9b4d-4147-8776-750797effaac-kube-api-access-lsjbl\" (UniqueName: \"kubernetes.io/projected/0b8e1b06-9b4d-4147-8776-750797effaac-kube-api-access-lsjbl\") pod \"0b8e1b06-9b4d-4147-8776-750797effaac\" (UID: \"0b8e1b06-9b4d-4147-8776-750797effaac\") "
Oct 31 08:24:42.012567 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:24:42.012539    4720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0b8e1b06-9b4d-4147-8776-750797effaac-tls-certs" pod "0b8e1b06-9b4d-4147-8776-750797effaac" (UID: "0b8e1b06-9b4d-4147-8776-750797effaac"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 08:24:42.012567 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:24:42.012550    4720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0b8e1b06-9b4d-4147-8776-750797effaac-kube-api-access-lsjbl" pod "0b8e1b06-9b4d-4147-8776-750797effaac" (UID: "0b8e1b06-9b4d-4147-8776-750797effaac"). InnerVolumeSpecName "kube-api-access-lsjbl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 08:24:42.111000 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:24:42.110953    4720 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0b8e1b06-9b4d-4147-8776-750797effaac-tls-certs\") on node \"ip-10-0-139-83.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 08:24:42.111000 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:24:42.110997    4720 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lsjbl\" (UniqueName: \"kubernetes.io/projected/0b8e1b06-9b4d-4147-8776-750797effaac-kube-api-access-lsjbl\") on node \"ip-10-0-139-83.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 08:24:42.224180 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:24:42.224142    4720 generic.go:356] "Generic (PLEG): container finished" podID="0b8e1b06-9b4d-4147-8776-750797effaac" containerID="84c7981e69db6f7337751e2dba6767f0bfef8e5d599f7a8dc951445dbc75b00a" exitCode=0
Oct 31 08:24:42.224369 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:24:42.224189    4720 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-57f69774b-czqxj"
Oct 31 08:24:42.224369 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:24:42.224229    4720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-57f69774b-czqxj" event={"ID":"0b8e1b06-9b4d-4147-8776-750797effaac","Type":"ContainerDied","Data":"84c7981e69db6f7337751e2dba6767f0bfef8e5d599f7a8dc951445dbc75b00a"}
Oct 31 08:24:42.224369 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:24:42.224288    4720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-57f69774b-czqxj" event={"ID":"0b8e1b06-9b4d-4147-8776-750797effaac","Type":"ContainerDied","Data":"35c885028011a89114405081f851c79717eed538606000120d0128b0a37e2fe4"}
Oct 31 08:24:42.224369 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:24:42.224311    4720 scope.go:122] "RemoveContainer" containerID="84c7981e69db6f7337751e2dba6767f0bfef8e5d599f7a8dc951445dbc75b00a"
Oct 31 08:24:42.233087 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:24:42.233066    4720 scope.go:122] "RemoveContainer" containerID="84c7981e69db6f7337751e2dba6767f0bfef8e5d599f7a8dc951445dbc75b00a"
Oct 31 08:24:42.233394 ip-10-0-139-83 kubenswrapper[4720]: E1031 08:24:42.233373    4720 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"84c7981e69db6f7337751e2dba6767f0bfef8e5d599f7a8dc951445dbc75b00a\": container with ID starting with 84c7981e69db6f7337751e2dba6767f0bfef8e5d599f7a8dc951445dbc75b00a not found: ID does not exist" containerID="84c7981e69db6f7337751e2dba6767f0bfef8e5d599f7a8dc951445dbc75b00a"
Oct 31 08:24:42.233457 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:24:42.233402    4720 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"84c7981e69db6f7337751e2dba6767f0bfef8e5d599f7a8dc951445dbc75b00a"} err="failed to get container status \"84c7981e69db6f7337751e2dba6767f0bfef8e5d599f7a8dc951445dbc75b00a\": rpc error: code = NotFound desc = could not find container \"84c7981e69db6f7337751e2dba6767f0bfef8e5d599f7a8dc951445dbc75b00a\": container with ID starting with 84c7981e69db6f7337751e2dba6767f0bfef8e5d599f7a8dc951445dbc75b00a not found: ID does not exist"
Oct 31 08:24:42.386151 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:24:42.386112    4720 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-57f69774b-czqxj"]
Oct 31 08:24:42.389553 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:24:42.389526    4720 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-57f69774b-czqxj"]
Oct 31 08:24:43.462257 ip-10-0-139-83 kubenswrapper[4720]: E1031 08:24:43.462217    4720 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8m768" containerName="extractor"
Oct 31 08:24:44.025913 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:24:44.025869    4720 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0b8e1b06-9b4d-4147-8776-750797effaac" path="/var/lib/kubelet/pods/0b8e1b06-9b4d-4147-8776-750797effaac/volumes"
Oct 31 08:24:46.059823 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:24:46.059788    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 08:24:46.219182 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:24:46.219146    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 08:24:51.084690 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:24:51.084654    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 08:24:58.078481 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:24:58.078438    4720 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5d887b9874jzx9t"]
Oct 31 08:24:58.078943 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:24:58.078812    4720 cpu_manager.go:414] "containerMap: removing container" podUID="c0febac0-957c-4001-8088-176900d4cfc6" containerName="main"
Oct 31 08:24:58.078943 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:24:58.078825    4720 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c0febac0-957c-4001-8088-176900d4cfc6" containerName="main"
Oct 31 08:24:58.078943 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:24:58.078838    4720 cpu_manager.go:414] "containerMap: removing container" podUID="0b8e1b06-9b4d-4147-8776-750797effaac" containerName="main"
Oct 31 08:24:58.078943 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:24:58.078843    4720 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0b8e1b06-9b4d-4147-8776-750797effaac" containerName="main"
Oct 31 08:24:58.078943 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:24:58.078859    4720 cpu_manager.go:414] "containerMap: removing container" podUID="c0febac0-957c-4001-8088-176900d4cfc6" containerName="storage-initializer"
Oct 31 08:24:58.078943 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:24:58.078864    4720 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c0febac0-957c-4001-8088-176900d4cfc6" containerName="storage-initializer"
Oct 31 08:24:58.078943 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:24:58.078940    4720 memory_manager.go:365] "RemoveStaleState removing state" podUID="c0febac0-957c-4001-8088-176900d4cfc6" containerName="main"
Oct 31 08:24:58.079153 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:24:58.078950    4720 memory_manager.go:365] "RemoveStaleState removing state" podUID="0b8e1b06-9b4d-4147-8776-750797effaac" containerName="main"
Oct 31 08:24:58.083686 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:24:58.083667    4720 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5d887b9874jzx9t"
Oct 31 08:24:58.136407 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:24:58.136372    4720 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5d887b9874jzx9t"]
Oct 31 08:24:58.156619 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:24:58.156575    4720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/83fff906-3b80-4fa2-9dcf-ebbb611580b6-dshm\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5d887b9874jzx9t\" (UID: \"83fff906-3b80-4fa2-9dcf-ebbb611580b6\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5d887b9874jzx9t"
Oct 31 08:24:58.156619 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:24:58.156618    4720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/83fff906-3b80-4fa2-9dcf-ebbb611580b6-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5d887b9874jzx9t\" (UID: \"83fff906-3b80-4fa2-9dcf-ebbb611580b6\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5d887b9874jzx9t"
Oct 31 08:24:58.157157 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:24:58.156711    4720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/83fff906-3b80-4fa2-9dcf-ebbb611580b6-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5d887b9874jzx9t\" (UID: \"83fff906-3b80-4fa2-9dcf-ebbb611580b6\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5d887b9874jzx9t"
Oct 31 08:24:58.157157 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:24:58.156759    4720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/83fff906-3b80-4fa2-9dcf-ebbb611580b6-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5d887b9874jzx9t\" (UID: \"83fff906-3b80-4fa2-9dcf-ebbb611580b6\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5d887b9874jzx9t"
Oct 31 08:24:58.157157 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:24:58.156782    4720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/83fff906-3b80-4fa2-9dcf-ebbb611580b6-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5d887b9874jzx9t\" (UID: \"83fff906-3b80-4fa2-9dcf-ebbb611580b6\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5d887b9874jzx9t"
Oct 31 08:24:58.157157 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:24:58.156863    4720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/83fff906-3b80-4fa2-9dcf-ebbb611580b6-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5d887b9874jzx9t\" (UID: \"83fff906-3b80-4fa2-9dcf-ebbb611580b6\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5d887b9874jzx9t"
Oct 31 08:24:58.157157 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:24:58.156918    4720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p4lq7\" (UniqueName: \"kubernetes.io/projected/83fff906-3b80-4fa2-9dcf-ebbb611580b6-kube-api-access-p4lq7\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5d887b9874jzx9t\" (UID: \"83fff906-3b80-4fa2-9dcf-ebbb611580b6\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5d887b9874jzx9t"
Oct 31 08:24:58.157157 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:24:58.156949    4720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/83fff906-3b80-4fa2-9dcf-ebbb611580b6-home\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5d887b9874jzx9t\" (UID: \"83fff906-3b80-4fa2-9dcf-ebbb611580b6\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5d887b9874jzx9t"
Oct 31 08:24:58.181891 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:24:58.181853    4720 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"seaweedfs-s3-creds\""
Oct 31 08:24:58.181891 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:24:58.181880    4720 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"odh-kserve-custom-ca-bundle\""
Oct 31 08:24:58.181891 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:24:58.181853    4720 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"openshift-service-ca.crt\""
Oct 31 08:24:58.181891 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:24:58.181853    4720 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"kube-root-ca.crt\""
Oct 31 08:24:58.182372 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:24:58.182104    4720 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-dockercfg-fzpcd\""
Oct 31 08:24:58.182372 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:24:58.182230    4720 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"default-dockercfg-wvx2n\""
Oct 31 08:24:58.182372 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:24:58.182293    4720 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"llmisv44d181485fad85e662eb092f3749502f-kserve-self-signed-certs\""
Oct 31 08:24:58.187029 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:24:58.187000    4720 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6cgq8lv"]
Oct 31 08:24:58.190599 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:24:58.190577    4720 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6cgq8lv"
Oct 31 08:24:58.251782 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:24:58.251752    4720 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6cgq8lv"]
Oct 31 08:24:58.258108 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:24:58.258075    4720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/83fff906-3b80-4fa2-9dcf-ebbb611580b6-dshm\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5d887b9874jzx9t\" (UID: \"83fff906-3b80-4fa2-9dcf-ebbb611580b6\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5d887b9874jzx9t"
Oct 31 08:24:58.258287 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:24:58.258124    4720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/83fff906-3b80-4fa2-9dcf-ebbb611580b6-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5d887b9874jzx9t\" (UID: \"83fff906-3b80-4fa2-9dcf-ebbb611580b6\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5d887b9874jzx9t"
Oct 31 08:24:58.258287 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:24:58.258174    4720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/83fff906-3b80-4fa2-9dcf-ebbb611580b6-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5d887b9874jzx9t\" (UID: \"83fff906-3b80-4fa2-9dcf-ebbb611580b6\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5d887b9874jzx9t"
Oct 31 08:24:58.258287 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:24:58.258216    4720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/83fff906-3b80-4fa2-9dcf-ebbb611580b6-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5d887b9874jzx9t\" (UID: \"83fff906-3b80-4fa2-9dcf-ebbb611580b6\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5d887b9874jzx9t"
Oct 31 08:24:58.258287 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:24:58.258248    4720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/83fff906-3b80-4fa2-9dcf-ebbb611580b6-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5d887b9874jzx9t\" (UID: \"83fff906-3b80-4fa2-9dcf-ebbb611580b6\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5d887b9874jzx9t"
Oct 31 08:24:58.258528 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:24:58.258295    4720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/83fff906-3b80-4fa2-9dcf-ebbb611580b6-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5d887b9874jzx9t\" (UID: \"83fff906-3b80-4fa2-9dcf-ebbb611580b6\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5d887b9874jzx9t"
Oct 31 08:24:58.258528 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:24:58.258411    4720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b13794bf-1eb8-4f5c-9485-9f99ffc8ac37-home\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6cgq8lv\" (UID: \"b13794bf-1eb8-4f5c-9485-9f99ffc8ac37\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6cgq8lv"
Oct 31 08:24:58.258528 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:24:58.258504    4720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b13794bf-1eb8-4f5c-9485-9f99ffc8ac37-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6cgq8lv\" (UID: \"b13794bf-1eb8-4f5c-9485-9f99ffc8ac37\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6cgq8lv"
Oct 31 08:24:58.258689 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:24:58.258553    4720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b13794bf-1eb8-4f5c-9485-9f99ffc8ac37-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6cgq8lv\" (UID: \"b13794bf-1eb8-4f5c-9485-9f99ffc8ac37\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6cgq8lv"
Oct 31 08:24:58.258689 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:24:58.258603    4720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/83fff906-3b80-4fa2-9dcf-ebbb611580b6-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5d887b9874jzx9t\" (UID: \"83fff906-3b80-4fa2-9dcf-ebbb611580b6\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5d887b9874jzx9t"
Oct 31 08:24:58.258689 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:24:58.258634    4720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p4lq7\" (UniqueName: \"kubernetes.io/projected/83fff906-3b80-4fa2-9dcf-ebbb611580b6-kube-api-access-p4lq7\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5d887b9874jzx9t\" (UID: \"83fff906-3b80-4fa2-9dcf-ebbb611580b6\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5d887b9874jzx9t"
Oct 31 08:24:58.258689 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:24:58.258668    4720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b13794bf-1eb8-4f5c-9485-9f99ffc8ac37-dshm\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6cgq8lv\" (UID: \"b13794bf-1eb8-4f5c-9485-9f99ffc8ac37\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6cgq8lv"
Oct 31 08:24:58.258913 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:24:58.258703    4720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8p2m7\" (UniqueName: \"kubernetes.io/projected/b13794bf-1eb8-4f5c-9485-9f99ffc8ac37-kube-api-access-8p2m7\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6cgq8lv\" (UID: \"b13794bf-1eb8-4f5c-9485-9f99ffc8ac37\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6cgq8lv"
Oct 31 08:24:58.258913 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:24:58.258743    4720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/83fff906-3b80-4fa2-9dcf-ebbb611580b6-home\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5d887b9874jzx9t\" (UID: \"83fff906-3b80-4fa2-9dcf-ebbb611580b6\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5d887b9874jzx9t"
Oct 31 08:24:58.258913 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:24:58.258770    4720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b13794bf-1eb8-4f5c-9485-9f99ffc8ac37-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6cgq8lv\" (UID: \"b13794bf-1eb8-4f5c-9485-9f99ffc8ac37\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6cgq8lv"
Oct 31 08:24:58.258913 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:24:58.258796    4720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b13794bf-1eb8-4f5c-9485-9f99ffc8ac37-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6cgq8lv\" (UID: \"b13794bf-1eb8-4f5c-9485-9f99ffc8ac37\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6cgq8lv"
Oct 31 08:24:58.258913 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:24:58.258820    4720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b13794bf-1eb8-4f5c-9485-9f99ffc8ac37-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6cgq8lv\" (UID: \"b13794bf-1eb8-4f5c-9485-9f99ffc8ac37\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6cgq8lv"
Oct 31 08:24:58.258913 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:24:58.258836    4720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/83fff906-3b80-4fa2-9dcf-ebbb611580b6-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5d887b9874jzx9t\" (UID: \"83fff906-3b80-4fa2-9dcf-ebbb611580b6\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5d887b9874jzx9t"
Oct 31 08:24:58.259226 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:24:58.259126    4720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/83fff906-3b80-4fa2-9dcf-ebbb611580b6-home\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5d887b9874jzx9t\" (UID: \"83fff906-3b80-4fa2-9dcf-ebbb611580b6\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5d887b9874jzx9t"
Oct 31 08:24:58.259262 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:24:58.259243    4720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/83fff906-3b80-4fa2-9dcf-ebbb611580b6-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5d887b9874jzx9t\" (UID: \"83fff906-3b80-4fa2-9dcf-ebbb611580b6\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5d887b9874jzx9t"
Oct 31 08:24:58.259443 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:24:58.259420    4720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/83fff906-3b80-4fa2-9dcf-ebbb611580b6-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5d887b9874jzx9t\" (UID: \"83fff906-3b80-4fa2-9dcf-ebbb611580b6\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5d887b9874jzx9t"
Oct 31 08:24:58.260714 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:24:58.260688    4720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/83fff906-3b80-4fa2-9dcf-ebbb611580b6-dshm\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5d887b9874jzx9t\" (UID: \"83fff906-3b80-4fa2-9dcf-ebbb611580b6\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5d887b9874jzx9t"
Oct 31 08:24:58.260988 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:24:58.260969    4720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/83fff906-3b80-4fa2-9dcf-ebbb611580b6-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5d887b9874jzx9t\" (UID: \"83fff906-3b80-4fa2-9dcf-ebbb611580b6\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5d887b9874jzx9t"
Oct 31 08:24:58.295721 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:24:58.295682    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 08:24:58.315059 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:24:58.315018    4720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p4lq7\" (UniqueName: \"kubernetes.io/projected/83fff906-3b80-4fa2-9dcf-ebbb611580b6-kube-api-access-p4lq7\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5d887b9874jzx9t\" (UID: \"83fff906-3b80-4fa2-9dcf-ebbb611580b6\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5d887b9874jzx9t"
Oct 31 08:24:58.367387 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:24:58.361499    4720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b13794bf-1eb8-4f5c-9485-9f99ffc8ac37-dshm\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6cgq8lv\" (UID: \"b13794bf-1eb8-4f5c-9485-9f99ffc8ac37\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6cgq8lv"
Oct 31 08:24:58.367387 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:24:58.361558    4720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8p2m7\" (UniqueName: \"kubernetes.io/projected/b13794bf-1eb8-4f5c-9485-9f99ffc8ac37-kube-api-access-8p2m7\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6cgq8lv\" (UID: \"b13794bf-1eb8-4f5c-9485-9f99ffc8ac37\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6cgq8lv"
Oct 31 08:24:58.367387 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:24:58.361621    4720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b13794bf-1eb8-4f5c-9485-9f99ffc8ac37-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6cgq8lv\" (UID: \"b13794bf-1eb8-4f5c-9485-9f99ffc8ac37\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6cgq8lv"
Oct 31 08:24:58.367387 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:24:58.361670    4720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b13794bf-1eb8-4f5c-9485-9f99ffc8ac37-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6cgq8lv\" (UID: \"b13794bf-1eb8-4f5c-9485-9f99ffc8ac37\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6cgq8lv"
Oct 31 08:24:58.367387 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:24:58.361705    4720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b13794bf-1eb8-4f5c-9485-9f99ffc8ac37-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6cgq8lv\" (UID: \"b13794bf-1eb8-4f5c-9485-9f99ffc8ac37\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6cgq8lv"
Oct 31 08:24:58.367387 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:24:58.361863    4720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b13794bf-1eb8-4f5c-9485-9f99ffc8ac37-home\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6cgq8lv\" (UID: \"b13794bf-1eb8-4f5c-9485-9f99ffc8ac37\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6cgq8lv"
Oct 31 08:24:58.367387 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:24:58.361918    4720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b13794bf-1eb8-4f5c-9485-9f99ffc8ac37-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6cgq8lv\" (UID: \"b13794bf-1eb8-4f5c-9485-9f99ffc8ac37\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6cgq8lv"
Oct 31 08:24:58.367387 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:24:58.361971    4720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b13794bf-1eb8-4f5c-9485-9f99ffc8ac37-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6cgq8lv\" (UID: \"b13794bf-1eb8-4f5c-9485-9f99ffc8ac37\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6cgq8lv"
Oct 31 08:24:58.367387 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:24:58.362578    4720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b13794bf-1eb8-4f5c-9485-9f99ffc8ac37-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6cgq8lv\" (UID: \"b13794bf-1eb8-4f5c-9485-9f99ffc8ac37\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6cgq8lv"
Oct 31 08:24:58.367387 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:24:58.364780    4720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b13794bf-1eb8-4f5c-9485-9f99ffc8ac37-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6cgq8lv\" (UID: \"b13794bf-1eb8-4f5c-9485-9f99ffc8ac37\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6cgq8lv"
Oct 31 08:24:58.367387 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:24:58.366515    4720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b13794bf-1eb8-4f5c-9485-9f99ffc8ac37-home\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6cgq8lv\" (UID: \"b13794bf-1eb8-4f5c-9485-9f99ffc8ac37\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6cgq8lv"
Oct 31 08:24:58.367387 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:24:58.366798    4720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b13794bf-1eb8-4f5c-9485-9f99ffc8ac37-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6cgq8lv\" (UID: \"b13794bf-1eb8-4f5c-9485-9f99ffc8ac37\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6cgq8lv"
Oct 31 08:24:58.368063 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:24:58.367573    4720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b13794bf-1eb8-4f5c-9485-9f99ffc8ac37-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6cgq8lv\" (UID: \"b13794bf-1eb8-4f5c-9485-9f99ffc8ac37\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6cgq8lv"
Oct 31 08:24:58.368063 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:24:58.367868    4720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b13794bf-1eb8-4f5c-9485-9f99ffc8ac37-dshm\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6cgq8lv\" (UID: \"b13794bf-1eb8-4f5c-9485-9f99ffc8ac37\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6cgq8lv"
Oct 31 08:24:58.369674 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:24:58.369651    4720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b13794bf-1eb8-4f5c-9485-9f99ffc8ac37-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6cgq8lv\" (UID: \"b13794bf-1eb8-4f5c-9485-9f99ffc8ac37\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6cgq8lv"
Oct 31 08:24:58.399905 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:24:58.399864    4720 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5d887b9874jzx9t"
Oct 31 08:24:58.420756 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:24:58.420723    4720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8p2m7\" (UniqueName: \"kubernetes.io/projected/b13794bf-1eb8-4f5c-9485-9f99ffc8ac37-kube-api-access-8p2m7\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6cgq8lv\" (UID: \"b13794bf-1eb8-4f5c-9485-9f99ffc8ac37\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6cgq8lv"
Oct 31 08:24:58.443014 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:24:58.442973    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 08:24:58.505871 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:24:58.505827    4720 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6cgq8lv"
Oct 31 08:24:58.606534 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:24:58.606497    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 08:24:58.766156 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:24:58.766069    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 08:24:58.833304 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:24:58.833263    4720 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5d887b9874jzx9t"]
Oct 31 08:24:58.835151 ip-10-0-139-83 kubenswrapper[4720]: W1031 08:24:58.835123    4720 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod83fff906_3b80_4fa2_9dcf_ebbb611580b6.slice/crio-057593ad2e037c9c5b2f6bf9dd88eee8178afe13d886aec1a93276d19d665a7d WatchSource:0}: Error finding container 057593ad2e037c9c5b2f6bf9dd88eee8178afe13d886aec1a93276d19d665a7d: Status 404 returned error can't find the container with id 057593ad2e037c9c5b2f6bf9dd88eee8178afe13d886aec1a93276d19d665a7d
Oct 31 08:24:58.890037 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:24:58.890006    4720 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6cgq8lv"]
Oct 31 08:24:58.891114 ip-10-0-139-83 kubenswrapper[4720]: W1031 08:24:58.891086    4720 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb13794bf_1eb8_4f5c_9485_9f99ffc8ac37.slice/crio-1cea409b78c42a61ebe83a241e09778b4e09b42453ec7dce26b567b1b2ab3a3c WatchSource:0}: Error finding container 1cea409b78c42a61ebe83a241e09778b4e09b42453ec7dce26b567b1b2ab3a3c: Status 404 returned error can't find the container with id 1cea409b78c42a61ebe83a241e09778b4e09b42453ec7dce26b567b1b2ab3a3c
Oct 31 08:24:58.918269 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:24:58.918239    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 08:24:59.315461 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:24:59.315424    4720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6cgq8lv" event={"ID":"b13794bf-1eb8-4f5c-9485-9f99ffc8ac37","Type":"ContainerStarted","Data":"bd2c662bc34e197472d80eb6f22c51f1dd3973461196a5f7b6c4d9f26ea4c362"}
Oct 31 08:24:59.316001 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:24:59.315488    4720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6cgq8lv" event={"ID":"b13794bf-1eb8-4f5c-9485-9f99ffc8ac37","Type":"ContainerStarted","Data":"1cea409b78c42a61ebe83a241e09778b4e09b42453ec7dce26b567b1b2ab3a3c"}
Oct 31 08:24:59.317137 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:24:59.317108    4720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5d887b9874jzx9t" event={"ID":"83fff906-3b80-4fa2-9dcf-ebbb611580b6","Type":"ContainerStarted","Data":"db15b0b0b44d8452c559f9e3712f88cf243305c77822a92a35b3550a625acb95"}
Oct 31 08:24:59.317311 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:24:59.317169    4720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5d887b9874jzx9t" event={"ID":"83fff906-3b80-4fa2-9dcf-ebbb611580b6","Type":"ContainerStarted","Data":"057593ad2e037c9c5b2f6bf9dd88eee8178afe13d886aec1a93276d19d665a7d"}
Oct 31 08:24:59.317311 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:24:59.317240    4720 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5d887b9874jzx9t"
Oct 31 08:25:00.323813 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:25:00.323764    4720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5d887b9874jzx9t" event={"ID":"83fff906-3b80-4fa2-9dcf-ebbb611580b6","Type":"ContainerStarted","Data":"9fa841e663068d80e8fe23765e01a51c5d675306e9cd270706846a9dc7a81455"}
Oct 31 08:25:02.342604 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:25:02.342567    4720 generic.go:356] "Generic (PLEG): container finished" podID="83fff906-3b80-4fa2-9dcf-ebbb611580b6" containerID="9fa841e663068d80e8fe23765e01a51c5d675306e9cd270706846a9dc7a81455" exitCode=0
Oct 31 08:25:02.343081 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:25:02.342639    4720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5d887b9874jzx9t" event={"ID":"83fff906-3b80-4fa2-9dcf-ebbb611580b6","Type":"ContainerDied","Data":"9fa841e663068d80e8fe23765e01a51c5d675306e9cd270706846a9dc7a81455"}
Oct 31 08:25:02.344443 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:25:02.344421    4720 generic.go:356] "Generic (PLEG): container finished" podID="b13794bf-1eb8-4f5c-9485-9f99ffc8ac37" containerID="bd2c662bc34e197472d80eb6f22c51f1dd3973461196a5f7b6c4d9f26ea4c362" exitCode=0
Oct 31 08:25:02.344558 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:25:02.344452    4720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6cgq8lv" event={"ID":"b13794bf-1eb8-4f5c-9485-9f99ffc8ac37","Type":"ContainerDied","Data":"bd2c662bc34e197472d80eb6f22c51f1dd3973461196a5f7b6c4d9f26ea4c362"}
Oct 31 08:25:03.351655 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:25:03.351604    4720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6cgq8lv" event={"ID":"b13794bf-1eb8-4f5c-9485-9f99ffc8ac37","Type":"ContainerStarted","Data":"ba15ea1f34c89ea3bf4d53f13577b383a23503091bef5266f57b206782ed6a37"}
Oct 31 08:25:03.354205 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:25:03.354172    4720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5d887b9874jzx9t" event={"ID":"83fff906-3b80-4fa2-9dcf-ebbb611580b6","Type":"ContainerStarted","Data":"1aafb77c9c4370df857002f234eb06092914bc2fb1793cf61063b4b2e994152c"}
Oct 31 08:25:03.518034 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:25:03.517971    4720 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6cgq8lv" podStartSLOduration=2.922925007 podStartE2EDuration="5.517956825s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.595031818s" isStatefulPod=false podCreationTimestamp="2026-10-31 08:24:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 08:25:03.517093152 +0000 UTC m=+3357.648161223" watchObservedRunningTime="2026-10-31 08:25:03.517956825 +0000 UTC m=+3357.649024924"
Oct 31 08:25:03.634558 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:25:03.634419    4720 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5d887b9874jzx9t" podStartSLOduration=4.03410449 podStartE2EDuration="6.634399452s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.600294962s" isStatefulPod=false podCreationTimestamp="2026-10-31 08:24:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 08:25:03.633664924 +0000 UTC m=+3357.764733018" watchObservedRunningTime="2026-10-31 08:25:03.634399452 +0000 UTC m=+3357.765467542"
Oct 31 08:25:05.186098 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:25:05.186056    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 08:25:08.400267 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:25:08.400225    4720 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5d887b9874jzx9t"
Oct 31 08:25:08.400773 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:25:08.400307    4720 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5d887b9874jzx9t"
Oct 31 08:25:08.402007 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:25:08.401969    4720 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5d887b9874jzx9t" podUID="83fff906-3b80-4fa2-9dcf-ebbb611580b6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8001/health\": dial tcp 10.132.0.69:8001: connect: connection refused"
Oct 31 08:25:08.415346 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:25:08.415288    4720 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5d887b9874jzx9t"
Oct 31 08:25:08.506476 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:25:08.506439    4720 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6cgq8lv"
Oct 31 08:25:08.506695 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:25:08.506510    4720 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6cgq8lv"
Oct 31 08:25:08.508165 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:25:08.508118    4720 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6cgq8lv" podUID="b13794bf-1eb8-4f5c-9485-9f99ffc8ac37" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/health\": dial tcp 10.132.0.70:8000: connect: connection refused"
Oct 31 08:25:12.321695 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:25:12.321656    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 08:25:13.231666 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:25:13.231604    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 08:25:13.415526 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:25:13.415481    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 08:25:13.462364 ip-10-0-139-83 kubenswrapper[4720]: E1031 08:25:13.462286    4720 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8m768" containerName="extractor"
Oct 31 08:25:18.400701 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:25:18.400592    4720 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5d887b9874jzx9t" podUID="83fff906-3b80-4fa2-9dcf-ebbb611580b6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8001/health\": dial tcp 10.132.0.69:8001: connect: connection refused"
Oct 31 08:25:18.507261 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:25:18.507220    4720 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6cgq8lv" podUID="b13794bf-1eb8-4f5c-9485-9f99ffc8ac37" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/health\": dial tcp 10.132.0.70:8000: connect: connection refused"
Oct 31 08:25:18.831234 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:25:18.831161    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 08:25:19.352297 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:25:19.352257    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 08:25:28.401259 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:25:28.401189    4720 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5d887b9874jzx9t" podUID="83fff906-3b80-4fa2-9dcf-ebbb611580b6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8001/health\": dial tcp 10.132.0.69:8001: connect: connection refused"
Oct 31 08:25:28.506315 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:25:28.506274    4720 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6cgq8lv" podUID="b13794bf-1eb8-4f5c-9485-9f99ffc8ac37" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/health\": dial tcp 10.132.0.70:8000: connect: connection refused"
Oct 31 08:25:29.443412 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:25:29.443356    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 08:25:32.772105 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:25:32.772069    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 08:25:32.929828 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:25:32.929788    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 08:25:33.970471 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:25:33.970420    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 08:25:38.400716 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:25:38.400648    4720 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5d887b9874jzx9t" podUID="83fff906-3b80-4fa2-9dcf-ebbb611580b6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8001/health\": dial tcp 10.132.0.69:8001: connect: connection refused"
Oct 31 08:25:38.506563 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:25:38.506509    4720 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6cgq8lv" podUID="b13794bf-1eb8-4f5c-9485-9f99ffc8ac37" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/health\": dial tcp 10.132.0.70:8000: connect: connection refused"
Oct 31 08:25:43.462558 ip-10-0-139-83 kubenswrapper[4720]: E1031 08:25:43.462521    4720 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8m768" containerName="extractor"
Oct 31 08:25:44.805574 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:25:44.805520    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 08:25:44.965047 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:25:44.965005    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 08:25:48.401458 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:25:48.401369    4720 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5d887b9874jzx9t" podUID="83fff906-3b80-4fa2-9dcf-ebbb611580b6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8001/health\": dial tcp 10.132.0.69:8001: connect: connection refused"
Oct 31 08:25:48.507142 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:25:48.507089    4720 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6cgq8lv" podUID="b13794bf-1eb8-4f5c-9485-9f99ffc8ac37" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/health\": dial tcp 10.132.0.70:8000: connect: connection refused"
Oct 31 08:25:50.817626 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:25:50.817575    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 08:25:51.772739 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:25:51.772696    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 08:25:52.228228 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:25:52.228189    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 08:25:56.450733 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:25:56.450696    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 08:25:56.604989 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:25:56.604944    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 08:25:58.400695 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:25:58.400641    4720 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5d887b9874jzx9t" podUID="83fff906-3b80-4fa2-9dcf-ebbb611580b6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8001/health\": dial tcp 10.132.0.69:8001: connect: connection refused"
Oct 31 08:25:58.506295 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:25:58.506246    4720 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6cgq8lv" podUID="b13794bf-1eb8-4f5c-9485-9f99ffc8ac37" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/health\": dial tcp 10.132.0.70:8000: connect: connection refused"
Oct 31 08:25:58.737240 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:25:58.737125    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 08:26:02.385503 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:26:02.385459    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 08:26:08.400898 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:26:08.400843    4720 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5d887b9874jzx9t" podUID="83fff906-3b80-4fa2-9dcf-ebbb611580b6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8001/health\": dial tcp 10.132.0.69:8001: connect: connection refused"
Oct 31 08:26:08.506506 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:26:08.506456    4720 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6cgq8lv" podUID="b13794bf-1eb8-4f5c-9485-9f99ffc8ac37" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/health\": dial tcp 10.132.0.70:8000: connect: connection refused"
Oct 31 08:26:13.463370 ip-10-0-139-83 kubenswrapper[4720]: E1031 08:26:13.463303    4720 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8m768" containerName="extractor"
Oct 31 08:26:18.400660 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:26:18.400615    4720 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5d887b9874jzx9t" podUID="83fff906-3b80-4fa2-9dcf-ebbb611580b6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8001/health\": dial tcp 10.132.0.69:8001: connect: connection refused"
Oct 31 08:26:18.507131 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:26:18.507093    4720 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6cgq8lv" podUID="b13794bf-1eb8-4f5c-9485-9f99ffc8ac37" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/health\": dial tcp 10.132.0.70:8000: connect: connection refused"
Oct 31 08:26:19.693575 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:26:19.693539    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 08:26:24.623879 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:26:24.623838    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 08:26:28.400798 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:26:28.400750    4720 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5d887b9874jzx9t" podUID="83fff906-3b80-4fa2-9dcf-ebbb611580b6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8001/health\": dial tcp 10.132.0.69:8001: connect: connection refused"
Oct 31 08:26:28.506445 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:26:28.506397    4720 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6cgq8lv" podUID="b13794bf-1eb8-4f5c-9485-9f99ffc8ac37" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/health\": dial tcp 10.132.0.70:8000: connect: connection refused"
Oct 31 08:26:34.311301 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:26:34.311262    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 08:26:38.400607 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:26:38.400554    4720 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5d887b9874jzx9t" podUID="83fff906-3b80-4fa2-9dcf-ebbb611580b6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8001/health\": dial tcp 10.132.0.69:8001: connect: connection refused"
Oct 31 08:26:38.506917 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:26:38.506867    4720 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6cgq8lv" podUID="b13794bf-1eb8-4f5c-9485-9f99ffc8ac37" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/health\": dial tcp 10.132.0.70:8000: connect: connection refused"
Oct 31 08:26:43.464162 ip-10-0-139-83 kubenswrapper[4720]: E1031 08:26:43.464113    4720 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8m768" containerName="extractor"
Oct 31 08:26:48.400928 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:26:48.400820    4720 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5d887b9874jzx9t" podUID="83fff906-3b80-4fa2-9dcf-ebbb611580b6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8001/health\": dial tcp 10.132.0.69:8001: connect: connection refused"
Oct 31 08:26:48.506604 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:26:48.506553    4720 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6cgq8lv" podUID="b13794bf-1eb8-4f5c-9485-9f99ffc8ac37" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/health\": dial tcp 10.132.0.70:8000: connect: connection refused"
Oct 31 08:26:58.401278 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:26:58.401218    4720 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5d887b9874jzx9t" podUID="83fff906-3b80-4fa2-9dcf-ebbb611580b6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8001/health\": dial tcp 10.132.0.69:8001: connect: connection refused"
Oct 31 08:26:58.506644 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:26:58.506601    4720 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6cgq8lv" podUID="b13794bf-1eb8-4f5c-9485-9f99ffc8ac37" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/health\": dial tcp 10.132.0.70:8000: connect: connection refused"
Oct 31 08:26:59.084748 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:26:59.084710    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 08:27:08.401148 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:27:08.401099    4720 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5d887b9874jzx9t" podUID="83fff906-3b80-4fa2-9dcf-ebbb611580b6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8001/health\": dial tcp 10.132.0.69:8001: connect: connection refused"
Oct 31 08:27:08.507396 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:27:08.507347    4720 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6cgq8lv" podUID="b13794bf-1eb8-4f5c-9485-9f99ffc8ac37" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/health\": dial tcp 10.132.0.70:8000: connect: connection refused"
Oct 31 08:27:13.464089 ip-10-0-139-83 kubenswrapper[4720]: E1031 08:27:13.464050    4720 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8m768" containerName="extractor"
Oct 31 08:27:18.401122 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:27:18.401074    4720 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5d887b9874jzx9t" podUID="83fff906-3b80-4fa2-9dcf-ebbb611580b6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8001/health\": dial tcp 10.132.0.69:8001: connect: connection refused"
Oct 31 08:27:18.506649 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:27:18.506606    4720 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6cgq8lv" podUID="b13794bf-1eb8-4f5c-9485-9f99ffc8ac37" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/health\": dial tcp 10.132.0.70:8000: connect: connection refused"
Oct 31 08:27:20.025059 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:27:20.025024    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 08:27:28.400817 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:27:28.400761    4720 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5d887b9874jzx9t" podUID="83fff906-3b80-4fa2-9dcf-ebbb611580b6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8001/health\": dial tcp 10.132.0.69:8001: connect: connection refused"
Oct 31 08:27:28.506995 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:27:28.506954    4720 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6cgq8lv" podUID="b13794bf-1eb8-4f5c-9485-9f99ffc8ac37" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/health\": dial tcp 10.132.0.70:8000: connect: connection refused"
Oct 31 08:27:34.652498 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:27:34.652458    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 08:27:34.874284 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:27:34.874252    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 08:27:35.016138 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:27:35.016052    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 08:27:38.400585 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:27:38.400533    4720 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5d887b9874jzx9t" podUID="83fff906-3b80-4fa2-9dcf-ebbb611580b6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8001/health\": dial tcp 10.132.0.69:8001: connect: connection refused"
Oct 31 08:27:38.507318 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:27:38.507267    4720 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6cgq8lv" podUID="b13794bf-1eb8-4f5c-9485-9f99ffc8ac37" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/health\": dial tcp 10.132.0.70:8000: connect: connection refused"
Oct 31 08:27:43.463144 ip-10-0-139-83 kubenswrapper[4720]: E1031 08:27:43.463097    4720 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8m768" containerName="extractor"
Oct 31 08:27:48.415863 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:27:48.415831    4720 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5d887b9874jzx9t"
Oct 31 08:27:48.433083 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:27:48.433056    4720 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5d887b9874jzx9t"
Oct 31 08:27:48.516425 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:27:48.516384    4720 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6cgq8lv"
Oct 31 08:27:48.524222 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:27:48.524180    4720 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6cgq8lv"
Oct 31 08:27:48.858697 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:27:48.858658    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 08:27:49.032281 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:27:49.032246    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 08:27:49.180881 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:27:49.180792    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 08:27:56.580934 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:27:56.580900    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 08:27:56.868577 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:27:56.868533    4720 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6c5444664c-f7kzr"]
Oct 31 08:27:56.879498 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:27:56.879460    4720 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6c5444664c-f7kzr"
Oct 31 08:27:56.931913 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:27:56.931876    4720 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6c5444664c-f7kzr"]
Oct 31 08:27:56.976770 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:27:56.976728    4720 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"kube-root-ca.crt\""
Oct 31 08:27:56.977043 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:27:56.976872    4720 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"openshift-service-ca.crt\""
Oct 31 08:27:56.977043 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:27:56.976939    4720 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"stop-feature-test-kserve-self-signed-certs\""
Oct 31 08:27:56.977276 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:27:56.977084    4720 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"stop-feature-test-epp-sa-dockercfg-f8cgq\""
Oct 31 08:27:56.980471 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:27:56.980444    4720 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"default-dockercfg-fwsxf\""
Oct 31 08:27:56.980994 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:27:56.980966    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 08:27:56.993961 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:27:56.993911    4720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xt2tt\" (UniqueName: \"kubernetes.io/projected/dc0cab61-6a9c-4260-9187-8f1088a62125-kube-api-access-xt2tt\") pod \"stop-feature-test-kserve-router-scheduler-6c5444664c-f7kzr\" (UID: \"dc0cab61-6a9c-4260-9187-8f1088a62125\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6c5444664c-f7kzr"
Oct 31 08:27:56.994176 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:27:56.994056    4720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dc0cab61-6a9c-4260-9187-8f1088a62125-tls-certs\") pod \"stop-feature-test-kserve-router-scheduler-6c5444664c-f7kzr\" (UID: \"dc0cab61-6a9c-4260-9187-8f1088a62125\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6c5444664c-f7kzr"
Oct 31 08:27:57.094870 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:27:57.094781    4720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xt2tt\" (UniqueName: \"kubernetes.io/projected/dc0cab61-6a9c-4260-9187-8f1088a62125-kube-api-access-xt2tt\") pod \"stop-feature-test-kserve-router-scheduler-6c5444664c-f7kzr\" (UID: \"dc0cab61-6a9c-4260-9187-8f1088a62125\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6c5444664c-f7kzr"
Oct 31 08:27:57.095210 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:27:57.095184    4720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dc0cab61-6a9c-4260-9187-8f1088a62125-tls-certs\") pod \"stop-feature-test-kserve-router-scheduler-6c5444664c-f7kzr\" (UID: \"dc0cab61-6a9c-4260-9187-8f1088a62125\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6c5444664c-f7kzr"
Oct 31 08:27:57.099309 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:27:57.099268    4720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dc0cab61-6a9c-4260-9187-8f1088a62125-tls-certs\") pod \"stop-feature-test-kserve-router-scheduler-6c5444664c-f7kzr\" (UID: \"dc0cab61-6a9c-4260-9187-8f1088a62125\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6c5444664c-f7kzr"
Oct 31 08:27:57.151151 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:27:57.151079    4720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xt2tt\" (UniqueName: \"kubernetes.io/projected/dc0cab61-6a9c-4260-9187-8f1088a62125-kube-api-access-xt2tt\") pod \"stop-feature-test-kserve-router-scheduler-6c5444664c-f7kzr\" (UID: \"dc0cab61-6a9c-4260-9187-8f1088a62125\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6c5444664c-f7kzr"
Oct 31 08:27:57.202052 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:27:57.201997    4720 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6c5444664c-f7kzr"
Oct 31 08:27:57.580672 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:27:57.580644    4720 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6c5444664c-f7kzr"]
Oct 31 08:27:57.583024 ip-10-0-139-83 kubenswrapper[4720]: W1031 08:27:57.582985    4720 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc0cab61_6a9c_4260_9187_8f1088a62125.slice/crio-e4a80b0e4e64da52b5dd41d42f10ceb5327b0ea2ca275bc8978aebb54d171b93 WatchSource:0}: Error finding container e4a80b0e4e64da52b5dd41d42f10ceb5327b0ea2ca275bc8978aebb54d171b93: Status 404 returned error can't find the container with id e4a80b0e4e64da52b5dd41d42f10ceb5327b0ea2ca275bc8978aebb54d171b93
Oct 31 08:27:58.247962 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:27:58.247917    4720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6c5444664c-f7kzr" event={"ID":"dc0cab61-6a9c-4260-9187-8f1088a62125","Type":"ContainerStarted","Data":"32ac09934702bf077c8ad7a392615b363164a1cbae12a7018f890626b9e25808"}
Oct 31 08:27:58.248180 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:27:58.247981    4720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6c5444664c-f7kzr" event={"ID":"dc0cab61-6a9c-4260-9187-8f1088a62125","Type":"ContainerStarted","Data":"e4a80b0e4e64da52b5dd41d42f10ceb5327b0ea2ca275bc8978aebb54d171b93"}
Oct 31 08:27:58.248180 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:27:58.248027    4720 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6c5444664c-f7kzr"
Oct 31 08:27:58.362375 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:27:58.362307    4720 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6c5444664c-f7kzr" podStartSLOduration=2.362292264 podStartE2EDuration="2.362292264s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 08:27:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 08:27:58.359606064 +0000 UTC m=+3532.490674147" watchObservedRunningTime="2026-10-31 08:27:58.362292264 +0000 UTC m=+3532.493360362"
Oct 31 08:27:59.430360 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:27:59.430286    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 08:28:06.102119 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:28:06.102086    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 08:28:06.114276 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:28:06.114245    4720 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6cgq8lv"]
Oct 31 08:28:06.139366 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:28:06.139316    4720 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5d887b9874jzx9t"]
Oct 31 08:28:06.256071 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:28:06.256005    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 08:28:06.979014 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:28:06.978975    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 08:28:13.461149 ip-10-0-139-83 kubenswrapper[4720]: E1031 08:28:13.461108    4720 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8m768" containerName="extractor"
Oct 31 08:28:20.370496 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:28:20.370412    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 08:28:21.138358 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:28:21.138285    4720 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6cgq8lv" podUID="b13794bf-1eb8-4f5c-9485-9f99ffc8ac37" containerName="main" containerID="cri-o://ba15ea1f34c89ea3bf4d53f13577b383a23503091bef5266f57b206782ed6a37" gracePeriod=45
Oct 31 08:28:21.163248 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:28:21.163204    4720 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5d887b9874jzx9t" podUID="83fff906-3b80-4fa2-9dcf-ebbb611580b6" containerName="main" containerID="cri-o://1aafb77c9c4370df857002f234eb06092914bc2fb1793cf61063b4b2e994152c" gracePeriod=45
Oct 31 08:28:21.981285 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:28:21.981239    4720 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5d887b9874jzx9t" podUID="83fff906-3b80-4fa2-9dcf-ebbb611580b6" containerName="llm-d-routing-sidecar" containerID="cri-o://db15b0b0b44d8452c559f9e3712f88cf243305c77822a92a35b3550a625acb95" gracePeriod=45
Oct 31 08:28:22.113377 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:28:22.113342    4720 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6cgq8lv"
Oct 31 08:28:22.228577 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:28:22.228549    4720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b13794bf-1eb8-4f5c-9485-9f99ffc8ac37-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b13794bf-1eb8-4f5c-9485-9f99ffc8ac37-cabundle-cert\") pod \"b13794bf-1eb8-4f5c-9485-9f99ffc8ac37\" (UID: \"b13794bf-1eb8-4f5c-9485-9f99ffc8ac37\") "
Oct 31 08:28:22.228741 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:28:22.228582    4720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b13794bf-1eb8-4f5c-9485-9f99ffc8ac37-tls-certs\" (UniqueName: \"kubernetes.io/secret/b13794bf-1eb8-4f5c-9485-9f99ffc8ac37-tls-certs\") pod \"b13794bf-1eb8-4f5c-9485-9f99ffc8ac37\" (UID: \"b13794bf-1eb8-4f5c-9485-9f99ffc8ac37\") "
Oct 31 08:28:22.228741 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:28:22.228607    4720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b13794bf-1eb8-4f5c-9485-9f99ffc8ac37-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b13794bf-1eb8-4f5c-9485-9f99ffc8ac37-model-cache\") pod \"b13794bf-1eb8-4f5c-9485-9f99ffc8ac37\" (UID: \"b13794bf-1eb8-4f5c-9485-9f99ffc8ac37\") "
Oct 31 08:28:22.228741 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:28:22.228622    4720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b13794bf-1eb8-4f5c-9485-9f99ffc8ac37-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b13794bf-1eb8-4f5c-9485-9f99ffc8ac37-tmp-dir\") pod \"b13794bf-1eb8-4f5c-9485-9f99ffc8ac37\" (UID: \"b13794bf-1eb8-4f5c-9485-9f99ffc8ac37\") "
Oct 31 08:28:22.228923 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:28:22.228819    4720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b13794bf-1eb8-4f5c-9485-9f99ffc8ac37-kube-api-access-8p2m7\" (UniqueName: \"kubernetes.io/projected/b13794bf-1eb8-4f5c-9485-9f99ffc8ac37-kube-api-access-8p2m7\") pod \"b13794bf-1eb8-4f5c-9485-9f99ffc8ac37\" (UID: \"b13794bf-1eb8-4f5c-9485-9f99ffc8ac37\") "
Oct 31 08:28:22.228923 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:28:22.228890    4720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b13794bf-1eb8-4f5c-9485-9f99ffc8ac37-model-cache" pod "b13794bf-1eb8-4f5c-9485-9f99ffc8ac37" (UID: "b13794bf-1eb8-4f5c-9485-9f99ffc8ac37"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 08:28:22.228923 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:28:22.228906    4720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b13794bf-1eb8-4f5c-9485-9f99ffc8ac37-home\" (UniqueName: \"kubernetes.io/empty-dir/b13794bf-1eb8-4f5c-9485-9f99ffc8ac37-home\") pod \"b13794bf-1eb8-4f5c-9485-9f99ffc8ac37\" (UID: \"b13794bf-1eb8-4f5c-9485-9f99ffc8ac37\") "
Oct 31 08:28:22.229070 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:28:22.228969    4720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b13794bf-1eb8-4f5c-9485-9f99ffc8ac37-cabundle-cert" pod "b13794bf-1eb8-4f5c-9485-9f99ffc8ac37" (UID: "b13794bf-1eb8-4f5c-9485-9f99ffc8ac37"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 08:28:22.229070 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:28:22.228974    4720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b13794bf-1eb8-4f5c-9485-9f99ffc8ac37-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b13794bf-1eb8-4f5c-9485-9f99ffc8ac37-kserve-provision-location\") pod \"b13794bf-1eb8-4f5c-9485-9f99ffc8ac37\" (UID: \"b13794bf-1eb8-4f5c-9485-9f99ffc8ac37\") "
Oct 31 08:28:22.229168 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:28:22.229035    4720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b13794bf-1eb8-4f5c-9485-9f99ffc8ac37-dshm\" (UniqueName: \"kubernetes.io/empty-dir/b13794bf-1eb8-4f5c-9485-9f99ffc8ac37-dshm\") pod \"b13794bf-1eb8-4f5c-9485-9f99ffc8ac37\" (UID: \"b13794bf-1eb8-4f5c-9485-9f99ffc8ac37\") "
Oct 31 08:28:22.229377 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:28:22.229360    4720 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b13794bf-1eb8-4f5c-9485-9f99ffc8ac37-cabundle-cert\") on node \"ip-10-0-139-83.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 08:28:22.229377 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:28:22.229380    4720 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b13794bf-1eb8-4f5c-9485-9f99ffc8ac37-model-cache\") on node \"ip-10-0-139-83.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 08:28:22.229626 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:28:22.229609    4720 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5d887b9874jzx9t"
Oct 31 08:28:22.229793 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:28:22.229765    4720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b13794bf-1eb8-4f5c-9485-9f99ffc8ac37-home" pod "b13794bf-1eb8-4f5c-9485-9f99ffc8ac37" (UID: "b13794bf-1eb8-4f5c-9485-9f99ffc8ac37"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 08:28:22.231261 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:28:22.231235    4720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b13794bf-1eb8-4f5c-9485-9f99ffc8ac37-kube-api-access-8p2m7" pod "b13794bf-1eb8-4f5c-9485-9f99ffc8ac37" (UID: "b13794bf-1eb8-4f5c-9485-9f99ffc8ac37"). InnerVolumeSpecName "kube-api-access-8p2m7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 08:28:22.231495 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:28:22.231472    4720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b13794bf-1eb8-4f5c-9485-9f99ffc8ac37-tls-certs" pod "b13794bf-1eb8-4f5c-9485-9f99ffc8ac37" (UID: "b13794bf-1eb8-4f5c-9485-9f99ffc8ac37"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 08:28:22.231763 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:28:22.231748    4720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b13794bf-1eb8-4f5c-9485-9f99ffc8ac37-dshm" pod "b13794bf-1eb8-4f5c-9485-9f99ffc8ac37" (UID: "b13794bf-1eb8-4f5c-9485-9f99ffc8ac37"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 08:28:22.240926 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:28:22.240888    4720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b13794bf-1eb8-4f5c-9485-9f99ffc8ac37-tmp-dir" pod "b13794bf-1eb8-4f5c-9485-9f99ffc8ac37" (UID: "b13794bf-1eb8-4f5c-9485-9f99ffc8ac37"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 08:28:22.255920 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:28:22.255860    4720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b13794bf-1eb8-4f5c-9485-9f99ffc8ac37-kserve-provision-location" pod "b13794bf-1eb8-4f5c-9485-9f99ffc8ac37" (UID: "b13794bf-1eb8-4f5c-9485-9f99ffc8ac37"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 08:28:22.330142 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:28:22.330104    4720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/83fff906-3b80-4fa2-9dcf-ebbb611580b6-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/83fff906-3b80-4fa2-9dcf-ebbb611580b6-cabundle-cert\") pod \"83fff906-3b80-4fa2-9dcf-ebbb611580b6\" (UID: \"83fff906-3b80-4fa2-9dcf-ebbb611580b6\") "
Oct 31 08:28:22.330142 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:28:22.330148    4720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/83fff906-3b80-4fa2-9dcf-ebbb611580b6-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/83fff906-3b80-4fa2-9dcf-ebbb611580b6-tmp-dir\") pod \"83fff906-3b80-4fa2-9dcf-ebbb611580b6\" (UID: \"83fff906-3b80-4fa2-9dcf-ebbb611580b6\") "
Oct 31 08:28:22.330473 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:28:22.330187    4720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/83fff906-3b80-4fa2-9dcf-ebbb611580b6-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/83fff906-3b80-4fa2-9dcf-ebbb611580b6-kserve-provision-location\") pod \"83fff906-3b80-4fa2-9dcf-ebbb611580b6\" (UID: \"83fff906-3b80-4fa2-9dcf-ebbb611580b6\") "
Oct 31 08:28:22.330473 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:28:22.330215    4720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/83fff906-3b80-4fa2-9dcf-ebbb611580b6-dshm\" (UniqueName: \"kubernetes.io/empty-dir/83fff906-3b80-4fa2-9dcf-ebbb611580b6-dshm\") pod \"83fff906-3b80-4fa2-9dcf-ebbb611580b6\" (UID: \"83fff906-3b80-4fa2-9dcf-ebbb611580b6\") "
Oct 31 08:28:22.330473 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:28:22.330236    4720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/83fff906-3b80-4fa2-9dcf-ebbb611580b6-tls-certs\" (UniqueName: \"kubernetes.io/secret/83fff906-3b80-4fa2-9dcf-ebbb611580b6-tls-certs\") pod \"83fff906-3b80-4fa2-9dcf-ebbb611580b6\" (UID: \"83fff906-3b80-4fa2-9dcf-ebbb611580b6\") "
Oct 31 08:28:22.330473 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:28:22.330303    4720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/83fff906-3b80-4fa2-9dcf-ebbb611580b6-home\" (UniqueName: \"kubernetes.io/empty-dir/83fff906-3b80-4fa2-9dcf-ebbb611580b6-home\") pod \"83fff906-3b80-4fa2-9dcf-ebbb611580b6\" (UID: \"83fff906-3b80-4fa2-9dcf-ebbb611580b6\") "
Oct 31 08:28:22.330473 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:28:22.330362    4720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/83fff906-3b80-4fa2-9dcf-ebbb611580b6-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/83fff906-3b80-4fa2-9dcf-ebbb611580b6-model-cache\") pod \"83fff906-3b80-4fa2-9dcf-ebbb611580b6\" (UID: \"83fff906-3b80-4fa2-9dcf-ebbb611580b6\") "
Oct 31 08:28:22.330473 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:28:22.330403    4720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/83fff906-3b80-4fa2-9dcf-ebbb611580b6-kube-api-access-p4lq7\" (UniqueName: \"kubernetes.io/projected/83fff906-3b80-4fa2-9dcf-ebbb611580b6-kube-api-access-p4lq7\") pod \"83fff906-3b80-4fa2-9dcf-ebbb611580b6\" (UID: \"83fff906-3b80-4fa2-9dcf-ebbb611580b6\") "
Oct 31 08:28:22.330738 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:28:22.330670    4720 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b13794bf-1eb8-4f5c-9485-9f99ffc8ac37-tls-certs\") on node \"ip-10-0-139-83.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 08:28:22.330738 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:28:22.330688    4720 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b13794bf-1eb8-4f5c-9485-9f99ffc8ac37-tmp-dir\") on node \"ip-10-0-139-83.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 08:28:22.330738 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:28:22.330700    4720 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8p2m7\" (UniqueName: \"kubernetes.io/projected/b13794bf-1eb8-4f5c-9485-9f99ffc8ac37-kube-api-access-8p2m7\") on node \"ip-10-0-139-83.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 08:28:22.330738 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:28:22.330712    4720 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b13794bf-1eb8-4f5c-9485-9f99ffc8ac37-home\") on node \"ip-10-0-139-83.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 08:28:22.330738 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:28:22.330724    4720 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b13794bf-1eb8-4f5c-9485-9f99ffc8ac37-kserve-provision-location\") on node \"ip-10-0-139-83.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 08:28:22.330738 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:28:22.330736    4720 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b13794bf-1eb8-4f5c-9485-9f99ffc8ac37-dshm\") on node \"ip-10-0-139-83.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 08:28:22.331589 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:28:22.331180    4720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/83fff906-3b80-4fa2-9dcf-ebbb611580b6-cabundle-cert" pod "83fff906-3b80-4fa2-9dcf-ebbb611580b6" (UID: "83fff906-3b80-4fa2-9dcf-ebbb611580b6"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 08:28:22.331589 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:28:22.331550    4720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/83fff906-3b80-4fa2-9dcf-ebbb611580b6-model-cache" pod "83fff906-3b80-4fa2-9dcf-ebbb611580b6" (UID: "83fff906-3b80-4fa2-9dcf-ebbb611580b6"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 08:28:22.332278 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:28:22.332166    4720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/83fff906-3b80-4fa2-9dcf-ebbb611580b6-home" pod "83fff906-3b80-4fa2-9dcf-ebbb611580b6" (UID: "83fff906-3b80-4fa2-9dcf-ebbb611580b6"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 08:28:22.333681 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:28:22.333652    4720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/83fff906-3b80-4fa2-9dcf-ebbb611580b6-dshm" pod "83fff906-3b80-4fa2-9dcf-ebbb611580b6" (UID: "83fff906-3b80-4fa2-9dcf-ebbb611580b6"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 08:28:22.333887 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:28:22.333866    4720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/83fff906-3b80-4fa2-9dcf-ebbb611580b6-tls-certs" pod "83fff906-3b80-4fa2-9dcf-ebbb611580b6" (UID: "83fff906-3b80-4fa2-9dcf-ebbb611580b6"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 08:28:22.335416 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:28:22.335388    4720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/83fff906-3b80-4fa2-9dcf-ebbb611580b6-kube-api-access-p4lq7" pod "83fff906-3b80-4fa2-9dcf-ebbb611580b6" (UID: "83fff906-3b80-4fa2-9dcf-ebbb611580b6"). InnerVolumeSpecName "kube-api-access-p4lq7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 08:28:22.344273 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:28:22.344240    4720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/83fff906-3b80-4fa2-9dcf-ebbb611580b6-tmp-dir" pod "83fff906-3b80-4fa2-9dcf-ebbb611580b6" (UID: "83fff906-3b80-4fa2-9dcf-ebbb611580b6"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 08:28:22.356483 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:28:22.356431    4720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/83fff906-3b80-4fa2-9dcf-ebbb611580b6-kserve-provision-location" pod "83fff906-3b80-4fa2-9dcf-ebbb611580b6" (UID: "83fff906-3b80-4fa2-9dcf-ebbb611580b6"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 08:28:22.364188 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:28:22.364149    4720 generic.go:356] "Generic (PLEG): container finished" podID="b13794bf-1eb8-4f5c-9485-9f99ffc8ac37" containerID="ba15ea1f34c89ea3bf4d53f13577b383a23503091bef5266f57b206782ed6a37" exitCode=0
Oct 31 08:28:22.364378 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:28:22.364245    4720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6cgq8lv" event={"ID":"b13794bf-1eb8-4f5c-9485-9f99ffc8ac37","Type":"ContainerDied","Data":"ba15ea1f34c89ea3bf4d53f13577b383a23503091bef5266f57b206782ed6a37"}
Oct 31 08:28:22.364378 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:28:22.364264    4720 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6cgq8lv"
Oct 31 08:28:22.364378 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:28:22.364306    4720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6cgq8lv" event={"ID":"b13794bf-1eb8-4f5c-9485-9f99ffc8ac37","Type":"ContainerDied","Data":"1cea409b78c42a61ebe83a241e09778b4e09b42453ec7dce26b567b1b2ab3a3c"}
Oct 31 08:28:22.364378 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:28:22.364370    4720 scope.go:122] "RemoveContainer" containerID="ba15ea1f34c89ea3bf4d53f13577b383a23503091bef5266f57b206782ed6a37"
Oct 31 08:28:22.366560 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:28:22.366529    4720 generic.go:356] "Generic (PLEG): container finished" podID="83fff906-3b80-4fa2-9dcf-ebbb611580b6" containerID="1aafb77c9c4370df857002f234eb06092914bc2fb1793cf61063b4b2e994152c" exitCode=0
Oct 31 08:28:22.366560 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:28:22.366558    4720 generic.go:356] "Generic (PLEG): container finished" podID="83fff906-3b80-4fa2-9dcf-ebbb611580b6" containerID="db15b0b0b44d8452c559f9e3712f88cf243305c77822a92a35b3550a625acb95" exitCode=0
Oct 31 08:28:22.366741 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:28:22.366588    4720 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5d887b9874jzx9t"
Oct 31 08:28:22.366741 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:28:22.366624    4720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5d887b9874jzx9t" event={"ID":"83fff906-3b80-4fa2-9dcf-ebbb611580b6","Type":"ContainerDied","Data":"1aafb77c9c4370df857002f234eb06092914bc2fb1793cf61063b4b2e994152c"}
Oct 31 08:28:22.366741 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:28:22.366691    4720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5d887b9874jzx9t" event={"ID":"83fff906-3b80-4fa2-9dcf-ebbb611580b6","Type":"ContainerDied","Data":"db15b0b0b44d8452c559f9e3712f88cf243305c77822a92a35b3550a625acb95"}
Oct 31 08:28:22.366741 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:28:22.366710    4720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5d887b9874jzx9t" event={"ID":"83fff906-3b80-4fa2-9dcf-ebbb611580b6","Type":"ContainerDied","Data":"057593ad2e037c9c5b2f6bf9dd88eee8178afe13d886aec1a93276d19d665a7d"}
Oct 31 08:28:22.374865 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:28:22.374843    4720 scope.go:122] "RemoveContainer" containerID="bd2c662bc34e197472d80eb6f22c51f1dd3973461196a5f7b6c4d9f26ea4c362"
Oct 31 08:28:22.383402 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:28:22.383380    4720 scope.go:122] "RemoveContainer" containerID="ba15ea1f34c89ea3bf4d53f13577b383a23503091bef5266f57b206782ed6a37"
Oct 31 08:28:22.383707 ip-10-0-139-83 kubenswrapper[4720]: E1031 08:28:22.383688    4720 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ba15ea1f34c89ea3bf4d53f13577b383a23503091bef5266f57b206782ed6a37\": container with ID starting with ba15ea1f34c89ea3bf4d53f13577b383a23503091bef5266f57b206782ed6a37 not found: ID does not exist" containerID="ba15ea1f34c89ea3bf4d53f13577b383a23503091bef5266f57b206782ed6a37"
Oct 31 08:28:22.383809 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:28:22.383717    4720 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ba15ea1f34c89ea3bf4d53f13577b383a23503091bef5266f57b206782ed6a37"} err="failed to get container status \"ba15ea1f34c89ea3bf4d53f13577b383a23503091bef5266f57b206782ed6a37\": rpc error: code = NotFound desc = could not find container \"ba15ea1f34c89ea3bf4d53f13577b383a23503091bef5266f57b206782ed6a37\": container with ID starting with ba15ea1f34c89ea3bf4d53f13577b383a23503091bef5266f57b206782ed6a37 not found: ID does not exist"
Oct 31 08:28:22.383809 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:28:22.383735    4720 scope.go:122] "RemoveContainer" containerID="bd2c662bc34e197472d80eb6f22c51f1dd3973461196a5f7b6c4d9f26ea4c362"
Oct 31 08:28:22.384042 ip-10-0-139-83 kubenswrapper[4720]: E1031 08:28:22.384023    4720 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bd2c662bc34e197472d80eb6f22c51f1dd3973461196a5f7b6c4d9f26ea4c362\": container with ID starting with bd2c662bc34e197472d80eb6f22c51f1dd3973461196a5f7b6c4d9f26ea4c362 not found: ID does not exist" containerID="bd2c662bc34e197472d80eb6f22c51f1dd3973461196a5f7b6c4d9f26ea4c362"
Oct 31 08:28:22.384092 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:28:22.384047    4720 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bd2c662bc34e197472d80eb6f22c51f1dd3973461196a5f7b6c4d9f26ea4c362"} err="failed to get container status \"bd2c662bc34e197472d80eb6f22c51f1dd3973461196a5f7b6c4d9f26ea4c362\": rpc error: code = NotFound desc = could not find container \"bd2c662bc34e197472d80eb6f22c51f1dd3973461196a5f7b6c4d9f26ea4c362\": container with ID starting with bd2c662bc34e197472d80eb6f22c51f1dd3973461196a5f7b6c4d9f26ea4c362 not found: ID does not exist"
Oct 31 08:28:22.384092 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:28:22.384062    4720 scope.go:122] "RemoveContainer" containerID="1aafb77c9c4370df857002f234eb06092914bc2fb1793cf61063b4b2e994152c"
Oct 31 08:28:22.392652 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:28:22.392619    4720 scope.go:122] "RemoveContainer" containerID="9fa841e663068d80e8fe23765e01a51c5d675306e9cd270706846a9dc7a81455"
Oct 31 08:28:22.402724 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:28:22.402674    4720 scope.go:122] "RemoveContainer" containerID="db15b0b0b44d8452c559f9e3712f88cf243305c77822a92a35b3550a625acb95"
Oct 31 08:28:22.411610 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:28:22.411589    4720 scope.go:122] "RemoveContainer" containerID="1aafb77c9c4370df857002f234eb06092914bc2fb1793cf61063b4b2e994152c"
Oct 31 08:28:22.412006 ip-10-0-139-83 kubenswrapper[4720]: E1031 08:28:22.411980    4720 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1aafb77c9c4370df857002f234eb06092914bc2fb1793cf61063b4b2e994152c\": container with ID starting with 1aafb77c9c4370df857002f234eb06092914bc2fb1793cf61063b4b2e994152c not found: ID does not exist" containerID="1aafb77c9c4370df857002f234eb06092914bc2fb1793cf61063b4b2e994152c"
Oct 31 08:28:22.412069 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:28:22.412014    4720 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1aafb77c9c4370df857002f234eb06092914bc2fb1793cf61063b4b2e994152c"} err="failed to get container status \"1aafb77c9c4370df857002f234eb06092914bc2fb1793cf61063b4b2e994152c\": rpc error: code = NotFound desc = could not find container \"1aafb77c9c4370df857002f234eb06092914bc2fb1793cf61063b4b2e994152c\": container with ID starting with 1aafb77c9c4370df857002f234eb06092914bc2fb1793cf61063b4b2e994152c not found: ID does not exist"
Oct 31 08:28:22.412069 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:28:22.412034    4720 scope.go:122] "RemoveContainer" containerID="9fa841e663068d80e8fe23765e01a51c5d675306e9cd270706846a9dc7a81455"
Oct 31 08:28:22.412291 ip-10-0-139-83 kubenswrapper[4720]: E1031 08:28:22.412275    4720 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9fa841e663068d80e8fe23765e01a51c5d675306e9cd270706846a9dc7a81455\": container with ID starting with 9fa841e663068d80e8fe23765e01a51c5d675306e9cd270706846a9dc7a81455 not found: ID does not exist" containerID="9fa841e663068d80e8fe23765e01a51c5d675306e9cd270706846a9dc7a81455"
Oct 31 08:28:22.412353 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:28:22.412294    4720 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9fa841e663068d80e8fe23765e01a51c5d675306e9cd270706846a9dc7a81455"} err="failed to get container status \"9fa841e663068d80e8fe23765e01a51c5d675306e9cd270706846a9dc7a81455\": rpc error: code = NotFound desc = could not find container \"9fa841e663068d80e8fe23765e01a51c5d675306e9cd270706846a9dc7a81455\": container with ID starting with 9fa841e663068d80e8fe23765e01a51c5d675306e9cd270706846a9dc7a81455 not found: ID does not exist"
Oct 31 08:28:22.412353 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:28:22.412306    4720 scope.go:122] "RemoveContainer" containerID="db15b0b0b44d8452c559f9e3712f88cf243305c77822a92a35b3550a625acb95"
Oct 31 08:28:22.412598 ip-10-0-139-83 kubenswrapper[4720]: E1031 08:28:22.412578    4720 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"db15b0b0b44d8452c559f9e3712f88cf243305c77822a92a35b3550a625acb95\": container with ID starting with db15b0b0b44d8452c559f9e3712f88cf243305c77822a92a35b3550a625acb95 not found: ID does not exist" containerID="db15b0b0b44d8452c559f9e3712f88cf243305c77822a92a35b3550a625acb95"
Oct 31 08:28:22.412666 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:28:22.412602    4720 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"db15b0b0b44d8452c559f9e3712f88cf243305c77822a92a35b3550a625acb95"} err="failed to get container status \"db15b0b0b44d8452c559f9e3712f88cf243305c77822a92a35b3550a625acb95\": rpc error: code = NotFound desc = could not find container \"db15b0b0b44d8452c559f9e3712f88cf243305c77822a92a35b3550a625acb95\": container with ID starting with db15b0b0b44d8452c559f9e3712f88cf243305c77822a92a35b3550a625acb95 not found: ID does not exist"
Oct 31 08:28:22.412666 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:28:22.412615    4720 scope.go:122] "RemoveContainer" containerID="1aafb77c9c4370df857002f234eb06092914bc2fb1793cf61063b4b2e994152c"
Oct 31 08:28:22.412888 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:28:22.412867    4720 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1aafb77c9c4370df857002f234eb06092914bc2fb1793cf61063b4b2e994152c"} err="failed to get container status \"1aafb77c9c4370df857002f234eb06092914bc2fb1793cf61063b4b2e994152c\": rpc error: code = NotFound desc = could not find container \"1aafb77c9c4370df857002f234eb06092914bc2fb1793cf61063b4b2e994152c\": container with ID starting with 1aafb77c9c4370df857002f234eb06092914bc2fb1793cf61063b4b2e994152c not found: ID does not exist"
Oct 31 08:28:22.412934 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:28:22.412889    4720 scope.go:122] "RemoveContainer" containerID="9fa841e663068d80e8fe23765e01a51c5d675306e9cd270706846a9dc7a81455"
Oct 31 08:28:22.413120 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:28:22.413106    4720 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9fa841e663068d80e8fe23765e01a51c5d675306e9cd270706846a9dc7a81455"} err="failed to get container status \"9fa841e663068d80e8fe23765e01a51c5d675306e9cd270706846a9dc7a81455\": rpc error: code = NotFound desc = could not find container \"9fa841e663068d80e8fe23765e01a51c5d675306e9cd270706846a9dc7a81455\": container with ID starting with 9fa841e663068d80e8fe23765e01a51c5d675306e9cd270706846a9dc7a81455 not found: ID does not exist"
Oct 31 08:28:22.413165 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:28:22.413120    4720 scope.go:122] "RemoveContainer" containerID="db15b0b0b44d8452c559f9e3712f88cf243305c77822a92a35b3550a625acb95"
Oct 31 08:28:22.413390 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:28:22.413318    4720 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"db15b0b0b44d8452c559f9e3712f88cf243305c77822a92a35b3550a625acb95"} err="failed to get container status \"db15b0b0b44d8452c559f9e3712f88cf243305c77822a92a35b3550a625acb95\": rpc error: code = NotFound desc = could not find container \"db15b0b0b44d8452c559f9e3712f88cf243305c77822a92a35b3550a625acb95\": container with ID starting with db15b0b0b44d8452c559f9e3712f88cf243305c77822a92a35b3550a625acb95 not found: ID does not exist"
Oct 31 08:28:22.431648 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:28:22.431611    4720 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/83fff906-3b80-4fa2-9dcf-ebbb611580b6-home\") on node \"ip-10-0-139-83.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 08:28:22.431648 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:28:22.431642    4720 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/83fff906-3b80-4fa2-9dcf-ebbb611580b6-model-cache\") on node \"ip-10-0-139-83.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 08:28:22.431648 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:28:22.431654    4720 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p4lq7\" (UniqueName: \"kubernetes.io/projected/83fff906-3b80-4fa2-9dcf-ebbb611580b6-kube-api-access-p4lq7\") on node \"ip-10-0-139-83.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 08:28:22.431843 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:28:22.431663    4720 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/83fff906-3b80-4fa2-9dcf-ebbb611580b6-cabundle-cert\") on node \"ip-10-0-139-83.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 08:28:22.431843 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:28:22.431672    4720 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/83fff906-3b80-4fa2-9dcf-ebbb611580b6-tmp-dir\") on node \"ip-10-0-139-83.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 08:28:22.431843 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:28:22.431682    4720 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/83fff906-3b80-4fa2-9dcf-ebbb611580b6-kserve-provision-location\") on node \"ip-10-0-139-83.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 08:28:22.431843 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:28:22.431690    4720 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/83fff906-3b80-4fa2-9dcf-ebbb611580b6-dshm\") on node \"ip-10-0-139-83.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 08:28:22.431843 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:28:22.431698    4720 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/83fff906-3b80-4fa2-9dcf-ebbb611580b6-tls-certs\") on node \"ip-10-0-139-83.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 08:28:22.511529 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:28:22.511494    4720 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5d887b9874jzx9t"]
Oct 31 08:28:22.514199 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:28:22.514172    4720 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5d887b9874jzx9t"]
Oct 31 08:28:22.622957 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:28:22.622914    4720 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6cgq8lv"]
Oct 31 08:28:22.627183 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:28:22.627154    4720 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6cgq8lv"]
Oct 31 08:28:24.025880 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:28:24.025835    4720 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="83fff906-3b80-4fa2-9dcf-ebbb611580b6" path="/var/lib/kubelet/pods/83fff906-3b80-4fa2-9dcf-ebbb611580b6/volumes"
Oct 31 08:28:24.026620 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:28:24.026599    4720 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b13794bf-1eb8-4f5c-9485-9f99ffc8ac37" path="/var/lib/kubelet/pods/b13794bf-1eb8-4f5c-9485-9f99ffc8ac37/volumes"
Oct 31 08:28:29.255031 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:28:29.254993    4720 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6c5444664c-f7kzr"
Oct 31 08:28:34.991424 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:28:34.991388    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 08:28:43.461957 ip-10-0-139-83 kubenswrapper[4720]: E1031 08:28:43.461916    4720 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8m768" containerName="extractor"
Oct 31 08:28:57.069824 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:28:57.069791    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 08:28:57.238227 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:28:57.238188    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 08:28:59.766930 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:28:59.766895    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 08:29:03.317263 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:29:03.317209    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 08:29:06.362493 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:29:06.362371    4720 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-4jszp_b1db0e3b-e85d-4769-8012-cf4b6f740d9d/console-operator/2.log"
Oct 31 08:29:06.370981 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:29:06.370957    4720 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-4jszp_b1db0e3b-e85d-4769-8012-cf4b6f740d9d/console-operator/2.log"
Oct 31 08:29:08.951682 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:29:08.951637    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 08:29:09.122049 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:29:09.122012    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 08:29:13.462010 ip-10-0-139-83 kubenswrapper[4720]: E1031 08:29:13.461976    4720 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8m768" containerName="extractor"
Oct 31 08:29:14.750803 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:29:14.750764    4720 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-7c457bbbbc-4gw8x"]
Oct 31 08:29:14.751207 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:29:14.751111    4720 cpu_manager.go:414] "containerMap: removing container" podUID="83fff906-3b80-4fa2-9dcf-ebbb611580b6" containerName="llm-d-routing-sidecar"
Oct 31 08:29:14.751207 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:29:14.751121    4720 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="83fff906-3b80-4fa2-9dcf-ebbb611580b6" containerName="llm-d-routing-sidecar"
Oct 31 08:29:14.751207 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:29:14.751137    4720 cpu_manager.go:414] "containerMap: removing container" podUID="b13794bf-1eb8-4f5c-9485-9f99ffc8ac37" containerName="main"
Oct 31 08:29:14.751207 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:29:14.751142    4720 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b13794bf-1eb8-4f5c-9485-9f99ffc8ac37" containerName="main"
Oct 31 08:29:14.751207 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:29:14.751152    4720 cpu_manager.go:414] "containerMap: removing container" podUID="83fff906-3b80-4fa2-9dcf-ebbb611580b6" containerName="storage-initializer"
Oct 31 08:29:14.751207 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:29:14.751158    4720 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="83fff906-3b80-4fa2-9dcf-ebbb611580b6" containerName="storage-initializer"
Oct 31 08:29:14.751207 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:29:14.751170    4720 cpu_manager.go:414] "containerMap: removing container" podUID="83fff906-3b80-4fa2-9dcf-ebbb611580b6" containerName="main"
Oct 31 08:29:14.751207 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:29:14.751175    4720 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="83fff906-3b80-4fa2-9dcf-ebbb611580b6" containerName="main"
Oct 31 08:29:14.751207 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:29:14.751181    4720 cpu_manager.go:414] "containerMap: removing container" podUID="b13794bf-1eb8-4f5c-9485-9f99ffc8ac37" containerName="storage-initializer"
Oct 31 08:29:14.751207 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:29:14.751186    4720 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b13794bf-1eb8-4f5c-9485-9f99ffc8ac37" containerName="storage-initializer"
Oct 31 08:29:14.751535 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:29:14.751239    4720 memory_manager.go:365] "RemoveStaleState removing state" podUID="83fff906-3b80-4fa2-9dcf-ebbb611580b6" containerName="main"
Oct 31 08:29:14.751535 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:29:14.751249    4720 memory_manager.go:365] "RemoveStaleState removing state" podUID="83fff906-3b80-4fa2-9dcf-ebbb611580b6" containerName="llm-d-routing-sidecar"
Oct 31 08:29:14.751535 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:29:14.751257    4720 memory_manager.go:365] "RemoveStaleState removing state" podUID="b13794bf-1eb8-4f5c-9485-9f99ffc8ac37" containerName="main"
Oct 31 08:29:14.754459 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:29:14.754440    4720 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-7c457bbbbc-4gw8x"
Oct 31 08:29:14.786769 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:29:14.786722    4720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/45202e5f-a0de-472d-9673-13fd99f9878b-cabundle-cert\") pod \"custom-route-timeout-pd-test-kserve-7c457bbbbc-4gw8x\" (UID: \"45202e5f-a0de-472d-9673-13fd99f9878b\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-7c457bbbbc-4gw8x"
Oct 31 08:29:14.786950 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:29:14.786792    4720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/45202e5f-a0de-472d-9673-13fd99f9878b-home\") pod \"custom-route-timeout-pd-test-kserve-7c457bbbbc-4gw8x\" (UID: \"45202e5f-a0de-472d-9673-13fd99f9878b\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-7c457bbbbc-4gw8x"
Oct 31 08:29:14.786950 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:29:14.786858    4720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/45202e5f-a0de-472d-9673-13fd99f9878b-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-7c457bbbbc-4gw8x\" (UID: \"45202e5f-a0de-472d-9673-13fd99f9878b\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-7c457bbbbc-4gw8x"
Oct 31 08:29:14.786950 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:29:14.786885    4720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/45202e5f-a0de-472d-9673-13fd99f9878b-kserve-provision-location\") pod \"custom-route-timeout-pd-test-kserve-7c457bbbbc-4gw8x\" (UID: \"45202e5f-a0de-472d-9673-13fd99f9878b\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-7c457bbbbc-4gw8x"
Oct 31 08:29:14.786950 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:29:14.786902    4720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vpwdz\" (UniqueName: \"kubernetes.io/projected/45202e5f-a0de-472d-9673-13fd99f9878b-kube-api-access-vpwdz\") pod \"custom-route-timeout-pd-test-kserve-7c457bbbbc-4gw8x\" (UID: \"45202e5f-a0de-472d-9673-13fd99f9878b\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-7c457bbbbc-4gw8x"
Oct 31 08:29:14.786950 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:29:14.786923    4720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/45202e5f-a0de-472d-9673-13fd99f9878b-tmp-dir\") pod \"custom-route-timeout-pd-test-kserve-7c457bbbbc-4gw8x\" (UID: \"45202e5f-a0de-472d-9673-13fd99f9878b\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-7c457bbbbc-4gw8x"
Oct 31 08:29:14.787147 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:29:14.787004    4720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/45202e5f-a0de-472d-9673-13fd99f9878b-dshm\") pod \"custom-route-timeout-pd-test-kserve-7c457bbbbc-4gw8x\" (UID: \"45202e5f-a0de-472d-9673-13fd99f9878b\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-7c457bbbbc-4gw8x"
Oct 31 08:29:14.787147 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:29:14.787027    4720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/45202e5f-a0de-472d-9673-13fd99f9878b-model-cache\") pod \"custom-route-timeout-pd-test-kserve-7c457bbbbc-4gw8x\" (UID: \"45202e5f-a0de-472d-9673-13fd99f9878b\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-7c457bbbbc-4gw8x"
Oct 31 08:29:14.806775 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:29:14.806740    4720 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"odh-kserve-custom-ca-bundle\""
Oct 31 08:29:14.806775 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:29:14.806769    4720 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"seaweedfs-s3-creds\""
Oct 31 08:29:14.806775 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:29:14.806799    4720 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"kube-root-ca.crt\""
Oct 31 08:29:14.807055 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:29:14.806851    4720 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"openshift-service-ca.crt\""
Oct 31 08:29:14.856108 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:29:14.856073    4720 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"custom-route-timeout-pd-test-kserve-self-signed-certs\""
Oct 31 08:29:14.856108 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:29:14.856093    4720 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"custom-route-timeout-pd-test-kserve-dockercfg-ms8m7\""
Oct 31 08:29:14.856108 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:29:14.856121    4720 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"default-dockercfg-zpxzg\""
Oct 31 08:29:14.861077 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:29:14.861045    4720 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-7c457bbbbc-4gw8x"]
Oct 31 08:29:14.887499 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:29:14.887449    4720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/45202e5f-a0de-472d-9673-13fd99f9878b-cabundle-cert\") pod \"custom-route-timeout-pd-test-kserve-7c457bbbbc-4gw8x\" (UID: \"45202e5f-a0de-472d-9673-13fd99f9878b\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-7c457bbbbc-4gw8x"
Oct 31 08:29:14.887701 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:29:14.887541    4720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/45202e5f-a0de-472d-9673-13fd99f9878b-home\") pod \"custom-route-timeout-pd-test-kserve-7c457bbbbc-4gw8x\" (UID: \"45202e5f-a0de-472d-9673-13fd99f9878b\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-7c457bbbbc-4gw8x"
Oct 31 08:29:14.887701 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:29:14.887566    4720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/45202e5f-a0de-472d-9673-13fd99f9878b-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-7c457bbbbc-4gw8x\" (UID: \"45202e5f-a0de-472d-9673-13fd99f9878b\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-7c457bbbbc-4gw8x"
Oct 31 08:29:14.887701 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:29:14.887583    4720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/45202e5f-a0de-472d-9673-13fd99f9878b-kserve-provision-location\") pod \"custom-route-timeout-pd-test-kserve-7c457bbbbc-4gw8x\" (UID: \"45202e5f-a0de-472d-9673-13fd99f9878b\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-7c457bbbbc-4gw8x"
Oct 31 08:29:14.887701 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:29:14.887598    4720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vpwdz\" (UniqueName: \"kubernetes.io/projected/45202e5f-a0de-472d-9673-13fd99f9878b-kube-api-access-vpwdz\") pod \"custom-route-timeout-pd-test-kserve-7c457bbbbc-4gw8x\" (UID: \"45202e5f-a0de-472d-9673-13fd99f9878b\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-7c457bbbbc-4gw8x"
Oct 31 08:29:14.887701 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:29:14.887615    4720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/45202e5f-a0de-472d-9673-13fd99f9878b-tmp-dir\") pod \"custom-route-timeout-pd-test-kserve-7c457bbbbc-4gw8x\" (UID: \"45202e5f-a0de-472d-9673-13fd99f9878b\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-7c457bbbbc-4gw8x"
Oct 31 08:29:14.887701 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:29:14.887636    4720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/45202e5f-a0de-472d-9673-13fd99f9878b-dshm\") pod \"custom-route-timeout-pd-test-kserve-7c457bbbbc-4gw8x\" (UID: \"45202e5f-a0de-472d-9673-13fd99f9878b\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-7c457bbbbc-4gw8x"
Oct 31 08:29:14.887701 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:29:14.887652    4720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/45202e5f-a0de-472d-9673-13fd99f9878b-model-cache\") pod \"custom-route-timeout-pd-test-kserve-7c457bbbbc-4gw8x\" (UID: \"45202e5f-a0de-472d-9673-13fd99f9878b\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-7c457bbbbc-4gw8x"
Oct 31 08:29:14.888136 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:29:14.888098    4720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/45202e5f-a0de-472d-9673-13fd99f9878b-home\") pod \"custom-route-timeout-pd-test-kserve-7c457bbbbc-4gw8x\" (UID: \"45202e5f-a0de-472d-9673-13fd99f9878b\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-7c457bbbbc-4gw8x"
Oct 31 08:29:14.888268 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:29:14.888167    4720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/45202e5f-a0de-472d-9673-13fd99f9878b-kserve-provision-location\") pod \"custom-route-timeout-pd-test-kserve-7c457bbbbc-4gw8x\" (UID: \"45202e5f-a0de-472d-9673-13fd99f9878b\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-7c457bbbbc-4gw8x"
Oct 31 08:29:14.888268 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:29:14.888215    4720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/45202e5f-a0de-472d-9673-13fd99f9878b-tmp-dir\") pod \"custom-route-timeout-pd-test-kserve-7c457bbbbc-4gw8x\" (UID: \"45202e5f-a0de-472d-9673-13fd99f9878b\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-7c457bbbbc-4gw8x"
Oct 31 08:29:14.888268 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:29:14.888231    4720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/45202e5f-a0de-472d-9673-13fd99f9878b-model-cache\") pod \"custom-route-timeout-pd-test-kserve-7c457bbbbc-4gw8x\" (UID: \"45202e5f-a0de-472d-9673-13fd99f9878b\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-7c457bbbbc-4gw8x"
Oct 31 08:29:14.888643 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:29:14.888314    4720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/45202e5f-a0de-472d-9673-13fd99f9878b-cabundle-cert\") pod \"custom-route-timeout-pd-test-kserve-7c457bbbbc-4gw8x\" (UID: \"45202e5f-a0de-472d-9673-13fd99f9878b\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-7c457bbbbc-4gw8x"
Oct 31 08:29:14.890135 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:29:14.890107    4720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/45202e5f-a0de-472d-9673-13fd99f9878b-dshm\") pod \"custom-route-timeout-pd-test-kserve-7c457bbbbc-4gw8x\" (UID: \"45202e5f-a0de-472d-9673-13fd99f9878b\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-7c457bbbbc-4gw8x"
Oct 31 08:29:14.890411 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:29:14.890392    4720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/45202e5f-a0de-472d-9673-13fd99f9878b-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-7c457bbbbc-4gw8x\" (UID: \"45202e5f-a0de-472d-9673-13fd99f9878b\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-7c457bbbbc-4gw8x"
Oct 31 08:29:14.947627 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:29:14.947596    4720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vpwdz\" (UniqueName: \"kubernetes.io/projected/45202e5f-a0de-472d-9673-13fd99f9878b-kube-api-access-vpwdz\") pod \"custom-route-timeout-pd-test-kserve-7c457bbbbc-4gw8x\" (UID: \"45202e5f-a0de-472d-9673-13fd99f9878b\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-7c457bbbbc-4gw8x"
Oct 31 08:29:15.037694 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:29:15.037601    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 08:29:15.071146 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:29:15.071111    4720 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-7c457bbbbc-4gw8x"
Oct 31 08:29:15.494472 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:29:15.494432    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 08:29:15.509572 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:29:15.509547    4720 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-7c457bbbbc-4gw8x"]
Oct 31 08:29:15.511371 ip-10-0-139-83 kubenswrapper[4720]: W1031 08:29:15.511317    4720 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod45202e5f_a0de_472d_9673_13fd99f9878b.slice/crio-e5f151fc075f7f3c0af8e805947f3bcaf1b3f7503539ff49b9c08b6da0fdf674 WatchSource:0}: Error finding container e5f151fc075f7f3c0af8e805947f3bcaf1b3f7503539ff49b9c08b6da0fdf674: Status 404 returned error can't find the container with id e5f151fc075f7f3c0af8e805947f3bcaf1b3f7503539ff49b9c08b6da0fdf674
Oct 31 08:29:15.615341 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:29:15.615286    4720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-7c457bbbbc-4gw8x" event={"ID":"45202e5f-a0de-472d-9673-13fd99f9878b","Type":"ContainerStarted","Data":"b61c1755f326b57ee291827a1744f7864b9175a6ba9e430aed7948b2c32ee9ed"}
Oct 31 08:29:15.615504 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:29:15.615382    4720 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-7c457bbbbc-4gw8x"
Oct 31 08:29:15.615504 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:29:15.615404    4720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-7c457bbbbc-4gw8x" event={"ID":"45202e5f-a0de-472d-9673-13fd99f9878b","Type":"ContainerStarted","Data":"e5f151fc075f7f3c0af8e805947f3bcaf1b3f7503539ff49b9c08b6da0fdf674"}
Oct 31 08:29:16.622629 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:29:16.622592    4720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-7c457bbbbc-4gw8x" event={"ID":"45202e5f-a0de-472d-9673-13fd99f9878b","Type":"ContainerStarted","Data":"fca476b048b15528f13911b753523f7d201381ee1365e63f94780a340ca1509e"}
Oct 31 08:29:18.635020 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:29:18.634985    4720 generic.go:356] "Generic (PLEG): container finished" podID="45202e5f-a0de-472d-9673-13fd99f9878b" containerID="fca476b048b15528f13911b753523f7d201381ee1365e63f94780a340ca1509e" exitCode=0
Oct 31 08:29:18.635451 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:29:18.635058    4720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-7c457bbbbc-4gw8x" event={"ID":"45202e5f-a0de-472d-9673-13fd99f9878b","Type":"ContainerDied","Data":"fca476b048b15528f13911b753523f7d201381ee1365e63f94780a340ca1509e"}
Oct 31 08:29:19.642892 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:29:19.642852    4720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-7c457bbbbc-4gw8x" event={"ID":"45202e5f-a0de-472d-9673-13fd99f9878b","Type":"ContainerStarted","Data":"25d5e6e9771f33a312fdcf2bc8bfc28ef30934aeccb57660873afa2f868af787"}
Oct 31 08:29:19.859124 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:29:19.859058    4720 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-7c457bbbbc-4gw8x" podStartSLOduration=3.499473889 podStartE2EDuration="5.859044985s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.359571096s" isStatefulPod=false podCreationTimestamp="2026-10-31 08:29:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 08:29:19.809129357 +0000 UTC m=+3613.940197425" watchObservedRunningTime="2026-10-31 08:29:19.859044985 +0000 UTC m=+3613.990113056"
Oct 31 08:29:20.704205 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:29:20.704171    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 08:29:25.071915 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:29:25.071875    4720 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-7c457bbbbc-4gw8x"
Oct 31 08:29:25.072412 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:29:25.071940    4720 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-7c457bbbbc-4gw8x"
Oct 31 08:29:25.073432 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:29:25.073397    4720 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-7c457bbbbc-4gw8x" podUID="45202e5f-a0de-472d-9673-13fd99f9878b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8001/health\": dial tcp 10.132.0.72:8001: connect: connection refused"
Oct 31 08:29:25.088877 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:29:25.088848    4720 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-7c457bbbbc-4gw8x"
Oct 31 08:29:25.651167 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:29:25.651132    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 08:29:35.071575 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:29:35.071530    4720 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-7c457bbbbc-4gw8x" podUID="45202e5f-a0de-472d-9673-13fd99f9878b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8001/health\": dial tcp 10.132.0.72:8001: connect: connection refused"
Oct 31 08:29:35.337837 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:29:35.337805    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 08:29:43.461743 ip-10-0-139-83 kubenswrapper[4720]: E1031 08:29:43.461704    4720 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8m768" containerName="extractor"
Oct 31 08:29:45.072547 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:29:45.072491    4720 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-7c457bbbbc-4gw8x" podUID="45202e5f-a0de-472d-9673-13fd99f9878b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8001/health\": dial tcp 10.132.0.72:8001: connect: connection refused"
Oct 31 08:29:55.072109 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:29:55.072055    4720 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-7c457bbbbc-4gw8x" podUID="45202e5f-a0de-472d-9673-13fd99f9878b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8001/health\": dial tcp 10.132.0.72:8001: connect: connection refused"
Oct 31 08:30:00.161945 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:30:00.161903    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 08:30:03.998614 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:30:03.998574    4720 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6c5444664c-f7kzr"]
Oct 31 08:30:04.098688 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:30:04.098653    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 08:30:04.304617 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:30:04.304527    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 08:30:05.071842 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:30:05.071796    4720 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-7c457bbbbc-4gw8x" podUID="45202e5f-a0de-472d-9673-13fd99f9878b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8001/health\": dial tcp 10.132.0.72:8001: connect: connection refused"
Oct 31 08:30:09.255605 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:30:09.255563    4720 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6c5444664c-f7kzr" podUID="dc0cab61-6a9c-4260-9187-8f1088a62125" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 08:30:13.461923 ip-10-0-139-83 kubenswrapper[4720]: E1031 08:30:13.461881    4720 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8m768" containerName="extractor"
Oct 31 08:30:15.072214 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:30:15.072160    4720 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-7c457bbbbc-4gw8x" podUID="45202e5f-a0de-472d-9673-13fd99f9878b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8001/health\": dial tcp 10.132.0.72:8001: connect: connection refused"
Oct 31 08:30:19.019588 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:30:19.019532    4720 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6c5444664c-f7kzr" podUID="dc0cab61-6a9c-4260-9187-8f1088a62125" containerName="main" containerID="cri-o://32ac09934702bf077c8ad7a392615b363164a1cbae12a7018f890626b9e25808" gracePeriod=45
Oct 31 08:30:19.255399 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:30:19.255359    4720 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6c5444664c-f7kzr" podUID="dc0cab61-6a9c-4260-9187-8f1088a62125" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 08:30:21.183456 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:30:21.183421    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 08:30:25.072009 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:30:25.071954    4720 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-7c457bbbbc-4gw8x" podUID="45202e5f-a0de-472d-9673-13fd99f9878b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8001/health\": dial tcp 10.132.0.72:8001: connect: connection refused"
Oct 31 08:30:29.255760 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:30:29.255709    4720 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6c5444664c-f7kzr" podUID="dc0cab61-6a9c-4260-9187-8f1088a62125" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 08:30:29.256176 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:30:29.255838    4720 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6c5444664c-f7kzr"
Oct 31 08:30:35.072282 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:30:35.072232    4720 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-7c457bbbbc-4gw8x" podUID="45202e5f-a0de-472d-9673-13fd99f9878b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8001/health\": dial tcp 10.132.0.72:8001: connect: connection refused"
Oct 31 08:30:35.683164 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:30:35.683117    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 08:30:39.255076 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:30:39.255033    4720 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6c5444664c-f7kzr" podUID="dc0cab61-6a9c-4260-9187-8f1088a62125" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 08:30:43.463058 ip-10-0-139-83 kubenswrapper[4720]: E1031 08:30:43.463020    4720 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8m768" containerName="extractor"
Oct 31 08:30:43.596349 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:30:43.596293    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 08:30:45.072447 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:30:45.072399    4720 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-7c457bbbbc-4gw8x" podUID="45202e5f-a0de-472d-9673-13fd99f9878b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8001/health\": dial tcp 10.132.0.72:8001: connect: connection refused"
Oct 31 08:30:49.292036 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:30:49.291958    4720 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6c5444664c-f7kzr"
Oct 31 08:30:49.457858 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:30:49.457818    4720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dc0cab61-6a9c-4260-9187-8f1088a62125-kube-api-access-xt2tt\" (UniqueName: \"kubernetes.io/projected/dc0cab61-6a9c-4260-9187-8f1088a62125-kube-api-access-xt2tt\") pod \"dc0cab61-6a9c-4260-9187-8f1088a62125\" (UID: \"dc0cab61-6a9c-4260-9187-8f1088a62125\") "
Oct 31 08:30:49.457858 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:30:49.457858    4720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dc0cab61-6a9c-4260-9187-8f1088a62125-tls-certs\" (UniqueName: \"kubernetes.io/secret/dc0cab61-6a9c-4260-9187-8f1088a62125-tls-certs\") pod \"dc0cab61-6a9c-4260-9187-8f1088a62125\" (UID: \"dc0cab61-6a9c-4260-9187-8f1088a62125\") "
Oct 31 08:30:49.460280 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:30:49.460250    4720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dc0cab61-6a9c-4260-9187-8f1088a62125-tls-certs" pod "dc0cab61-6a9c-4260-9187-8f1088a62125" (UID: "dc0cab61-6a9c-4260-9187-8f1088a62125"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 08:30:49.460427 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:30:49.460374    4720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dc0cab61-6a9c-4260-9187-8f1088a62125-kube-api-access-xt2tt" pod "dc0cab61-6a9c-4260-9187-8f1088a62125" (UID: "dc0cab61-6a9c-4260-9187-8f1088a62125"). InnerVolumeSpecName "kube-api-access-xt2tt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 08:30:49.558938 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:30:49.558848    4720 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xt2tt\" (UniqueName: \"kubernetes.io/projected/dc0cab61-6a9c-4260-9187-8f1088a62125-kube-api-access-xt2tt\") on node \"ip-10-0-139-83.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 08:30:49.558938 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:30:49.558885    4720 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dc0cab61-6a9c-4260-9187-8f1088a62125-tls-certs\") on node \"ip-10-0-139-83.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 08:30:50.090286 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:30:50.090247    4720 generic.go:356] "Generic (PLEG): container finished" podID="dc0cab61-6a9c-4260-9187-8f1088a62125" containerID="32ac09934702bf077c8ad7a392615b363164a1cbae12a7018f890626b9e25808" exitCode=0
Oct 31 08:30:50.090482 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:30:50.090346    4720 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6c5444664c-f7kzr"
Oct 31 08:30:50.090482 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:30:50.090345    4720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6c5444664c-f7kzr" event={"ID":"dc0cab61-6a9c-4260-9187-8f1088a62125","Type":"ContainerDied","Data":"32ac09934702bf077c8ad7a392615b363164a1cbae12a7018f890626b9e25808"}
Oct 31 08:30:50.090482 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:30:50.090414    4720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6c5444664c-f7kzr" event={"ID":"dc0cab61-6a9c-4260-9187-8f1088a62125","Type":"ContainerDied","Data":"e4a80b0e4e64da52b5dd41d42f10ceb5327b0ea2ca275bc8978aebb54d171b93"}
Oct 31 08:30:50.090482 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:30:50.090446    4720 scope.go:122] "RemoveContainer" containerID="32ac09934702bf077c8ad7a392615b363164a1cbae12a7018f890626b9e25808"
Oct 31 08:30:50.101211 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:30:50.101183    4720 scope.go:122] "RemoveContainer" containerID="32ac09934702bf077c8ad7a392615b363164a1cbae12a7018f890626b9e25808"
Oct 31 08:30:50.101579 ip-10-0-139-83 kubenswrapper[4720]: E1031 08:30:50.101552    4720 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"32ac09934702bf077c8ad7a392615b363164a1cbae12a7018f890626b9e25808\": container with ID starting with 32ac09934702bf077c8ad7a392615b363164a1cbae12a7018f890626b9e25808 not found: ID does not exist" containerID="32ac09934702bf077c8ad7a392615b363164a1cbae12a7018f890626b9e25808"
Oct 31 08:30:50.101660 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:30:50.101586    4720 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"32ac09934702bf077c8ad7a392615b363164a1cbae12a7018f890626b9e25808"} err="failed to get container status \"32ac09934702bf077c8ad7a392615b363164a1cbae12a7018f890626b9e25808\": rpc error: code = NotFound desc = could not find container \"32ac09934702bf077c8ad7a392615b363164a1cbae12a7018f890626b9e25808\": container with ID starting with 32ac09934702bf077c8ad7a392615b363164a1cbae12a7018f890626b9e25808 not found: ID does not exist"
Oct 31 08:30:50.254229 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:30:50.254182    4720 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6c5444664c-f7kzr" podUID="dc0cab61-6a9c-4260-9187-8f1088a62125" containerName="main" probeResult="failure" output="timeout: failed to connect service \"10.132.0.71:9003\" within 1s: context deadline exceeded"
Oct 31 08:30:50.254443 ip-10-0-139-83 kubenswrapper[4720]: W1031 08:30:50.254285    4720 logging.go:55] [core] [Channel #581 SubChannel #582]grpc: addrConn.createTransport failed to connect to {Addr: "10.132.0.71:9003", ServerName: "10.132.0.71:9003", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.132.0.71:9003: operation was canceled"
Oct 31 08:30:50.255242 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:30:50.255219    4720 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6c5444664c-f7kzr"]
Oct 31 08:30:50.304532 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:30:50.304450    4720 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6c5444664c-f7kzr"]
Oct 31 08:30:51.727961 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:30:51.727899    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 08:30:51.907722 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:30:51.907680    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 08:30:52.026465 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:30:52.026380    4720 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dc0cab61-6a9c-4260-9187-8f1088a62125" path="/var/lib/kubelet/pods/dc0cab61-6a9c-4260-9187-8f1088a62125/volumes"
Oct 31 08:30:55.072461 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:30:55.072409    4720 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-7c457bbbbc-4gw8x" podUID="45202e5f-a0de-472d-9673-13fd99f9878b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8001/health\": dial tcp 10.132.0.72:8001: connect: connection refused"
Oct 31 08:30:57.850496 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:30:57.850462    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 08:31:00.255475 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:31:00.255441    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 08:31:00.415262 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:31:00.415215    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 08:31:00.577853 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:31:00.577762    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 08:31:01.614358 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:31:01.614301    4720 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-7d79587f48-2js5x"]
Oct 31 08:31:01.614892 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:31:01.614874    4720 cpu_manager.go:414] "containerMap: removing container" podUID="dc0cab61-6a9c-4260-9187-8f1088a62125" containerName="main"
Oct 31 08:31:01.614942 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:31:01.614895    4720 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dc0cab61-6a9c-4260-9187-8f1088a62125" containerName="main"
Oct 31 08:31:01.615044 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:31:01.615032    4720 memory_manager.go:365] "RemoveStaleState removing state" podUID="dc0cab61-6a9c-4260-9187-8f1088a62125" containerName="main"
Oct 31 08:31:01.621163 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:31:01.621133    4720 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-7d79587f48-2js5x"
Oct 31 08:31:01.668501 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:31:01.668459    4720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8358054c-61a6-4ac7-b479-b9e9da9724f2-dshm\") pod \"lora-single-adapter-test-kserve-7d79587f48-2js5x\" (UID: \"8358054c-61a6-4ac7-b479-b9e9da9724f2\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-7d79587f48-2js5x"
Oct 31 08:31:01.668704 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:31:01.668513    4720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8358054c-61a6-4ac7-b479-b9e9da9724f2-tls-certs\") pod \"lora-single-adapter-test-kserve-7d79587f48-2js5x\" (UID: \"8358054c-61a6-4ac7-b479-b9e9da9724f2\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-7d79587f48-2js5x"
Oct 31 08:31:01.668704 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:31:01.668613    4720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8358054c-61a6-4ac7-b479-b9e9da9724f2-kserve-provision-location\") pod \"lora-single-adapter-test-kserve-7d79587f48-2js5x\" (UID: \"8358054c-61a6-4ac7-b479-b9e9da9724f2\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-7d79587f48-2js5x"
Oct 31 08:31:01.668704 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:31:01.668678    4720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8358054c-61a6-4ac7-b479-b9e9da9724f2-home\") pod \"lora-single-adapter-test-kserve-7d79587f48-2js5x\" (UID: \"8358054c-61a6-4ac7-b479-b9e9da9724f2\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-7d79587f48-2js5x"
Oct 31 08:31:01.668704 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:31:01.668703    4720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8358054c-61a6-4ac7-b479-b9e9da9724f2-cabundle-cert\") pod \"lora-single-adapter-test-kserve-7d79587f48-2js5x\" (UID: \"8358054c-61a6-4ac7-b479-b9e9da9724f2\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-7d79587f48-2js5x"
Oct 31 08:31:01.668913 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:31:01.668740    4720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8358054c-61a6-4ac7-b479-b9e9da9724f2-tmp-dir\") pod \"lora-single-adapter-test-kserve-7d79587f48-2js5x\" (UID: \"8358054c-61a6-4ac7-b479-b9e9da9724f2\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-7d79587f48-2js5x"
Oct 31 08:31:01.668913 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:31:01.668802    4720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8358054c-61a6-4ac7-b479-b9e9da9724f2-model-cache\") pod \"lora-single-adapter-test-kserve-7d79587f48-2js5x\" (UID: \"8358054c-61a6-4ac7-b479-b9e9da9724f2\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-7d79587f48-2js5x"
Oct 31 08:31:01.668913 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:31:01.668834    4720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wmwcj\" (UniqueName: \"kubernetes.io/projected/8358054c-61a6-4ac7-b479-b9e9da9724f2-kube-api-access-wmwcj\") pod \"lora-single-adapter-test-kserve-7d79587f48-2js5x\" (UID: \"8358054c-61a6-4ac7-b479-b9e9da9724f2\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-7d79587f48-2js5x"
Oct 31 08:31:01.670273 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:31:01.670252    4720 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-7d79587f48-2js5x"]
Oct 31 08:31:01.675581 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:31:01.675552    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 08:31:01.716236 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:31:01.716191    4720 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-with-lora-adapters-3cbf5e9c\"/\"odh-kserve-custom-ca-bundle\""
Oct 31 08:31:01.716236 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:31:01.716212    4720 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-with-lora-adapters-3cbf5e9c\"/\"openshift-service-ca.crt\""
Oct 31 08:31:01.716531 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:31:01.716258    4720 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-with-lora-adapters-3cbf5e9c\"/\"lora-single-adapter-test-kserve-self-signed-certs\""
Oct 31 08:31:01.716531 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:31:01.716305    4720 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-with-lora-adapters-3cbf5e9c\"/\"kube-root-ca.crt\""
Oct 31 08:31:01.716531 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:31:01.716321    4720 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-with-lora-adapters-3cbf5e9c\"/\"seaweedfs-s3-creds\""
Oct 31 08:31:01.716531 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:31:01.716415    4720 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-with-lora-adapters-3cbf5e9c\"/\"default-dockercfg-vczlv\""
Oct 31 08:31:01.769418 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:31:01.769376    4720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8358054c-61a6-4ac7-b479-b9e9da9724f2-dshm\") pod \"lora-single-adapter-test-kserve-7d79587f48-2js5x\" (UID: \"8358054c-61a6-4ac7-b479-b9e9da9724f2\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-7d79587f48-2js5x"
Oct 31 08:31:01.769418 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:31:01.769419    4720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8358054c-61a6-4ac7-b479-b9e9da9724f2-tls-certs\") pod \"lora-single-adapter-test-kserve-7d79587f48-2js5x\" (UID: \"8358054c-61a6-4ac7-b479-b9e9da9724f2\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-7d79587f48-2js5x"
Oct 31 08:31:01.769701 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:31:01.769443    4720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8358054c-61a6-4ac7-b479-b9e9da9724f2-kserve-provision-location\") pod \"lora-single-adapter-test-kserve-7d79587f48-2js5x\" (UID: \"8358054c-61a6-4ac7-b479-b9e9da9724f2\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-7d79587f48-2js5x"
Oct 31 08:31:01.769701 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:31:01.769462    4720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8358054c-61a6-4ac7-b479-b9e9da9724f2-home\") pod \"lora-single-adapter-test-kserve-7d79587f48-2js5x\" (UID: \"8358054c-61a6-4ac7-b479-b9e9da9724f2\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-7d79587f48-2js5x"
Oct 31 08:31:01.769701 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:31:01.769477    4720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8358054c-61a6-4ac7-b479-b9e9da9724f2-cabundle-cert\") pod \"lora-single-adapter-test-kserve-7d79587f48-2js5x\" (UID: \"8358054c-61a6-4ac7-b479-b9e9da9724f2\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-7d79587f48-2js5x"
Oct 31 08:31:01.769701 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:31:01.769504    4720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8358054c-61a6-4ac7-b479-b9e9da9724f2-tmp-dir\") pod \"lora-single-adapter-test-kserve-7d79587f48-2js5x\" (UID: \"8358054c-61a6-4ac7-b479-b9e9da9724f2\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-7d79587f48-2js5x"
Oct 31 08:31:01.769701 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:31:01.769555    4720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8358054c-61a6-4ac7-b479-b9e9da9724f2-model-cache\") pod \"lora-single-adapter-test-kserve-7d79587f48-2js5x\" (UID: \"8358054c-61a6-4ac7-b479-b9e9da9724f2\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-7d79587f48-2js5x"
Oct 31 08:31:01.769701 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:31:01.769661    4720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wmwcj\" (UniqueName: \"kubernetes.io/projected/8358054c-61a6-4ac7-b479-b9e9da9724f2-kube-api-access-wmwcj\") pod \"lora-single-adapter-test-kserve-7d79587f48-2js5x\" (UID: \"8358054c-61a6-4ac7-b479-b9e9da9724f2\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-7d79587f48-2js5x"
Oct 31 08:31:01.770028 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:31:01.770004    4720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8358054c-61a6-4ac7-b479-b9e9da9724f2-home\") pod \"lora-single-adapter-test-kserve-7d79587f48-2js5x\" (UID: \"8358054c-61a6-4ac7-b479-b9e9da9724f2\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-7d79587f48-2js5x"
Oct 31 08:31:01.770089 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:31:01.770051    4720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8358054c-61a6-4ac7-b479-b9e9da9724f2-kserve-provision-location\") pod \"lora-single-adapter-test-kserve-7d79587f48-2js5x\" (UID: \"8358054c-61a6-4ac7-b479-b9e9da9724f2\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-7d79587f48-2js5x"
Oct 31 08:31:01.770089 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:31:01.770079    4720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8358054c-61a6-4ac7-b479-b9e9da9724f2-model-cache\") pod \"lora-single-adapter-test-kserve-7d79587f48-2js5x\" (UID: \"8358054c-61a6-4ac7-b479-b9e9da9724f2\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-7d79587f48-2js5x"
Oct 31 08:31:01.770402 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:31:01.770316    4720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8358054c-61a6-4ac7-b479-b9e9da9724f2-tmp-dir\") pod \"lora-single-adapter-test-kserve-7d79587f48-2js5x\" (UID: \"8358054c-61a6-4ac7-b479-b9e9da9724f2\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-7d79587f48-2js5x"
Oct 31 08:31:01.770402 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:31:01.770352    4720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8358054c-61a6-4ac7-b479-b9e9da9724f2-cabundle-cert\") pod \"lora-single-adapter-test-kserve-7d79587f48-2js5x\" (UID: \"8358054c-61a6-4ac7-b479-b9e9da9724f2\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-7d79587f48-2js5x"
Oct 31 08:31:01.772043 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:31:01.772020    4720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8358054c-61a6-4ac7-b479-b9e9da9724f2-dshm\") pod \"lora-single-adapter-test-kserve-7d79587f48-2js5x\" (UID: \"8358054c-61a6-4ac7-b479-b9e9da9724f2\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-7d79587f48-2js5x"
Oct 31 08:31:01.772615 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:31:01.772594    4720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8358054c-61a6-4ac7-b479-b9e9da9724f2-tls-certs\") pod \"lora-single-adapter-test-kserve-7d79587f48-2js5x\" (UID: \"8358054c-61a6-4ac7-b479-b9e9da9724f2\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-7d79587f48-2js5x"
Oct 31 08:31:01.827412 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:31:01.827377    4720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wmwcj\" (UniqueName: \"kubernetes.io/projected/8358054c-61a6-4ac7-b479-b9e9da9724f2-kube-api-access-wmwcj\") pod \"lora-single-adapter-test-kserve-7d79587f48-2js5x\" (UID: \"8358054c-61a6-4ac7-b479-b9e9da9724f2\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-7d79587f48-2js5x"
Oct 31 08:31:01.940595 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:31:01.940500    4720 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-7d79587f48-2js5x"
Oct 31 08:31:02.377508 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:31:02.377469    4720 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-7d79587f48-2js5x"]
Oct 31 08:31:02.381636 ip-10-0-139-83 kubenswrapper[4720]: W1031 08:31:02.381605    4720 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8358054c_61a6_4ac7_b479_b9e9da9724f2.slice/crio-d9182c8997efaf48a91f9627b33e355b049864505692facaf3b67a02c5f6ce42 WatchSource:0}: Error finding container d9182c8997efaf48a91f9627b33e355b049864505692facaf3b67a02c5f6ce42: Status 404 returned error can't find the container with id d9182c8997efaf48a91f9627b33e355b049864505692facaf3b67a02c5f6ce42
Oct 31 08:31:03.162781 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:31:03.162736    4720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-7d79587f48-2js5x" event={"ID":"8358054c-61a6-4ac7-b479-b9e9da9724f2","Type":"ContainerStarted","Data":"87b14dc1df211ffc745b227bb0929efac9760cc1d0565bbbcee56d9e029d32a2"}
Oct 31 08:31:03.163255 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:31:03.162810    4720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-7d79587f48-2js5x" event={"ID":"8358054c-61a6-4ac7-b479-b9e9da9724f2","Type":"ContainerStarted","Data":"d9182c8997efaf48a91f9627b33e355b049864505692facaf3b67a02c5f6ce42"}
Oct 31 08:31:03.294115 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:31:03.294079    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 08:31:03.495061 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:31:03.494980    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 08:31:05.072573 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:31:05.072528    4720 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-7c457bbbbc-4gw8x" podUID="45202e5f-a0de-472d-9673-13fd99f9878b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8001/health\": dial tcp 10.132.0.72:8001: connect: connection refused"
Oct 31 08:31:07.185016 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:31:07.184979    4720 generic.go:356] "Generic (PLEG): container finished" podID="8358054c-61a6-4ac7-b479-b9e9da9724f2" containerID="87b14dc1df211ffc745b227bb0929efac9760cc1d0565bbbcee56d9e029d32a2" exitCode=0
Oct 31 08:31:07.185552 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:31:07.185056    4720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-7d79587f48-2js5x" event={"ID":"8358054c-61a6-4ac7-b479-b9e9da9724f2","Type":"ContainerDied","Data":"87b14dc1df211ffc745b227bb0929efac9760cc1d0565bbbcee56d9e029d32a2"}
Oct 31 08:31:08.192475 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:31:08.192434    4720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-7d79587f48-2js5x" event={"ID":"8358054c-61a6-4ac7-b479-b9e9da9724f2","Type":"ContainerStarted","Data":"53285d4d8195f0846e86cc450f8296579813a8bc40d48e8323ba52c04ea33de9"}
Oct 31 08:31:08.411849 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:31:08.411783    4720 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-7d79587f48-2js5x" podStartSLOduration=3.605860431 podStartE2EDuration="7.41176454s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.805904109s" isStatefulPod=false podCreationTimestamp="2026-10-31 08:31:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 08:31:08.410421339 +0000 UTC m=+3722.541489410" watchObservedRunningTime="2026-10-31 08:31:08.41176454 +0000 UTC m=+3722.542832612"
Oct 31 08:31:08.940683 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:31:08.940646    4720 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-7d79587f48-2js5x"
Oct 31 08:31:11.941724 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:31:11.941676    4720 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-7d79587f48-2js5x"
Oct 31 08:31:11.943687 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:31:11.943650    4720 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-7d79587f48-2js5x" podUID="8358054c-61a6-4ac7-b479-b9e9da9724f2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/health\": dial tcp 10.132.0.73:8000: connect: connection refused"
Oct 31 08:31:13.464846 ip-10-0-139-83 kubenswrapper[4720]: E1031 08:31:13.464801    4720 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8m768" containerName="extractor"
Oct 31 08:31:15.087359 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:31:15.087297    4720 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-7c457bbbbc-4gw8x"
Oct 31 08:31:15.099508 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:31:15.099479    4720 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-7c457bbbbc-4gw8x"
Oct 31 08:31:21.488313 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:31:21.488273    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 08:31:21.941579 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:31:21.941527    4720 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-7d79587f48-2js5x" podUID="8358054c-61a6-4ac7-b479-b9e9da9724f2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/health\": dial tcp 10.132.0.73:8000: connect: connection refused"
Oct 31 08:31:29.158444 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:31:29.158412    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 08:31:29.181570 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:31:29.181530    4720 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-7c457bbbbc-4gw8x"]
Oct 31 08:31:29.309437 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:31:29.309398    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 08:31:31.941344 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:31:31.941287    4720 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-7d79587f48-2js5x" podUID="8358054c-61a6-4ac7-b479-b9e9da9724f2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/health\": dial tcp 10.132.0.73:8000: connect: connection refused"
Oct 31 08:31:36.004655 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:31:36.004612    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 08:31:41.941608 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:31:41.941563    4720 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-7d79587f48-2js5x" podUID="8358054c-61a6-4ac7-b479-b9e9da9724f2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/health\": dial tcp 10.132.0.73:8000: connect: connection refused"
Oct 31 08:31:43.462211 ip-10-0-139-83 kubenswrapper[4720]: E1031 08:31:43.462173    4720 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8m768" containerName="extractor"
Oct 31 08:31:44.205911 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:31:44.205849    4720 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-7c457bbbbc-4gw8x" podUID="45202e5f-a0de-472d-9673-13fd99f9878b" containerName="main" containerID="cri-o://25d5e6e9771f33a312fdcf2bc8bfc28ef30934aeccb57660873afa2f868af787" gracePeriod=45
Oct 31 08:31:44.818819 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:31:44.818780    4720 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-7c457bbbbc-4gw8x" podUID="45202e5f-a0de-472d-9673-13fd99f9878b" containerName="llm-d-routing-sidecar" containerID="cri-o://b61c1755f326b57ee291827a1744f7864b9175a6ba9e430aed7948b2c32ee9ed" gracePeriod=45
Oct 31 08:31:45.084794 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:31:45.084766    4720 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-7c457bbbbc-4gw8x"
Oct 31 08:31:45.275508 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:31:45.275465    4720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/45202e5f-a0de-472d-9673-13fd99f9878b-tls-certs\" (UniqueName: \"kubernetes.io/secret/45202e5f-a0de-472d-9673-13fd99f9878b-tls-certs\") pod \"45202e5f-a0de-472d-9673-13fd99f9878b\" (UID: \"45202e5f-a0de-472d-9673-13fd99f9878b\") "
Oct 31 08:31:45.275736 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:31:45.275555    4720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/45202e5f-a0de-472d-9673-13fd99f9878b-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/45202e5f-a0de-472d-9673-13fd99f9878b-kserve-provision-location\") pod \"45202e5f-a0de-472d-9673-13fd99f9878b\" (UID: \"45202e5f-a0de-472d-9673-13fd99f9878b\") "
Oct 31 08:31:45.275736 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:31:45.275591    4720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/45202e5f-a0de-472d-9673-13fd99f9878b-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/45202e5f-a0de-472d-9673-13fd99f9878b-tmp-dir\") pod \"45202e5f-a0de-472d-9673-13fd99f9878b\" (UID: \"45202e5f-a0de-472d-9673-13fd99f9878b\") "
Oct 31 08:31:45.275736 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:31:45.275665    4720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/45202e5f-a0de-472d-9673-13fd99f9878b-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/45202e5f-a0de-472d-9673-13fd99f9878b-cabundle-cert\") pod \"45202e5f-a0de-472d-9673-13fd99f9878b\" (UID: \"45202e5f-a0de-472d-9673-13fd99f9878b\") "
Oct 31 08:31:45.275736 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:31:45.275700    4720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/45202e5f-a0de-472d-9673-13fd99f9878b-kube-api-access-vpwdz\" (UniqueName: \"kubernetes.io/projected/45202e5f-a0de-472d-9673-13fd99f9878b-kube-api-access-vpwdz\") pod \"45202e5f-a0de-472d-9673-13fd99f9878b\" (UID: \"45202e5f-a0de-472d-9673-13fd99f9878b\") "
Oct 31 08:31:45.275969 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:31:45.275758    4720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/45202e5f-a0de-472d-9673-13fd99f9878b-home\" (UniqueName: \"kubernetes.io/empty-dir/45202e5f-a0de-472d-9673-13fd99f9878b-home\") pod \"45202e5f-a0de-472d-9673-13fd99f9878b\" (UID: \"45202e5f-a0de-472d-9673-13fd99f9878b\") "
Oct 31 08:31:45.275969 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:31:45.275798    4720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/45202e5f-a0de-472d-9673-13fd99f9878b-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/45202e5f-a0de-472d-9673-13fd99f9878b-model-cache\") pod \"45202e5f-a0de-472d-9673-13fd99f9878b\" (UID: \"45202e5f-a0de-472d-9673-13fd99f9878b\") "
Oct 31 08:31:45.275969 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:31:45.275836    4720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/45202e5f-a0de-472d-9673-13fd99f9878b-dshm\" (UniqueName: \"kubernetes.io/empty-dir/45202e5f-a0de-472d-9673-13fd99f9878b-dshm\") pod \"45202e5f-a0de-472d-9673-13fd99f9878b\" (UID: \"45202e5f-a0de-472d-9673-13fd99f9878b\") "
Oct 31 08:31:45.276648 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:31:45.276575    4720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/45202e5f-a0de-472d-9673-13fd99f9878b-model-cache" pod "45202e5f-a0de-472d-9673-13fd99f9878b" (UID: "45202e5f-a0de-472d-9673-13fd99f9878b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 08:31:45.276778 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:31:45.276684    4720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/45202e5f-a0de-472d-9673-13fd99f9878b-cabundle-cert" pod "45202e5f-a0de-472d-9673-13fd99f9878b" (UID: "45202e5f-a0de-472d-9673-13fd99f9878b"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 08:31:45.277286 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:31:45.277255    4720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/45202e5f-a0de-472d-9673-13fd99f9878b-home" pod "45202e5f-a0de-472d-9673-13fd99f9878b" (UID: "45202e5f-a0de-472d-9673-13fd99f9878b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 08:31:45.278781 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:31:45.278741    4720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/45202e5f-a0de-472d-9673-13fd99f9878b-kube-api-access-vpwdz" pod "45202e5f-a0de-472d-9673-13fd99f9878b" (UID: "45202e5f-a0de-472d-9673-13fd99f9878b"). InnerVolumeSpecName "kube-api-access-vpwdz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 08:31:45.278897 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:31:45.278829    4720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/45202e5f-a0de-472d-9673-13fd99f9878b-tls-certs" pod "45202e5f-a0de-472d-9673-13fd99f9878b" (UID: "45202e5f-a0de-472d-9673-13fd99f9878b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 08:31:45.279077 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:31:45.279048    4720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/45202e5f-a0de-472d-9673-13fd99f9878b-dshm" pod "45202e5f-a0de-472d-9673-13fd99f9878b" (UID: "45202e5f-a0de-472d-9673-13fd99f9878b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 08:31:45.289998 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:31:45.289964    4720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/45202e5f-a0de-472d-9673-13fd99f9878b-tmp-dir" pod "45202e5f-a0de-472d-9673-13fd99f9878b" (UID: "45202e5f-a0de-472d-9673-13fd99f9878b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 08:31:45.304665 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:31:45.304623    4720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/45202e5f-a0de-472d-9673-13fd99f9878b-kserve-provision-location" pod "45202e5f-a0de-472d-9673-13fd99f9878b" (UID: "45202e5f-a0de-472d-9673-13fd99f9878b"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 08:31:45.377493 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:31:45.377450    4720 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/45202e5f-a0de-472d-9673-13fd99f9878b-home\") on node \"ip-10-0-139-83.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 08:31:45.377493 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:31:45.377480    4720 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/45202e5f-a0de-472d-9673-13fd99f9878b-model-cache\") on node \"ip-10-0-139-83.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 08:31:45.377493 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:31:45.377490    4720 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/45202e5f-a0de-472d-9673-13fd99f9878b-dshm\") on node \"ip-10-0-139-83.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 08:31:45.377493 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:31:45.377500    4720 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/45202e5f-a0de-472d-9673-13fd99f9878b-tls-certs\") on node \"ip-10-0-139-83.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 08:31:45.377885 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:31:45.377509    4720 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/45202e5f-a0de-472d-9673-13fd99f9878b-kserve-provision-location\") on node \"ip-10-0-139-83.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 08:31:45.377885 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:31:45.377518    4720 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/45202e5f-a0de-472d-9673-13fd99f9878b-tmp-dir\") on node \"ip-10-0-139-83.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 08:31:45.377885 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:31:45.377526    4720 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/45202e5f-a0de-472d-9673-13fd99f9878b-cabundle-cert\") on node \"ip-10-0-139-83.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 08:31:45.377885 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:31:45.377534    4720 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vpwdz\" (UniqueName: \"kubernetes.io/projected/45202e5f-a0de-472d-9673-13fd99f9878b-kube-api-access-vpwdz\") on node \"ip-10-0-139-83.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 08:31:45.388782 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:31:45.388741    4720 generic.go:356] "Generic (PLEG): container finished" podID="45202e5f-a0de-472d-9673-13fd99f9878b" containerID="25d5e6e9771f33a312fdcf2bc8bfc28ef30934aeccb57660873afa2f868af787" exitCode=0
Oct 31 08:31:45.388782 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:31:45.388781    4720 generic.go:356] "Generic (PLEG): container finished" podID="45202e5f-a0de-472d-9673-13fd99f9878b" containerID="b61c1755f326b57ee291827a1744f7864b9175a6ba9e430aed7948b2c32ee9ed" exitCode=0
Oct 31 08:31:45.388782 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:31:45.388793    4720 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-7c457bbbbc-4gw8x"
Oct 31 08:31:45.389035 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:31:45.388824    4720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-7c457bbbbc-4gw8x" event={"ID":"45202e5f-a0de-472d-9673-13fd99f9878b","Type":"ContainerDied","Data":"25d5e6e9771f33a312fdcf2bc8bfc28ef30934aeccb57660873afa2f868af787"}
Oct 31 08:31:45.389035 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:31:45.388885    4720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-7c457bbbbc-4gw8x" event={"ID":"45202e5f-a0de-472d-9673-13fd99f9878b","Type":"ContainerDied","Data":"b61c1755f326b57ee291827a1744f7864b9175a6ba9e430aed7948b2c32ee9ed"}
Oct 31 08:31:45.389035 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:31:45.388902    4720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-7c457bbbbc-4gw8x" event={"ID":"45202e5f-a0de-472d-9673-13fd99f9878b","Type":"ContainerDied","Data":"e5f151fc075f7f3c0af8e805947f3bcaf1b3f7503539ff49b9c08b6da0fdf674"}
Oct 31 08:31:45.389035 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:31:45.388922    4720 scope.go:122] "RemoveContainer" containerID="25d5e6e9771f33a312fdcf2bc8bfc28ef30934aeccb57660873afa2f868af787"
Oct 31 08:31:45.398886 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:31:45.398861    4720 scope.go:122] "RemoveContainer" containerID="fca476b048b15528f13911b753523f7d201381ee1365e63f94780a340ca1509e"
Oct 31 08:31:45.407727 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:31:45.407705    4720 scope.go:122] "RemoveContainer" containerID="b61c1755f326b57ee291827a1744f7864b9175a6ba9e430aed7948b2c32ee9ed"
Oct 31 08:31:45.416106 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:31:45.416073    4720 scope.go:122] "RemoveContainer" containerID="25d5e6e9771f33a312fdcf2bc8bfc28ef30934aeccb57660873afa2f868af787"
Oct 31 08:31:45.416561 ip-10-0-139-83 kubenswrapper[4720]: E1031 08:31:45.416536    4720 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"25d5e6e9771f33a312fdcf2bc8bfc28ef30934aeccb57660873afa2f868af787\": container with ID starting with 25d5e6e9771f33a312fdcf2bc8bfc28ef30934aeccb57660873afa2f868af787 not found: ID does not exist" containerID="25d5e6e9771f33a312fdcf2bc8bfc28ef30934aeccb57660873afa2f868af787"
Oct 31 08:31:45.416650 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:31:45.416572    4720 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"25d5e6e9771f33a312fdcf2bc8bfc28ef30934aeccb57660873afa2f868af787"} err="failed to get container status \"25d5e6e9771f33a312fdcf2bc8bfc28ef30934aeccb57660873afa2f868af787\": rpc error: code = NotFound desc = could not find container \"25d5e6e9771f33a312fdcf2bc8bfc28ef30934aeccb57660873afa2f868af787\": container with ID starting with 25d5e6e9771f33a312fdcf2bc8bfc28ef30934aeccb57660873afa2f868af787 not found: ID does not exist"
Oct 31 08:31:45.416650 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:31:45.416591    4720 scope.go:122] "RemoveContainer" containerID="fca476b048b15528f13911b753523f7d201381ee1365e63f94780a340ca1509e"
Oct 31 08:31:45.416953 ip-10-0-139-83 kubenswrapper[4720]: E1031 08:31:45.416926    4720 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fca476b048b15528f13911b753523f7d201381ee1365e63f94780a340ca1509e\": container with ID starting with fca476b048b15528f13911b753523f7d201381ee1365e63f94780a340ca1509e not found: ID does not exist" containerID="fca476b048b15528f13911b753523f7d201381ee1365e63f94780a340ca1509e"
Oct 31 08:31:45.417008 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:31:45.416967    4720 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fca476b048b15528f13911b753523f7d201381ee1365e63f94780a340ca1509e"} err="failed to get container status \"fca476b048b15528f13911b753523f7d201381ee1365e63f94780a340ca1509e\": rpc error: code = NotFound desc = could not find container \"fca476b048b15528f13911b753523f7d201381ee1365e63f94780a340ca1509e\": container with ID starting with fca476b048b15528f13911b753523f7d201381ee1365e63f94780a340ca1509e not found: ID does not exist"
Oct 31 08:31:45.417008 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:31:45.416990    4720 scope.go:122] "RemoveContainer" containerID="b61c1755f326b57ee291827a1744f7864b9175a6ba9e430aed7948b2c32ee9ed"
Oct 31 08:31:45.417276 ip-10-0-139-83 kubenswrapper[4720]: E1031 08:31:45.417250    4720 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b61c1755f326b57ee291827a1744f7864b9175a6ba9e430aed7948b2c32ee9ed\": container with ID starting with b61c1755f326b57ee291827a1744f7864b9175a6ba9e430aed7948b2c32ee9ed not found: ID does not exist" containerID="b61c1755f326b57ee291827a1744f7864b9175a6ba9e430aed7948b2c32ee9ed"
Oct 31 08:31:45.417357 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:31:45.417280    4720 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b61c1755f326b57ee291827a1744f7864b9175a6ba9e430aed7948b2c32ee9ed"} err="failed to get container status \"b61c1755f326b57ee291827a1744f7864b9175a6ba9e430aed7948b2c32ee9ed\": rpc error: code = NotFound desc = could not find container \"b61c1755f326b57ee291827a1744f7864b9175a6ba9e430aed7948b2c32ee9ed\": container with ID starting with b61c1755f326b57ee291827a1744f7864b9175a6ba9e430aed7948b2c32ee9ed not found: ID does not exist"
Oct 31 08:31:45.417357 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:31:45.417300    4720 scope.go:122] "RemoveContainer" containerID="25d5e6e9771f33a312fdcf2bc8bfc28ef30934aeccb57660873afa2f868af787"
Oct 31 08:31:45.417637 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:31:45.417618    4720 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"25d5e6e9771f33a312fdcf2bc8bfc28ef30934aeccb57660873afa2f868af787"} err="failed to get container status \"25d5e6e9771f33a312fdcf2bc8bfc28ef30934aeccb57660873afa2f868af787\": rpc error: code = NotFound desc = could not find container \"25d5e6e9771f33a312fdcf2bc8bfc28ef30934aeccb57660873afa2f868af787\": container with ID starting with 25d5e6e9771f33a312fdcf2bc8bfc28ef30934aeccb57660873afa2f868af787 not found: ID does not exist"
Oct 31 08:31:45.417690 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:31:45.417637    4720 scope.go:122] "RemoveContainer" containerID="fca476b048b15528f13911b753523f7d201381ee1365e63f94780a340ca1509e"
Oct 31 08:31:45.417896 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:31:45.417874    4720 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fca476b048b15528f13911b753523f7d201381ee1365e63f94780a340ca1509e"} err="failed to get container status \"fca476b048b15528f13911b753523f7d201381ee1365e63f94780a340ca1509e\": rpc error: code = NotFound desc = could not find container \"fca476b048b15528f13911b753523f7d201381ee1365e63f94780a340ca1509e\": container with ID starting with fca476b048b15528f13911b753523f7d201381ee1365e63f94780a340ca1509e not found: ID does not exist"
Oct 31 08:31:45.417943 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:31:45.417899    4720 scope.go:122] "RemoveContainer" containerID="b61c1755f326b57ee291827a1744f7864b9175a6ba9e430aed7948b2c32ee9ed"
Oct 31 08:31:45.418139 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:31:45.418122    4720 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b61c1755f326b57ee291827a1744f7864b9175a6ba9e430aed7948b2c32ee9ed"} err="failed to get container status \"b61c1755f326b57ee291827a1744f7864b9175a6ba9e430aed7948b2c32ee9ed\": rpc error: code = NotFound desc = could not find container \"b61c1755f326b57ee291827a1744f7864b9175a6ba9e430aed7948b2c32ee9ed\": container with ID starting with b61c1755f326b57ee291827a1744f7864b9175a6ba9e430aed7948b2c32ee9ed not found: ID does not exist"
Oct 31 08:31:45.554306 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:31:45.554267    4720 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-7c457bbbbc-4gw8x"]
Oct 31 08:31:45.560151 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:31:45.560123    4720 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-7c457bbbbc-4gw8x"]
Oct 31 08:31:46.026531 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:31:46.026485    4720 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="45202e5f-a0de-472d-9673-13fd99f9878b" path="/var/lib/kubelet/pods/45202e5f-a0de-472d-9673-13fd99f9878b/volumes"
Oct 31 08:31:50.073037 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:31:50.072986    4720 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-7c457bbbbc-4gw8x" podUID="45202e5f-a0de-472d-9673-13fd99f9878b" containerName="llm-d-routing-sidecar" probeResult="failure" output="Get \"https://10.132.0.72:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 08:31:51.941041 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:31:51.941000    4720 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-7d79587f48-2js5x" podUID="8358054c-61a6-4ac7-b479-b9e9da9724f2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/health\": dial tcp 10.132.0.73:8000: connect: connection refused"
Oct 31 08:31:55.071740 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:31:55.071686    4720 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-7c457bbbbc-4gw8x" podUID="45202e5f-a0de-472d-9673-13fd99f9878b" containerName="llm-d-routing-sidecar" probeResult="failure" output="Get \"https://10.132.0.72:8000/health\": context deadline exceeded"
Oct 31 08:32:00.781137 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:32:00.781102    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 08:32:01.941573 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:32:01.941525    4720 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-7d79587f48-2js5x" podUID="8358054c-61a6-4ac7-b479-b9e9da9724f2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/health\": dial tcp 10.132.0.73:8000: connect: connection refused"
Oct 31 08:32:11.941472 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:32:11.941428    4720 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-7d79587f48-2js5x" podUID="8358054c-61a6-4ac7-b479-b9e9da9724f2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/health\": dial tcp 10.132.0.73:8000: connect: connection refused"
Oct 31 08:32:13.461552 ip-10-0-139-83 kubenswrapper[4720]: E1031 08:32:13.461501    4720 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8m768" containerName="extractor"
Oct 31 08:32:18.648714 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:32:18.648678    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 08:32:18.811895 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:32:18.811843    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 08:32:19.245802 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:32:19.245770    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 08:32:19.393208 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:32:19.393172    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 08:32:21.753763 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:32:21.753726    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 08:32:21.941671 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:32:21.941623    4720 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-7d79587f48-2js5x" podUID="8358054c-61a6-4ac7-b479-b9e9da9724f2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/health\": dial tcp 10.132.0.73:8000: connect: connection refused"
Oct 31 08:32:23.295025 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:32:23.294944    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 08:32:24.779082 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:32:24.779045    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 08:32:30.245045 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:32:30.245005    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 08:32:30.395131 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:32:30.395094    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 08:32:31.951756 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:32:31.951718    4720 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-7d79587f48-2js5x"
Oct 31 08:32:31.959753 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:32:31.959728    4720 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-7d79587f48-2js5x"
Oct 31 08:32:34.070133 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:32:34.070089    4720 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-59bb995bd9-ddng5"]
Oct 31 08:32:34.070574 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:32:34.070479    4720 cpu_manager.go:414] "containerMap: removing container" podUID="45202e5f-a0de-472d-9673-13fd99f9878b" containerName="storage-initializer"
Oct 31 08:32:34.070574 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:32:34.070491    4720 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="45202e5f-a0de-472d-9673-13fd99f9878b" containerName="storage-initializer"
Oct 31 08:32:34.070574 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:32:34.070506    4720 cpu_manager.go:414] "containerMap: removing container" podUID="45202e5f-a0de-472d-9673-13fd99f9878b" containerName="main"
Oct 31 08:32:34.070574 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:32:34.070514    4720 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="45202e5f-a0de-472d-9673-13fd99f9878b" containerName="main"
Oct 31 08:32:34.070574 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:32:34.070539    4720 cpu_manager.go:414] "containerMap: removing container" podUID="45202e5f-a0de-472d-9673-13fd99f9878b" containerName="llm-d-routing-sidecar"
Oct 31 08:32:34.070574 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:32:34.070544    4720 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="45202e5f-a0de-472d-9673-13fd99f9878b" containerName="llm-d-routing-sidecar"
Oct 31 08:32:34.070771 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:32:34.070600    4720 memory_manager.go:365] "RemoveStaleState removing state" podUID="45202e5f-a0de-472d-9673-13fd99f9878b" containerName="llm-d-routing-sidecar"
Oct 31 08:32:34.070771 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:32:34.070611    4720 memory_manager.go:365] "RemoveStaleState removing state" podUID="45202e5f-a0de-472d-9673-13fd99f9878b" containerName="main"
Oct 31 08:32:34.073808 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:32:34.073782    4720 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-59bb995bd9-ddng5"
Oct 31 08:32:34.110635 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:32:34.110601    4720 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-59bb995bd9-ddng5"]
Oct 31 08:32:34.123570 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:32:34.123536    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 08:32:34.125081 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:32:34.125065    4720 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"seaweedfs-s3-creds\""
Oct 31 08:32:34.129479 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:32:34.129453    4720 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"openshift-service-ca.crt\""
Oct 31 08:32:34.129617 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:32:34.129490    4720 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"router-with-refs-pd-test-kserve-self-signed-certs\""
Oct 31 08:32:34.129617 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:32:34.129549    4720 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"odh-kserve-custom-ca-bundle\""
Oct 31 08:32:34.129617 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:32:34.129558    4720 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"kube-root-ca.crt\""
Oct 31 08:32:34.129617 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:32:34.129562    4720 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"default-dockercfg-5zqt7\""
Oct 31 08:32:34.222595 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:32:34.222559    4720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/eb737fc6-3a46-41f0-ba3b-ef2a410f3543-home\") pod \"router-with-refs-pd-test-kserve-prefill-59bb995bd9-ddng5\" (UID: \"eb737fc6-3a46-41f0-ba3b-ef2a410f3543\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-59bb995bd9-ddng5"
Oct 31 08:32:34.222595 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:32:34.222599    4720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/eb737fc6-3a46-41f0-ba3b-ef2a410f3543-dshm\") pod \"router-with-refs-pd-test-kserve-prefill-59bb995bd9-ddng5\" (UID: \"eb737fc6-3a46-41f0-ba3b-ef2a410f3543\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-59bb995bd9-ddng5"
Oct 31 08:32:34.222854 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:32:34.222620    4720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5xnnt\" (UniqueName: \"kubernetes.io/projected/eb737fc6-3a46-41f0-ba3b-ef2a410f3543-kube-api-access-5xnnt\") pod \"router-with-refs-pd-test-kserve-prefill-59bb995bd9-ddng5\" (UID: \"eb737fc6-3a46-41f0-ba3b-ef2a410f3543\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-59bb995bd9-ddng5"
Oct 31 08:32:34.222854 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:32:34.222770    4720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eb737fc6-3a46-41f0-ba3b-ef2a410f3543-model-cache\") pod \"router-with-refs-pd-test-kserve-prefill-59bb995bd9-ddng5\" (UID: \"eb737fc6-3a46-41f0-ba3b-ef2a410f3543\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-59bb995bd9-ddng5"
Oct 31 08:32:34.222854 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:32:34.222811    4720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eb737fc6-3a46-41f0-ba3b-ef2a410f3543-tmp-dir\") pod \"router-with-refs-pd-test-kserve-prefill-59bb995bd9-ddng5\" (UID: \"eb737fc6-3a46-41f0-ba3b-ef2a410f3543\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-59bb995bd9-ddng5"
Oct 31 08:32:34.223026 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:32:34.222859    4720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eb737fc6-3a46-41f0-ba3b-ef2a410f3543-tls-certs\") pod \"router-with-refs-pd-test-kserve-prefill-59bb995bd9-ddng5\" (UID: \"eb737fc6-3a46-41f0-ba3b-ef2a410f3543\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-59bb995bd9-ddng5"
Oct 31 08:32:34.223026 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:32:34.222890    4720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/eb737fc6-3a46-41f0-ba3b-ef2a410f3543-cabundle-cert\") pod \"router-with-refs-pd-test-kserve-prefill-59bb995bd9-ddng5\" (UID: \"eb737fc6-3a46-41f0-ba3b-ef2a410f3543\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-59bb995bd9-ddng5"
Oct 31 08:32:34.223129 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:32:34.223058    4720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/eb737fc6-3a46-41f0-ba3b-ef2a410f3543-kserve-provision-location\") pod \"router-with-refs-pd-test-kserve-prefill-59bb995bd9-ddng5\" (UID: \"eb737fc6-3a46-41f0-ba3b-ef2a410f3543\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-59bb995bd9-ddng5"
Oct 31 08:32:34.323815 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:32:34.323712    4720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/eb737fc6-3a46-41f0-ba3b-ef2a410f3543-kserve-provision-location\") pod \"router-with-refs-pd-test-kserve-prefill-59bb995bd9-ddng5\" (UID: \"eb737fc6-3a46-41f0-ba3b-ef2a410f3543\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-59bb995bd9-ddng5"
Oct 31 08:32:34.323815 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:32:34.323753    4720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/eb737fc6-3a46-41f0-ba3b-ef2a410f3543-home\") pod \"router-with-refs-pd-test-kserve-prefill-59bb995bd9-ddng5\" (UID: \"eb737fc6-3a46-41f0-ba3b-ef2a410f3543\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-59bb995bd9-ddng5"
Oct 31 08:32:34.323815 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:32:34.323777    4720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/eb737fc6-3a46-41f0-ba3b-ef2a410f3543-dshm\") pod \"router-with-refs-pd-test-kserve-prefill-59bb995bd9-ddng5\" (UID: \"eb737fc6-3a46-41f0-ba3b-ef2a410f3543\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-59bb995bd9-ddng5"
Oct 31 08:32:34.323815 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:32:34.323794    4720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5xnnt\" (UniqueName: \"kubernetes.io/projected/eb737fc6-3a46-41f0-ba3b-ef2a410f3543-kube-api-access-5xnnt\") pod \"router-with-refs-pd-test-kserve-prefill-59bb995bd9-ddng5\" (UID: \"eb737fc6-3a46-41f0-ba3b-ef2a410f3543\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-59bb995bd9-ddng5"
Oct 31 08:32:34.323815 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:32:34.323816    4720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eb737fc6-3a46-41f0-ba3b-ef2a410f3543-model-cache\") pod \"router-with-refs-pd-test-kserve-prefill-59bb995bd9-ddng5\" (UID: \"eb737fc6-3a46-41f0-ba3b-ef2a410f3543\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-59bb995bd9-ddng5"
Oct 31 08:32:34.324243 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:32:34.323839    4720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eb737fc6-3a46-41f0-ba3b-ef2a410f3543-tmp-dir\") pod \"router-with-refs-pd-test-kserve-prefill-59bb995bd9-ddng5\" (UID: \"eb737fc6-3a46-41f0-ba3b-ef2a410f3543\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-59bb995bd9-ddng5"
Oct 31 08:32:34.324243 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:32:34.323857    4720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eb737fc6-3a46-41f0-ba3b-ef2a410f3543-tls-certs\") pod \"router-with-refs-pd-test-kserve-prefill-59bb995bd9-ddng5\" (UID: \"eb737fc6-3a46-41f0-ba3b-ef2a410f3543\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-59bb995bd9-ddng5"
Oct 31 08:32:34.324243 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:32:34.323885    4720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/eb737fc6-3a46-41f0-ba3b-ef2a410f3543-cabundle-cert\") pod \"router-with-refs-pd-test-kserve-prefill-59bb995bd9-ddng5\" (UID: \"eb737fc6-3a46-41f0-ba3b-ef2a410f3543\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-59bb995bd9-ddng5"
Oct 31 08:32:34.324243 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:32:34.324204    4720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/eb737fc6-3a46-41f0-ba3b-ef2a410f3543-kserve-provision-location\") pod \"router-with-refs-pd-test-kserve-prefill-59bb995bd9-ddng5\" (UID: \"eb737fc6-3a46-41f0-ba3b-ef2a410f3543\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-59bb995bd9-ddng5"
Oct 31 08:32:34.324243 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:32:34.324225    4720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/eb737fc6-3a46-41f0-ba3b-ef2a410f3543-home\") pod \"router-with-refs-pd-test-kserve-prefill-59bb995bd9-ddng5\" (UID: \"eb737fc6-3a46-41f0-ba3b-ef2a410f3543\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-59bb995bd9-ddng5"
Oct 31 08:32:34.324568 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:32:34.324296    4720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eb737fc6-3a46-41f0-ba3b-ef2a410f3543-model-cache\") pod \"router-with-refs-pd-test-kserve-prefill-59bb995bd9-ddng5\" (UID: \"eb737fc6-3a46-41f0-ba3b-ef2a410f3543\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-59bb995bd9-ddng5"
Oct 31 08:32:34.324568 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:32:34.324369    4720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eb737fc6-3a46-41f0-ba3b-ef2a410f3543-tmp-dir\") pod \"router-with-refs-pd-test-kserve-prefill-59bb995bd9-ddng5\" (UID: \"eb737fc6-3a46-41f0-ba3b-ef2a410f3543\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-59bb995bd9-ddng5"
Oct 31 08:32:34.324717 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:32:34.324698    4720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/eb737fc6-3a46-41f0-ba3b-ef2a410f3543-cabundle-cert\") pod \"router-with-refs-pd-test-kserve-prefill-59bb995bd9-ddng5\" (UID: \"eb737fc6-3a46-41f0-ba3b-ef2a410f3543\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-59bb995bd9-ddng5"
Oct 31 08:32:34.326582 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:32:34.326560    4720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/eb737fc6-3a46-41f0-ba3b-ef2a410f3543-dshm\") pod \"router-with-refs-pd-test-kserve-prefill-59bb995bd9-ddng5\" (UID: \"eb737fc6-3a46-41f0-ba3b-ef2a410f3543\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-59bb995bd9-ddng5"
Oct 31 08:32:34.326706 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:32:34.326663    4720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eb737fc6-3a46-41f0-ba3b-ef2a410f3543-tls-certs\") pod \"router-with-refs-pd-test-kserve-prefill-59bb995bd9-ddng5\" (UID: \"eb737fc6-3a46-41f0-ba3b-ef2a410f3543\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-59bb995bd9-ddng5"
Oct 31 08:32:34.380194 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:32:34.380158    4720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5xnnt\" (UniqueName: \"kubernetes.io/projected/eb737fc6-3a46-41f0-ba3b-ef2a410f3543-kube-api-access-5xnnt\") pod \"router-with-refs-pd-test-kserve-prefill-59bb995bd9-ddng5\" (UID: \"eb737fc6-3a46-41f0-ba3b-ef2a410f3543\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-59bb995bd9-ddng5"
Oct 31 08:32:34.389967 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:32:34.389937    4720 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-59bb995bd9-ddng5"
Oct 31 08:32:34.575075 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:32:34.574996    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 08:32:34.734044 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:32:34.734006    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 08:32:34.819400 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:32:34.819374    4720 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-59bb995bd9-ddng5"]
Oct 31 08:32:34.822321 ip-10-0-139-83 kubenswrapper[4720]: W1031 08:32:34.822296    4720 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeb737fc6_3a46_41f0_ba3b_ef2a410f3543.slice/crio-0c4ba94308c15c24976f5fc001a3ee5b40a368b86519dce5608c7177643580de WatchSource:0}: Error finding container 0c4ba94308c15c24976f5fc001a3ee5b40a368b86519dce5608c7177643580de: Status 404 returned error can't find the container with id 0c4ba94308c15c24976f5fc001a3ee5b40a368b86519dce5608c7177643580de
Oct 31 08:32:35.641520 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:32:35.641467    4720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-59bb995bd9-ddng5" event={"ID":"eb737fc6-3a46-41f0-ba3b-ef2a410f3543","Type":"ContainerStarted","Data":"62aef6b7986e58a989442d38eb00908c04684b43e0bb80420be8e18c2901efc3"}
Oct 31 08:32:35.642005 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:32:35.641584    4720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-59bb995bd9-ddng5" event={"ID":"eb737fc6-3a46-41f0-ba3b-ef2a410f3543","Type":"ContainerStarted","Data":"0c4ba94308c15c24976f5fc001a3ee5b40a368b86519dce5608c7177643580de"}
Oct 31 08:32:36.347030 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:32:36.346982    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 08:32:37.651905 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:32:37.651818    4720 generic.go:356] "Generic (PLEG): container finished" podID="eb737fc6-3a46-41f0-ba3b-ef2a410f3543" containerID="62aef6b7986e58a989442d38eb00908c04684b43e0bb80420be8e18c2901efc3" exitCode=0
Oct 31 08:32:37.651905 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:32:37.651884    4720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-59bb995bd9-ddng5" event={"ID":"eb737fc6-3a46-41f0-ba3b-ef2a410f3543","Type":"ContainerDied","Data":"62aef6b7986e58a989442d38eb00908c04684b43e0bb80420be8e18c2901efc3"}
Oct 31 08:32:38.659256 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:32:38.659219    4720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-59bb995bd9-ddng5" event={"ID":"eb737fc6-3a46-41f0-ba3b-ef2a410f3543","Type":"ContainerStarted","Data":"846425dc26929c11ac64014b209b2f96d382cf35ad0370efeed7510ede6be205"}
Oct 31 08:32:38.880784 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:32:38.880724    4720 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-59bb995bd9-ddng5" podStartSLOduration=3.399904962 podStartE2EDuration="5.88070818s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.480803218s" isStatefulPod=false podCreationTimestamp="2026-10-31 08:32:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 08:32:38.877605181 +0000 UTC m=+3813.008673274" watchObservedRunningTime="2026-10-31 08:32:38.88070818 +0000 UTC m=+3813.011776241"
Oct 31 08:32:43.461979 ip-10-0-139-83 kubenswrapper[4720]: E1031 08:32:43.461931    4720 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8m768" containerName="extractor"
Oct 31 08:32:44.390221 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:32:44.390177    4720 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-59bb995bd9-ddng5"
Oct 31 08:32:44.390461 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:32:44.390260    4720 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-59bb995bd9-ddng5"
Oct 31 08:32:44.391839 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:32:44.391806    4720 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-59bb995bd9-ddng5" podUID="eb737fc6-3a46-41f0-ba3b-ef2a410f3543" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/health\": dial tcp 10.132.0.74:8000: connect: connection refused"
Oct 31 08:32:54.391267 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:32:54.391214    4720 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-59bb995bd9-ddng5" podUID="eb737fc6-3a46-41f0-ba3b-ef2a410f3543" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/health\": dial tcp 10.132.0.74:8000: connect: connection refused"
Oct 31 08:33:01.143748 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:33:01.143707    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 08:33:01.400373 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:33:01.400261    4720 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-with-lora-adapters-3cbf5e9c_lora-single-adapter-test-kserve-7d79587f48-2js5x_8358054c-61a6-4ac7-b479-b9e9da9724f2/storage-initializer/0.log"
Oct 31 08:33:01.518976 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:33:01.518937    4720 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-with-lora-adapters-3cbf5e9c_lora-single-adapter-test-kserve-7d79587f48-2js5x_8358054c-61a6-4ac7-b479-b9e9da9724f2/main/0.log"
Oct 31 08:33:04.390480 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:33:04.390441    4720 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-59bb995bd9-ddng5" podUID="eb737fc6-3a46-41f0-ba3b-ef2a410f3543" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/health\": dial tcp 10.132.0.74:8000: connect: connection refused"
Oct 31 08:33:07.654058 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:33:07.654016    4720 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-7d79587f48-2js5x"]
Oct 31 08:33:07.690257 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:33:07.690213    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 08:33:13.462409 ip-10-0-139-83 kubenswrapper[4720]: E1031 08:33:13.462362    4720 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8m768" containerName="extractor"
Oct 31 08:33:14.391531 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:33:14.391485    4720 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-59bb995bd9-ddng5" podUID="eb737fc6-3a46-41f0-ba3b-ef2a410f3543" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/health\": dial tcp 10.132.0.74:8000: connect: connection refused"
Oct 31 08:33:19.317870 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:33:19.317828    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 08:33:22.079849 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:33:22.079820    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 08:33:22.174928 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:33:22.174893    4720 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-56f46b8cdd-9hmnc"]
Oct 31 08:33:22.175184 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:33:22.175159    4720 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/openshift-ai-inference-openshift-default-56f46b8cdd-9hmnc" podUID="4d6179bf-191a-4085-8742-fe4ba603e788" containerName="istio-proxy" containerID="cri-o://044ba974d734f848bacbca636742c7617e24931e4e790195866362b59f31b875" gracePeriod=30
Oct 31 08:33:22.677475 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:33:22.677432    4720 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-7d79587f48-2js5x" podUID="8358054c-61a6-4ac7-b479-b9e9da9724f2" containerName="main" containerID="cri-o://53285d4d8195f0846e86cc450f8296579813a8bc40d48e8323ba52c04ea33de9" gracePeriod=45
Oct 31 08:33:22.953141 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:33:22.953040    4720 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-7d79587f48-2js5x" podUID="8358054c-61a6-4ac7-b479-b9e9da9724f2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/health\": dial tcp 10.132.0.73:8000: connect: connection refused"
Oct 31 08:33:23.357697 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:33:23.357663    4720 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-7d79587f48-2js5x"
Oct 31 08:33:23.380650 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:33:23.380621    4720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8358054c-61a6-4ac7-b479-b9e9da9724f2-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8358054c-61a6-4ac7-b479-b9e9da9724f2-tmp-dir\") pod \"8358054c-61a6-4ac7-b479-b9e9da9724f2\" (UID: \"8358054c-61a6-4ac7-b479-b9e9da9724f2\") "
Oct 31 08:33:23.380785 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:33:23.380674    4720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8358054c-61a6-4ac7-b479-b9e9da9724f2-tls-certs\" (UniqueName: \"kubernetes.io/secret/8358054c-61a6-4ac7-b479-b9e9da9724f2-tls-certs\") pod \"8358054c-61a6-4ac7-b479-b9e9da9724f2\" (UID: \"8358054c-61a6-4ac7-b479-b9e9da9724f2\") "
Oct 31 08:33:23.380785 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:33:23.380720    4720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8358054c-61a6-4ac7-b479-b9e9da9724f2-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8358054c-61a6-4ac7-b479-b9e9da9724f2-kserve-provision-location\") pod \"8358054c-61a6-4ac7-b479-b9e9da9724f2\" (UID: \"8358054c-61a6-4ac7-b479-b9e9da9724f2\") "
Oct 31 08:33:23.380785 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:33:23.380746    4720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8358054c-61a6-4ac7-b479-b9e9da9724f2-home\" (UniqueName: \"kubernetes.io/empty-dir/8358054c-61a6-4ac7-b479-b9e9da9724f2-home\") pod \"8358054c-61a6-4ac7-b479-b9e9da9724f2\" (UID: \"8358054c-61a6-4ac7-b479-b9e9da9724f2\") "
Oct 31 08:33:23.380785 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:33:23.380768    4720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8358054c-61a6-4ac7-b479-b9e9da9724f2-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8358054c-61a6-4ac7-b479-b9e9da9724f2-cabundle-cert\") pod \"8358054c-61a6-4ac7-b479-b9e9da9724f2\" (UID: \"8358054c-61a6-4ac7-b479-b9e9da9724f2\") "
Oct 31 08:33:23.381006 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:33:23.380861    4720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8358054c-61a6-4ac7-b479-b9e9da9724f2-kube-api-access-wmwcj\" (UniqueName: \"kubernetes.io/projected/8358054c-61a6-4ac7-b479-b9e9da9724f2-kube-api-access-wmwcj\") pod \"8358054c-61a6-4ac7-b479-b9e9da9724f2\" (UID: \"8358054c-61a6-4ac7-b479-b9e9da9724f2\") "
Oct 31 08:33:23.381006 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:33:23.380924    4720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8358054c-61a6-4ac7-b479-b9e9da9724f2-dshm\" (UniqueName: \"kubernetes.io/empty-dir/8358054c-61a6-4ac7-b479-b9e9da9724f2-dshm\") pod \"8358054c-61a6-4ac7-b479-b9e9da9724f2\" (UID: \"8358054c-61a6-4ac7-b479-b9e9da9724f2\") "
Oct 31 08:33:23.381006 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:33:23.380952    4720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8358054c-61a6-4ac7-b479-b9e9da9724f2-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8358054c-61a6-4ac7-b479-b9e9da9724f2-model-cache\") pod \"8358054c-61a6-4ac7-b479-b9e9da9724f2\" (UID: \"8358054c-61a6-4ac7-b479-b9e9da9724f2\") "
Oct 31 08:33:23.381358 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:33:23.381184    4720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8358054c-61a6-4ac7-b479-b9e9da9724f2-cabundle-cert" pod "8358054c-61a6-4ac7-b479-b9e9da9724f2" (UID: "8358054c-61a6-4ac7-b479-b9e9da9724f2"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 08:33:23.381358 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:33:23.381288    4720 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8358054c-61a6-4ac7-b479-b9e9da9724f2-cabundle-cert\") on node \"ip-10-0-139-83.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 08:33:23.381358 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:33:23.381184    4720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8358054c-61a6-4ac7-b479-b9e9da9724f2-home" pod "8358054c-61a6-4ac7-b479-b9e9da9724f2" (UID: "8358054c-61a6-4ac7-b479-b9e9da9724f2"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 08:33:23.381829 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:33:23.381672    4720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8358054c-61a6-4ac7-b479-b9e9da9724f2-model-cache" pod "8358054c-61a6-4ac7-b479-b9e9da9724f2" (UID: "8358054c-61a6-4ac7-b479-b9e9da9724f2"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 08:33:23.383440 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:33:23.383412    4720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8358054c-61a6-4ac7-b479-b9e9da9724f2-tls-certs" pod "8358054c-61a6-4ac7-b479-b9e9da9724f2" (UID: "8358054c-61a6-4ac7-b479-b9e9da9724f2"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 08:33:23.383530 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:33:23.383482    4720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8358054c-61a6-4ac7-b479-b9e9da9724f2-dshm" pod "8358054c-61a6-4ac7-b479-b9e9da9724f2" (UID: "8358054c-61a6-4ac7-b479-b9e9da9724f2"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 08:33:23.383949 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:33:23.383910    4720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8358054c-61a6-4ac7-b479-b9e9da9724f2-kube-api-access-wmwcj" pod "8358054c-61a6-4ac7-b479-b9e9da9724f2" (UID: "8358054c-61a6-4ac7-b479-b9e9da9724f2"). InnerVolumeSpecName "kube-api-access-wmwcj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 08:33:23.392784 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:33:23.392756    4720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8358054c-61a6-4ac7-b479-b9e9da9724f2-tmp-dir" pod "8358054c-61a6-4ac7-b479-b9e9da9724f2" (UID: "8358054c-61a6-4ac7-b479-b9e9da9724f2"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 08:33:23.414223 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:33:23.414177    4720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8358054c-61a6-4ac7-b479-b9e9da9724f2-kserve-provision-location" pod "8358054c-61a6-4ac7-b479-b9e9da9724f2" (UID: "8358054c-61a6-4ac7-b479-b9e9da9724f2"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 08:33:23.481996 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:33:23.481961    4720 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8358054c-61a6-4ac7-b479-b9e9da9724f2-kserve-provision-location\") on node \"ip-10-0-139-83.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 08:33:23.481996 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:33:23.481990    4720 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8358054c-61a6-4ac7-b479-b9e9da9724f2-home\") on node \"ip-10-0-139-83.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 08:33:23.481996 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:33:23.482000    4720 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wmwcj\" (UniqueName: \"kubernetes.io/projected/8358054c-61a6-4ac7-b479-b9e9da9724f2-kube-api-access-wmwcj\") on node \"ip-10-0-139-83.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 08:33:23.482231 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:33:23.482008    4720 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8358054c-61a6-4ac7-b479-b9e9da9724f2-dshm\") on node \"ip-10-0-139-83.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 08:33:23.482231 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:33:23.482017    4720 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8358054c-61a6-4ac7-b479-b9e9da9724f2-model-cache\") on node \"ip-10-0-139-83.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 08:33:23.482231 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:33:23.482026    4720 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8358054c-61a6-4ac7-b479-b9e9da9724f2-tmp-dir\") on node \"ip-10-0-139-83.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 08:33:23.482231 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:33:23.482035    4720 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8358054c-61a6-4ac7-b479-b9e9da9724f2-tls-certs\") on node \"ip-10-0-139-83.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 08:33:23.890954 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:33:23.890919    4720 generic.go:356] "Generic (PLEG): container finished" podID="8358054c-61a6-4ac7-b479-b9e9da9724f2" containerID="53285d4d8195f0846e86cc450f8296579813a8bc40d48e8323ba52c04ea33de9" exitCode=0
Oct 31 08:33:23.891162 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:33:23.890986    4720 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-7d79587f48-2js5x"
Oct 31 08:33:23.891162 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:33:23.890989    4720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-7d79587f48-2js5x" event={"ID":"8358054c-61a6-4ac7-b479-b9e9da9724f2","Type":"ContainerDied","Data":"53285d4d8195f0846e86cc450f8296579813a8bc40d48e8323ba52c04ea33de9"}
Oct 31 08:33:23.891162 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:33:23.891049    4720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-7d79587f48-2js5x" event={"ID":"8358054c-61a6-4ac7-b479-b9e9da9724f2","Type":"ContainerDied","Data":"d9182c8997efaf48a91f9627b33e355b049864505692facaf3b67a02c5f6ce42"}
Oct 31 08:33:23.891162 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:33:23.891075    4720 scope.go:122] "RemoveContainer" containerID="53285d4d8195f0846e86cc450f8296579813a8bc40d48e8323ba52c04ea33de9"
Oct 31 08:33:23.905159 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:33:23.905133    4720 scope.go:122] "RemoveContainer" containerID="87b14dc1df211ffc745b227bb0929efac9760cc1d0565bbbcee56d9e029d32a2"
Oct 31 08:33:23.914166 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:33:23.914140    4720 scope.go:122] "RemoveContainer" containerID="53285d4d8195f0846e86cc450f8296579813a8bc40d48e8323ba52c04ea33de9"
Oct 31 08:33:23.914490 ip-10-0-139-83 kubenswrapper[4720]: E1031 08:33:23.914467    4720 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"53285d4d8195f0846e86cc450f8296579813a8bc40d48e8323ba52c04ea33de9\": container with ID starting with 53285d4d8195f0846e86cc450f8296579813a8bc40d48e8323ba52c04ea33de9 not found: ID does not exist" containerID="53285d4d8195f0846e86cc450f8296579813a8bc40d48e8323ba52c04ea33de9"
Oct 31 08:33:23.914544 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:33:23.914500    4720 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"53285d4d8195f0846e86cc450f8296579813a8bc40d48e8323ba52c04ea33de9"} err="failed to get container status \"53285d4d8195f0846e86cc450f8296579813a8bc40d48e8323ba52c04ea33de9\": rpc error: code = NotFound desc = could not find container \"53285d4d8195f0846e86cc450f8296579813a8bc40d48e8323ba52c04ea33de9\": container with ID starting with 53285d4d8195f0846e86cc450f8296579813a8bc40d48e8323ba52c04ea33de9 not found: ID does not exist"
Oct 31 08:33:23.914544 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:33:23.914520    4720 scope.go:122] "RemoveContainer" containerID="87b14dc1df211ffc745b227bb0929efac9760cc1d0565bbbcee56d9e029d32a2"
Oct 31 08:33:23.914827 ip-10-0-139-83 kubenswrapper[4720]: E1031 08:33:23.914807    4720 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"87b14dc1df211ffc745b227bb0929efac9760cc1d0565bbbcee56d9e029d32a2\": container with ID starting with 87b14dc1df211ffc745b227bb0929efac9760cc1d0565bbbcee56d9e029d32a2 not found: ID does not exist" containerID="87b14dc1df211ffc745b227bb0929efac9760cc1d0565bbbcee56d9e029d32a2"
Oct 31 08:33:23.914891 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:33:23.914833    4720 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"87b14dc1df211ffc745b227bb0929efac9760cc1d0565bbbcee56d9e029d32a2"} err="failed to get container status \"87b14dc1df211ffc745b227bb0929efac9760cc1d0565bbbcee56d9e029d32a2\": rpc error: code = NotFound desc = could not find container \"87b14dc1df211ffc745b227bb0929efac9760cc1d0565bbbcee56d9e029d32a2\": container with ID starting with 87b14dc1df211ffc745b227bb0929efac9760cc1d0565bbbcee56d9e029d32a2 not found: ID does not exist"
Oct 31 08:33:24.055674 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:33:24.055633    4720 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-7d79587f48-2js5x"]
Oct 31 08:33:24.058893 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:33:24.058864    4720 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-7d79587f48-2js5x"]
Oct 31 08:33:24.390633 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:33:24.390588    4720 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-59bb995bd9-ddng5" podUID="eb737fc6-3a46-41f0-ba3b-ef2a410f3543" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/health\": dial tcp 10.132.0.74:8000: connect: connection refused"
Oct 31 08:33:26.026559 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:33:26.026527    4720 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8358054c-61a6-4ac7-b479-b9e9da9724f2" path="/var/lib/kubelet/pods/8358054c-61a6-4ac7-b479-b9e9da9724f2/volumes"
Oct 31 08:33:27.450217 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:33:27.450153    4720 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-default-56f46b8cdd-9hmnc"
Oct 31 08:33:27.515021 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:33:27.514984    4720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4d6179bf-191a-4085-8742-fe4ba603e788-istio-token\" (UniqueName: \"kubernetes.io/projected/4d6179bf-191a-4085-8742-fe4ba603e788-istio-token\") pod \"4d6179bf-191a-4085-8742-fe4ba603e788\" (UID: \"4d6179bf-191a-4085-8742-fe4ba603e788\") "
Oct 31 08:33:27.515188 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:33:27.515039    4720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4d6179bf-191a-4085-8742-fe4ba603e788-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d6179bf-191a-4085-8742-fe4ba603e788-workload-socket\") pod \"4d6179bf-191a-4085-8742-fe4ba603e788\" (UID: \"4d6179bf-191a-4085-8742-fe4ba603e788\") "
Oct 31 08:33:27.515188 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:33:27.515073    4720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4d6179bf-191a-4085-8742-fe4ba603e788-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4d6179bf-191a-4085-8742-fe4ba603e788-istio-envoy\") pod \"4d6179bf-191a-4085-8742-fe4ba603e788\" (UID: \"4d6179bf-191a-4085-8742-fe4ba603e788\") "
Oct 31 08:33:27.515188 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:33:27.515111    4720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4d6179bf-191a-4085-8742-fe4ba603e788-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4d6179bf-191a-4085-8742-fe4ba603e788-istio-podinfo\") pod \"4d6179bf-191a-4085-8742-fe4ba603e788\" (UID: \"4d6179bf-191a-4085-8742-fe4ba603e788\") "
Oct 31 08:33:27.515188 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:33:27.515129    4720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4d6179bf-191a-4085-8742-fe4ba603e788-kube-api-access-vr8sd\" (UniqueName: \"kubernetes.io/projected/4d6179bf-191a-4085-8742-fe4ba603e788-kube-api-access-vr8sd\") pod \"4d6179bf-191a-4085-8742-fe4ba603e788\" (UID: \"4d6179bf-191a-4085-8742-fe4ba603e788\") "
Oct 31 08:33:27.515188 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:33:27.515159    4720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4d6179bf-191a-4085-8742-fe4ba603e788-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4d6179bf-191a-4085-8742-fe4ba603e788-istiod-ca-cert\") pod \"4d6179bf-191a-4085-8742-fe4ba603e788\" (UID: \"4d6179bf-191a-4085-8742-fe4ba603e788\") "
Oct 31 08:33:27.515497 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:33:27.515209    4720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4d6179bf-191a-4085-8742-fe4ba603e788-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4d6179bf-191a-4085-8742-fe4ba603e788-istio-data\") pod \"4d6179bf-191a-4085-8742-fe4ba603e788\" (UID: \"4d6179bf-191a-4085-8742-fe4ba603e788\") "
Oct 31 08:33:27.515497 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:33:27.515265    4720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4d6179bf-191a-4085-8742-fe4ba603e788-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d6179bf-191a-4085-8742-fe4ba603e788-credential-socket\") pod \"4d6179bf-191a-4085-8742-fe4ba603e788\" (UID: \"4d6179bf-191a-4085-8742-fe4ba603e788\") "
Oct 31 08:33:27.515497 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:33:27.515306    4720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4d6179bf-191a-4085-8742-fe4ba603e788-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4d6179bf-191a-4085-8742-fe4ba603e788-workload-certs\") pod \"4d6179bf-191a-4085-8742-fe4ba603e788\" (UID: \"4d6179bf-191a-4085-8742-fe4ba603e788\") "
Oct 31 08:33:27.515658 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:33:27.515498    4720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d6179bf-191a-4085-8742-fe4ba603e788-workload-socket" pod "4d6179bf-191a-4085-8742-fe4ba603e788" (UID: "4d6179bf-191a-4085-8742-fe4ba603e788"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 08:33:27.515779 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:33:27.515759    4720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d6179bf-191a-4085-8742-fe4ba603e788-credential-socket" pod "4d6179bf-191a-4085-8742-fe4ba603e788" (UID: "4d6179bf-191a-4085-8742-fe4ba603e788"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 08:33:27.515949 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:33:27.515856    4720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4d6179bf-191a-4085-8742-fe4ba603e788-istiod-ca-cert" pod "4d6179bf-191a-4085-8742-fe4ba603e788" (UID: "4d6179bf-191a-4085-8742-fe4ba603e788"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 08:33:27.516472 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:33:27.516398    4720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d6179bf-191a-4085-8742-fe4ba603e788-istio-data" pod "4d6179bf-191a-4085-8742-fe4ba603e788" (UID: "4d6179bf-191a-4085-8742-fe4ba603e788"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 08:33:27.516472 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:33:27.516435    4720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d6179bf-191a-4085-8742-fe4ba603e788-workload-certs" pod "4d6179bf-191a-4085-8742-fe4ba603e788" (UID: "4d6179bf-191a-4085-8742-fe4ba603e788"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 08:33:27.517824 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:33:27.517795    4720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4d6179bf-191a-4085-8742-fe4ba603e788-istio-podinfo" pod "4d6179bf-191a-4085-8742-fe4ba603e788" (UID: "4d6179bf-191a-4085-8742-fe4ba603e788"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 08:33:27.517824 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:33:27.517802    4720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d6179bf-191a-4085-8742-fe4ba603e788-istio-envoy" pod "4d6179bf-191a-4085-8742-fe4ba603e788" (UID: "4d6179bf-191a-4085-8742-fe4ba603e788"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 08:33:27.517977 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:33:27.517914    4720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4d6179bf-191a-4085-8742-fe4ba603e788-istio-token" pod "4d6179bf-191a-4085-8742-fe4ba603e788" (UID: "4d6179bf-191a-4085-8742-fe4ba603e788"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 08:33:27.518149 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:33:27.518134    4720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4d6179bf-191a-4085-8742-fe4ba603e788-kube-api-access-vr8sd" pod "4d6179bf-191a-4085-8742-fe4ba603e788" (UID: "4d6179bf-191a-4085-8742-fe4ba603e788"). InnerVolumeSpecName "kube-api-access-vr8sd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 08:33:27.616828 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:33:27.616673    4720 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4d6179bf-191a-4085-8742-fe4ba603e788-istio-data\") on node \"ip-10-0-139-83.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 08:33:27.616828 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:33:27.616710    4720 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d6179bf-191a-4085-8742-fe4ba603e788-credential-socket\") on node \"ip-10-0-139-83.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 08:33:27.616828 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:33:27.616725    4720 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4d6179bf-191a-4085-8742-fe4ba603e788-workload-certs\") on node \"ip-10-0-139-83.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 08:33:27.616828 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:33:27.616739    4720 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4d6179bf-191a-4085-8742-fe4ba603e788-istio-token\") on node \"ip-10-0-139-83.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 08:33:27.616828 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:33:27.616752    4720 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d6179bf-191a-4085-8742-fe4ba603e788-workload-socket\") on node \"ip-10-0-139-83.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 08:33:27.616828 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:33:27.616765    4720 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4d6179bf-191a-4085-8742-fe4ba603e788-istio-envoy\") on node \"ip-10-0-139-83.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 08:33:27.616828 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:33:27.616776    4720 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4d6179bf-191a-4085-8742-fe4ba603e788-istio-podinfo\") on node \"ip-10-0-139-83.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 08:33:27.616828 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:33:27.616788    4720 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vr8sd\" (UniqueName: \"kubernetes.io/projected/4d6179bf-191a-4085-8742-fe4ba603e788-kube-api-access-vr8sd\") on node \"ip-10-0-139-83.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 08:33:27.616828 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:33:27.616803    4720 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4d6179bf-191a-4085-8742-fe4ba603e788-istiod-ca-cert\") on node \"ip-10-0-139-83.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 08:33:27.911116 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:33:27.911079    4720 generic.go:356] "Generic (PLEG): container finished" podID="4d6179bf-191a-4085-8742-fe4ba603e788" containerID="044ba974d734f848bacbca636742c7617e24931e4e790195866362b59f31b875" exitCode=0
Oct 31 08:33:27.911285 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:33:27.911127    4720 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-default-56f46b8cdd-9hmnc"
Oct 31 08:33:27.911285 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:33:27.911151    4720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-56f46b8cdd-9hmnc" event={"ID":"4d6179bf-191a-4085-8742-fe4ba603e788","Type":"ContainerDied","Data":"044ba974d734f848bacbca636742c7617e24931e4e790195866362b59f31b875"}
Oct 31 08:33:27.911285 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:33:27.911207    4720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-56f46b8cdd-9hmnc" event={"ID":"4d6179bf-191a-4085-8742-fe4ba603e788","Type":"ContainerDied","Data":"27ac8be4c0c5c3dc929f07f7a212d95e594ac9498341d7c3ea760a1bf3033fcf"}
Oct 31 08:33:27.911285 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:33:27.911232    4720 scope.go:122] "RemoveContainer" containerID="044ba974d734f848bacbca636742c7617e24931e4e790195866362b59f31b875"
Oct 31 08:33:27.924405 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:33:27.924374    4720 scope.go:122] "RemoveContainer" containerID="044ba974d734f848bacbca636742c7617e24931e4e790195866362b59f31b875"
Oct 31 08:33:27.924788 ip-10-0-139-83 kubenswrapper[4720]: E1031 08:33:27.924763    4720 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"044ba974d734f848bacbca636742c7617e24931e4e790195866362b59f31b875\": container with ID starting with 044ba974d734f848bacbca636742c7617e24931e4e790195866362b59f31b875 not found: ID does not exist" containerID="044ba974d734f848bacbca636742c7617e24931e4e790195866362b59f31b875"
Oct 31 08:33:27.924858 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:33:27.924796    4720 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"044ba974d734f848bacbca636742c7617e24931e4e790195866362b59f31b875"} err="failed to get container status \"044ba974d734f848bacbca636742c7617e24931e4e790195866362b59f31b875\": rpc error: code = NotFound desc = could not find container \"044ba974d734f848bacbca636742c7617e24931e4e790195866362b59f31b875\": container with ID starting with 044ba974d734f848bacbca636742c7617e24931e4e790195866362b59f31b875 not found: ID does not exist"
Oct 31 08:33:27.933887 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:33:27.933856    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 08:33:28.079311 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:33:28.079271    4720 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-56f46b8cdd-9hmnc"]
Oct 31 08:33:28.103938 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:33:28.103905    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 08:33:28.128241 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:33:28.128205    4720 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-56f46b8cdd-9hmnc"]
Oct 31 08:33:30.025748 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:33:30.025716    4720 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4d6179bf-191a-4085-8742-fe4ba603e788" path="/var/lib/kubelet/pods/4d6179bf-191a-4085-8742-fe4ba603e788/volumes"
Oct 31 08:33:32.814730 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:33:32.814695    4720 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-769c5d65fc-jm59l"]
Oct 31 08:33:32.815235 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:33:32.815212    4720 cpu_manager.go:414] "containerMap: removing container" podUID="8358054c-61a6-4ac7-b479-b9e9da9724f2" containerName="storage-initializer"
Oct 31 08:33:32.815316 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:33:32.815240    4720 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8358054c-61a6-4ac7-b479-b9e9da9724f2" containerName="storage-initializer"
Oct 31 08:33:32.815316 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:33:32.815258    4720 cpu_manager.go:414] "containerMap: removing container" podUID="4d6179bf-191a-4085-8742-fe4ba603e788" containerName="istio-proxy"
Oct 31 08:33:32.815316 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:33:32.815266    4720 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4d6179bf-191a-4085-8742-fe4ba603e788" containerName="istio-proxy"
Oct 31 08:33:32.815316 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:33:32.815277    4720 cpu_manager.go:414] "containerMap: removing container" podUID="8358054c-61a6-4ac7-b479-b9e9da9724f2" containerName="main"
Oct 31 08:33:32.815316 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:33:32.815285    4720 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8358054c-61a6-4ac7-b479-b9e9da9724f2" containerName="main"
Oct 31 08:33:32.815596 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:33:32.815423    4720 memory_manager.go:365] "RemoveStaleState removing state" podUID="4d6179bf-191a-4085-8742-fe4ba603e788" containerName="istio-proxy"
Oct 31 08:33:32.815596 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:33:32.815437    4720 memory_manager.go:365] "RemoveStaleState removing state" podUID="8358054c-61a6-4ac7-b479-b9e9da9724f2" containerName="main"
Oct 31 08:33:32.820192 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:33:32.820164    4720 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-769c5d65fc-jm59l"
Oct 31 08:33:32.865048 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:33:32.865012    4720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c43713ad-58fa-403c-ac6f-792caa306ab5-model-cache\") pod \"lora-multiple-adapters-test-kserve-769c5d65fc-jm59l\" (UID: \"c43713ad-58fa-403c-ac6f-792caa306ab5\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-769c5d65fc-jm59l"
Oct 31 08:33:32.865211 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:33:32.865051    4720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c43713ad-58fa-403c-ac6f-792caa306ab5-tls-certs\") pod \"lora-multiple-adapters-test-kserve-769c5d65fc-jm59l\" (UID: \"c43713ad-58fa-403c-ac6f-792caa306ab5\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-769c5d65fc-jm59l"
Oct 31 08:33:32.865211 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:33:32.865080    4720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c43713ad-58fa-403c-ac6f-792caa306ab5-dshm\") pod \"lora-multiple-adapters-test-kserve-769c5d65fc-jm59l\" (UID: \"c43713ad-58fa-403c-ac6f-792caa306ab5\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-769c5d65fc-jm59l"
Oct 31 08:33:32.865211 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:33:32.865186    4720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c43713ad-58fa-403c-ac6f-792caa306ab5-kserve-provision-location\") pod \"lora-multiple-adapters-test-kserve-769c5d65fc-jm59l\" (UID: \"c43713ad-58fa-403c-ac6f-792caa306ab5\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-769c5d65fc-jm59l"
Oct 31 08:33:32.865405 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:33:32.865378    4720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c43713ad-58fa-403c-ac6f-792caa306ab5-home\") pod \"lora-multiple-adapters-test-kserve-769c5d65fc-jm59l\" (UID: \"c43713ad-58fa-403c-ac6f-792caa306ab5\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-769c5d65fc-jm59l"
Oct 31 08:33:32.865450 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:33:32.865422    4720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c43713ad-58fa-403c-ac6f-792caa306ab5-tmp-dir\") pod \"lora-multiple-adapters-test-kserve-769c5d65fc-jm59l\" (UID: \"c43713ad-58fa-403c-ac6f-792caa306ab5\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-769c5d65fc-jm59l"
Oct 31 08:33:32.865530 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:33:32.865512    4720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c43713ad-58fa-403c-ac6f-792caa306ab5-cabundle-cert\") pod \"lora-multiple-adapters-test-kserve-769c5d65fc-jm59l\" (UID: \"c43713ad-58fa-403c-ac6f-792caa306ab5\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-769c5d65fc-jm59l"
Oct 31 08:33:32.865590 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:33:32.865573    4720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8kpwq\" (UniqueName: \"kubernetes.io/projected/c43713ad-58fa-403c-ac6f-792caa306ab5-kube-api-access-8kpwq\") pod \"lora-multiple-adapters-test-kserve-769c5d65fc-jm59l\" (UID: \"c43713ad-58fa-403c-ac6f-792caa306ab5\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-769c5d65fc-jm59l"
Oct 31 08:33:32.873553 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:33:32.873524    4720 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-769c5d65fc-jm59l"]
Oct 31 08:33:32.889051 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:33:32.889015    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 08:33:32.919215 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:33:32.919137    4720 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-with-lora-adapters-bbeb2681\"/\"seaweedfs-s3-creds\""
Oct 31 08:33:32.919215 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:33:32.919180    4720 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-with-lora-adapters-bbeb2681\"/\"odh-kserve-custom-ca-bundle\""
Oct 31 08:33:32.919215 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:33:32.919190    4720 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-with-lora-adapters-bbeb2681\"/\"default-dockercfg-cddrx\""
Oct 31 08:33:32.919482 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:33:32.919216    4720 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-with-lora-adapters-bbeb2681\"/\"kube-root-ca.crt\""
Oct 31 08:33:32.919482 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:33:32.919229    4720 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-with-lora-adapters-bbeb2681\"/\"lora-multiple-adapters-test-kserve-self-signed-certs\""
Oct 31 08:33:32.919482 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:33:32.919259    4720 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-with-lora-adapters-bbeb2681\"/\"openshift-service-ca.crt\""
Oct 31 08:33:32.967008 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:33:32.966977    4720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c43713ad-58fa-403c-ac6f-792caa306ab5-home\") pod \"lora-multiple-adapters-test-kserve-769c5d65fc-jm59l\" (UID: \"c43713ad-58fa-403c-ac6f-792caa306ab5\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-769c5d65fc-jm59l"
Oct 31 08:33:32.967008 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:33:32.967011    4720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c43713ad-58fa-403c-ac6f-792caa306ab5-tmp-dir\") pod \"lora-multiple-adapters-test-kserve-769c5d65fc-jm59l\" (UID: \"c43713ad-58fa-403c-ac6f-792caa306ab5\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-769c5d65fc-jm59l"
Oct 31 08:33:32.967262 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:33:32.967041    4720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c43713ad-58fa-403c-ac6f-792caa306ab5-cabundle-cert\") pod \"lora-multiple-adapters-test-kserve-769c5d65fc-jm59l\" (UID: \"c43713ad-58fa-403c-ac6f-792caa306ab5\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-769c5d65fc-jm59l"
Oct 31 08:33:32.967262 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:33:32.967088    4720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8kpwq\" (UniqueName: \"kubernetes.io/projected/c43713ad-58fa-403c-ac6f-792caa306ab5-kube-api-access-8kpwq\") pod \"lora-multiple-adapters-test-kserve-769c5d65fc-jm59l\" (UID: \"c43713ad-58fa-403c-ac6f-792caa306ab5\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-769c5d65fc-jm59l"
Oct 31 08:33:32.967262 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:33:32.967118    4720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c43713ad-58fa-403c-ac6f-792caa306ab5-model-cache\") pod \"lora-multiple-adapters-test-kserve-769c5d65fc-jm59l\" (UID: \"c43713ad-58fa-403c-ac6f-792caa306ab5\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-769c5d65fc-jm59l"
Oct 31 08:33:32.967262 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:33:32.967143    4720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c43713ad-58fa-403c-ac6f-792caa306ab5-tls-certs\") pod \"lora-multiple-adapters-test-kserve-769c5d65fc-jm59l\" (UID: \"c43713ad-58fa-403c-ac6f-792caa306ab5\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-769c5d65fc-jm59l"
Oct 31 08:33:32.967262 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:33:32.967164    4720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c43713ad-58fa-403c-ac6f-792caa306ab5-dshm\") pod \"lora-multiple-adapters-test-kserve-769c5d65fc-jm59l\" (UID: \"c43713ad-58fa-403c-ac6f-792caa306ab5\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-769c5d65fc-jm59l"
Oct 31 08:33:32.967262 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:33:32.967201    4720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c43713ad-58fa-403c-ac6f-792caa306ab5-kserve-provision-location\") pod \"lora-multiple-adapters-test-kserve-769c5d65fc-jm59l\" (UID: \"c43713ad-58fa-403c-ac6f-792caa306ab5\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-769c5d65fc-jm59l"
Oct 31 08:33:32.967613 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:33:32.967569    4720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c43713ad-58fa-403c-ac6f-792caa306ab5-tmp-dir\") pod \"lora-multiple-adapters-test-kserve-769c5d65fc-jm59l\" (UID: \"c43713ad-58fa-403c-ac6f-792caa306ab5\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-769c5d65fc-jm59l"
Oct 31 08:33:32.967613 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:33:32.967581    4720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c43713ad-58fa-403c-ac6f-792caa306ab5-home\") pod \"lora-multiple-adapters-test-kserve-769c5d65fc-jm59l\" (UID: \"c43713ad-58fa-403c-ac6f-792caa306ab5\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-769c5d65fc-jm59l"
Oct 31 08:33:32.967700 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:33:32.967636    4720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c43713ad-58fa-403c-ac6f-792caa306ab5-model-cache\") pod \"lora-multiple-adapters-test-kserve-769c5d65fc-jm59l\" (UID: \"c43713ad-58fa-403c-ac6f-792caa306ab5\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-769c5d65fc-jm59l"
Oct 31 08:33:32.967742 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:33:32.967722    4720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c43713ad-58fa-403c-ac6f-792caa306ab5-kserve-provision-location\") pod \"lora-multiple-adapters-test-kserve-769c5d65fc-jm59l\" (UID: \"c43713ad-58fa-403c-ac6f-792caa306ab5\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-769c5d65fc-jm59l"
Oct 31 08:33:32.967958 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:33:32.967940    4720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c43713ad-58fa-403c-ac6f-792caa306ab5-cabundle-cert\") pod \"lora-multiple-adapters-test-kserve-769c5d65fc-jm59l\" (UID: \"c43713ad-58fa-403c-ac6f-792caa306ab5\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-769c5d65fc-jm59l"
Oct 31 08:33:32.969962 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:33:32.969940    4720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c43713ad-58fa-403c-ac6f-792caa306ab5-dshm\") pod \"lora-multiple-adapters-test-kserve-769c5d65fc-jm59l\" (UID: \"c43713ad-58fa-403c-ac6f-792caa306ab5\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-769c5d65fc-jm59l"
Oct 31 08:33:32.970296 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:33:32.970271    4720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c43713ad-58fa-403c-ac6f-792caa306ab5-tls-certs\") pod \"lora-multiple-adapters-test-kserve-769c5d65fc-jm59l\" (UID: \"c43713ad-58fa-403c-ac6f-792caa306ab5\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-769c5d65fc-jm59l"
Oct 31 08:33:33.023221 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:33:33.023189    4720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8kpwq\" (UniqueName: \"kubernetes.io/projected/c43713ad-58fa-403c-ac6f-792caa306ab5-kube-api-access-8kpwq\") pod \"lora-multiple-adapters-test-kserve-769c5d65fc-jm59l\" (UID: \"c43713ad-58fa-403c-ac6f-792caa306ab5\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-769c5d65fc-jm59l"
Oct 31 08:33:33.136506 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:33:33.136466    4720 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-769c5d65fc-jm59l"
Oct 31 08:33:33.568687 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:33:33.568657    4720 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-769c5d65fc-jm59l"]
Oct 31 08:33:33.571340 ip-10-0-139-83 kubenswrapper[4720]: W1031 08:33:33.571288    4720 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc43713ad_58fa_403c_ac6f_792caa306ab5.slice/crio-d9741757a02ee5159e61877b4be2acaf78e2e2c714e090c2c393fde20593bafe WatchSource:0}: Error finding container d9741757a02ee5159e61877b4be2acaf78e2e2c714e090c2c393fde20593bafe: Status 404 returned error can't find the container with id d9741757a02ee5159e61877b4be2acaf78e2e2c714e090c2c393fde20593bafe
Oct 31 08:33:33.945198 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:33:33.945158    4720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-769c5d65fc-jm59l" event={"ID":"c43713ad-58fa-403c-ac6f-792caa306ab5","Type":"ContainerStarted","Data":"438410222df91a7750983f83fc97ad556815adec14acde5508c56bf680946538"}
Oct 31 08:33:33.945634 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:33:33.945214    4720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-769c5d65fc-jm59l" event={"ID":"c43713ad-58fa-403c-ac6f-792caa306ab5","Type":"ContainerStarted","Data":"d9741757a02ee5159e61877b4be2acaf78e2e2c714e090c2c393fde20593bafe"}
Oct 31 08:33:34.328161 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:33:34.328073    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 08:33:34.391561 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:33:34.391511    4720 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-59bb995bd9-ddng5" podUID="eb737fc6-3a46-41f0-ba3b-ef2a410f3543" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/health\": dial tcp 10.132.0.74:8000: connect: connection refused"
Oct 31 08:33:34.496210 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:33:34.496173    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 08:33:36.692948 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:33:36.692908    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 08:33:38.979176 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:33:38.979135    4720 generic.go:356] "Generic (PLEG): container finished" podID="c43713ad-58fa-403c-ac6f-792caa306ab5" containerID="438410222df91a7750983f83fc97ad556815adec14acde5508c56bf680946538" exitCode=0
Oct 31 08:33:38.979644 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:33:38.979208    4720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-769c5d65fc-jm59l" event={"ID":"c43713ad-58fa-403c-ac6f-792caa306ab5","Type":"ContainerDied","Data":"438410222df91a7750983f83fc97ad556815adec14acde5508c56bf680946538"}
Oct 31 08:33:39.986693 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:33:39.986657    4720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-769c5d65fc-jm59l" event={"ID":"c43713ad-58fa-403c-ac6f-792caa306ab5","Type":"ContainerStarted","Data":"4c6695ba6f78d7ae4187c56262e44b463f6f49108b5ed41c143a6c83788b6427"}
Oct 31 08:33:40.136805 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:33:40.136770    4720 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-769c5d65fc-jm59l"
Oct 31 08:33:40.203786 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:33:40.203733    4720 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-769c5d65fc-jm59l" podStartSLOduration=3.8665329269999997 podStartE2EDuration="8.203717657s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.33718473s" isStatefulPod=false podCreationTimestamp="2026-10-31 08:33:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 08:33:40.15256139 +0000 UTC m=+3874.283629471" watchObservedRunningTime="2026-10-31 08:33:40.203717657 +0000 UTC m=+3874.334785728"
Oct 31 08:33:43.137028 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:33:43.136982    4720 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-769c5d65fc-jm59l"
Oct 31 08:33:43.138815 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:33:43.138781    4720 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-769c5d65fc-jm59l" podUID="c43713ad-58fa-403c-ac6f-792caa306ab5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/health\": dial tcp 10.132.0.75:8000: connect: connection refused"
Oct 31 08:33:43.463109 ip-10-0-139-83 kubenswrapper[4720]: E1031 08:33:43.463014    4720 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8m768" containerName="extractor"
Oct 31 08:33:44.391092 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:33:44.391039    4720 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-59bb995bd9-ddng5" podUID="eb737fc6-3a46-41f0-ba3b-ef2a410f3543" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/health\": dial tcp 10.132.0.74:8000: connect: connection refused"
Oct 31 08:33:53.137476 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:33:53.137432    4720 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-769c5d65fc-jm59l" podUID="c43713ad-58fa-403c-ac6f-792caa306ab5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/health\": dial tcp 10.132.0.75:8000: connect: connection refused"
Oct 31 08:33:54.390507 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:33:54.390469    4720 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-59bb995bd9-ddng5" podUID="eb737fc6-3a46-41f0-ba3b-ef2a410f3543" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/health\": dial tcp 10.132.0.74:8000: connect: connection refused"
Oct 31 08:34:01.489278 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:34:01.489237    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 08:34:03.137773 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:34:03.137728    4720 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-769c5d65fc-jm59l" podUID="c43713ad-58fa-403c-ac6f-792caa306ab5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/health\": dial tcp 10.132.0.75:8000: connect: connection refused"
Oct 31 08:34:04.390827 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:34:04.390779    4720 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-59bb995bd9-ddng5" podUID="eb737fc6-3a46-41f0-ba3b-ef2a410f3543" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/health\": dial tcp 10.132.0.74:8000: connect: connection refused"
Oct 31 08:34:06.394744 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:34:06.394590    4720 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-4jszp_b1db0e3b-e85d-4769-8012-cf4b6f740d9d/console-operator/2.log"
Oct 31 08:34:06.403390 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:34:06.403358    4720 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-4jszp_b1db0e3b-e85d-4769-8012-cf4b6f740d9d/console-operator/2.log"
Oct 31 08:34:13.137713 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:34:13.137663    4720 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-769c5d65fc-jm59l" podUID="c43713ad-58fa-403c-ac6f-792caa306ab5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/health\": dial tcp 10.132.0.75:8000: connect: connection refused"
Oct 31 08:34:13.465621 ip-10-0-139-83 kubenswrapper[4720]: E1031 08:34:13.465520    4720 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8m768" containerName="extractor"
Oct 31 08:34:14.390384 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:34:14.390342    4720 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-59bb995bd9-ddng5" podUID="eb737fc6-3a46-41f0-ba3b-ef2a410f3543" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/health\": dial tcp 10.132.0.74:8000: connect: connection refused"
Oct 31 08:34:22.430656 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:34:22.430554    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 08:34:23.137819 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:34:23.137765    4720 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-769c5d65fc-jm59l" podUID="c43713ad-58fa-403c-ac6f-792caa306ab5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/health\": dial tcp 10.132.0.75:8000: connect: connection refused"
Oct 31 08:34:24.390703 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:34:24.390651    4720 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-59bb995bd9-ddng5" podUID="eb737fc6-3a46-41f0-ba3b-ef2a410f3543" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/health\": dial tcp 10.132.0.74:8000: connect: connection refused"
Oct 31 08:34:33.137847 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:34:33.137797    4720 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-769c5d65fc-jm59l" podUID="c43713ad-58fa-403c-ac6f-792caa306ab5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/health\": dial tcp 10.132.0.75:8000: connect: connection refused"
Oct 31 08:34:34.401365 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:34:34.401301    4720 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-59bb995bd9-ddng5"
Oct 31 08:34:34.409450 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:34:34.409416    4720 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-59bb995bd9-ddng5"
Oct 31 08:34:37.045347 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:34:37.045282    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 08:34:37.179736 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:34:37.179703    4720 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-56f46b8cdd-gd6lh"]
Oct 31 08:34:37.180007 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:34:37.179958    4720 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/openshift-ai-inference-openshift-default-56f46b8cdd-gd6lh" podUID="a4330a4d-4a3f-40ea-b824-5b5c8c1fb0ee" containerName="istio-proxy" containerID="cri-o://7cf00be1f39377820191da1c0b6c3f34d6cdf182db37676709ded2c2518ddcc4" gracePeriod=30
Oct 31 08:34:42.322482 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:34:42.322438    4720 generic.go:356] "Generic (PLEG): container finished" podID="a4330a4d-4a3f-40ea-b824-5b5c8c1fb0ee" containerID="7cf00be1f39377820191da1c0b6c3f34d6cdf182db37676709ded2c2518ddcc4" exitCode=0
Oct 31 08:34:42.322931 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:34:42.322594    4720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-56f46b8cdd-gd6lh" event={"ID":"a4330a4d-4a3f-40ea-b824-5b5c8c1fb0ee","Type":"ContainerDied","Data":"7cf00be1f39377820191da1c0b6c3f34d6cdf182db37676709ded2c2518ddcc4"}
Oct 31 08:34:42.440132 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:34:42.440097    4720 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-default-56f46b8cdd-gd6lh"
Oct 31 08:34:42.530898 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:34:42.530801    4720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a4330a4d-4a3f-40ea-b824-5b5c8c1fb0ee-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a4330a4d-4a3f-40ea-b824-5b5c8c1fb0ee-istiod-ca-cert\") pod \"a4330a4d-4a3f-40ea-b824-5b5c8c1fb0ee\" (UID: \"a4330a4d-4a3f-40ea-b824-5b5c8c1fb0ee\") "
Oct 31 08:34:42.530898 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:34:42.530883    4720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a4330a4d-4a3f-40ea-b824-5b5c8c1fb0ee-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a4330a4d-4a3f-40ea-b824-5b5c8c1fb0ee-istio-podinfo\") pod \"a4330a4d-4a3f-40ea-b824-5b5c8c1fb0ee\" (UID: \"a4330a4d-4a3f-40ea-b824-5b5c8c1fb0ee\") "
Oct 31 08:34:42.531138 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:34:42.530924    4720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a4330a4d-4a3f-40ea-b824-5b5c8c1fb0ee-istio-token\" (UniqueName: \"kubernetes.io/projected/a4330a4d-4a3f-40ea-b824-5b5c8c1fb0ee-istio-token\") pod \"a4330a4d-4a3f-40ea-b824-5b5c8c1fb0ee\" (UID: \"a4330a4d-4a3f-40ea-b824-5b5c8c1fb0ee\") "
Oct 31 08:34:42.531138 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:34:42.530987    4720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a4330a4d-4a3f-40ea-b824-5b5c8c1fb0ee-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4330a4d-4a3f-40ea-b824-5b5c8c1fb0ee-workload-socket\") pod \"a4330a4d-4a3f-40ea-b824-5b5c8c1fb0ee\" (UID: \"a4330a4d-4a3f-40ea-b824-5b5c8c1fb0ee\") "
Oct 31 08:34:42.531138 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:34:42.531021    4720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a4330a4d-4a3f-40ea-b824-5b5c8c1fb0ee-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a4330a4d-4a3f-40ea-b824-5b5c8c1fb0ee-istio-data\") pod \"a4330a4d-4a3f-40ea-b824-5b5c8c1fb0ee\" (UID: \"a4330a4d-4a3f-40ea-b824-5b5c8c1fb0ee\") "
Oct 31 08:34:42.531138 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:34:42.531053    4720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a4330a4d-4a3f-40ea-b824-5b5c8c1fb0ee-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a4330a4d-4a3f-40ea-b824-5b5c8c1fb0ee-istio-envoy\") pod \"a4330a4d-4a3f-40ea-b824-5b5c8c1fb0ee\" (UID: \"a4330a4d-4a3f-40ea-b824-5b5c8c1fb0ee\") "
Oct 31 08:34:42.531362 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:34:42.531136    4720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a4330a4d-4a3f-40ea-b824-5b5c8c1fb0ee-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4330a4d-4a3f-40ea-b824-5b5c8c1fb0ee-credential-socket\") pod \"a4330a4d-4a3f-40ea-b824-5b5c8c1fb0ee\" (UID: \"a4330a4d-4a3f-40ea-b824-5b5c8c1fb0ee\") "
Oct 31 08:34:42.531362 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:34:42.531192    4720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a4330a4d-4a3f-40ea-b824-5b5c8c1fb0ee-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a4330a4d-4a3f-40ea-b824-5b5c8c1fb0ee-workload-certs\") pod \"a4330a4d-4a3f-40ea-b824-5b5c8c1fb0ee\" (UID: \"a4330a4d-4a3f-40ea-b824-5b5c8c1fb0ee\") "
Oct 31 08:34:42.531362 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:34:42.531220    4720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a4330a4d-4a3f-40ea-b824-5b5c8c1fb0ee-kube-api-access-vxpk2\" (UniqueName: \"kubernetes.io/projected/a4330a4d-4a3f-40ea-b824-5b5c8c1fb0ee-kube-api-access-vxpk2\") pod \"a4330a4d-4a3f-40ea-b824-5b5c8c1fb0ee\" (UID: \"a4330a4d-4a3f-40ea-b824-5b5c8c1fb0ee\") "
Oct 31 08:34:42.531362 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:34:42.531186    4720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a4330a4d-4a3f-40ea-b824-5b5c8c1fb0ee-istiod-ca-cert" pod "a4330a4d-4a3f-40ea-b824-5b5c8c1fb0ee" (UID: "a4330a4d-4a3f-40ea-b824-5b5c8c1fb0ee"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 08:34:42.531362 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:34:42.531293    4720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a4330a4d-4a3f-40ea-b824-5b5c8c1fb0ee-workload-socket" pod "a4330a4d-4a3f-40ea-b824-5b5c8c1fb0ee" (UID: "a4330a4d-4a3f-40ea-b824-5b5c8c1fb0ee"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 08:34:42.531623 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:34:42.531509    4720 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4330a4d-4a3f-40ea-b824-5b5c8c1fb0ee-workload-socket\") on node \"ip-10-0-139-83.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 08:34:42.531623 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:34:42.531522    4720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a4330a4d-4a3f-40ea-b824-5b5c8c1fb0ee-credential-socket" pod "a4330a4d-4a3f-40ea-b824-5b5c8c1fb0ee" (UID: "a4330a4d-4a3f-40ea-b824-5b5c8c1fb0ee"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 08:34:42.531623 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:34:42.531529    4720 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a4330a4d-4a3f-40ea-b824-5b5c8c1fb0ee-istiod-ca-cert\") on node \"ip-10-0-139-83.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 08:34:42.531944 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:34:42.531784    4720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a4330a4d-4a3f-40ea-b824-5b5c8c1fb0ee-workload-certs" pod "a4330a4d-4a3f-40ea-b824-5b5c8c1fb0ee" (UID: "a4330a4d-4a3f-40ea-b824-5b5c8c1fb0ee"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 08:34:42.532007 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:34:42.531932    4720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a4330a4d-4a3f-40ea-b824-5b5c8c1fb0ee-istio-data" pod "a4330a4d-4a3f-40ea-b824-5b5c8c1fb0ee" (UID: "a4330a4d-4a3f-40ea-b824-5b5c8c1fb0ee"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 08:34:42.533476 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:34:42.533449    4720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a4330a4d-4a3f-40ea-b824-5b5c8c1fb0ee-istio-podinfo" pod "a4330a4d-4a3f-40ea-b824-5b5c8c1fb0ee" (UID: "a4330a4d-4a3f-40ea-b824-5b5c8c1fb0ee"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 08:34:42.534225 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:34:42.534198    4720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a4330a4d-4a3f-40ea-b824-5b5c8c1fb0ee-istio-envoy" pod "a4330a4d-4a3f-40ea-b824-5b5c8c1fb0ee" (UID: "a4330a4d-4a3f-40ea-b824-5b5c8c1fb0ee"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 08:34:42.534225 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:34:42.534219    4720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a4330a4d-4a3f-40ea-b824-5b5c8c1fb0ee-istio-token" pod "a4330a4d-4a3f-40ea-b824-5b5c8c1fb0ee" (UID: "a4330a4d-4a3f-40ea-b824-5b5c8c1fb0ee"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 08:34:42.534434 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:34:42.534237    4720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a4330a4d-4a3f-40ea-b824-5b5c8c1fb0ee-kube-api-access-vxpk2" pod "a4330a4d-4a3f-40ea-b824-5b5c8c1fb0ee" (UID: "a4330a4d-4a3f-40ea-b824-5b5c8c1fb0ee"). InnerVolumeSpecName "kube-api-access-vxpk2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 08:34:42.632598 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:34:42.632561    4720 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a4330a4d-4a3f-40ea-b824-5b5c8c1fb0ee-istio-podinfo\") on node \"ip-10-0-139-83.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 08:34:42.632598 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:34:42.632598    4720 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a4330a4d-4a3f-40ea-b824-5b5c8c1fb0ee-istio-token\") on node \"ip-10-0-139-83.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 08:34:42.632796 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:34:42.632611    4720 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a4330a4d-4a3f-40ea-b824-5b5c8c1fb0ee-istio-data\") on node \"ip-10-0-139-83.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 08:34:42.632796 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:34:42.632624    4720 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a4330a4d-4a3f-40ea-b824-5b5c8c1fb0ee-istio-envoy\") on node \"ip-10-0-139-83.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 08:34:42.632796 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:34:42.632637    4720 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4330a4d-4a3f-40ea-b824-5b5c8c1fb0ee-credential-socket\") on node \"ip-10-0-139-83.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 08:34:42.632796 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:34:42.632650    4720 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a4330a4d-4a3f-40ea-b824-5b5c8c1fb0ee-workload-certs\") on node \"ip-10-0-139-83.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 08:34:42.632796 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:34:42.632662    4720 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vxpk2\" (UniqueName: \"kubernetes.io/projected/a4330a4d-4a3f-40ea-b824-5b5c8c1fb0ee-kube-api-access-vxpk2\") on node \"ip-10-0-139-83.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 08:34:43.137067 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:34:43.137025    4720 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-769c5d65fc-jm59l" podUID="c43713ad-58fa-403c-ac6f-792caa306ab5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/health\": dial tcp 10.132.0.75:8000: connect: connection refused"
Oct 31 08:34:43.329699 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:34:43.329661    4720 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-default-56f46b8cdd-gd6lh"
Oct 31 08:34:43.329699 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:34:43.329662    4720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-56f46b8cdd-gd6lh" event={"ID":"a4330a4d-4a3f-40ea-b824-5b5c8c1fb0ee","Type":"ContainerDied","Data":"a2959305e3ffa192d9470ef39cff9858554bc2afeffe68c7f89160a3b5e6634e"}
Oct 31 08:34:43.330266 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:34:43.329738    4720 scope.go:122] "RemoveContainer" containerID="7cf00be1f39377820191da1c0b6c3f34d6cdf182db37676709ded2c2518ddcc4"
Oct 31 08:34:43.461944 ip-10-0-139-83 kubenswrapper[4720]: E1031 08:34:43.461851    4720 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8m768" containerName="extractor"
Oct 31 08:34:43.542226 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:34:43.542196    4720 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-56f46b8cdd-gd6lh"]
Oct 31 08:34:43.591544 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:34:43.591506    4720 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-56f46b8cdd-gd6lh"]
Oct 31 08:34:44.025559 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:34:44.025521    4720 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a4330a4d-4a3f-40ea-b824-5b5c8c1fb0ee" path="/var/lib/kubelet/pods/a4330a4d-4a3f-40ea-b824-5b5c8c1fb0ee/volumes"
Oct 31 08:34:48.062918 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:34:48.062879    4720 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-59bb995bd9-ddng5"]
Oct 31 08:34:48.098166 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:34:48.098135    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 08:34:48.312075 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:34:48.312043    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 08:34:53.137419 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:34:53.137378    4720 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-769c5d65fc-jm59l" podUID="c43713ad-58fa-403c-ac6f-792caa306ab5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/health\": dial tcp 10.132.0.75:8000: connect: connection refused"
Oct 31 08:35:01.829876 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:35:01.829829    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 08:35:03.086865 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:35:03.086819    4720 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-59bb995bd9-ddng5" podUID="eb737fc6-3a46-41f0-ba3b-ef2a410f3543" containerName="main" containerID="cri-o://846425dc26929c11ac64014b209b2f96d382cf35ad0370efeed7510ede6be205" gracePeriod=45
Oct 31 08:35:03.147932 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:35:03.147896    4720 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-769c5d65fc-jm59l"
Oct 31 08:35:03.155877 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:35:03.155844    4720 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-769c5d65fc-jm59l"
Oct 31 08:35:03.845945 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:35:03.845916    4720 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-59bb995bd9-ddng5"
Oct 31 08:35:03.917380 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:35:03.917254    4720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/eb737fc6-3a46-41f0-ba3b-ef2a410f3543-tls-certs\" (UniqueName: \"kubernetes.io/secret/eb737fc6-3a46-41f0-ba3b-ef2a410f3543-tls-certs\") pod \"eb737fc6-3a46-41f0-ba3b-ef2a410f3543\" (UID: \"eb737fc6-3a46-41f0-ba3b-ef2a410f3543\") "
Oct 31 08:35:03.917380 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:35:03.917299    4720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eb737fc6-3a46-41f0-ba3b-ef2a410f3543-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eb737fc6-3a46-41f0-ba3b-ef2a410f3543-tmp-dir\") pod \"eb737fc6-3a46-41f0-ba3b-ef2a410f3543\" (UID: \"eb737fc6-3a46-41f0-ba3b-ef2a410f3543\") "
Oct 31 08:35:03.917380 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:35:03.917374    4720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eb737fc6-3a46-41f0-ba3b-ef2a410f3543-kube-api-access-5xnnt\" (UniqueName: \"kubernetes.io/projected/eb737fc6-3a46-41f0-ba3b-ef2a410f3543-kube-api-access-5xnnt\") pod \"eb737fc6-3a46-41f0-ba3b-ef2a410f3543\" (UID: \"eb737fc6-3a46-41f0-ba3b-ef2a410f3543\") "
Oct 31 08:35:03.917667 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:35:03.917517    4720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eb737fc6-3a46-41f0-ba3b-ef2a410f3543-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eb737fc6-3a46-41f0-ba3b-ef2a410f3543-model-cache\") pod \"eb737fc6-3a46-41f0-ba3b-ef2a410f3543\" (UID: \"eb737fc6-3a46-41f0-ba3b-ef2a410f3543\") "
Oct 31 08:35:03.917667 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:35:03.917580    4720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eb737fc6-3a46-41f0-ba3b-ef2a410f3543-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/eb737fc6-3a46-41f0-ba3b-ef2a410f3543-kserve-provision-location\") pod \"eb737fc6-3a46-41f0-ba3b-ef2a410f3543\" (UID: \"eb737fc6-3a46-41f0-ba3b-ef2a410f3543\") "
Oct 31 08:35:03.917667 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:35:03.917633    4720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eb737fc6-3a46-41f0-ba3b-ef2a410f3543-dshm\" (UniqueName: \"kubernetes.io/empty-dir/eb737fc6-3a46-41f0-ba3b-ef2a410f3543-dshm\") pod \"eb737fc6-3a46-41f0-ba3b-ef2a410f3543\" (UID: \"eb737fc6-3a46-41f0-ba3b-ef2a410f3543\") "
Oct 31 08:35:03.917827 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:35:03.917672    4720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eb737fc6-3a46-41f0-ba3b-ef2a410f3543-home\" (UniqueName: \"kubernetes.io/empty-dir/eb737fc6-3a46-41f0-ba3b-ef2a410f3543-home\") pod \"eb737fc6-3a46-41f0-ba3b-ef2a410f3543\" (UID: \"eb737fc6-3a46-41f0-ba3b-ef2a410f3543\") "
Oct 31 08:35:03.917827 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:35:03.917706    4720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eb737fc6-3a46-41f0-ba3b-ef2a410f3543-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/eb737fc6-3a46-41f0-ba3b-ef2a410f3543-cabundle-cert\") pod \"eb737fc6-3a46-41f0-ba3b-ef2a410f3543\" (UID: \"eb737fc6-3a46-41f0-ba3b-ef2a410f3543\") "
Oct 31 08:35:03.917827 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:35:03.917781    4720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb737fc6-3a46-41f0-ba3b-ef2a410f3543-model-cache" pod "eb737fc6-3a46-41f0-ba3b-ef2a410f3543" (UID: "eb737fc6-3a46-41f0-ba3b-ef2a410f3543"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 08:35:03.917990 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:35:03.917978    4720 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eb737fc6-3a46-41f0-ba3b-ef2a410f3543-model-cache\") on node \"ip-10-0-139-83.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 08:35:03.918374 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:35:03.918322    4720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eb737fc6-3a46-41f0-ba3b-ef2a410f3543-cabundle-cert" pod "eb737fc6-3a46-41f0-ba3b-ef2a410f3543" (UID: "eb737fc6-3a46-41f0-ba3b-ef2a410f3543"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 08:35:03.918759 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:35:03.918732    4720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb737fc6-3a46-41f0-ba3b-ef2a410f3543-home" pod "eb737fc6-3a46-41f0-ba3b-ef2a410f3543" (UID: "eb737fc6-3a46-41f0-ba3b-ef2a410f3543"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 08:35:03.920093 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:35:03.920068    4720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eb737fc6-3a46-41f0-ba3b-ef2a410f3543-kube-api-access-5xnnt" pod "eb737fc6-3a46-41f0-ba3b-ef2a410f3543" (UID: "eb737fc6-3a46-41f0-ba3b-ef2a410f3543"). InnerVolumeSpecName "kube-api-access-5xnnt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 08:35:03.920093 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:35:03.920072    4720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eb737fc6-3a46-41f0-ba3b-ef2a410f3543-tls-certs" pod "eb737fc6-3a46-41f0-ba3b-ef2a410f3543" (UID: "eb737fc6-3a46-41f0-ba3b-ef2a410f3543"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 08:35:03.920226 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:35:03.920160    4720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb737fc6-3a46-41f0-ba3b-ef2a410f3543-dshm" pod "eb737fc6-3a46-41f0-ba3b-ef2a410f3543" (UID: "eb737fc6-3a46-41f0-ba3b-ef2a410f3543"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 08:35:03.936198 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:35:03.936127    4720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb737fc6-3a46-41f0-ba3b-ef2a410f3543-tmp-dir" pod "eb737fc6-3a46-41f0-ba3b-ef2a410f3543" (UID: "eb737fc6-3a46-41f0-ba3b-ef2a410f3543"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 08:35:03.945856 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:35:03.945817    4720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb737fc6-3a46-41f0-ba3b-ef2a410f3543-kserve-provision-location" pod "eb737fc6-3a46-41f0-ba3b-ef2a410f3543" (UID: "eb737fc6-3a46-41f0-ba3b-ef2a410f3543"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 08:35:04.018979 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:35:04.018937    4720 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/eb737fc6-3a46-41f0-ba3b-ef2a410f3543-cabundle-cert\") on node \"ip-10-0-139-83.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 08:35:04.018979 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:35:04.018970    4720 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eb737fc6-3a46-41f0-ba3b-ef2a410f3543-tls-certs\") on node \"ip-10-0-139-83.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 08:35:04.018979 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:35:04.018981    4720 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eb737fc6-3a46-41f0-ba3b-ef2a410f3543-tmp-dir\") on node \"ip-10-0-139-83.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 08:35:04.018979 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:35:04.018989    4720 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5xnnt\" (UniqueName: \"kubernetes.io/projected/eb737fc6-3a46-41f0-ba3b-ef2a410f3543-kube-api-access-5xnnt\") on node \"ip-10-0-139-83.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 08:35:04.019261 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:35:04.018998    4720 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/eb737fc6-3a46-41f0-ba3b-ef2a410f3543-kserve-provision-location\") on node \"ip-10-0-139-83.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 08:35:04.019261 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:35:04.019008    4720 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/eb737fc6-3a46-41f0-ba3b-ef2a410f3543-dshm\") on node \"ip-10-0-139-83.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 08:35:04.019261 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:35:04.019016    4720 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/eb737fc6-3a46-41f0-ba3b-ef2a410f3543-home\") on node \"ip-10-0-139-83.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 08:35:04.434810 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:35:04.434779    4720 generic.go:356] "Generic (PLEG): container finished" podID="eb737fc6-3a46-41f0-ba3b-ef2a410f3543" containerID="846425dc26929c11ac64014b209b2f96d382cf35ad0370efeed7510ede6be205" exitCode=0
Oct 31 08:35:04.435284 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:35:04.434830    4720 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-59bb995bd9-ddng5"
Oct 31 08:35:04.435284 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:35:04.434864    4720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-59bb995bd9-ddng5" event={"ID":"eb737fc6-3a46-41f0-ba3b-ef2a410f3543","Type":"ContainerDied","Data":"846425dc26929c11ac64014b209b2f96d382cf35ad0370efeed7510ede6be205"}
Oct 31 08:35:04.435284 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:35:04.434921    4720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-59bb995bd9-ddng5" event={"ID":"eb737fc6-3a46-41f0-ba3b-ef2a410f3543","Type":"ContainerDied","Data":"0c4ba94308c15c24976f5fc001a3ee5b40a368b86519dce5608c7177643580de"}
Oct 31 08:35:04.435284 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:35:04.434944    4720 scope.go:122] "RemoveContainer" containerID="846425dc26929c11ac64014b209b2f96d382cf35ad0370efeed7510ede6be205"
Oct 31 08:35:04.443915 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:35:04.443892    4720 scope.go:122] "RemoveContainer" containerID="62aef6b7986e58a989442d38eb00908c04684b43e0bb80420be8e18c2901efc3"
Oct 31 08:35:04.452139 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:35:04.452117    4720 scope.go:122] "RemoveContainer" containerID="846425dc26929c11ac64014b209b2f96d382cf35ad0370efeed7510ede6be205"
Oct 31 08:35:04.452433 ip-10-0-139-83 kubenswrapper[4720]: E1031 08:35:04.452414    4720 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"846425dc26929c11ac64014b209b2f96d382cf35ad0370efeed7510ede6be205\": container with ID starting with 846425dc26929c11ac64014b209b2f96d382cf35ad0370efeed7510ede6be205 not found: ID does not exist" containerID="846425dc26929c11ac64014b209b2f96d382cf35ad0370efeed7510ede6be205"
Oct 31 08:35:04.452493 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:35:04.452442    4720 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"846425dc26929c11ac64014b209b2f96d382cf35ad0370efeed7510ede6be205"} err="failed to get container status \"846425dc26929c11ac64014b209b2f96d382cf35ad0370efeed7510ede6be205\": rpc error: code = NotFound desc = could not find container \"846425dc26929c11ac64014b209b2f96d382cf35ad0370efeed7510ede6be205\": container with ID starting with 846425dc26929c11ac64014b209b2f96d382cf35ad0370efeed7510ede6be205 not found: ID does not exist"
Oct 31 08:35:04.452493 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:35:04.452462    4720 scope.go:122] "RemoveContainer" containerID="62aef6b7986e58a989442d38eb00908c04684b43e0bb80420be8e18c2901efc3"
Oct 31 08:35:04.452715 ip-10-0-139-83 kubenswrapper[4720]: E1031 08:35:04.452698    4720 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"62aef6b7986e58a989442d38eb00908c04684b43e0bb80420be8e18c2901efc3\": container with ID starting with 62aef6b7986e58a989442d38eb00908c04684b43e0bb80420be8e18c2901efc3 not found: ID does not exist" containerID="62aef6b7986e58a989442d38eb00908c04684b43e0bb80420be8e18c2901efc3"
Oct 31 08:35:04.452755 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:35:04.452724    4720 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"62aef6b7986e58a989442d38eb00908c04684b43e0bb80420be8e18c2901efc3"} err="failed to get container status \"62aef6b7986e58a989442d38eb00908c04684b43e0bb80420be8e18c2901efc3\": rpc error: code = NotFound desc = could not find container \"62aef6b7986e58a989442d38eb00908c04684b43e0bb80420be8e18c2901efc3\": container with ID starting with 62aef6b7986e58a989442d38eb00908c04684b43e0bb80420be8e18c2901efc3 not found: ID does not exist"
Oct 31 08:35:04.489123 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:35:04.489088    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 08:35:04.599675 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:35:04.599637    4720 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-59bb995bd9-ddng5"]
Oct 31 08:35:04.649057 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:35:04.649018    4720 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-59bb995bd9-ddng5"]
Oct 31 08:35:06.025614 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:35:06.025573    4720 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eb737fc6-3a46-41f0-ba3b-ef2a410f3543" path="/var/lib/kubelet/pods/eb737fc6-3a46-41f0-ba3b-ef2a410f3543/volumes"
Oct 31 08:35:13.463071 ip-10-0-139-83 kubenswrapper[4720]: E1031 08:35:13.463023    4720 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8m768" containerName="extractor"
Oct 31 08:35:22.803519 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:35:22.803476    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 08:35:33.136679 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:35:33.136635    4720 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-with-lora-adapters-bbeb2681_lora-multiple-adapters-test-kserve-769c5d65fc-jm59l_c43713ad-58fa-403c-ac6f-792caa306ab5/storage-initializer/0.log"
Oct 31 08:35:33.254242 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:35:33.254202    4720 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-with-lora-adapters-bbeb2681_lora-multiple-adapters-test-kserve-769c5d65fc-jm59l_c43713ad-58fa-403c-ac6f-792caa306ab5/main/0.log"
Oct 31 08:35:37.371701 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:35:37.371662    4720 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-769c5d65fc-jm59l"]
Oct 31 08:35:37.380189 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:35:37.380150    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 08:35:38.594010 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:35:38.593969    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 08:35:42.622615 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:35:42.622581    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 08:35:43.462080 ip-10-0-139-83 kubenswrapper[4720]: E1031 08:35:43.462043    4720 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8m768" containerName="extractor"
Oct 31 08:35:48.329580 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:35:48.329491    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 08:35:52.394892 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:35:52.394849    4720 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-769c5d65fc-jm59l" podUID="c43713ad-58fa-403c-ac6f-792caa306ab5" containerName="main" containerID="cri-o://4c6695ba6f78d7ae4187c56262e44b463f6f49108b5ed41c143a6c83788b6427" gracePeriod=45
Oct 31 08:35:53.049574 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:35:53.049546    4720 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-769c5d65fc-jm59l"
Oct 31 08:35:53.143580 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:35:53.143493    4720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c43713ad-58fa-403c-ac6f-792caa306ab5-dshm\" (UniqueName: \"kubernetes.io/empty-dir/c43713ad-58fa-403c-ac6f-792caa306ab5-dshm\") pod \"c43713ad-58fa-403c-ac6f-792caa306ab5\" (UID: \"c43713ad-58fa-403c-ac6f-792caa306ab5\") "
Oct 31 08:35:53.143580 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:35:53.143549    4720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c43713ad-58fa-403c-ac6f-792caa306ab5-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c43713ad-58fa-403c-ac6f-792caa306ab5-model-cache\") pod \"c43713ad-58fa-403c-ac6f-792caa306ab5\" (UID: \"c43713ad-58fa-403c-ac6f-792caa306ab5\") "
Oct 31 08:35:53.143580 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:35:53.143575    4720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c43713ad-58fa-403c-ac6f-792caa306ab5-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c43713ad-58fa-403c-ac6f-792caa306ab5-kserve-provision-location\") pod \"c43713ad-58fa-403c-ac6f-792caa306ab5\" (UID: \"c43713ad-58fa-403c-ac6f-792caa306ab5\") "
Oct 31 08:35:53.143851 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:35:53.143599    4720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c43713ad-58fa-403c-ac6f-792caa306ab5-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c43713ad-58fa-403c-ac6f-792caa306ab5-cabundle-cert\") pod \"c43713ad-58fa-403c-ac6f-792caa306ab5\" (UID: \"c43713ad-58fa-403c-ac6f-792caa306ab5\") "
Oct 31 08:35:53.143851 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:35:53.143619    4720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c43713ad-58fa-403c-ac6f-792caa306ab5-tls-certs\" (UniqueName: \"kubernetes.io/secret/c43713ad-58fa-403c-ac6f-792caa306ab5-tls-certs\") pod \"c43713ad-58fa-403c-ac6f-792caa306ab5\" (UID: \"c43713ad-58fa-403c-ac6f-792caa306ab5\") "
Oct 31 08:35:53.143851 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:35:53.143662    4720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c43713ad-58fa-403c-ac6f-792caa306ab5-home\" (UniqueName: \"kubernetes.io/empty-dir/c43713ad-58fa-403c-ac6f-792caa306ab5-home\") pod \"c43713ad-58fa-403c-ac6f-792caa306ab5\" (UID: \"c43713ad-58fa-403c-ac6f-792caa306ab5\") "
Oct 31 08:35:53.143851 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:35:53.143698    4720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c43713ad-58fa-403c-ac6f-792caa306ab5-kube-api-access-8kpwq\" (UniqueName: \"kubernetes.io/projected/c43713ad-58fa-403c-ac6f-792caa306ab5-kube-api-access-8kpwq\") pod \"c43713ad-58fa-403c-ac6f-792caa306ab5\" (UID: \"c43713ad-58fa-403c-ac6f-792caa306ab5\") "
Oct 31 08:35:53.143851 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:35:53.143733    4720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c43713ad-58fa-403c-ac6f-792caa306ab5-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c43713ad-58fa-403c-ac6f-792caa306ab5-tmp-dir\") pod \"c43713ad-58fa-403c-ac6f-792caa306ab5\" (UID: \"c43713ad-58fa-403c-ac6f-792caa306ab5\") "
Oct 31 08:35:53.144116 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:35:53.143924    4720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c43713ad-58fa-403c-ac6f-792caa306ab5-model-cache" pod "c43713ad-58fa-403c-ac6f-792caa306ab5" (UID: "c43713ad-58fa-403c-ac6f-792caa306ab5"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 08:35:53.144116 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:35:53.144007    4720 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c43713ad-58fa-403c-ac6f-792caa306ab5-model-cache\") on node \"ip-10-0-139-83.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 08:35:53.144116 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:35:53.144029    4720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c43713ad-58fa-403c-ac6f-792caa306ab5-cabundle-cert" pod "c43713ad-58fa-403c-ac6f-792caa306ab5" (UID: "c43713ad-58fa-403c-ac6f-792caa306ab5"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 08:35:53.144282 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:35:53.144128    4720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c43713ad-58fa-403c-ac6f-792caa306ab5-home" pod "c43713ad-58fa-403c-ac6f-792caa306ab5" (UID: "c43713ad-58fa-403c-ac6f-792caa306ab5"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 08:35:53.146168 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:35:53.146138    4720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c43713ad-58fa-403c-ac6f-792caa306ab5-tls-certs" pod "c43713ad-58fa-403c-ac6f-792caa306ab5" (UID: "c43713ad-58fa-403c-ac6f-792caa306ab5"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 08:35:53.146305 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:35:53.146186    4720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c43713ad-58fa-403c-ac6f-792caa306ab5-dshm" pod "c43713ad-58fa-403c-ac6f-792caa306ab5" (UID: "c43713ad-58fa-403c-ac6f-792caa306ab5"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 08:35:53.146305 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:35:53.146259    4720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c43713ad-58fa-403c-ac6f-792caa306ab5-kube-api-access-8kpwq" pod "c43713ad-58fa-403c-ac6f-792caa306ab5" (UID: "c43713ad-58fa-403c-ac6f-792caa306ab5"). InnerVolumeSpecName "kube-api-access-8kpwq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 08:35:53.164701 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:35:53.164654    4720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c43713ad-58fa-403c-ac6f-792caa306ab5-tmp-dir" pod "c43713ad-58fa-403c-ac6f-792caa306ab5" (UID: "c43713ad-58fa-403c-ac6f-792caa306ab5"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 08:35:53.175382 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:35:53.175355    4720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c43713ad-58fa-403c-ac6f-792caa306ab5-kserve-provision-location" pod "c43713ad-58fa-403c-ac6f-792caa306ab5" (UID: "c43713ad-58fa-403c-ac6f-792caa306ab5"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 08:35:53.245017 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:35:53.244981    4720 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c43713ad-58fa-403c-ac6f-792caa306ab5-dshm\") on node \"ip-10-0-139-83.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 08:35:53.245017 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:35:53.245009    4720 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c43713ad-58fa-403c-ac6f-792caa306ab5-kserve-provision-location\") on node \"ip-10-0-139-83.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 08:35:53.245017 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:35:53.245019    4720 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c43713ad-58fa-403c-ac6f-792caa306ab5-cabundle-cert\") on node \"ip-10-0-139-83.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 08:35:53.245258 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:35:53.245029    4720 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c43713ad-58fa-403c-ac6f-792caa306ab5-tls-certs\") on node \"ip-10-0-139-83.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 08:35:53.245258 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:35:53.245038    4720 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c43713ad-58fa-403c-ac6f-792caa306ab5-home\") on node \"ip-10-0-139-83.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 08:35:53.245258 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:35:53.245046    4720 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8kpwq\" (UniqueName: \"kubernetes.io/projected/c43713ad-58fa-403c-ac6f-792caa306ab5-kube-api-access-8kpwq\") on node \"ip-10-0-139-83.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 08:35:53.245258 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:35:53.245055    4720 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c43713ad-58fa-403c-ac6f-792caa306ab5-tmp-dir\") on node \"ip-10-0-139-83.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 08:35:53.670122 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:35:53.670086    4720 generic.go:356] "Generic (PLEG): container finished" podID="c43713ad-58fa-403c-ac6f-792caa306ab5" containerID="4c6695ba6f78d7ae4187c56262e44b463f6f49108b5ed41c143a6c83788b6427" exitCode=0
Oct 31 08:35:53.670617 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:35:53.670140    4720 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-769c5d65fc-jm59l"
Oct 31 08:35:53.670617 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:35:53.670149    4720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-769c5d65fc-jm59l" event={"ID":"c43713ad-58fa-403c-ac6f-792caa306ab5","Type":"ContainerDied","Data":"4c6695ba6f78d7ae4187c56262e44b463f6f49108b5ed41c143a6c83788b6427"}
Oct 31 08:35:53.670617 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:35:53.670196    4720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-769c5d65fc-jm59l" event={"ID":"c43713ad-58fa-403c-ac6f-792caa306ab5","Type":"ContainerDied","Data":"d9741757a02ee5159e61877b4be2acaf78e2e2c714e090c2c393fde20593bafe"}
Oct 31 08:35:53.670617 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:35:53.670219    4720 scope.go:122] "RemoveContainer" containerID="4c6695ba6f78d7ae4187c56262e44b463f6f49108b5ed41c143a6c83788b6427"
Oct 31 08:35:53.679298 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:35:53.679274    4720 scope.go:122] "RemoveContainer" containerID="438410222df91a7750983f83fc97ad556815adec14acde5508c56bf680946538"
Oct 31 08:35:53.687752 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:35:53.687729    4720 scope.go:122] "RemoveContainer" containerID="4c6695ba6f78d7ae4187c56262e44b463f6f49108b5ed41c143a6c83788b6427"
Oct 31 08:35:53.688044 ip-10-0-139-83 kubenswrapper[4720]: E1031 08:35:53.688023    4720 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4c6695ba6f78d7ae4187c56262e44b463f6f49108b5ed41c143a6c83788b6427\": container with ID starting with 4c6695ba6f78d7ae4187c56262e44b463f6f49108b5ed41c143a6c83788b6427 not found: ID does not exist" containerID="4c6695ba6f78d7ae4187c56262e44b463f6f49108b5ed41c143a6c83788b6427"
Oct 31 08:35:53.688084 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:35:53.688054    4720 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4c6695ba6f78d7ae4187c56262e44b463f6f49108b5ed41c143a6c83788b6427"} err="failed to get container status \"4c6695ba6f78d7ae4187c56262e44b463f6f49108b5ed41c143a6c83788b6427\": rpc error: code = NotFound desc = could not find container \"4c6695ba6f78d7ae4187c56262e44b463f6f49108b5ed41c143a6c83788b6427\": container with ID starting with 4c6695ba6f78d7ae4187c56262e44b463f6f49108b5ed41c143a6c83788b6427 not found: ID does not exist"
Oct 31 08:35:53.688084 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:35:53.688072    4720 scope.go:122] "RemoveContainer" containerID="438410222df91a7750983f83fc97ad556815adec14acde5508c56bf680946538"
Oct 31 08:35:53.688356 ip-10-0-139-83 kubenswrapper[4720]: E1031 08:35:53.688311    4720 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"438410222df91a7750983f83fc97ad556815adec14acde5508c56bf680946538\": container with ID starting with 438410222df91a7750983f83fc97ad556815adec14acde5508c56bf680946538 not found: ID does not exist" containerID="438410222df91a7750983f83fc97ad556815adec14acde5508c56bf680946538"
Oct 31 08:35:53.688398 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:35:53.688356    4720 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"438410222df91a7750983f83fc97ad556815adec14acde5508c56bf680946538"} err="failed to get container status \"438410222df91a7750983f83fc97ad556815adec14acde5508c56bf680946538\": rpc error: code = NotFound desc = could not find container \"438410222df91a7750983f83fc97ad556815adec14acde5508c56bf680946538\": container with ID starting with 438410222df91a7750983f83fc97ad556815adec14acde5508c56bf680946538 not found: ID does not exist"
Oct 31 08:35:53.834194 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:35:53.834155    4720 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-769c5d65fc-jm59l"]
Oct 31 08:35:53.884094 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:35:53.884060    4720 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-769c5d65fc-jm59l"]
Oct 31 08:35:54.025451 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:35:54.025371    4720 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c43713ad-58fa-403c-ac6f-792caa306ab5" path="/var/lib/kubelet/pods/c43713ad-58fa-403c-ac6f-792caa306ab5/volumes"
Oct 31 08:35:57.699013 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:35:57.698975    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 08:35:57.916648 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:35:57.916617    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 08:35:58.174542 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:35:58.174504    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 08:35:58.406428 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:35:58.406393    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 08:35:58.572108 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:35:58.572029    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 08:36:01.521059 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:36:01.521020    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 08:36:01.670383 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:36:01.670347    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 08:36:02.166729 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:36:02.166695    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 08:36:02.326164 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:36:02.326125    4720 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-qlmsn"]
Oct 31 08:36:02.326609 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:36:02.326592    4720 cpu_manager.go:414] "containerMap: removing container" podUID="eb737fc6-3a46-41f0-ba3b-ef2a410f3543" containerName="main"
Oct 31 08:36:02.326669 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:36:02.326611    4720 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eb737fc6-3a46-41f0-ba3b-ef2a410f3543" containerName="main"
Oct 31 08:36:02.326669 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:36:02.326631    4720 cpu_manager.go:414] "containerMap: removing container" podUID="eb737fc6-3a46-41f0-ba3b-ef2a410f3543" containerName="storage-initializer"
Oct 31 08:36:02.326669 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:36:02.326636    4720 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eb737fc6-3a46-41f0-ba3b-ef2a410f3543" containerName="storage-initializer"
Oct 31 08:36:02.326669 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:36:02.326645    4720 cpu_manager.go:414] "containerMap: removing container" podUID="c43713ad-58fa-403c-ac6f-792caa306ab5" containerName="storage-initializer"
Oct 31 08:36:02.326669 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:36:02.326650    4720 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c43713ad-58fa-403c-ac6f-792caa306ab5" containerName="storage-initializer"
Oct 31 08:36:02.326669 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:36:02.326664    4720 cpu_manager.go:414] "containerMap: removing container" podUID="a4330a4d-4a3f-40ea-b824-5b5c8c1fb0ee" containerName="istio-proxy"
Oct 31 08:36:02.326669 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:36:02.326669    4720 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a4330a4d-4a3f-40ea-b824-5b5c8c1fb0ee" containerName="istio-proxy"
Oct 31 08:36:02.326953 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:36:02.326678    4720 cpu_manager.go:414] "containerMap: removing container" podUID="c43713ad-58fa-403c-ac6f-792caa306ab5" containerName="main"
Oct 31 08:36:02.326953 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:36:02.326682    4720 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c43713ad-58fa-403c-ac6f-792caa306ab5" containerName="main"
Oct 31 08:36:02.326953 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:36:02.326738    4720 memory_manager.go:365] "RemoveStaleState removing state" podUID="a4330a4d-4a3f-40ea-b824-5b5c8c1fb0ee" containerName="istio-proxy"
Oct 31 08:36:02.326953 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:36:02.326746    4720 memory_manager.go:365] "RemoveStaleState removing state" podUID="c43713ad-58fa-403c-ac6f-792caa306ab5" containerName="main"
Oct 31 08:36:02.326953 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:36:02.326754    4720 memory_manager.go:365] "RemoveStaleState removing state" podUID="eb737fc6-3a46-41f0-ba3b-ef2a410f3543" containerName="main"
Oct 31 08:36:02.331156 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:36:02.331136    4720 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-qlmsn"
Oct 31 08:36:02.383238 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:36:02.383134    4720 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-with-lora-adapters-4412bdb9\"/\"openshift-service-ca.crt\""
Oct 31 08:36:02.383495 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:36:02.383265    4720 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-with-lora-adapters-4412bdb9\"/\"kube-root-ca.crt\""
Oct 31 08:36:02.386049 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:36:02.386021    4720 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-qlmsn"]
Oct 31 08:36:02.432659 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:36:02.432567    4720 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-with-lora-adapters-4412bdb9\"/\"seaweedfs-s3-creds\""
Oct 31 08:36:02.432831 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:36:02.432742    4720 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-with-lora-adapters-4412bdb9\"/\"default-dockercfg-bd7h9\""
Oct 31 08:36:02.487426 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:36:02.487394    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 08:36:02.527815 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:36:02.527770    4720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fhszf\" (UniqueName: \"kubernetes.io/projected/07da9eb3-de85-4bc9-a5a1-2a3cf25765f3-kube-api-access-fhszf\") pod \"e2e-pvc-lora-download-qlmsn\" (UID: \"07da9eb3-de85-4bc9-a5a1-2a3cf25765f3\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-qlmsn"
Oct 31 08:36:02.628698 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:36:02.628658    4720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fhszf\" (UniqueName: \"kubernetes.io/projected/07da9eb3-de85-4bc9-a5a1-2a3cf25765f3-kube-api-access-fhszf\") pod \"e2e-pvc-lora-download-qlmsn\" (UID: \"07da9eb3-de85-4bc9-a5a1-2a3cf25765f3\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-qlmsn"
Oct 31 08:36:02.686043 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:36:02.685973    4720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fhszf\" (UniqueName: \"kubernetes.io/projected/07da9eb3-de85-4bc9-a5a1-2a3cf25765f3-kube-api-access-fhszf\") pod \"e2e-pvc-lora-download-qlmsn\" (UID: \"07da9eb3-de85-4bc9-a5a1-2a3cf25765f3\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-qlmsn"
Oct 31 08:36:03.787659 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:36:03.787623    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 08:36:03.952940 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:36:03.952896    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 08:36:05.305755 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:36:05.305717    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 08:36:11.019006 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:36:11.018969    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 08:36:12.010412 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:36:12.010372    4720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-4bf6a61c-17dd-42d7-94fb-6807246354e3\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-066f50edccf52d23d\") pod \"e2e-pvc-lora-download-qlmsn\" (UID: \"07da9eb3-de85-4bc9-a5a1-2a3cf25765f3\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-qlmsn"
Oct 31 08:36:12.113928 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:36:12.113896    4720 operation_generator.go:1464] "Controller attach succeeded for volume \"pvc-4bf6a61c-17dd-42d7-94fb-6807246354e3\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-066f50edccf52d23d\") pod \"e2e-pvc-lora-download-qlmsn\" (UID: \"07da9eb3-de85-4bc9-a5a1-2a3cf25765f3\") device path: \"\"" pod="e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-qlmsn"
Oct 31 08:36:12.212619 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:36:12.212580    4720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"pvc-4bf6a61c-17dd-42d7-94fb-6807246354e3\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-066f50edccf52d23d\") pod \"e2e-pvc-lora-download-qlmsn\" (UID: \"07da9eb3-de85-4bc9-a5a1-2a3cf25765f3\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-qlmsn"
Oct 31 08:36:12.212798 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:36:12.212736    4720 operation_generator.go:515] "MountVolume.WaitForAttach entering for volume \"pvc-4bf6a61c-17dd-42d7-94fb-6807246354e3\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-066f50edccf52d23d\") pod \"e2e-pvc-lora-download-qlmsn\" (UID: \"07da9eb3-de85-4bc9-a5a1-2a3cf25765f3\") DevicePath \"\"" pod="e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-qlmsn"
Oct 31 08:36:12.264280 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:36:12.264196    4720 operation_generator.go:525] "MountVolume.WaitForAttach succeeded for volume \"pvc-4bf6a61c-17dd-42d7-94fb-6807246354e3\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-066f50edccf52d23d\") pod \"e2e-pvc-lora-download-qlmsn\" (UID: \"07da9eb3-de85-4bc9-a5a1-2a3cf25765f3\") DevicePath \"csi-088cc0e5c12cd8d68a78f81b27a0f5d315ce78944d37334dab4342f9d0c7c07f\"" pod="e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-qlmsn"
Oct 31 08:36:12.520212 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:36:12.520130    4720 operation_generator.go:557] "MountVolume.MountDevice succeeded for volume \"pvc-4bf6a61c-17dd-42d7-94fb-6807246354e3\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-066f50edccf52d23d\") pod \"e2e-pvc-lora-download-qlmsn\" (UID: \"07da9eb3-de85-4bc9-a5a1-2a3cf25765f3\") device mount path \"/var/lib/kubelet/plugins/kubernetes.io/csi/ebs.csi.aws.com/43df6d0400fb77580adebabb960e943450a9774f25d4fd15bd5e128a7002d449/globalmount\"" pod="e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-qlmsn"
Oct 31 08:36:12.582823 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:36:12.582793    4720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"pvc-4bf6a61c-17dd-42d7-94fb-6807246354e3\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-066f50edccf52d23d\") pod \"e2e-pvc-lora-download-qlmsn\" (UID: \"07da9eb3-de85-4bc9-a5a1-2a3cf25765f3\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-qlmsn"
Oct 31 08:36:12.894283 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:36:12.894248    4720 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-with-lora-adapters-4412bdb9\"/\"default-dockercfg-bd7h9\""
Oct 31 08:36:12.903231 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:36:12.903208    4720 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-qlmsn"
Oct 31 08:36:13.281577 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:36:13.281542    4720 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-qlmsn"]
Oct 31 08:36:13.284056 ip-10-0-139-83 kubenswrapper[4720]: W1031 08:36:13.284018    4720 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod07da9eb3_de85_4bc9_a5a1_2a3cf25765f3.slice/crio-ce3f7f2f459d274246f50c4eb7a11504b12c3542ed7574dcce44ded2b2caee8d WatchSource:0}: Error finding container ce3f7f2f459d274246f50c4eb7a11504b12c3542ed7574dcce44ded2b2caee8d: Status 404 returned error can't find the container with id ce3f7f2f459d274246f50c4eb7a11504b12c3542ed7574dcce44ded2b2caee8d
Oct 31 08:36:13.337431 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:36:13.337394    4720 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-with-lora-adapters-4412bdb9\"/\"seaweedfs-s3-creds\""
Oct 31 08:36:13.462948 ip-10-0-139-83 kubenswrapper[4720]: E1031 08:36:13.462907    4720 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8m768" containerName="extractor"
Oct 31 08:36:13.768220 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:36:13.768184    4720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-qlmsn" event={"ID":"07da9eb3-de85-4bc9-a5a1-2a3cf25765f3","Type":"ContainerStarted","Data":"abc986dcd32bd4fa17dd70939d19ad69cdcaeb28ebfefa0638219cba7ede2bfa"}
Oct 31 08:36:13.768220 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:36:13.768227    4720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-qlmsn" event={"ID":"07da9eb3-de85-4bc9-a5a1-2a3cf25765f3","Type":"ContainerStarted","Data":"ce3f7f2f459d274246f50c4eb7a11504b12c3542ed7574dcce44ded2b2caee8d"}
Oct 31 08:36:13.883537 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:36:13.883474    4720 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-qlmsn" podStartSLOduration=15.883459613 podStartE2EDuration="15.883459613s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 08:35:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 08:36:13.881477883 +0000 UTC m=+4028.012545953" watchObservedRunningTime="2026-10-31 08:36:13.883459613 +0000 UTC m=+4028.014527683"
Oct 31 08:36:16.783752 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:36:16.783714    4720 generic.go:356] "Generic (PLEG): container finished" podID="07da9eb3-de85-4bc9-a5a1-2a3cf25765f3" containerID="abc986dcd32bd4fa17dd70939d19ad69cdcaeb28ebfefa0638219cba7ede2bfa" exitCode=0
Oct 31 08:36:16.784205 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:36:16.783776    4720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-qlmsn" event={"ID":"07da9eb3-de85-4bc9-a5a1-2a3cf25765f3","Type":"ContainerDied","Data":"abc986dcd32bd4fa17dd70939d19ad69cdcaeb28ebfefa0638219cba7ede2bfa"}
Oct 31 08:36:17.927078 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:36:17.927051    4720 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-qlmsn"
Oct 31 08:36:18.061255 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:36:18.061179    4720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/csi/ebs.csi.aws.com^vol-066f50edccf52d23d\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-066f50edccf52d23d\") pod \"07da9eb3-de85-4bc9-a5a1-2a3cf25765f3\" (UID: \"07da9eb3-de85-4bc9-a5a1-2a3cf25765f3\") "
Oct 31 08:36:18.061462 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:36:18.061280    4720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/07da9eb3-de85-4bc9-a5a1-2a3cf25765f3-kube-api-access-fhszf\" (UniqueName: \"kubernetes.io/projected/07da9eb3-de85-4bc9-a5a1-2a3cf25765f3-kube-api-access-fhszf\") pod \"07da9eb3-de85-4bc9-a5a1-2a3cf25765f3\" (UID: \"07da9eb3-de85-4bc9-a5a1-2a3cf25765f3\") "
Oct 31 08:36:18.063687 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:36:18.063658    4720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/07da9eb3-de85-4bc9-a5a1-2a3cf25765f3-kube-api-access-fhszf" pod "07da9eb3-de85-4bc9-a5a1-2a3cf25765f3" (UID: "07da9eb3-de85-4bc9-a5a1-2a3cf25765f3"). InnerVolumeSpecName "kube-api-access-fhszf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 08:36:18.065669 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:36:18.065649    4720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/ebs.csi.aws.com^vol-066f50edccf52d23d" pod "07da9eb3-de85-4bc9-a5a1-2a3cf25765f3" (UID: "07da9eb3-de85-4bc9-a5a1-2a3cf25765f3"). InnerVolumeSpecName "pvc-4bf6a61c-17dd-42d7-94fb-6807246354e3". PluginName "kubernetes.io/csi", VolumeGIDValue ""
Oct 31 08:36:18.162513 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:36:18.162475    4720 reconciler_common.go:292] "operationExecutor.UnmountDevice started for volume \"pvc-4bf6a61c-17dd-42d7-94fb-6807246354e3\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-066f50edccf52d23d\") on node \"ip-10-0-139-83.us-west-1.compute.internal\" "
Oct 31 08:36:18.162513 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:36:18.162509    4720 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fhszf\" (UniqueName: \"kubernetes.io/projected/07da9eb3-de85-4bc9-a5a1-2a3cf25765f3-kube-api-access-fhszf\") on node \"ip-10-0-139-83.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 08:36:18.793796 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:36:18.793768    4720 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-qlmsn"
Oct 31 08:36:18.793970 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:36:18.793765    4720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-qlmsn" event={"ID":"07da9eb3-de85-4bc9-a5a1-2a3cf25765f3","Type":"ContainerDied","Data":"ce3f7f2f459d274246f50c4eb7a11504b12c3542ed7574dcce44ded2b2caee8d"}
Oct 31 08:36:18.793970 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:36:18.793881    4720 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ce3f7f2f459d274246f50c4eb7a11504b12c3542ed7574dcce44ded2b2caee8d"
Oct 31 08:36:19.133984 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:36:19.133959    4720 operation_generator.go:893] UnmountDevice succeeded for volume "pvc-4bf6a61c-17dd-42d7-94fb-6807246354e3" (UniqueName: "kubernetes.io/csi/ebs.csi.aws.com^vol-066f50edccf52d23d") on node "ip-10-0-139-83.us-west-1.compute.internal"
Oct 31 08:36:19.170997 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:36:19.170959    4720 reconciler_common.go:299] "Volume detached for volume \"pvc-4bf6a61c-17dd-42d7-94fb-6807246354e3\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-066f50edccf52d23d\") on node \"ip-10-0-139-83.us-west-1.compute.internal\" DevicePath \"csi-088cc0e5c12cd8d68a78f81b27a0f5d315ce78944d37334dab4342f9d0c7c07f\""
Oct 31 08:36:20.883981 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:36:20.883942    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 08:36:23.100622 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:36:23.100581    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 08:36:25.199272 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:36:25.199238    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 08:36:25.359451 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:36:25.359413    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 08:36:32.293587 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:36:32.293554    4720 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-86bf898848-7q2nm"]
Oct 31 08:36:32.294102 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:36:32.294079    4720 cpu_manager.go:414] "containerMap: removing container" podUID="07da9eb3-de85-4bc9-a5a1-2a3cf25765f3" containerName="storage-initializer"
Oct 31 08:36:32.294145 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:36:32.294106    4720 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="07da9eb3-de85-4bc9-a5a1-2a3cf25765f3" containerName="storage-initializer"
Oct 31 08:36:32.294243 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:36:32.294229    4720 memory_manager.go:365] "RemoveStaleState removing state" podUID="07da9eb3-de85-4bc9-a5a1-2a3cf25765f3" containerName="storage-initializer"
Oct 31 08:36:32.298113 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:36:32.298088    4720 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-86bf898848-7q2nm"
Oct 31 08:36:32.347704 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:36:32.347665    4720 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-86bf898848-7q2nm"]
Oct 31 08:36:32.350001 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:36:32.349966    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 08:36:32.393639 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:36:32.393596    4720 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-with-lora-adapters-4412bdb9\"/\"default-dockercfg-bd7h9\""
Oct 31 08:36:32.393639 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:36:32.393596    4720 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-with-lora-adapters-4412bdb9\"/\"lora-pvc-single-adapter-test-kserve-self-signed-certs\""
Oct 31 08:36:32.393866 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:36:32.393599    4720 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-with-lora-adapters-4412bdb9\"/\"seaweedfs-s3-creds\""
Oct 31 08:36:32.394048 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:36:32.394034    4720 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-with-lora-adapters-4412bdb9\"/\"openshift-service-ca.crt\""
Oct 31 08:36:32.394088 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:36:32.394047    4720 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-with-lora-adapters-4412bdb9\"/\"kube-root-ca.crt\""
Oct 31 08:36:32.394122 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:36:32.394106    4720 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-with-lora-adapters-4412bdb9\"/\"odh-kserve-custom-ca-bundle\""
Oct 31 08:36:32.485354 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:36:32.485285    4720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6751475b-156f-4016-b488-6577e4964ea9-cabundle-cert\") pod \"lora-pvc-single-adapter-test-kserve-86bf898848-7q2nm\" (UID: \"6751475b-156f-4016-b488-6577e4964ea9\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-86bf898848-7q2nm"
Oct 31 08:36:32.485538 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:36:32.485355    4720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6751475b-156f-4016-b488-6577e4964ea9-tls-certs\") pod \"lora-pvc-single-adapter-test-kserve-86bf898848-7q2nm\" (UID: \"6751475b-156f-4016-b488-6577e4964ea9\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-86bf898848-7q2nm"
Oct 31 08:36:32.485538 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:36:32.485467    4720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6751475b-156f-4016-b488-6577e4964ea9-kserve-provision-location\") pod \"lora-pvc-single-adapter-test-kserve-86bf898848-7q2nm\" (UID: \"6751475b-156f-4016-b488-6577e4964ea9\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-86bf898848-7q2nm"
Oct 31 08:36:32.485538 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:36:32.485525    4720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6751475b-156f-4016-b488-6577e4964ea9-model-cache\") pod \"lora-pvc-single-adapter-test-kserve-86bf898848-7q2nm\" (UID: \"6751475b-156f-4016-b488-6577e4964ea9\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-86bf898848-7q2nm"
Oct 31 08:36:32.485645 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:36:32.485587    4720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6751475b-156f-4016-b488-6577e4964ea9-tmp-dir\") pod \"lora-pvc-single-adapter-test-kserve-86bf898848-7q2nm\" (UID: \"6751475b-156f-4016-b488-6577e4964ea9\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-86bf898848-7q2nm"
Oct 31 08:36:32.485645 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:36:32.485617    4720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6751475b-156f-4016-b488-6577e4964ea9-home\") pod \"lora-pvc-single-adapter-test-kserve-86bf898848-7q2nm\" (UID: \"6751475b-156f-4016-b488-6577e4964ea9\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-86bf898848-7q2nm"
Oct 31 08:36:32.485715 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:36:32.485681    4720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6751475b-156f-4016-b488-6577e4964ea9-dshm\") pod \"lora-pvc-single-adapter-test-kserve-86bf898848-7q2nm\" (UID: \"6751475b-156f-4016-b488-6577e4964ea9\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-86bf898848-7q2nm"
Oct 31 08:36:32.586638 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:36:32.586605    4720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6751475b-156f-4016-b488-6577e4964ea9-model-cache\") pod \"lora-pvc-single-adapter-test-kserve-86bf898848-7q2nm\" (UID: \"6751475b-156f-4016-b488-6577e4964ea9\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-86bf898848-7q2nm"
Oct 31 08:36:32.586825 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:36:32.586645    4720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6751475b-156f-4016-b488-6577e4964ea9-tmp-dir\") pod \"lora-pvc-single-adapter-test-kserve-86bf898848-7q2nm\" (UID: \"6751475b-156f-4016-b488-6577e4964ea9\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-86bf898848-7q2nm"
Oct 31 08:36:32.586825 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:36:32.586671    4720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6751475b-156f-4016-b488-6577e4964ea9-home\") pod \"lora-pvc-single-adapter-test-kserve-86bf898848-7q2nm\" (UID: \"6751475b-156f-4016-b488-6577e4964ea9\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-86bf898848-7q2nm"
Oct 31 08:36:32.586825 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:36:32.586724    4720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6751475b-156f-4016-b488-6577e4964ea9-dshm\") pod \"lora-pvc-single-adapter-test-kserve-86bf898848-7q2nm\" (UID: \"6751475b-156f-4016-b488-6577e4964ea9\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-86bf898848-7q2nm"
Oct 31 08:36:32.586825 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:36:32.586771    4720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6751475b-156f-4016-b488-6577e4964ea9-cabundle-cert\") pod \"lora-pvc-single-adapter-test-kserve-86bf898848-7q2nm\" (UID: \"6751475b-156f-4016-b488-6577e4964ea9\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-86bf898848-7q2nm"
Oct 31 08:36:32.586825 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:36:32.586795    4720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6751475b-156f-4016-b488-6577e4964ea9-tls-certs\") pod \"lora-pvc-single-adapter-test-kserve-86bf898848-7q2nm\" (UID: \"6751475b-156f-4016-b488-6577e4964ea9\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-86bf898848-7q2nm"
Oct 31 08:36:32.586825 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:36:32.586823    4720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6751475b-156f-4016-b488-6577e4964ea9-kserve-provision-location\") pod \"lora-pvc-single-adapter-test-kserve-86bf898848-7q2nm\" (UID: \"6751475b-156f-4016-b488-6577e4964ea9\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-86bf898848-7q2nm"
Oct 31 08:36:32.587173 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:36:32.586858    4720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xdpwx\" (UniqueName: \"kubernetes.io/projected/6751475b-156f-4016-b488-6577e4964ea9-kube-api-access-xdpwx\") pod \"lora-pvc-single-adapter-test-kserve-86bf898848-7q2nm\" (UID: \"6751475b-156f-4016-b488-6577e4964ea9\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-86bf898848-7q2nm"
Oct 31 08:36:32.587173 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:36:32.587093    4720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6751475b-156f-4016-b488-6577e4964ea9-tmp-dir\") pod \"lora-pvc-single-adapter-test-kserve-86bf898848-7q2nm\" (UID: \"6751475b-156f-4016-b488-6577e4964ea9\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-86bf898848-7q2nm"
Oct 31 08:36:32.587280 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:36:32.587174    4720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6751475b-156f-4016-b488-6577e4964ea9-model-cache\") pod \"lora-pvc-single-adapter-test-kserve-86bf898848-7q2nm\" (UID: \"6751475b-156f-4016-b488-6577e4964ea9\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-86bf898848-7q2nm"
Oct 31 08:36:32.587280 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:36:32.587189    4720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6751475b-156f-4016-b488-6577e4964ea9-home\") pod \"lora-pvc-single-adapter-test-kserve-86bf898848-7q2nm\" (UID: \"6751475b-156f-4016-b488-6577e4964ea9\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-86bf898848-7q2nm"
Oct 31 08:36:32.587575 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:36:32.587556    4720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6751475b-156f-4016-b488-6577e4964ea9-kserve-provision-location\") pod \"lora-pvc-single-adapter-test-kserve-86bf898848-7q2nm\" (UID: \"6751475b-156f-4016-b488-6577e4964ea9\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-86bf898848-7q2nm"
Oct 31 08:36:32.587697 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:36:32.587676    4720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6751475b-156f-4016-b488-6577e4964ea9-cabundle-cert\") pod \"lora-pvc-single-adapter-test-kserve-86bf898848-7q2nm\" (UID: \"6751475b-156f-4016-b488-6577e4964ea9\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-86bf898848-7q2nm"
Oct 31 08:36:32.589638 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:36:32.589610    4720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6751475b-156f-4016-b488-6577e4964ea9-dshm\") pod \"lora-pvc-single-adapter-test-kserve-86bf898848-7q2nm\" (UID: \"6751475b-156f-4016-b488-6577e4964ea9\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-86bf898848-7q2nm"
Oct 31 08:36:32.590051 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:36:32.590031    4720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6751475b-156f-4016-b488-6577e4964ea9-tls-certs\") pod \"lora-pvc-single-adapter-test-kserve-86bf898848-7q2nm\" (UID: \"6751475b-156f-4016-b488-6577e4964ea9\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-86bf898848-7q2nm"
Oct 31 08:36:32.687993 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:36:32.687954    4720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xdpwx\" (UniqueName: \"kubernetes.io/projected/6751475b-156f-4016-b488-6577e4964ea9-kube-api-access-xdpwx\") pod \"lora-pvc-single-adapter-test-kserve-86bf898848-7q2nm\" (UID: \"6751475b-156f-4016-b488-6577e4964ea9\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-86bf898848-7q2nm"
Oct 31 08:36:32.749052 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:36:32.749020    4720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xdpwx\" (UniqueName: \"kubernetes.io/projected/6751475b-156f-4016-b488-6577e4964ea9-kube-api-access-xdpwx\") pod \"lora-pvc-single-adapter-test-kserve-86bf898848-7q2nm\" (UID: \"6751475b-156f-4016-b488-6577e4964ea9\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-86bf898848-7q2nm"
Oct 31 08:36:34.806661 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:36:34.806618    4720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-4bf6a61c-17dd-42d7-94fb-6807246354e3\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-066f50edccf52d23d\") pod \"lora-pvc-single-adapter-test-kserve-86bf898848-7q2nm\" (UID: \"6751475b-156f-4016-b488-6577e4964ea9\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-86bf898848-7q2nm"
Oct 31 08:36:34.909271 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:36:34.909231    4720 operation_generator.go:1464] "Controller attach succeeded for volume \"pvc-4bf6a61c-17dd-42d7-94fb-6807246354e3\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-066f50edccf52d23d\") pod \"lora-pvc-single-adapter-test-kserve-86bf898848-7q2nm\" (UID: \"6751475b-156f-4016-b488-6577e4964ea9\") device path: \"\"" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-86bf898848-7q2nm"
Oct 31 08:36:35.007773 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:36:35.007730    4720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"pvc-4bf6a61c-17dd-42d7-94fb-6807246354e3\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-066f50edccf52d23d\") pod \"lora-pvc-single-adapter-test-kserve-86bf898848-7q2nm\" (UID: \"6751475b-156f-4016-b488-6577e4964ea9\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-86bf898848-7q2nm"
Oct 31 08:36:35.007992 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:36:35.007898    4720 operation_generator.go:515] "MountVolume.WaitForAttach entering for volume \"pvc-4bf6a61c-17dd-42d7-94fb-6807246354e3\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-066f50edccf52d23d\") pod \"lora-pvc-single-adapter-test-kserve-86bf898848-7q2nm\" (UID: \"6751475b-156f-4016-b488-6577e4964ea9\") DevicePath \"\"" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-86bf898848-7q2nm"
Oct 31 08:36:35.059287 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:36:35.059189    4720 operation_generator.go:525] "MountVolume.WaitForAttach succeeded for volume \"pvc-4bf6a61c-17dd-42d7-94fb-6807246354e3\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-066f50edccf52d23d\") pod \"lora-pvc-single-adapter-test-kserve-86bf898848-7q2nm\" (UID: \"6751475b-156f-4016-b488-6577e4964ea9\") DevicePath \"csi-088cc0e5c12cd8d68a78f81b27a0f5d315ce78944d37334dab4342f9d0c7c07f\"" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-86bf898848-7q2nm"
Oct 31 08:36:35.196666 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:36:35.196632    4720 operation_generator.go:557] "MountVolume.MountDevice succeeded for volume \"pvc-4bf6a61c-17dd-42d7-94fb-6807246354e3\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-066f50edccf52d23d\") pod \"lora-pvc-single-adapter-test-kserve-86bf898848-7q2nm\" (UID: \"6751475b-156f-4016-b488-6577e4964ea9\") device mount path \"/var/lib/kubelet/plugins/kubernetes.io/csi/ebs.csi.aws.com/43df6d0400fb77580adebabb960e943450a9774f25d4fd15bd5e128a7002d449/globalmount\"" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-86bf898848-7q2nm"
Oct 31 08:36:35.267453 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:36:35.267414    4720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"pvc-4bf6a61c-17dd-42d7-94fb-6807246354e3\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-066f50edccf52d23d\") pod \"lora-pvc-single-adapter-test-kserve-86bf898848-7q2nm\" (UID: \"6751475b-156f-4016-b488-6577e4964ea9\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-86bf898848-7q2nm"
Oct 31 08:36:35.614791 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:36:35.614752    4720 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-86bf898848-7q2nm"
Oct 31 08:36:36.111546 ip-10-0-139-83 kubenswrapper[4720]: W1031 08:36:36.111500    4720 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6751475b_156f_4016_b488_6577e4964ea9.slice/crio-745212f51f051f2ba9ebcc8173fd0a856ac22daaea6b946c411b0b27c706a777 WatchSource:0}: Error finding container 745212f51f051f2ba9ebcc8173fd0a856ac22daaea6b946c411b0b27c706a777: Status 404 returned error can't find the container with id 745212f51f051f2ba9ebcc8173fd0a856ac22daaea6b946c411b0b27c706a777
Oct 31 08:36:36.159826 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:36:36.159796    4720 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-86bf898848-7q2nm"]
Oct 31 08:36:36.874846 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:36:36.874805    4720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-86bf898848-7q2nm" event={"ID":"6751475b-156f-4016-b488-6577e4964ea9","Type":"ContainerStarted","Data":"f45a8ea5adb9b5718935b58663c63c6f1ee952a4fc4597c69d833a079e322844"}
Oct 31 08:36:36.875056 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:36:36.874867    4720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-86bf898848-7q2nm" event={"ID":"6751475b-156f-4016-b488-6577e4964ea9","Type":"ContainerStarted","Data":"745212f51f051f2ba9ebcc8173fd0a856ac22daaea6b946c411b0b27c706a777"}
Oct 31 08:36:37.713882 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:36:37.713852    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 08:36:38.886547 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:36:38.886512    4720 generic.go:356] "Generic (PLEG): container finished" podID="6751475b-156f-4016-b488-6577e4964ea9" containerID="f45a8ea5adb9b5718935b58663c63c6f1ee952a4fc4597c69d833a079e322844" exitCode=0
Oct 31 08:36:38.886944 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:36:38.886553    4720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-86bf898848-7q2nm" event={"ID":"6751475b-156f-4016-b488-6577e4964ea9","Type":"ContainerDied","Data":"f45a8ea5adb9b5718935b58663c63c6f1ee952a4fc4597c69d833a079e322844"}
Oct 31 08:36:39.892696 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:36:39.892664    4720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-86bf898848-7q2nm" event={"ID":"6751475b-156f-4016-b488-6577e4964ea9","Type":"ContainerStarted","Data":"fe204cf653b3654bdc3187846607819c4808577bde5c747b1af72d52ed5a449e"}
Oct 31 08:36:40.098689 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:36:40.098616    4720 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-86bf898848-7q2nm" podStartSLOduration=5.735642949 podStartE2EDuration="8.098596456s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.362953507s" isStatefulPod=true podCreationTimestamp="2026-10-31 08:36:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 08:36:40.095493109 +0000 UTC m=+4054.226561179" watchObservedRunningTime="2026-10-31 08:36:40.098596456 +0000 UTC m=+4054.229664527"
Oct 31 08:36:40.615371 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:36:40.615302    4720 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-86bf898848-7q2nm"
Oct 31 08:36:43.461682 ip-10-0-139-83 kubenswrapper[4720]: E1031 08:36:43.461621    4720 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8m768" containerName="extractor"
Oct 31 08:36:45.614925 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:36:45.614885    4720 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-86bf898848-7q2nm"
Oct 31 08:36:45.616753 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:36:45.616720    4720 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-86bf898848-7q2nm" podUID="6751475b-156f-4016-b488-6577e4964ea9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/health\": dial tcp 10.132.0.77:8000: connect: connection refused"
Oct 31 08:36:52.298187 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:36:52.298146    4720 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-56f46b8cdd-fwl5k"]
Oct 31 08:36:52.298743 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:36:52.298385    4720 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/openshift-ai-inference-openshift-default-56f46b8cdd-fwl5k" podUID="d133cdef-3e19-49da-86bd-1e75e4191ab2" containerName="istio-proxy" containerID="cri-o://8122081ffda3101946c6f766ddfda39e79bcb9ad7dbb5994551822a606b7019f" gracePeriod=30
Oct 31 08:36:55.615733 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:36:55.615689    4720 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-86bf898848-7q2nm" podUID="6751475b-156f-4016-b488-6577e4964ea9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/health\": dial tcp 10.132.0.77:8000: connect: connection refused"
Oct 31 08:36:57.548822 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:36:57.548793    4720 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-default-56f46b8cdd-fwl5k"
Oct 31 08:36:57.607872 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:36:57.607832    4720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d133cdef-3e19-49da-86bd-1e75e4191ab2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d133cdef-3e19-49da-86bd-1e75e4191ab2-istio-podinfo\") pod \"d133cdef-3e19-49da-86bd-1e75e4191ab2\" (UID: \"d133cdef-3e19-49da-86bd-1e75e4191ab2\") "
Oct 31 08:36:57.608071 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:36:57.607897    4720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d133cdef-3e19-49da-86bd-1e75e4191ab2-kube-api-access-79qv7\" (UniqueName: \"kubernetes.io/projected/d133cdef-3e19-49da-86bd-1e75e4191ab2-kube-api-access-79qv7\") pod \"d133cdef-3e19-49da-86bd-1e75e4191ab2\" (UID: \"d133cdef-3e19-49da-86bd-1e75e4191ab2\") "
Oct 31 08:36:57.608071 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:36:57.607931    4720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d133cdef-3e19-49da-86bd-1e75e4191ab2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d133cdef-3e19-49da-86bd-1e75e4191ab2-workload-socket\") pod \"d133cdef-3e19-49da-86bd-1e75e4191ab2\" (UID: \"d133cdef-3e19-49da-86bd-1e75e4191ab2\") "
Oct 31 08:36:57.608071 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:36:57.607971    4720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d133cdef-3e19-49da-86bd-1e75e4191ab2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d133cdef-3e19-49da-86bd-1e75e4191ab2-istiod-ca-cert\") pod \"d133cdef-3e19-49da-86bd-1e75e4191ab2\" (UID: \"d133cdef-3e19-49da-86bd-1e75e4191ab2\") "
Oct 31 08:36:57.608071 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:36:57.608000    4720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d133cdef-3e19-49da-86bd-1e75e4191ab2-istio-token\" (UniqueName: \"kubernetes.io/projected/d133cdef-3e19-49da-86bd-1e75e4191ab2-istio-token\") pod \"d133cdef-3e19-49da-86bd-1e75e4191ab2\" (UID: \"d133cdef-3e19-49da-86bd-1e75e4191ab2\") "
Oct 31 08:36:57.608071 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:36:57.608045    4720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d133cdef-3e19-49da-86bd-1e75e4191ab2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d133cdef-3e19-49da-86bd-1e75e4191ab2-workload-certs\") pod \"d133cdef-3e19-49da-86bd-1e75e4191ab2\" (UID: \"d133cdef-3e19-49da-86bd-1e75e4191ab2\") "
Oct 31 08:36:57.608364 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:36:57.608071    4720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d133cdef-3e19-49da-86bd-1e75e4191ab2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d133cdef-3e19-49da-86bd-1e75e4191ab2-istio-envoy\") pod \"d133cdef-3e19-49da-86bd-1e75e4191ab2\" (UID: \"d133cdef-3e19-49da-86bd-1e75e4191ab2\") "
Oct 31 08:36:57.608364 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:36:57.608094    4720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d133cdef-3e19-49da-86bd-1e75e4191ab2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d133cdef-3e19-49da-86bd-1e75e4191ab2-istio-data\") pod \"d133cdef-3e19-49da-86bd-1e75e4191ab2\" (UID: \"d133cdef-3e19-49da-86bd-1e75e4191ab2\") "
Oct 31 08:36:57.608364 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:36:57.608127    4720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d133cdef-3e19-49da-86bd-1e75e4191ab2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d133cdef-3e19-49da-86bd-1e75e4191ab2-credential-socket\") pod \"d133cdef-3e19-49da-86bd-1e75e4191ab2\" (UID: \"d133cdef-3e19-49da-86bd-1e75e4191ab2\") "
Oct 31 08:36:57.608533 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:36:57.608319    4720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d133cdef-3e19-49da-86bd-1e75e4191ab2-workload-socket" pod "d133cdef-3e19-49da-86bd-1e75e4191ab2" (UID: "d133cdef-3e19-49da-86bd-1e75e4191ab2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 08:36:57.608533 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:36:57.608388    4720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d133cdef-3e19-49da-86bd-1e75e4191ab2-workload-certs" pod "d133cdef-3e19-49da-86bd-1e75e4191ab2" (UID: "d133cdef-3e19-49da-86bd-1e75e4191ab2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 08:36:57.608533 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:36:57.608458    4720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d133cdef-3e19-49da-86bd-1e75e4191ab2-istiod-ca-cert" pod "d133cdef-3e19-49da-86bd-1e75e4191ab2" (UID: "d133cdef-3e19-49da-86bd-1e75e4191ab2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 08:36:57.608685 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:36:57.608589    4720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d133cdef-3e19-49da-86bd-1e75e4191ab2-credential-socket" pod "d133cdef-3e19-49da-86bd-1e75e4191ab2" (UID: "d133cdef-3e19-49da-86bd-1e75e4191ab2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 08:36:57.608685 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:36:57.608636    4720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d133cdef-3e19-49da-86bd-1e75e4191ab2-istio-data" pod "d133cdef-3e19-49da-86bd-1e75e4191ab2" (UID: "d133cdef-3e19-49da-86bd-1e75e4191ab2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 08:36:57.610638 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:36:57.610608    4720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d133cdef-3e19-49da-86bd-1e75e4191ab2-istio-podinfo" pod "d133cdef-3e19-49da-86bd-1e75e4191ab2" (UID: "d133cdef-3e19-49da-86bd-1e75e4191ab2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 08:36:57.610761 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:36:57.610741    4720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d133cdef-3e19-49da-86bd-1e75e4191ab2-istio-token" pod "d133cdef-3e19-49da-86bd-1e75e4191ab2" (UID: "d133cdef-3e19-49da-86bd-1e75e4191ab2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 08:36:57.610914 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:36:57.610886    4720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d133cdef-3e19-49da-86bd-1e75e4191ab2-kube-api-access-79qv7" pod "d133cdef-3e19-49da-86bd-1e75e4191ab2" (UID: "d133cdef-3e19-49da-86bd-1e75e4191ab2"). InnerVolumeSpecName "kube-api-access-79qv7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 08:36:57.610974 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:36:57.610922    4720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d133cdef-3e19-49da-86bd-1e75e4191ab2-istio-envoy" pod "d133cdef-3e19-49da-86bd-1e75e4191ab2" (UID: "d133cdef-3e19-49da-86bd-1e75e4191ab2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 08:36:57.709136 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:36:57.709096    4720 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d133cdef-3e19-49da-86bd-1e75e4191ab2-istio-podinfo\") on node \"ip-10-0-139-83.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 08:36:57.709136 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:36:57.709128    4720 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-79qv7\" (UniqueName: \"kubernetes.io/projected/d133cdef-3e19-49da-86bd-1e75e4191ab2-kube-api-access-79qv7\") on node \"ip-10-0-139-83.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 08:36:57.709136 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:36:57.709138    4720 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d133cdef-3e19-49da-86bd-1e75e4191ab2-workload-socket\") on node \"ip-10-0-139-83.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 08:36:57.709418 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:36:57.709149    4720 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d133cdef-3e19-49da-86bd-1e75e4191ab2-istiod-ca-cert\") on node \"ip-10-0-139-83.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 08:36:57.709418 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:36:57.709157    4720 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d133cdef-3e19-49da-86bd-1e75e4191ab2-istio-token\") on node \"ip-10-0-139-83.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 08:36:57.709418 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:36:57.709165    4720 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d133cdef-3e19-49da-86bd-1e75e4191ab2-workload-certs\") on node \"ip-10-0-139-83.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 08:36:57.709418 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:36:57.709173    4720 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d133cdef-3e19-49da-86bd-1e75e4191ab2-istio-envoy\") on node \"ip-10-0-139-83.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 08:36:57.709418 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:36:57.709181    4720 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d133cdef-3e19-49da-86bd-1e75e4191ab2-istio-data\") on node \"ip-10-0-139-83.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 08:36:57.709418 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:36:57.709189    4720 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d133cdef-3e19-49da-86bd-1e75e4191ab2-credential-socket\") on node \"ip-10-0-139-83.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 08:36:57.983511 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:36:57.983406    4720 generic.go:356] "Generic (PLEG): container finished" podID="d133cdef-3e19-49da-86bd-1e75e4191ab2" containerID="8122081ffda3101946c6f766ddfda39e79bcb9ad7dbb5994551822a606b7019f" exitCode=0
Oct 31 08:36:57.983511 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:36:57.983467    4720 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-default-56f46b8cdd-fwl5k"
Oct 31 08:36:57.983511 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:36:57.983490    4720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-56f46b8cdd-fwl5k" event={"ID":"d133cdef-3e19-49da-86bd-1e75e4191ab2","Type":"ContainerDied","Data":"8122081ffda3101946c6f766ddfda39e79bcb9ad7dbb5994551822a606b7019f"}
Oct 31 08:36:57.983791 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:36:57.983554    4720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-56f46b8cdd-fwl5k" event={"ID":"d133cdef-3e19-49da-86bd-1e75e4191ab2","Type":"ContainerDied","Data":"eb3dbaad1158afa9a31224758dcb5dc1254ab7f28fb999331f4e8ce2e963c1f8"}
Oct 31 08:36:57.983791 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:36:57.983577    4720 scope.go:122] "RemoveContainer" containerID="8122081ffda3101946c6f766ddfda39e79bcb9ad7dbb5994551822a606b7019f"
Oct 31 08:36:57.993427 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:36:57.993407    4720 scope.go:122] "RemoveContainer" containerID="8122081ffda3101946c6f766ddfda39e79bcb9ad7dbb5994551822a606b7019f"
Oct 31 08:36:57.993756 ip-10-0-139-83 kubenswrapper[4720]: E1031 08:36:57.993736    4720 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8122081ffda3101946c6f766ddfda39e79bcb9ad7dbb5994551822a606b7019f\": container with ID starting with 8122081ffda3101946c6f766ddfda39e79bcb9ad7dbb5994551822a606b7019f not found: ID does not exist" containerID="8122081ffda3101946c6f766ddfda39e79bcb9ad7dbb5994551822a606b7019f"
Oct 31 08:36:57.993804 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:36:57.993765    4720 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8122081ffda3101946c6f766ddfda39e79bcb9ad7dbb5994551822a606b7019f"} err="failed to get container status \"8122081ffda3101946c6f766ddfda39e79bcb9ad7dbb5994551822a606b7019f\": rpc error: code = NotFound desc = could not find container \"8122081ffda3101946c6f766ddfda39e79bcb9ad7dbb5994551822a606b7019f\": container with ID starting with 8122081ffda3101946c6f766ddfda39e79bcb9ad7dbb5994551822a606b7019f not found: ID does not exist"
Oct 31 08:36:58.148457 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:36:58.148415    4720 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-56f46b8cdd-fwl5k"]
Oct 31 08:36:58.196515 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:36:58.196473    4720 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-56f46b8cdd-fwl5k"]
Oct 31 08:37:00.025032 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:37:00.024986    4720 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d133cdef-3e19-49da-86bd-1e75e4191ab2" path="/var/lib/kubelet/pods/d133cdef-3e19-49da-86bd-1e75e4191ab2/volumes"
Oct 31 08:37:02.512742 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:37:02.512700    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 08:37:05.615749 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:37:05.615708    4720 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-86bf898848-7q2nm" podUID="6751475b-156f-4016-b488-6577e4964ea9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/health\": dial tcp 10.132.0.77:8000: connect: connection refused"
Oct 31 08:37:13.097626 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:37:13.097583    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 08:37:13.249965 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:37:13.249925    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 08:37:13.462674 ip-10-0-139-83 kubenswrapper[4720]: E1031 08:37:13.462576    4720 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8m768" containerName="extractor"
Oct 31 08:37:15.615748 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:37:15.615662    4720 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-86bf898848-7q2nm" podUID="6751475b-156f-4016-b488-6577e4964ea9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/health\": dial tcp 10.132.0.77:8000: connect: connection refused"
Oct 31 08:37:23.456559 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:37:23.456521    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 08:37:25.615850 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:37:25.615803    4720 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-86bf898848-7q2nm" podUID="6751475b-156f-4016-b488-6577e4964ea9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/health\": dial tcp 10.132.0.77:8000: connect: connection refused"
Oct 31 08:37:35.615752 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:37:35.615707    4720 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-86bf898848-7q2nm" podUID="6751475b-156f-4016-b488-6577e4964ea9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/health\": dial tcp 10.132.0.77:8000: connect: connection refused"
Oct 31 08:37:38.056062 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:37:38.056027    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 08:37:41.105513 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:37:41.105462    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 08:37:43.463696 ip-10-0-139-83 kubenswrapper[4720]: E1031 08:37:43.463644    4720 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8m768" containerName="extractor"
Oct 31 08:37:45.616011 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:37:45.615964    4720 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-86bf898848-7q2nm" podUID="6751475b-156f-4016-b488-6577e4964ea9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/health\": dial tcp 10.132.0.77:8000: connect: connection refused"
Oct 31 08:37:55.626124 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:37:55.626088    4720 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-86bf898848-7q2nm"
Oct 31 08:37:55.634418 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:37:55.634379    4720 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-86bf898848-7q2nm"
Oct 31 08:38:00.573810 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:38:00.573772    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 08:38:00.747610 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:38:00.747575    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 08:38:02.844780 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:38:02.844746    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 08:38:06.650265 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:38:06.650230    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 08:38:08.884399 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:38:08.884351    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 08:38:10.127382 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:38:10.127318    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 08:38:10.359915 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:38:10.359881    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 08:38:10.568901 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:38:10.568824    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 08:38:10.784253 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:38:10.784212    4720 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"]
Oct 31 08:38:10.784642 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:38:10.784624    4720 cpu_manager.go:414] "containerMap: removing container" podUID="d133cdef-3e19-49da-86bd-1e75e4191ab2" containerName="istio-proxy"
Oct 31 08:38:10.784728 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:38:10.784645    4720 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d133cdef-3e19-49da-86bd-1e75e4191ab2" containerName="istio-proxy"
Oct 31 08:38:10.784794 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:38:10.784779    4720 memory_manager.go:365] "RemoveStaleState removing state" podUID="d133cdef-3e19-49da-86bd-1e75e4191ab2" containerName="istio-proxy"
Oct 31 08:38:10.788097 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:38:10.788071    4720 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Oct 31 08:38:10.849068 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:38:10.849029    4720 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"]
Oct 31 08:38:10.878345 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:38:10.878297    4720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a9851ffc-5ef5-49a7-93a5-02032024e205-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"a9851ffc-5ef5-49a7-93a5-02032024e205\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Oct 31 08:38:10.878536 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:38:10.878434    4720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sxgnl\" (UniqueName: \"kubernetes.io/projected/a9851ffc-5ef5-49a7-93a5-02032024e205-kube-api-access-sxgnl\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"a9851ffc-5ef5-49a7-93a5-02032024e205\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Oct 31 08:38:10.878536 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:38:10.878495    4720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a9851ffc-5ef5-49a7-93a5-02032024e205-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"a9851ffc-5ef5-49a7-93a5-02032024e205\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Oct 31 08:38:10.878536 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:38:10.878523    4720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a9851ffc-5ef5-49a7-93a5-02032024e205-home\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"a9851ffc-5ef5-49a7-93a5-02032024e205\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Oct 31 08:38:10.878647 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:38:10.878572    4720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a9851ffc-5ef5-49a7-93a5-02032024e205-dshm\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"a9851ffc-5ef5-49a7-93a5-02032024e205\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Oct 31 08:38:10.878681 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:38:10.878634    4720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a9851ffc-5ef5-49a7-93a5-02032024e205-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"a9851ffc-5ef5-49a7-93a5-02032024e205\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Oct 31 08:38:10.878752 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:38:10.878735    4720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a9851ffc-5ef5-49a7-93a5-02032024e205-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"a9851ffc-5ef5-49a7-93a5-02032024e205\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Oct 31 08:38:10.878789 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:38:10.878764    4720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a9851ffc-5ef5-49a7-93a5-02032024e205-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"a9851ffc-5ef5-49a7-93a5-02032024e205\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Oct 31 08:38:10.890155 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:38:10.890079    4720 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"seaweedfs-s3-creds\""
Oct 31 08:38:10.890155 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:38:10.890084    4720 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"llmisvc-model-fb-opt-125m-route-dc21cb-57cf7736-dockercfg-x5bhj\""
Oct 31 08:38:10.890155 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:38:10.890138    4720 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"default-dockercfg-khhlv\""
Oct 31 08:38:10.890452 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:38:10.890185    4720 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"llmisv32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-self-signed-certs\""
Oct 31 08:38:10.893646 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:38:10.893621    4720 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"odh-kserve-custom-ca-bundle\""
Oct 31 08:38:10.893646 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:38:10.893629    4720 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"kube-root-ca.crt\""
Oct 31 08:38:10.893884 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:38:10.893854    4720 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"openshift-service-ca.crt\""
Oct 31 08:38:10.980066 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:38:10.980020    4720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a9851ffc-5ef5-49a7-93a5-02032024e205-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"a9851ffc-5ef5-49a7-93a5-02032024e205\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Oct 31 08:38:10.980277 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:38:10.980079    4720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a9851ffc-5ef5-49a7-93a5-02032024e205-home\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"a9851ffc-5ef5-49a7-93a5-02032024e205\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Oct 31 08:38:10.980277 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:38:10.980115    4720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a9851ffc-5ef5-49a7-93a5-02032024e205-dshm\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"a9851ffc-5ef5-49a7-93a5-02032024e205\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Oct 31 08:38:10.980277 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:38:10.980143    4720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a9851ffc-5ef5-49a7-93a5-02032024e205-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"a9851ffc-5ef5-49a7-93a5-02032024e205\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Oct 31 08:38:10.980277 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:38:10.980175    4720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a9851ffc-5ef5-49a7-93a5-02032024e205-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"a9851ffc-5ef5-49a7-93a5-02032024e205\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Oct 31 08:38:10.980277 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:38:10.980206    4720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a9851ffc-5ef5-49a7-93a5-02032024e205-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"a9851ffc-5ef5-49a7-93a5-02032024e205\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Oct 31 08:38:10.980572 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:38:10.980277    4720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a9851ffc-5ef5-49a7-93a5-02032024e205-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"a9851ffc-5ef5-49a7-93a5-02032024e205\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Oct 31 08:38:10.980572 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:38:10.980349    4720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sxgnl\" (UniqueName: \"kubernetes.io/projected/a9851ffc-5ef5-49a7-93a5-02032024e205-kube-api-access-sxgnl\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"a9851ffc-5ef5-49a7-93a5-02032024e205\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Oct 31 08:38:10.980688 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:38:10.980589    4720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a9851ffc-5ef5-49a7-93a5-02032024e205-home\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"a9851ffc-5ef5-49a7-93a5-02032024e205\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Oct 31 08:38:10.980747 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:38:10.980699    4720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a9851ffc-5ef5-49a7-93a5-02032024e205-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"a9851ffc-5ef5-49a7-93a5-02032024e205\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Oct 31 08:38:10.980855 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:38:10.980836    4720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a9851ffc-5ef5-49a7-93a5-02032024e205-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"a9851ffc-5ef5-49a7-93a5-02032024e205\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Oct 31 08:38:10.980926 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:38:10.980833    4720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a9851ffc-5ef5-49a7-93a5-02032024e205-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"a9851ffc-5ef5-49a7-93a5-02032024e205\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Oct 31 08:38:10.980976 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:38:10.980939    4720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a9851ffc-5ef5-49a7-93a5-02032024e205-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"a9851ffc-5ef5-49a7-93a5-02032024e205\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Oct 31 08:38:10.983317 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:38:10.983286    4720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a9851ffc-5ef5-49a7-93a5-02032024e205-dshm\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"a9851ffc-5ef5-49a7-93a5-02032024e205\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Oct 31 08:38:10.983432 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:38:10.983407    4720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a9851ffc-5ef5-49a7-93a5-02032024e205-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"a9851ffc-5ef5-49a7-93a5-02032024e205\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Oct 31 08:38:11.037134 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:38:11.037101    4720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sxgnl\" (UniqueName: \"kubernetes.io/projected/a9851ffc-5ef5-49a7-93a5-02032024e205-kube-api-access-sxgnl\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"a9851ffc-5ef5-49a7-93a5-02032024e205\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Oct 31 08:38:11.103508 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:38:11.103415    4720 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Oct 31 08:38:11.541578 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:38:11.541553    4720 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"]
Oct 31 08:38:11.544688 ip-10-0-139-83 kubenswrapper[4720]: W1031 08:38:11.544654    4720 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda9851ffc_5ef5_49a7_93a5_02032024e205.slice/crio-a4bdcf165b87df388df54b3d9592129ac6514b9098c7121fe30fde08608ca391 WatchSource:0}: Error finding container a4bdcf165b87df388df54b3d9592129ac6514b9098c7121fe30fde08608ca391: Status 404 returned error can't find the container with id a4bdcf165b87df388df54b3d9592129ac6514b9098c7121fe30fde08608ca391
Oct 31 08:38:12.125534 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:38:12.125494    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 08:38:12.280246 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:38:12.280207    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 08:38:12.341501 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:38:12.341467    4720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" event={"ID":"a9851ffc-5ef5-49a7-93a5-02032024e205","Type":"ContainerStarted","Data":"5c8153830ad86cd6c2b143241b5189f511f2dd2b104c0277f77a482de18a8a96"}
Oct 31 08:38:12.341711 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:38:12.341538    4720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" event={"ID":"a9851ffc-5ef5-49a7-93a5-02032024e205","Type":"ContainerStarted","Data":"a4bdcf165b87df388df54b3d9592129ac6514b9098c7121fe30fde08608ca391"}
Oct 31 08:38:13.462514 ip-10-0-139-83 kubenswrapper[4720]: E1031 08:38:13.462471    4720 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8m768" containerName="extractor"
Oct 31 08:38:14.353537 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:38:14.353494    4720 generic.go:356] "Generic (PLEG): container finished" podID="a9851ffc-5ef5-49a7-93a5-02032024e205" containerID="5c8153830ad86cd6c2b143241b5189f511f2dd2b104c0277f77a482de18a8a96" exitCode=0
Oct 31 08:38:14.353714 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:38:14.353549    4720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" event={"ID":"a9851ffc-5ef5-49a7-93a5-02032024e205","Type":"ContainerDied","Data":"5c8153830ad86cd6c2b143241b5189f511f2dd2b104c0277f77a482de18a8a96"}
Oct 31 08:38:15.361307 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:38:15.361270    4720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" event={"ID":"a9851ffc-5ef5-49a7-93a5-02032024e205","Type":"ContainerStarted","Data":"71607751ab3ee87089c7dd7988c2319ab7f164a16b5620e7d9b32db184c94807"}
Oct 31 08:38:15.528750 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:38:15.528690    4720 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" podStartSLOduration=2.996511463 podStartE2EDuration="5.528676253s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.53216479s" isStatefulPod=false podCreationTimestamp="2026-10-31 08:38:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 08:38:15.526148812 +0000 UTC m=+4149.657216882" watchObservedRunningTime="2026-10-31 08:38:15.528676253 +0000 UTC m=+4149.659744323"
Oct 31 08:38:19.041487 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:38:19.041423    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 08:38:23.769632 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:38:23.769594    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 08:38:24.475013 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:38:24.474980    4720 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-with-lora-adapters-4412bdb9_lora-pvc-single-adapter-test-kserve-86bf898848-7q2nm_6751475b-156f-4016-b488-6577e4964ea9/storage-initializer/0.log"
Oct 31 08:38:24.592963 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:38:24.592925    4720 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-with-lora-adapters-4412bdb9_lora-pvc-single-adapter-test-kserve-86bf898848-7q2nm_6751475b-156f-4016-b488-6577e4964ea9/main/0.log"
Oct 31 08:38:28.842238 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:38:28.842197    4720 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-86bf898848-7q2nm"]
Oct 31 08:38:28.868362 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:38:28.868295    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 08:38:38.391608 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:38:38.391560    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 08:38:43.461958 ip-10-0-139-83 kubenswrapper[4720]: E1031 08:38:43.461904    4720 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8m768" containerName="extractor"
Oct 31 08:38:43.866348 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:38:43.866269    4720 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-86bf898848-7q2nm" podUID="6751475b-156f-4016-b488-6577e4964ea9" containerName="main" containerID="cri-o://fe204cf653b3654bdc3187846607819c4808577bde5c747b1af72d52ed5a449e" gracePeriod=45
Oct 31 08:38:44.513900 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:38:44.513867    4720 generic.go:356] "Generic (PLEG): container finished" podID="6751475b-156f-4016-b488-6577e4964ea9" containerID="fe204cf653b3654bdc3187846607819c4808577bde5c747b1af72d52ed5a449e" exitCode=0
Oct 31 08:38:44.514372 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:38:44.513936    4720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-86bf898848-7q2nm" event={"ID":"6751475b-156f-4016-b488-6577e4964ea9","Type":"ContainerDied","Data":"fe204cf653b3654bdc3187846607819c4808577bde5c747b1af72d52ed5a449e"}
Oct 31 08:38:44.535075 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:38:44.535044    4720 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-86bf898848-7q2nm"
Oct 31 08:38:44.597121 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:38:44.597087    4720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6751475b-156f-4016-b488-6577e4964ea9-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6751475b-156f-4016-b488-6577e4964ea9-cabundle-cert\") pod \"6751475b-156f-4016-b488-6577e4964ea9\" (UID: \"6751475b-156f-4016-b488-6577e4964ea9\") "
Oct 31 08:38:44.597121 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:38:44.597131    4720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6751475b-156f-4016-b488-6577e4964ea9-kube-api-access-xdpwx\" (UniqueName: \"kubernetes.io/projected/6751475b-156f-4016-b488-6577e4964ea9-kube-api-access-xdpwx\") pod \"6751475b-156f-4016-b488-6577e4964ea9\" (UID: \"6751475b-156f-4016-b488-6577e4964ea9\") "
Oct 31 08:38:44.597421 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:38:44.597184    4720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6751475b-156f-4016-b488-6577e4964ea9-home\" (UniqueName: \"kubernetes.io/empty-dir/6751475b-156f-4016-b488-6577e4964ea9-home\") pod \"6751475b-156f-4016-b488-6577e4964ea9\" (UID: \"6751475b-156f-4016-b488-6577e4964ea9\") "
Oct 31 08:38:44.597421 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:38:44.597229    4720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6751475b-156f-4016-b488-6577e4964ea9-tls-certs\" (UniqueName: \"kubernetes.io/secret/6751475b-156f-4016-b488-6577e4964ea9-tls-certs\") pod \"6751475b-156f-4016-b488-6577e4964ea9\" (UID: \"6751475b-156f-4016-b488-6577e4964ea9\") "
Oct 31 08:38:44.597421 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:38:44.597271    4720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6751475b-156f-4016-b488-6577e4964ea9-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6751475b-156f-4016-b488-6577e4964ea9-kserve-provision-location\") pod \"6751475b-156f-4016-b488-6577e4964ea9\" (UID: \"6751475b-156f-4016-b488-6577e4964ea9\") "
Oct 31 08:38:44.597421 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:38:44.597352    4720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6751475b-156f-4016-b488-6577e4964ea9-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6751475b-156f-4016-b488-6577e4964ea9-model-cache\") pod \"6751475b-156f-4016-b488-6577e4964ea9\" (UID: \"6751475b-156f-4016-b488-6577e4964ea9\") "
Oct 31 08:38:44.598095 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:38:44.597477    4720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/csi/ebs.csi.aws.com^vol-066f50edccf52d23d\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-066f50edccf52d23d\") pod \"6751475b-156f-4016-b488-6577e4964ea9\" (UID: \"6751475b-156f-4016-b488-6577e4964ea9\") "
Oct 31 08:38:44.598095 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:38:44.597518    4720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6751475b-156f-4016-b488-6577e4964ea9-dshm\" (UniqueName: \"kubernetes.io/empty-dir/6751475b-156f-4016-b488-6577e4964ea9-dshm\") pod \"6751475b-156f-4016-b488-6577e4964ea9\" (UID: \"6751475b-156f-4016-b488-6577e4964ea9\") "
Oct 31 08:38:44.598095 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:38:44.597571    4720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6751475b-156f-4016-b488-6577e4964ea9-cabundle-cert" pod "6751475b-156f-4016-b488-6577e4964ea9" (UID: "6751475b-156f-4016-b488-6577e4964ea9"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 08:38:44.598095 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:38:44.597582    4720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6751475b-156f-4016-b488-6577e4964ea9-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6751475b-156f-4016-b488-6577e4964ea9-tmp-dir\") pod \"6751475b-156f-4016-b488-6577e4964ea9\" (UID: \"6751475b-156f-4016-b488-6577e4964ea9\") "
Oct 31 08:38:44.598095 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:38:44.597611    4720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6751475b-156f-4016-b488-6577e4964ea9-home" pod "6751475b-156f-4016-b488-6577e4964ea9" (UID: "6751475b-156f-4016-b488-6577e4964ea9"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 08:38:44.598095 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:38:44.597708    4720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6751475b-156f-4016-b488-6577e4964ea9-model-cache" pod "6751475b-156f-4016-b488-6577e4964ea9" (UID: "6751475b-156f-4016-b488-6577e4964ea9"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 08:38:44.598095 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:38:44.597974    4720 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6751475b-156f-4016-b488-6577e4964ea9-model-cache\") on node \"ip-10-0-139-83.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 08:38:44.598095 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:38:44.597994    4720 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6751475b-156f-4016-b488-6577e4964ea9-cabundle-cert\") on node \"ip-10-0-139-83.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 08:38:44.598095 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:38:44.598006    4720 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6751475b-156f-4016-b488-6577e4964ea9-home\") on node \"ip-10-0-139-83.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 08:38:44.600065 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:38:44.600026    4720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6751475b-156f-4016-b488-6577e4964ea9-kube-api-access-xdpwx" pod "6751475b-156f-4016-b488-6577e4964ea9" (UID: "6751475b-156f-4016-b488-6577e4964ea9"). InnerVolumeSpecName "kube-api-access-xdpwx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 08:38:44.600655 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:38:44.600626    4720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6751475b-156f-4016-b488-6577e4964ea9-dshm" pod "6751475b-156f-4016-b488-6577e4964ea9" (UID: "6751475b-156f-4016-b488-6577e4964ea9"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 08:38:44.600976 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:38:44.600947    4720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6751475b-156f-4016-b488-6577e4964ea9-tls-certs" pod "6751475b-156f-4016-b488-6577e4964ea9" (UID: "6751475b-156f-4016-b488-6577e4964ea9"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 08:38:44.603870 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:38:44.603849    4720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/ebs.csi.aws.com^vol-066f50edccf52d23d" pod "6751475b-156f-4016-b488-6577e4964ea9" (UID: "6751475b-156f-4016-b488-6577e4964ea9"). InnerVolumeSpecName "pvc-4bf6a61c-17dd-42d7-94fb-6807246354e3". PluginName "kubernetes.io/csi", VolumeGIDValue ""
Oct 31 08:38:44.615106 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:38:44.615068    4720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6751475b-156f-4016-b488-6577e4964ea9-tmp-dir" pod "6751475b-156f-4016-b488-6577e4964ea9" (UID: "6751475b-156f-4016-b488-6577e4964ea9"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 08:38:44.626157 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:38:44.626098    4720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6751475b-156f-4016-b488-6577e4964ea9-kserve-provision-location" pod "6751475b-156f-4016-b488-6577e4964ea9" (UID: "6751475b-156f-4016-b488-6577e4964ea9"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 08:38:44.698886 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:38:44.698841    4720 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6751475b-156f-4016-b488-6577e4964ea9-tmp-dir\") on node \"ip-10-0-139-83.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 08:38:44.698886 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:38:44.698884    4720 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xdpwx\" (UniqueName: \"kubernetes.io/projected/6751475b-156f-4016-b488-6577e4964ea9-kube-api-access-xdpwx\") on node \"ip-10-0-139-83.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 08:38:44.699123 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:38:44.698898    4720 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6751475b-156f-4016-b488-6577e4964ea9-tls-certs\") on node \"ip-10-0-139-83.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 08:38:44.699123 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:38:44.698911    4720 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6751475b-156f-4016-b488-6577e4964ea9-kserve-provision-location\") on node \"ip-10-0-139-83.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 08:38:44.699123 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:38:44.698949    4720 reconciler_common.go:292] "operationExecutor.UnmountDevice started for volume \"pvc-4bf6a61c-17dd-42d7-94fb-6807246354e3\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-066f50edccf52d23d\") on node \"ip-10-0-139-83.us-west-1.compute.internal\" "
Oct 31 08:38:44.699123 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:38:44.698962    4720 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6751475b-156f-4016-b488-6577e4964ea9-dshm\") on node \"ip-10-0-139-83.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 08:38:44.709572 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:38:44.709550    4720 operation_generator.go:893] UnmountDevice succeeded for volume "pvc-4bf6a61c-17dd-42d7-94fb-6807246354e3" (UniqueName: "kubernetes.io/csi/ebs.csi.aws.com^vol-066f50edccf52d23d") on node "ip-10-0-139-83.us-west-1.compute.internal"
Oct 31 08:38:44.800003 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:38:44.799968    4720 reconciler_common.go:299] "Volume detached for volume \"pvc-4bf6a61c-17dd-42d7-94fb-6807246354e3\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-066f50edccf52d23d\") on node \"ip-10-0-139-83.us-west-1.compute.internal\" DevicePath \"csi-088cc0e5c12cd8d68a78f81b27a0f5d315ce78944d37334dab4342f9d0c7c07f\""
Oct 31 08:38:45.522075 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:38:45.522036    4720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-86bf898848-7q2nm" event={"ID":"6751475b-156f-4016-b488-6577e4964ea9","Type":"ContainerDied","Data":"745212f51f051f2ba9ebcc8173fd0a856ac22daaea6b946c411b0b27c706a777"}
Oct 31 08:38:45.522075 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:38:45.522059    4720 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-86bf898848-7q2nm"
Oct 31 08:38:45.522874 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:38:45.522104    4720 scope.go:122] "RemoveContainer" containerID="fe204cf653b3654bdc3187846607819c4808577bde5c747b1af72d52ed5a449e"
Oct 31 08:38:45.532743 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:38:45.532716    4720 scope.go:122] "RemoveContainer" containerID="f45a8ea5adb9b5718935b58663c63c6f1ee952a4fc4597c69d833a079e322844"
Oct 31 08:38:45.736164 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:38:45.736058    4720 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-86bf898848-7q2nm"]
Oct 31 08:38:45.785535 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:38:45.785501    4720 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-86bf898848-7q2nm"]
Oct 31 08:38:46.026485 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:38:46.026389    4720 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6751475b-156f-4016-b488-6577e4964ea9" path="/var/lib/kubelet/pods/6751475b-156f-4016-b488-6577e4964ea9/volumes"
Oct 31 08:38:49.145642 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:38:49.145604    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 08:38:49.308571 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:38:49.308531    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 08:38:49.476304 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:38:49.476206    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 08:38:49.635094 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:38:49.635059    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 08:38:49.802395 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:38:49.802294    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 08:38:49.958712 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:38:49.958675    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 08:38:52.065226 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:38:52.065191    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 08:38:52.214879 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:38:52.214834    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 08:38:52.738798 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:38:52.738758    4720 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-zqwkd"]
Oct 31 08:38:52.739100 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:38:52.739088    4720 cpu_manager.go:414] "containerMap: removing container" podUID="6751475b-156f-4016-b488-6577e4964ea9" containerName="storage-initializer"
Oct 31 08:38:52.739168 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:38:52.739102    4720 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6751475b-156f-4016-b488-6577e4964ea9" containerName="storage-initializer"
Oct 31 08:38:52.739168 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:38:52.739121    4720 cpu_manager.go:414] "containerMap: removing container" podUID="6751475b-156f-4016-b488-6577e4964ea9" containerName="main"
Oct 31 08:38:52.739168 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:38:52.739126    4720 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6751475b-156f-4016-b488-6577e4964ea9" containerName="main"
Oct 31 08:38:52.739286 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:38:52.739188    4720 memory_manager.go:365] "RemoveStaleState removing state" podUID="6751475b-156f-4016-b488-6577e4964ea9" containerName="main"
Oct 31 08:38:52.743483 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:38:52.743463    4720 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-zqwkd"
Oct 31 08:38:52.798453 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:38:52.798417    4720 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-zqwkd"]
Oct 31 08:38:52.844862 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:38:52.844824    4720 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-with-lora-adapters-a290d4b0\"/\"openshift-service-ca.crt\""
Oct 31 08:38:52.844862 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:38:52.844838    4720 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-with-lora-adapters-a290d4b0\"/\"seaweedfs-s3-creds\""
Oct 31 08:38:52.845081 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:38:52.844870    4720 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-with-lora-adapters-a290d4b0\"/\"kube-root-ca.crt\""
Oct 31 08:38:52.845081 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:38:52.844922    4720 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-with-lora-adapters-a290d4b0\"/\"default-dockercfg-th4wv\""
Oct 31 08:38:52.859955 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:38:52.859920    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 08:38:52.968688 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:38:52.968655    4720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zscxq\" (UniqueName: \"kubernetes.io/projected/6d2844ac-8483-400b-8044-73a9f1d6461f-kube-api-access-zscxq\") pod \"e2e-pvc-lora-download-zqwkd\" (UID: \"6d2844ac-8483-400b-8044-73a9f1d6461f\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-zqwkd"
Oct 31 08:38:53.069913 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:38:53.069825    4720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zscxq\" (UniqueName: \"kubernetes.io/projected/6d2844ac-8483-400b-8044-73a9f1d6461f-kube-api-access-zscxq\") pod \"e2e-pvc-lora-download-zqwkd\" (UID: \"6d2844ac-8483-400b-8044-73a9f1d6461f\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-zqwkd"
Oct 31 08:38:53.126818 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:38:53.126783    4720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zscxq\" (UniqueName: \"kubernetes.io/projected/6d2844ac-8483-400b-8044-73a9f1d6461f-kube-api-access-zscxq\") pod \"e2e-pvc-lora-download-zqwkd\" (UID: \"6d2844ac-8483-400b-8044-73a9f1d6461f\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-zqwkd"
Oct 31 08:38:54.079386 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:38:54.079350    4720 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-qlmsn"]
Oct 31 08:38:54.082830 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:38:54.082800    4720 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-qlmsn"]
Oct 31 08:38:55.018772 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:38:55.018732    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 08:38:55.176494 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:38:55.176463    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 08:38:55.334250 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:38:55.334208    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 08:38:56.025260 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:38:56.025220    4720 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="07da9eb3-de85-4bc9-a5a1-2a3cf25765f3" path="/var/lib/kubelet/pods/07da9eb3-de85-4bc9-a5a1-2a3cf25765f3/volumes"
Oct 31 08:38:56.300987 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:38:56.300895    4720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-43dec6d8-5073-49ff-9a60-668d1ef5314e\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0a0ae6d4bc3a7e09e\") pod \"e2e-pvc-lora-download-zqwkd\" (UID: \"6d2844ac-8483-400b-8044-73a9f1d6461f\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-zqwkd"
Oct 31 08:38:56.403649 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:38:56.403612    4720 operation_generator.go:1464] "Controller attach succeeded for volume \"pvc-43dec6d8-5073-49ff-9a60-668d1ef5314e\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0a0ae6d4bc3a7e09e\") pod \"e2e-pvc-lora-download-zqwkd\" (UID: \"6d2844ac-8483-400b-8044-73a9f1d6461f\") device path: \"\"" pod="e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-zqwkd"
Oct 31 08:38:56.502807 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:38:56.502763    4720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"pvc-43dec6d8-5073-49ff-9a60-668d1ef5314e\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0a0ae6d4bc3a7e09e\") pod \"e2e-pvc-lora-download-zqwkd\" (UID: \"6d2844ac-8483-400b-8044-73a9f1d6461f\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-zqwkd"
Oct 31 08:38:56.503011 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:38:56.502912    4720 operation_generator.go:515] "MountVolume.WaitForAttach entering for volume \"pvc-43dec6d8-5073-49ff-9a60-668d1ef5314e\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0a0ae6d4bc3a7e09e\") pod \"e2e-pvc-lora-download-zqwkd\" (UID: \"6d2844ac-8483-400b-8044-73a9f1d6461f\") DevicePath \"\"" pod="e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-zqwkd"
Oct 31 08:38:56.554799 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:38:56.554711    4720 operation_generator.go:525] "MountVolume.WaitForAttach succeeded for volume \"pvc-43dec6d8-5073-49ff-9a60-668d1ef5314e\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0a0ae6d4bc3a7e09e\") pod \"e2e-pvc-lora-download-zqwkd\" (UID: \"6d2844ac-8483-400b-8044-73a9f1d6461f\") DevicePath \"csi-dc8f82727fb2f3dcf5bd31e6e4d9824f484fc6bab7fd24d19f2c8e10d9c5d013\"" pod="e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-zqwkd"
Oct 31 08:38:56.808054 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:38:56.807982    4720 operation_generator.go:557] "MountVolume.MountDevice succeeded for volume \"pvc-43dec6d8-5073-49ff-9a60-668d1ef5314e\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0a0ae6d4bc3a7e09e\") pod \"e2e-pvc-lora-download-zqwkd\" (UID: \"6d2844ac-8483-400b-8044-73a9f1d6461f\") device mount path \"/var/lib/kubelet/plugins/kubernetes.io/csi/ebs.csi.aws.com/610c7b521ea5c930d2863ede1e3d8aa55e0e337945c3b806d42d0c59e251b146/globalmount\"" pod="e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-zqwkd"
Oct 31 08:38:56.871913 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:38:56.871875    4720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"pvc-43dec6d8-5073-49ff-9a60-668d1ef5314e\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0a0ae6d4bc3a7e09e\") pod \"e2e-pvc-lora-download-zqwkd\" (UID: \"6d2844ac-8483-400b-8044-73a9f1d6461f\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-zqwkd"
Oct 31 08:38:56.954907 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:38:56.954872    4720 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-zqwkd"
Oct 31 08:38:57.327395 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:38:57.327358    4720 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-zqwkd"]
Oct 31 08:38:57.330869 ip-10-0-139-83 kubenswrapper[4720]: W1031 08:38:57.330836    4720 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6d2844ac_8483_400b_8044_73a9f1d6461f.slice/crio-e24fe539db4f9cda7b31e9ac23a837075e203487e23d3af087f4592653392e97 WatchSource:0}: Error finding container e24fe539db4f9cda7b31e9ac23a837075e203487e23d3af087f4592653392e97: Status 404 returned error can't find the container with id e24fe539db4f9cda7b31e9ac23a837075e203487e23d3af087f4592653392e97
Oct 31 08:38:57.581938 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:38:57.581891    4720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-zqwkd" event={"ID":"6d2844ac-8483-400b-8044-73a9f1d6461f","Type":"ContainerStarted","Data":"6dd718e6cdb43e1aa7b646bb37635e4783cbdd0d115a05e124aa5ab9ef49ae72"}
Oct 31 08:38:57.582107 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:38:57.581951    4720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-zqwkd" event={"ID":"6d2844ac-8483-400b-8044-73a9f1d6461f","Type":"ContainerStarted","Data":"e24fe539db4f9cda7b31e9ac23a837075e203487e23d3af087f4592653392e97"}
Oct 31 08:38:57.694058 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:38:57.693793    4720 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-zqwkd" podStartSLOduration=8.693775408 podStartE2EDuration="8.693775408s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 08:38:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 08:38:57.693669988 +0000 UTC m=+4191.824738057" watchObservedRunningTime="2026-10-31 08:38:57.693775408 +0000 UTC m=+4191.824843480"
Oct 31 08:39:00.598956 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:39:00.598917    4720 generic.go:356] "Generic (PLEG): container finished" podID="6d2844ac-8483-400b-8044-73a9f1d6461f" containerID="6dd718e6cdb43e1aa7b646bb37635e4783cbdd0d115a05e124aa5ab9ef49ae72" exitCode=0
Oct 31 08:39:00.599390 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:39:00.598989    4720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-zqwkd" event={"ID":"6d2844ac-8483-400b-8044-73a9f1d6461f","Type":"ContainerDied","Data":"6dd718e6cdb43e1aa7b646bb37635e4783cbdd0d115a05e124aa5ab9ef49ae72"}
Oct 31 08:39:00.613053 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:39:00.613022    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 08:39:00.876094 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:39:00.876008    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 08:39:01.733599 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:39:01.733568    4720 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-zqwkd"
Oct 31 08:39:01.854488 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:39:01.854417    4720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/csi/ebs.csi.aws.com^vol-0a0ae6d4bc3a7e09e\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0a0ae6d4bc3a7e09e\") pod \"6d2844ac-8483-400b-8044-73a9f1d6461f\" (UID: \"6d2844ac-8483-400b-8044-73a9f1d6461f\") "
Oct 31 08:39:01.854653 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:39:01.854509    4720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6d2844ac-8483-400b-8044-73a9f1d6461f-kube-api-access-zscxq\" (UniqueName: \"kubernetes.io/projected/6d2844ac-8483-400b-8044-73a9f1d6461f-kube-api-access-zscxq\") pod \"6d2844ac-8483-400b-8044-73a9f1d6461f\" (UID: \"6d2844ac-8483-400b-8044-73a9f1d6461f\") "
Oct 31 08:39:01.857059 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:39:01.857024    4720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6d2844ac-8483-400b-8044-73a9f1d6461f-kube-api-access-zscxq" pod "6d2844ac-8483-400b-8044-73a9f1d6461f" (UID: "6d2844ac-8483-400b-8044-73a9f1d6461f"). InnerVolumeSpecName "kube-api-access-zscxq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 08:39:01.857627 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:39:01.857604    4720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/ebs.csi.aws.com^vol-0a0ae6d4bc3a7e09e" pod "6d2844ac-8483-400b-8044-73a9f1d6461f" (UID: "6d2844ac-8483-400b-8044-73a9f1d6461f"). InnerVolumeSpecName "pvc-43dec6d8-5073-49ff-9a60-668d1ef5314e". PluginName "kubernetes.io/csi", VolumeGIDValue ""
Oct 31 08:39:01.955533 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:39:01.955500    4720 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zscxq\" (UniqueName: \"kubernetes.io/projected/6d2844ac-8483-400b-8044-73a9f1d6461f-kube-api-access-zscxq\") on node \"ip-10-0-139-83.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 08:39:01.955716 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:39:01.955552    4720 reconciler_common.go:292] "operationExecutor.UnmountDevice started for volume \"pvc-43dec6d8-5073-49ff-9a60-668d1ef5314e\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0a0ae6d4bc3a7e09e\") on node \"ip-10-0-139-83.us-west-1.compute.internal\" "
Oct 31 08:39:02.609712 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:39:02.609674    4720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-zqwkd" event={"ID":"6d2844ac-8483-400b-8044-73a9f1d6461f","Type":"ContainerDied","Data":"e24fe539db4f9cda7b31e9ac23a837075e203487e23d3af087f4592653392e97"}
Oct 31 08:39:02.609902 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:39:02.609721    4720 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e24fe539db4f9cda7b31e9ac23a837075e203487e23d3af087f4592653392e97"
Oct 31 08:39:02.609902 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:39:02.609684    4720 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-zqwkd"
Oct 31 08:39:02.937743 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:39:02.937716    4720 operation_generator.go:893] UnmountDevice succeeded for volume "pvc-43dec6d8-5073-49ff-9a60-668d1ef5314e" (UniqueName: "kubernetes.io/csi/ebs.csi.aws.com^vol-0a0ae6d4bc3a7e09e") on node "ip-10-0-139-83.us-west-1.compute.internal"
Oct 31 08:39:02.964319 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:39:02.964275    4720 reconciler_common.go:299] "Volume detached for volume \"pvc-43dec6d8-5073-49ff-9a60-668d1ef5314e\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0a0ae6d4bc3a7e09e\") on node \"ip-10-0-139-83.us-west-1.compute.internal\" DevicePath \"csi-dc8f82727fb2f3dcf5bd31e6e4d9824f484fc6bab7fd24d19f2c8e10d9c5d013\""
Oct 31 08:39:03.201934 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:39:03.201849    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 08:39:06.426624 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:39:06.426505    4720 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-4jszp_b1db0e3b-e85d-4769-8012-cf4b6f740d9d/console-operator/2.log"
Oct 31 08:39:06.435824 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:39:06.435802    4720 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-4jszp_b1db0e3b-e85d-4769-8012-cf4b6f740d9d/console-operator/2.log"
Oct 31 08:39:13.463839 ip-10-0-139-83 kubenswrapper[4720]: E1031 08:39:13.463798    4720 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8m768" containerName="extractor"
Oct 31 08:39:20.460974 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:39:20.460929    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 08:39:24.115852 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:39:24.115815    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 08:39:38.740580 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:39:38.740537    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 08:39:43.462456 ip-10-0-139-83 kubenswrapper[4720]: E1031 08:39:43.462415    4720 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8m768" containerName="extractor"
Oct 31 08:40:00.965057 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:40:00.965021    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 08:40:01.132548 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:40:01.132508    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 08:40:03.513980 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:40:03.513942    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 08:40:13.462803 ip-10-0-139-83 kubenswrapper[4720]: E1031 08:40:13.462766    4720 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8m768" containerName="extractor"
Oct 31 08:40:17.425230 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:40:17.425149    4720 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"]
Oct 31 08:40:24.506170 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:40:24.506130    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 08:40:32.448630 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:40:32.448561    4720 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" podUID="a9851ffc-5ef5-49a7-93a5-02032024e205" containerName="main" containerID="cri-o://71607751ab3ee87089c7dd7988c2319ab7f164a16b5620e7d9b32db184c94807" gracePeriod=45
Oct 31 08:40:33.197511 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:40:33.197471    4720 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Oct 31 08:40:33.284782 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:40:33.284683    4720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a9851ffc-5ef5-49a7-93a5-02032024e205-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a9851ffc-5ef5-49a7-93a5-02032024e205-cabundle-cert\") pod \"a9851ffc-5ef5-49a7-93a5-02032024e205\" (UID: \"a9851ffc-5ef5-49a7-93a5-02032024e205\") "
Oct 31 08:40:33.284782 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:40:33.284730    4720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a9851ffc-5ef5-49a7-93a5-02032024e205-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a9851ffc-5ef5-49a7-93a5-02032024e205-kserve-provision-location\") pod \"a9851ffc-5ef5-49a7-93a5-02032024e205\" (UID: \"a9851ffc-5ef5-49a7-93a5-02032024e205\") "
Oct 31 08:40:33.284782 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:40:33.284757    4720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a9851ffc-5ef5-49a7-93a5-02032024e205-kube-api-access-sxgnl\" (UniqueName: \"kubernetes.io/projected/a9851ffc-5ef5-49a7-93a5-02032024e205-kube-api-access-sxgnl\") pod \"a9851ffc-5ef5-49a7-93a5-02032024e205\" (UID: \"a9851ffc-5ef5-49a7-93a5-02032024e205\") "
Oct 31 08:40:33.284782 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:40:33.284782    4720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a9851ffc-5ef5-49a7-93a5-02032024e205-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a9851ffc-5ef5-49a7-93a5-02032024e205-tmp-dir\") pod \"a9851ffc-5ef5-49a7-93a5-02032024e205\" (UID: \"a9851ffc-5ef5-49a7-93a5-02032024e205\") "
Oct 31 08:40:33.285160 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:40:33.284801    4720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a9851ffc-5ef5-49a7-93a5-02032024e205-home\" (UniqueName: \"kubernetes.io/empty-dir/a9851ffc-5ef5-49a7-93a5-02032024e205-home\") pod \"a9851ffc-5ef5-49a7-93a5-02032024e205\" (UID: \"a9851ffc-5ef5-49a7-93a5-02032024e205\") "
Oct 31 08:40:33.285160 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:40:33.284821    4720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a9851ffc-5ef5-49a7-93a5-02032024e205-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a9851ffc-5ef5-49a7-93a5-02032024e205-model-cache\") pod \"a9851ffc-5ef5-49a7-93a5-02032024e205\" (UID: \"a9851ffc-5ef5-49a7-93a5-02032024e205\") "
Oct 31 08:40:33.285160 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:40:33.284874    4720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a9851ffc-5ef5-49a7-93a5-02032024e205-dshm\" (UniqueName: \"kubernetes.io/empty-dir/a9851ffc-5ef5-49a7-93a5-02032024e205-dshm\") pod \"a9851ffc-5ef5-49a7-93a5-02032024e205\" (UID: \"a9851ffc-5ef5-49a7-93a5-02032024e205\") "
Oct 31 08:40:33.285160 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:40:33.284901    4720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a9851ffc-5ef5-49a7-93a5-02032024e205-tls-certs\" (UniqueName: \"kubernetes.io/secret/a9851ffc-5ef5-49a7-93a5-02032024e205-tls-certs\") pod \"a9851ffc-5ef5-49a7-93a5-02032024e205\" (UID: \"a9851ffc-5ef5-49a7-93a5-02032024e205\") "
Oct 31 08:40:33.285420 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:40:33.285153    4720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a9851ffc-5ef5-49a7-93a5-02032024e205-model-cache" pod "a9851ffc-5ef5-49a7-93a5-02032024e205" (UID: "a9851ffc-5ef5-49a7-93a5-02032024e205"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 08:40:33.285420 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:40:33.285212    4720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a9851ffc-5ef5-49a7-93a5-02032024e205-cabundle-cert" pod "a9851ffc-5ef5-49a7-93a5-02032024e205" (UID: "a9851ffc-5ef5-49a7-93a5-02032024e205"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 08:40:33.285857 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:40:33.285809    4720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a9851ffc-5ef5-49a7-93a5-02032024e205-home" pod "a9851ffc-5ef5-49a7-93a5-02032024e205" (UID: "a9851ffc-5ef5-49a7-93a5-02032024e205"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 08:40:33.287555 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:40:33.287526    4720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a9851ffc-5ef5-49a7-93a5-02032024e205-dshm" pod "a9851ffc-5ef5-49a7-93a5-02032024e205" (UID: "a9851ffc-5ef5-49a7-93a5-02032024e205"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 08:40:33.287766 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:40:33.287743    4720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a9851ffc-5ef5-49a7-93a5-02032024e205-tls-certs" pod "a9851ffc-5ef5-49a7-93a5-02032024e205" (UID: "a9851ffc-5ef5-49a7-93a5-02032024e205"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 08:40:33.287766 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:40:33.287750    4720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a9851ffc-5ef5-49a7-93a5-02032024e205-kube-api-access-sxgnl" pod "a9851ffc-5ef5-49a7-93a5-02032024e205" (UID: "a9851ffc-5ef5-49a7-93a5-02032024e205"). InnerVolumeSpecName "kube-api-access-sxgnl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 08:40:33.306373 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:40:33.306310    4720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a9851ffc-5ef5-49a7-93a5-02032024e205-tmp-dir" pod "a9851ffc-5ef5-49a7-93a5-02032024e205" (UID: "a9851ffc-5ef5-49a7-93a5-02032024e205"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 08:40:33.311021 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:40:33.310992    4720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a9851ffc-5ef5-49a7-93a5-02032024e205-kserve-provision-location" pod "a9851ffc-5ef5-49a7-93a5-02032024e205" (UID: "a9851ffc-5ef5-49a7-93a5-02032024e205"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 08:40:33.386076 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:40:33.386033    4720 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sxgnl\" (UniqueName: \"kubernetes.io/projected/a9851ffc-5ef5-49a7-93a5-02032024e205-kube-api-access-sxgnl\") on node \"ip-10-0-139-83.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 08:40:33.386076 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:40:33.386072    4720 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a9851ffc-5ef5-49a7-93a5-02032024e205-tmp-dir\") on node \"ip-10-0-139-83.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 08:40:33.386300 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:40:33.386086    4720 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a9851ffc-5ef5-49a7-93a5-02032024e205-home\") on node \"ip-10-0-139-83.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 08:40:33.386300 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:40:33.386097    4720 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a9851ffc-5ef5-49a7-93a5-02032024e205-model-cache\") on node \"ip-10-0-139-83.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 08:40:33.386300 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:40:33.386110    4720 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a9851ffc-5ef5-49a7-93a5-02032024e205-dshm\") on node \"ip-10-0-139-83.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 08:40:33.386300 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:40:33.386123    4720 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a9851ffc-5ef5-49a7-93a5-02032024e205-tls-certs\") on node \"ip-10-0-139-83.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 08:40:33.386300 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:40:33.386133    4720 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a9851ffc-5ef5-49a7-93a5-02032024e205-cabundle-cert\") on node \"ip-10-0-139-83.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 08:40:33.386300 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:40:33.386144    4720 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a9851ffc-5ef5-49a7-93a5-02032024e205-kserve-provision-location\") on node \"ip-10-0-139-83.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 08:40:34.040796 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:40:34.040763    4720 generic.go:356] "Generic (PLEG): container finished" podID="a9851ffc-5ef5-49a7-93a5-02032024e205" containerID="71607751ab3ee87089c7dd7988c2319ab7f164a16b5620e7d9b32db184c94807" exitCode=0
Oct 31 08:40:34.041160 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:40:34.040820    4720 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Oct 31 08:40:34.041160 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:40:34.040842    4720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" event={"ID":"a9851ffc-5ef5-49a7-93a5-02032024e205","Type":"ContainerDied","Data":"71607751ab3ee87089c7dd7988c2319ab7f164a16b5620e7d9b32db184c94807"}
Oct 31 08:40:34.041160 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:40:34.040887    4720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" event={"ID":"a9851ffc-5ef5-49a7-93a5-02032024e205","Type":"ContainerDied","Data":"a4bdcf165b87df388df54b3d9592129ac6514b9098c7121fe30fde08608ca391"}
Oct 31 08:40:34.041160 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:40:34.040907    4720 scope.go:122] "RemoveContainer" containerID="71607751ab3ee87089c7dd7988c2319ab7f164a16b5620e7d9b32db184c94807"
Oct 31 08:40:34.052015 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:40:34.051952    4720 scope.go:122] "RemoveContainer" containerID="5c8153830ad86cd6c2b143241b5189f511f2dd2b104c0277f77a482de18a8a96"
Oct 31 08:40:34.060456 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:40:34.060436    4720 scope.go:122] "RemoveContainer" containerID="71607751ab3ee87089c7dd7988c2319ab7f164a16b5620e7d9b32db184c94807"
Oct 31 08:40:34.060738 ip-10-0-139-83 kubenswrapper[4720]: E1031 08:40:34.060715    4720 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"71607751ab3ee87089c7dd7988c2319ab7f164a16b5620e7d9b32db184c94807\": container with ID starting with 71607751ab3ee87089c7dd7988c2319ab7f164a16b5620e7d9b32db184c94807 not found: ID does not exist" containerID="71607751ab3ee87089c7dd7988c2319ab7f164a16b5620e7d9b32db184c94807"
Oct 31 08:40:34.060825 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:40:34.060749    4720 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"71607751ab3ee87089c7dd7988c2319ab7f164a16b5620e7d9b32db184c94807"} err="failed to get container status \"71607751ab3ee87089c7dd7988c2319ab7f164a16b5620e7d9b32db184c94807\": rpc error: code = NotFound desc = could not find container \"71607751ab3ee87089c7dd7988c2319ab7f164a16b5620e7d9b32db184c94807\": container with ID starting with 71607751ab3ee87089c7dd7988c2319ab7f164a16b5620e7d9b32db184c94807 not found: ID does not exist"
Oct 31 08:40:34.060825 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:40:34.060770    4720 scope.go:122] "RemoveContainer" containerID="5c8153830ad86cd6c2b143241b5189f511f2dd2b104c0277f77a482de18a8a96"
Oct 31 08:40:34.061020 ip-10-0-139-83 kubenswrapper[4720]: E1031 08:40:34.061000    4720 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5c8153830ad86cd6c2b143241b5189f511f2dd2b104c0277f77a482de18a8a96\": container with ID starting with 5c8153830ad86cd6c2b143241b5189f511f2dd2b104c0277f77a482de18a8a96 not found: ID does not exist" containerID="5c8153830ad86cd6c2b143241b5189f511f2dd2b104c0277f77a482de18a8a96"
Oct 31 08:40:34.061068 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:40:34.061026    4720 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5c8153830ad86cd6c2b143241b5189f511f2dd2b104c0277f77a482de18a8a96"} err="failed to get container status \"5c8153830ad86cd6c2b143241b5189f511f2dd2b104c0277f77a482de18a8a96\": rpc error: code = NotFound desc = could not find container \"5c8153830ad86cd6c2b143241b5189f511f2dd2b104c0277f77a482de18a8a96\": container with ID starting with 5c8153830ad86cd6c2b143241b5189f511f2dd2b104c0277f77a482de18a8a96 not found: ID does not exist"
Oct 31 08:40:34.253770 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:40:34.253729    4720 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"]
Oct 31 08:40:34.285662 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:40:34.285580    4720 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"]
Oct 31 08:40:36.026142 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:40:36.026104    4720 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a9851ffc-5ef5-49a7-93a5-02032024e205" path="/var/lib/kubelet/pods/a9851ffc-5ef5-49a7-93a5-02032024e205/volumes"
Oct 31 08:40:39.075253 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:40:39.075214    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 08:40:41.134945 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:40:41.134911    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 08:40:43.462776 ip-10-0-139-83 kubenswrapper[4720]: E1031 08:40:43.462735    4720 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8m768" containerName="extractor"
Oct 31 08:40:50.364741 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:40:50.364703    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 08:40:50.607539 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:40:50.607500    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 08:40:56.710262 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:40:56.710220    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 08:40:57.495929 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:40:57.495892    4720 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-prefill-577b5bd485-5gk9r"]
Oct 31 08:40:57.496240 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:40:57.496228    4720 cpu_manager.go:414] "containerMap: removing container" podUID="a9851ffc-5ef5-49a7-93a5-02032024e205" containerName="main"
Oct 31 08:40:57.496288 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:40:57.496242    4720 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a9851ffc-5ef5-49a7-93a5-02032024e205" containerName="main"
Oct 31 08:40:57.496288 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:40:57.496255    4720 cpu_manager.go:414] "containerMap: removing container" podUID="a9851ffc-5ef5-49a7-93a5-02032024e205" containerName="storage-initializer"
Oct 31 08:40:57.496288 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:40:57.496260    4720 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a9851ffc-5ef5-49a7-93a5-02032024e205" containerName="storage-initializer"
Oct 31 08:40:57.496288 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:40:57.496274    4720 cpu_manager.go:414] "containerMap: removing container" podUID="6d2844ac-8483-400b-8044-73a9f1d6461f" containerName="storage-initializer"
Oct 31 08:40:57.496288 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:40:57.496279    4720 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6d2844ac-8483-400b-8044-73a9f1d6461f" containerName="storage-initializer"
Oct 31 08:40:57.496461 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:40:57.496371    4720 memory_manager.go:365] "RemoveStaleState removing state" podUID="a9851ffc-5ef5-49a7-93a5-02032024e205" containerName="main"
Oct 31 08:40:57.496461 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:40:57.496381    4720 memory_manager.go:365] "RemoveStaleState removing state" podUID="6d2844ac-8483-400b-8044-73a9f1d6461f" containerName="storage-initializer"
Oct 31 08:40:57.499543 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:40:57.499519    4720 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-prefill-577b5bd485-5gk9r"
Oct 31 08:40:57.512321 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:40:57.512289    4720 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-prefill-577b5bd485-5gk9r"]
Oct 31 08:40:57.551661 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:40:57.551624    4720 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-tls-resources-e4cad0c3\"/\"seaweedfs-s3-creds\""
Oct 31 08:40:57.558147 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:40:57.558116    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 08:40:57.559157 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:40:57.559127    4720 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-tls-resources-e4cad0c3\"/\"openshift-service-ca.crt\""
Oct 31 08:40:57.559293 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:40:57.559153    4720 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-tls-resources-e4cad0c3\"/\"tls-verification-test-kserve-self-signed-certs\""
Oct 31 08:40:57.559293 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:40:57.559134    4720 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-tls-resources-e4cad0c3\"/\"odh-kserve-custom-ca-bundle\""
Oct 31 08:40:57.559293 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:40:57.559134    4720 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-tls-resources-e4cad0c3\"/\"kube-root-ca.crt\""
Oct 31 08:40:57.559293 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:40:57.559190    4720 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-tls-resources-e4cad0c3\"/\"default-dockercfg-drb5d\""
Oct 31 08:40:57.582543 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:40:57.582504    4720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f9cc8749-3865-468d-b250-02cac0c7d050-cabundle-cert\") pod \"tls-verification-test-kserve-prefill-577b5bd485-5gk9r\" (UID: \"f9cc8749-3865-468d-b250-02cac0c7d050\") " pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-prefill-577b5bd485-5gk9r"
Oct 31 08:40:57.582699 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:40:57.582554    4720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f9cc8749-3865-468d-b250-02cac0c7d050-home\") pod \"tls-verification-test-kserve-prefill-577b5bd485-5gk9r\" (UID: \"f9cc8749-3865-468d-b250-02cac0c7d050\") " pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-prefill-577b5bd485-5gk9r"
Oct 31 08:40:57.582699 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:40:57.582571    4720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f9cc8749-3865-468d-b250-02cac0c7d050-tls-certs\") pod \"tls-verification-test-kserve-prefill-577b5bd485-5gk9r\" (UID: \"f9cc8749-3865-468d-b250-02cac0c7d050\") " pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-prefill-577b5bd485-5gk9r"
Oct 31 08:40:57.582699 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:40:57.582680    4720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f9cc8749-3865-468d-b250-02cac0c7d050-tmp-dir\") pod \"tls-verification-test-kserve-prefill-577b5bd485-5gk9r\" (UID: \"f9cc8749-3865-468d-b250-02cac0c7d050\") " pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-prefill-577b5bd485-5gk9r"
Oct 31 08:40:57.582821 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:40:57.582719    4720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f9cc8749-3865-468d-b250-02cac0c7d050-model-cache\") pod \"tls-verification-test-kserve-prefill-577b5bd485-5gk9r\" (UID: \"f9cc8749-3865-468d-b250-02cac0c7d050\") " pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-prefill-577b5bd485-5gk9r"
Oct 31 08:40:57.582821 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:40:57.582748    4720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-45fmp\" (UniqueName: \"kubernetes.io/projected/f9cc8749-3865-468d-b250-02cac0c7d050-kube-api-access-45fmp\") pod \"tls-verification-test-kserve-prefill-577b5bd485-5gk9r\" (UID: \"f9cc8749-3865-468d-b250-02cac0c7d050\") " pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-prefill-577b5bd485-5gk9r"
Oct 31 08:40:57.582821 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:40:57.582774    4720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f9cc8749-3865-468d-b250-02cac0c7d050-kserve-provision-location\") pod \"tls-verification-test-kserve-prefill-577b5bd485-5gk9r\" (UID: \"f9cc8749-3865-468d-b250-02cac0c7d050\") " pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-prefill-577b5bd485-5gk9r"
Oct 31 08:40:57.582926 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:40:57.582849    4720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f9cc8749-3865-468d-b250-02cac0c7d050-dshm\") pod \"tls-verification-test-kserve-prefill-577b5bd485-5gk9r\" (UID: \"f9cc8749-3865-468d-b250-02cac0c7d050\") " pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-prefill-577b5bd485-5gk9r"
Oct 31 08:40:57.683506 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:40:57.683464    4720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f9cc8749-3865-468d-b250-02cac0c7d050-cabundle-cert\") pod \"tls-verification-test-kserve-prefill-577b5bd485-5gk9r\" (UID: \"f9cc8749-3865-468d-b250-02cac0c7d050\") " pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-prefill-577b5bd485-5gk9r"
Oct 31 08:40:57.683506 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:40:57.683513    4720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f9cc8749-3865-468d-b250-02cac0c7d050-home\") pod \"tls-verification-test-kserve-prefill-577b5bd485-5gk9r\" (UID: \"f9cc8749-3865-468d-b250-02cac0c7d050\") " pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-prefill-577b5bd485-5gk9r"
Oct 31 08:40:57.683759 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:40:57.683528    4720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f9cc8749-3865-468d-b250-02cac0c7d050-tls-certs\") pod \"tls-verification-test-kserve-prefill-577b5bd485-5gk9r\" (UID: \"f9cc8749-3865-468d-b250-02cac0c7d050\") " pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-prefill-577b5bd485-5gk9r"
Oct 31 08:40:57.683759 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:40:57.683559    4720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f9cc8749-3865-468d-b250-02cac0c7d050-tmp-dir\") pod \"tls-verification-test-kserve-prefill-577b5bd485-5gk9r\" (UID: \"f9cc8749-3865-468d-b250-02cac0c7d050\") " pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-prefill-577b5bd485-5gk9r"
Oct 31 08:40:57.683759 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:40:57.683587    4720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f9cc8749-3865-468d-b250-02cac0c7d050-model-cache\") pod \"tls-verification-test-kserve-prefill-577b5bd485-5gk9r\" (UID: \"f9cc8749-3865-468d-b250-02cac0c7d050\") " pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-prefill-577b5bd485-5gk9r"
Oct 31 08:40:57.683759 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:40:57.683620    4720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-45fmp\" (UniqueName: \"kubernetes.io/projected/f9cc8749-3865-468d-b250-02cac0c7d050-kube-api-access-45fmp\") pod \"tls-verification-test-kserve-prefill-577b5bd485-5gk9r\" (UID: \"f9cc8749-3865-468d-b250-02cac0c7d050\") " pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-prefill-577b5bd485-5gk9r"
Oct 31 08:40:57.683759 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:40:57.683654    4720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f9cc8749-3865-468d-b250-02cac0c7d050-kserve-provision-location\") pod \"tls-verification-test-kserve-prefill-577b5bd485-5gk9r\" (UID: \"f9cc8749-3865-468d-b250-02cac0c7d050\") " pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-prefill-577b5bd485-5gk9r"
Oct 31 08:40:57.683759 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:40:57.683687    4720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f9cc8749-3865-468d-b250-02cac0c7d050-dshm\") pod \"tls-verification-test-kserve-prefill-577b5bd485-5gk9r\" (UID: \"f9cc8749-3865-468d-b250-02cac0c7d050\") " pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-prefill-577b5bd485-5gk9r"
Oct 31 08:40:57.684065 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:40:57.683954    4720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f9cc8749-3865-468d-b250-02cac0c7d050-home\") pod \"tls-verification-test-kserve-prefill-577b5bd485-5gk9r\" (UID: \"f9cc8749-3865-468d-b250-02cac0c7d050\") " pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-prefill-577b5bd485-5gk9r"
Oct 31 08:40:57.684065 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:40:57.683984    4720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f9cc8749-3865-468d-b250-02cac0c7d050-tmp-dir\") pod \"tls-verification-test-kserve-prefill-577b5bd485-5gk9r\" (UID: \"f9cc8749-3865-468d-b250-02cac0c7d050\") " pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-prefill-577b5bd485-5gk9r"
Oct 31 08:40:57.684173 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:40:57.684134    4720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f9cc8749-3865-468d-b250-02cac0c7d050-model-cache\") pod \"tls-verification-test-kserve-prefill-577b5bd485-5gk9r\" (UID: \"f9cc8749-3865-468d-b250-02cac0c7d050\") " pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-prefill-577b5bd485-5gk9r"
Oct 31 08:40:57.684226 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:40:57.684169    4720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f9cc8749-3865-468d-b250-02cac0c7d050-kserve-provision-location\") pod \"tls-verification-test-kserve-prefill-577b5bd485-5gk9r\" (UID: \"f9cc8749-3865-468d-b250-02cac0c7d050\") " pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-prefill-577b5bd485-5gk9r"
Oct 31 08:40:57.684369 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:40:57.684316    4720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f9cc8749-3865-468d-b250-02cac0c7d050-cabundle-cert\") pod \"tls-verification-test-kserve-prefill-577b5bd485-5gk9r\" (UID: \"f9cc8749-3865-468d-b250-02cac0c7d050\") " pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-prefill-577b5bd485-5gk9r"
Oct 31 08:40:57.686257 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:40:57.686225    4720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f9cc8749-3865-468d-b250-02cac0c7d050-dshm\") pod \"tls-verification-test-kserve-prefill-577b5bd485-5gk9r\" (UID: \"f9cc8749-3865-468d-b250-02cac0c7d050\") " pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-prefill-577b5bd485-5gk9r"
Oct 31 08:40:57.686534 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:40:57.686517    4720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f9cc8749-3865-468d-b250-02cac0c7d050-tls-certs\") pod \"tls-verification-test-kserve-prefill-577b5bd485-5gk9r\" (UID: \"f9cc8749-3865-468d-b250-02cac0c7d050\") " pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-prefill-577b5bd485-5gk9r"
Oct 31 08:40:57.744771 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:40:57.744731    4720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-45fmp\" (UniqueName: \"kubernetes.io/projected/f9cc8749-3865-468d-b250-02cac0c7d050-kube-api-access-45fmp\") pod \"tls-verification-test-kserve-prefill-577b5bd485-5gk9r\" (UID: \"f9cc8749-3865-468d-b250-02cac0c7d050\") " pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-prefill-577b5bd485-5gk9r"
Oct 31 08:40:57.817173 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:40:57.817079    4720 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-prefill-577b5bd485-5gk9r"
Oct 31 08:40:57.894111 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:40:57.894066    4720 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-router-scheduler-69f7c679f8-nnp58"]
Oct 31 08:40:57.898187 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:40:57.898164    4720 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-router-scheduler-69f7c679f8-nnp58"
Oct 31 08:40:57.952073 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:40:57.952011    4720 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-tls-resources-e4cad0c3\"/\"tls-verification-test-epp-sa-dockercfg-6sd8c\""
Oct 31 08:40:57.955055 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:40:57.955023    4720 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-router-scheduler-69f7c679f8-nnp58"]
Oct 31 08:40:57.986748 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:40:57.986711    4720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sszrd\" (UniqueName: \"kubernetes.io/projected/38a428b4-4abb-4176-b421-1719e7993740-kube-api-access-sszrd\") pod \"tls-verification-test-kserve-router-scheduler-69f7c679f8-nnp58\" (UID: \"38a428b4-4abb-4176-b421-1719e7993740\") " pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-router-scheduler-69f7c679f8-nnp58"
Oct 31 08:40:57.986934 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:40:57.986760    4720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/38a428b4-4abb-4176-b421-1719e7993740-tls-certs\") pod \"tls-verification-test-kserve-router-scheduler-69f7c679f8-nnp58\" (UID: \"38a428b4-4abb-4176-b421-1719e7993740\") " pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-router-scheduler-69f7c679f8-nnp58"
Oct 31 08:40:58.019749 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:40:58.019712    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 08:40:58.087722 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:40:58.087687    4720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sszrd\" (UniqueName: \"kubernetes.io/projected/38a428b4-4abb-4176-b421-1719e7993740-kube-api-access-sszrd\") pod \"tls-verification-test-kserve-router-scheduler-69f7c679f8-nnp58\" (UID: \"38a428b4-4abb-4176-b421-1719e7993740\") " pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-router-scheduler-69f7c679f8-nnp58"
Oct 31 08:40:58.087914 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:40:58.087746    4720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/38a428b4-4abb-4176-b421-1719e7993740-tls-certs\") pod \"tls-verification-test-kserve-router-scheduler-69f7c679f8-nnp58\" (UID: \"38a428b4-4abb-4176-b421-1719e7993740\") " pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-router-scheduler-69f7c679f8-nnp58"
Oct 31 08:40:58.090476 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:40:58.090447    4720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/38a428b4-4abb-4176-b421-1719e7993740-tls-certs\") pod \"tls-verification-test-kserve-router-scheduler-69f7c679f8-nnp58\" (UID: \"38a428b4-4abb-4176-b421-1719e7993740\") " pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-router-scheduler-69f7c679f8-nnp58"
Oct 31 08:40:58.144315 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:40:58.144275    4720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sszrd\" (UniqueName: \"kubernetes.io/projected/38a428b4-4abb-4176-b421-1719e7993740-kube-api-access-sszrd\") pod \"tls-verification-test-kserve-router-scheduler-69f7c679f8-nnp58\" (UID: \"38a428b4-4abb-4176-b421-1719e7993740\") " pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-router-scheduler-69f7c679f8-nnp58"
Oct 31 08:40:58.213640 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:40:58.213593    4720 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-router-scheduler-69f7c679f8-nnp58"
Oct 31 08:40:58.247480 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:40:58.247452    4720 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-prefill-577b5bd485-5gk9r"]
Oct 31 08:40:58.250477 ip-10-0-139-83 kubenswrapper[4720]: W1031 08:40:58.250448    4720 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf9cc8749_3865_468d_b250_02cac0c7d050.slice/crio-206c318ce7718577958928f7b24477cc69279a11736173943ef1a55d395f4aa4 WatchSource:0}: Error finding container 206c318ce7718577958928f7b24477cc69279a11736173943ef1a55d395f4aa4: Status 404 returned error can't find the container with id 206c318ce7718577958928f7b24477cc69279a11736173943ef1a55d395f4aa4
Oct 31 08:40:58.599654 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:40:58.599619    4720 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-router-scheduler-69f7c679f8-nnp58"]
Oct 31 08:40:58.603900 ip-10-0-139-83 kubenswrapper[4720]: W1031 08:40:58.603867    4720 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod38a428b4_4abb_4176_b421_1719e7993740.slice/crio-2719fbf7c4d6a83aa220fa1ab5c0fd4fba1289551857a354085492b9b776748f WatchSource:0}: Error finding container 2719fbf7c4d6a83aa220fa1ab5c0fd4fba1289551857a354085492b9b776748f: Status 404 returned error can't find the container with id 2719fbf7c4d6a83aa220fa1ab5c0fd4fba1289551857a354085492b9b776748f
Oct 31 08:40:59.152408 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:40:59.152359    4720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-prefill-577b5bd485-5gk9r" event={"ID":"f9cc8749-3865-468d-b250-02cac0c7d050","Type":"ContainerStarted","Data":"88f66af69123ea58261f9eb3722badf1188522a4fd4e0e031f2e4a95b19573b9"}
Oct 31 08:40:59.152922 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:40:59.152429    4720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-prefill-577b5bd485-5gk9r" event={"ID":"f9cc8749-3865-468d-b250-02cac0c7d050","Type":"ContainerStarted","Data":"206c318ce7718577958928f7b24477cc69279a11736173943ef1a55d395f4aa4"}
Oct 31 08:40:59.154099 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:40:59.154071    4720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-router-scheduler-69f7c679f8-nnp58" event={"ID":"38a428b4-4abb-4176-b421-1719e7993740","Type":"ContainerStarted","Data":"0757ea2e97dc1fb105d91b740a6fc0d4d2ea8d4528db101bf64e5e6512b8fa55"}
Oct 31 08:40:59.154236 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:40:59.154122    4720 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-router-scheduler-69f7c679f8-nnp58"
Oct 31 08:40:59.154236 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:40:59.154148    4720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-router-scheduler-69f7c679f8-nnp58" event={"ID":"38a428b4-4abb-4176-b421-1719e7993740","Type":"ContainerStarted","Data":"2719fbf7c4d6a83aa220fa1ab5c0fd4fba1289551857a354085492b9b776748f"}
Oct 31 08:40:59.393380 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:40:59.393267    4720 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-router-scheduler-69f7c679f8-nnp58" podStartSLOduration=2.393244493 podStartE2EDuration="2.393244493s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 08:40:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 08:40:59.391284351 +0000 UTC m=+4313.522352424" watchObservedRunningTime="2026-10-31 08:40:59.393244493 +0000 UTC m=+4313.524312565"
Oct 31 08:41:01.164729 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:41:01.164694    4720 generic.go:356] "Generic (PLEG): container finished" podID="f9cc8749-3865-468d-b250-02cac0c7d050" containerID="88f66af69123ea58261f9eb3722badf1188522a4fd4e0e031f2e4a95b19573b9" exitCode=0
Oct 31 08:41:01.165156 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:41:01.164774    4720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-prefill-577b5bd485-5gk9r" event={"ID":"f9cc8749-3865-468d-b250-02cac0c7d050","Type":"ContainerDied","Data":"88f66af69123ea58261f9eb3722badf1188522a4fd4e0e031f2e4a95b19573b9"}
Oct 31 08:41:02.171429 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:41:02.171389    4720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-prefill-577b5bd485-5gk9r" event={"ID":"f9cc8749-3865-468d-b250-02cac0c7d050","Type":"ContainerStarted","Data":"24295abdc23fcd222ac5485a7af3153f684283ff009b020ad2c205487598493d"}
Oct 31 08:41:02.391512 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:41:02.391459    4720 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-prefill-577b5bd485-5gk9r" podStartSLOduration=3.042384184 podStartE2EDuration="5.391445584s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.3490614s" isStatefulPod=false podCreationTimestamp="2026-10-31 08:40:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 08:41:02.390306504 +0000 UTC m=+4316.521374587" watchObservedRunningTime="2026-10-31 08:41:02.391445584 +0000 UTC m=+4316.522513654"
Oct 31 08:41:02.627194 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:41:02.627148    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 08:41:02.789366 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:41:02.789299    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 08:41:03.858136 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:41:03.858099    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 08:41:07.818244 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:41:07.818195    4720 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-prefill-577b5bd485-5gk9r"
Oct 31 08:41:07.818692 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:41:07.818280    4720 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-prefill-577b5bd485-5gk9r"
Oct 31 08:41:07.819639 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:41:07.819607    4720 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-prefill-577b5bd485-5gk9r" podUID="f9cc8749-3865-468d-b250-02cac0c7d050" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/health\": dial tcp 10.132.0.80:8000: connect: connection refused"
Oct 31 08:41:09.884580 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:41:09.884524    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 08:41:13.462257 ip-10-0-139-83 kubenswrapper[4720]: E1031 08:41:13.462213    4720 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8m768" containerName="extractor"
Oct 31 08:41:17.817924 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:41:17.817886    4720 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-prefill-577b5bd485-5gk9r" podUID="f9cc8749-3865-468d-b250-02cac0c7d050" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/health\": dial tcp 10.132.0.80:8000: connect: connection refused"
Oct 31 08:41:24.799681 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:41:24.799643    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 08:41:27.818158 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:41:27.818100    4720 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-prefill-577b5bd485-5gk9r" podUID="f9cc8749-3865-468d-b250-02cac0c7d050" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/health\": dial tcp 10.132.0.80:8000: connect: connection refused"
Oct 31 08:41:30.161351 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:41:30.161302    4720 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-router-scheduler-69f7c679f8-nnp58"
Oct 31 08:41:37.818341 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:41:37.818286    4720 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-prefill-577b5bd485-5gk9r" podUID="f9cc8749-3865-468d-b250-02cac0c7d050" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/health\": dial tcp 10.132.0.80:8000: connect: connection refused"
Oct 31 08:41:39.472917 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:41:39.472874    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 08:41:43.462487 ip-10-0-139-83 kubenswrapper[4720]: E1031 08:41:43.462443    4720 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8m768" containerName="extractor"
Oct 31 08:41:47.818638 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:41:47.818546    4720 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-prefill-577b5bd485-5gk9r" podUID="f9cc8749-3865-468d-b250-02cac0c7d050" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/health\": dial tcp 10.132.0.80:8000: connect: connection refused"
Oct 31 08:41:57.818449 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:41:57.818398    4720 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-prefill-577b5bd485-5gk9r" podUID="f9cc8749-3865-468d-b250-02cac0c7d050" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/health\": dial tcp 10.132.0.80:8000: connect: connection refused"
Oct 31 08:42:04.200064 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:42:04.200030    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 08:42:07.817807 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:42:07.817759    4720 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-prefill-577b5bd485-5gk9r" podUID="f9cc8749-3865-468d-b250-02cac0c7d050" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/health\": dial tcp 10.132.0.80:8000: connect: connection refused"
Oct 31 08:42:12.405780 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:42:12.405742    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 08:42:13.462792 ip-10-0-139-83 kubenswrapper[4720]: E1031 08:42:13.462749    4720 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8m768" containerName="extractor"
Oct 31 08:42:17.817993 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:42:17.817940    4720 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-prefill-577b5bd485-5gk9r" podUID="f9cc8749-3865-468d-b250-02cac0c7d050" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/health\": dial tcp 10.132.0.80:8000: connect: connection refused"
Oct 31 08:42:24.955643 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:42:24.955604    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 08:42:25.129892 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:42:25.129857    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 08:42:27.817958 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:42:27.817914    4720 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-prefill-577b5bd485-5gk9r" podUID="f9cc8749-3865-468d-b250-02cac0c7d050" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/health\": dial tcp 10.132.0.80:8000: connect: connection refused"
Oct 31 08:42:37.817571 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:42:37.817528    4720 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-prefill-577b5bd485-5gk9r" podUID="f9cc8749-3865-468d-b250-02cac0c7d050" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/health\": dial tcp 10.132.0.80:8000: connect: connection refused"
Oct 31 08:42:39.766954 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:42:39.766914    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 08:42:43.461771 ip-10-0-139-83 kubenswrapper[4720]: E1031 08:42:43.461732    4720 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8m768" containerName="extractor"
Oct 31 08:42:47.818555 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:42:47.818511    4720 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-prefill-577b5bd485-5gk9r" podUID="f9cc8749-3865-468d-b250-02cac0c7d050" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/health\": dial tcp 10.132.0.80:8000: connect: connection refused"
Oct 31 08:42:57.827952 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:42:57.827913    4720 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-prefill-577b5bd485-5gk9r"
Oct 31 08:42:57.836189 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:42:57.836158    4720 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-prefill-577b5bd485-5gk9r"
Oct 31 08:43:04.529279 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:43:04.529232    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 08:43:06.723815 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:43:06.723778    4720 scope.go:122] "RemoveContainer" containerID="abc986dcd32bd4fa17dd70939d19ad69cdcaeb28ebfefa0638219cba7ede2bfa"
Oct 31 08:43:13.460426 ip-10-0-139-83 kubenswrapper[4720]: E1031 08:43:13.460390    4720 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8m768" containerName="extractor"
Oct 31 08:43:25.484853 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:43:25.484763    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 08:43:40.209080 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:43:40.209033    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 08:43:42.823533 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:43:42.823495    4720 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-router-scheduler-69f7c679f8-nnp58"]
Oct 31 08:43:42.841596 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:43:42.841556    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 08:43:42.862893 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:43:42.862846    4720 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-prefill-577b5bd485-5gk9r"]
Oct 31 08:43:43.001227 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:43:43.001194    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 08:43:43.460180 ip-10-0-139-83 kubenswrapper[4720]: E1031 08:43:43.460146    4720 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8m768" containerName="extractor"
Oct 31 08:43:50.161781 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:43:50.161742    4720 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-router-scheduler-69f7c679f8-nnp58" podUID="38a428b4-4abb-4176-b421-1719e7993740" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 08:43:56.918501 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:43:56.918464    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 08:43:57.843860 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:43:57.843827    4720 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-router-scheduler-69f7c679f8-nnp58" podUID="38a428b4-4abb-4176-b421-1719e7993740" containerName="main" containerID="cri-o://0757ea2e97dc1fb105d91b740a6fc0d4d2ea8d4528db101bf64e5e6512b8fa55" gracePeriod=45
Oct 31 08:43:57.891596 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:43:57.891560    4720 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-prefill-577b5bd485-5gk9r" podUID="f9cc8749-3865-468d-b250-02cac0c7d050" containerName="main" containerID="cri-o://24295abdc23fcd222ac5485a7af3153f684283ff009b020ad2c205487598493d" gracePeriod=45
Oct 31 08:43:58.634556 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:43:58.634532    4720 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-prefill-577b5bd485-5gk9r"
Oct 31 08:43:58.746148 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:43:58.746092    4720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f9cc8749-3865-468d-b250-02cac0c7d050-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f9cc8749-3865-468d-b250-02cac0c7d050-cabundle-cert\") pod \"f9cc8749-3865-468d-b250-02cac0c7d050\" (UID: \"f9cc8749-3865-468d-b250-02cac0c7d050\") "
Oct 31 08:43:58.746262 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:43:58.746144    4720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f9cc8749-3865-468d-b250-02cac0c7d050-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f9cc8749-3865-468d-b250-02cac0c7d050-model-cache\") pod \"f9cc8749-3865-468d-b250-02cac0c7d050\" (UID: \"f9cc8749-3865-468d-b250-02cac0c7d050\") "
Oct 31 08:43:58.746262 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:43:58.746176    4720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f9cc8749-3865-468d-b250-02cac0c7d050-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f9cc8749-3865-468d-b250-02cac0c7d050-tmp-dir\") pod \"f9cc8749-3865-468d-b250-02cac0c7d050\" (UID: \"f9cc8749-3865-468d-b250-02cac0c7d050\") "
Oct 31 08:43:58.746262 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:43:58.746204    4720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f9cc8749-3865-468d-b250-02cac0c7d050-kube-api-access-45fmp\" (UniqueName: \"kubernetes.io/projected/f9cc8749-3865-468d-b250-02cac0c7d050-kube-api-access-45fmp\") pod \"f9cc8749-3865-468d-b250-02cac0c7d050\" (UID: \"f9cc8749-3865-468d-b250-02cac0c7d050\") "
Oct 31 08:43:58.746262 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:43:58.746244    4720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f9cc8749-3865-468d-b250-02cac0c7d050-home\" (UniqueName: \"kubernetes.io/empty-dir/f9cc8749-3865-468d-b250-02cac0c7d050-home\") pod \"f9cc8749-3865-468d-b250-02cac0c7d050\" (UID: \"f9cc8749-3865-468d-b250-02cac0c7d050\") "
Oct 31 08:43:58.746570 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:43:58.746278    4720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f9cc8749-3865-468d-b250-02cac0c7d050-dshm\" (UniqueName: \"kubernetes.io/empty-dir/f9cc8749-3865-468d-b250-02cac0c7d050-dshm\") pod \"f9cc8749-3865-468d-b250-02cac0c7d050\" (UID: \"f9cc8749-3865-468d-b250-02cac0c7d050\") "
Oct 31 08:43:58.746570 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:43:58.746315    4720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f9cc8749-3865-468d-b250-02cac0c7d050-tls-certs\" (UniqueName: \"kubernetes.io/secret/f9cc8749-3865-468d-b250-02cac0c7d050-tls-certs\") pod \"f9cc8749-3865-468d-b250-02cac0c7d050\" (UID: \"f9cc8749-3865-468d-b250-02cac0c7d050\") "
Oct 31 08:43:58.746570 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:43:58.746411    4720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f9cc8749-3865-468d-b250-02cac0c7d050-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f9cc8749-3865-468d-b250-02cac0c7d050-kserve-provision-location\") pod \"f9cc8749-3865-468d-b250-02cac0c7d050\" (UID: \"f9cc8749-3865-468d-b250-02cac0c7d050\") "
Oct 31 08:43:58.746570 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:43:58.746531    4720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f9cc8749-3865-468d-b250-02cac0c7d050-cabundle-cert" pod "f9cc8749-3865-468d-b250-02cac0c7d050" (UID: "f9cc8749-3865-468d-b250-02cac0c7d050"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 08:43:58.746570 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:43:58.746535    4720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9cc8749-3865-468d-b250-02cac0c7d050-model-cache" pod "f9cc8749-3865-468d-b250-02cac0c7d050" (UID: "f9cc8749-3865-468d-b250-02cac0c7d050"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 08:43:58.746856 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:43:58.746730    4720 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f9cc8749-3865-468d-b250-02cac0c7d050-cabundle-cert\") on node \"ip-10-0-139-83.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 08:43:58.746856 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:43:58.746752    4720 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f9cc8749-3865-468d-b250-02cac0c7d050-model-cache\") on node \"ip-10-0-139-83.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 08:43:58.747061 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:43:58.747028    4720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9cc8749-3865-468d-b250-02cac0c7d050-home" pod "f9cc8749-3865-468d-b250-02cac0c7d050" (UID: "f9cc8749-3865-468d-b250-02cac0c7d050"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 08:43:58.751759 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:43:58.751720    4720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f9cc8749-3865-468d-b250-02cac0c7d050-kube-api-access-45fmp" pod "f9cc8749-3865-468d-b250-02cac0c7d050" (UID: "f9cc8749-3865-468d-b250-02cac0c7d050"). InnerVolumeSpecName "kube-api-access-45fmp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 08:43:58.751759 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:43:58.751744    4720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f9cc8749-3865-468d-b250-02cac0c7d050-tls-certs" pod "f9cc8749-3865-468d-b250-02cac0c7d050" (UID: "f9cc8749-3865-468d-b250-02cac0c7d050"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 08:43:58.751879 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:43:58.751751    4720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9cc8749-3865-468d-b250-02cac0c7d050-dshm" pod "f9cc8749-3865-468d-b250-02cac0c7d050" (UID: "f9cc8749-3865-468d-b250-02cac0c7d050"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 08:43:58.761088 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:43:58.761064    4720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9cc8749-3865-468d-b250-02cac0c7d050-tmp-dir" pod "f9cc8749-3865-468d-b250-02cac0c7d050" (UID: "f9cc8749-3865-468d-b250-02cac0c7d050"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 08:43:58.771372 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:43:58.771320    4720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9cc8749-3865-468d-b250-02cac0c7d050-kserve-provision-location" pod "f9cc8749-3865-468d-b250-02cac0c7d050" (UID: "f9cc8749-3865-468d-b250-02cac0c7d050"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 08:43:58.847959 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:43:58.847935    4720 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f9cc8749-3865-468d-b250-02cac0c7d050-tmp-dir\") on node \"ip-10-0-139-83.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 08:43:58.847959 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:43:58.847956    4720 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-45fmp\" (UniqueName: \"kubernetes.io/projected/f9cc8749-3865-468d-b250-02cac0c7d050-kube-api-access-45fmp\") on node \"ip-10-0-139-83.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 08:43:58.848096 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:43:58.847968    4720 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f9cc8749-3865-468d-b250-02cac0c7d050-home\") on node \"ip-10-0-139-83.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 08:43:58.848096 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:43:58.847977    4720 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f9cc8749-3865-468d-b250-02cac0c7d050-dshm\") on node \"ip-10-0-139-83.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 08:43:58.848096 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:43:58.847985    4720 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f9cc8749-3865-468d-b250-02cac0c7d050-tls-certs\") on node \"ip-10-0-139-83.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 08:43:58.848096 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:43:58.847993    4720 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f9cc8749-3865-468d-b250-02cac0c7d050-kserve-provision-location\") on node \"ip-10-0-139-83.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 08:43:59.010417 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:43:59.010320    4720 generic.go:356] "Generic (PLEG): container finished" podID="f9cc8749-3865-468d-b250-02cac0c7d050" containerID="24295abdc23fcd222ac5485a7af3153f684283ff009b020ad2c205487598493d" exitCode=0
Oct 31 08:43:59.010417 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:43:59.010395    4720 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-prefill-577b5bd485-5gk9r"
Oct 31 08:43:59.010417 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:43:59.010400    4720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-prefill-577b5bd485-5gk9r" event={"ID":"f9cc8749-3865-468d-b250-02cac0c7d050","Type":"ContainerDied","Data":"24295abdc23fcd222ac5485a7af3153f684283ff009b020ad2c205487598493d"}
Oct 31 08:43:59.010656 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:43:59.010458    4720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-prefill-577b5bd485-5gk9r" event={"ID":"f9cc8749-3865-468d-b250-02cac0c7d050","Type":"ContainerDied","Data":"206c318ce7718577958928f7b24477cc69279a11736173943ef1a55d395f4aa4"}
Oct 31 08:43:59.010656 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:43:59.010481    4720 scope.go:122] "RemoveContainer" containerID="24295abdc23fcd222ac5485a7af3153f684283ff009b020ad2c205487598493d"
Oct 31 08:43:59.020058 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:43:59.020039    4720 scope.go:122] "RemoveContainer" containerID="88f66af69123ea58261f9eb3722badf1188522a4fd4e0e031f2e4a95b19573b9"
Oct 31 08:43:59.027733 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:43:59.027717    4720 scope.go:122] "RemoveContainer" containerID="24295abdc23fcd222ac5485a7af3153f684283ff009b020ad2c205487598493d"
Oct 31 08:43:59.027988 ip-10-0-139-83 kubenswrapper[4720]: E1031 08:43:59.027967    4720 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"24295abdc23fcd222ac5485a7af3153f684283ff009b020ad2c205487598493d\": container with ID starting with 24295abdc23fcd222ac5485a7af3153f684283ff009b020ad2c205487598493d not found: ID does not exist" containerID="24295abdc23fcd222ac5485a7af3153f684283ff009b020ad2c205487598493d"
Oct 31 08:43:59.028051 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:43:59.027997    4720 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"24295abdc23fcd222ac5485a7af3153f684283ff009b020ad2c205487598493d"} err="failed to get container status \"24295abdc23fcd222ac5485a7af3153f684283ff009b020ad2c205487598493d\": rpc error: code = NotFound desc = could not find container \"24295abdc23fcd222ac5485a7af3153f684283ff009b020ad2c205487598493d\": container with ID starting with 24295abdc23fcd222ac5485a7af3153f684283ff009b020ad2c205487598493d not found: ID does not exist"
Oct 31 08:43:59.028051 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:43:59.028015    4720 scope.go:122] "RemoveContainer" containerID="88f66af69123ea58261f9eb3722badf1188522a4fd4e0e031f2e4a95b19573b9"
Oct 31 08:43:59.028260 ip-10-0-139-83 kubenswrapper[4720]: E1031 08:43:59.028241    4720 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"88f66af69123ea58261f9eb3722badf1188522a4fd4e0e031f2e4a95b19573b9\": container with ID starting with 88f66af69123ea58261f9eb3722badf1188522a4fd4e0e031f2e4a95b19573b9 not found: ID does not exist" containerID="88f66af69123ea58261f9eb3722badf1188522a4fd4e0e031f2e4a95b19573b9"
Oct 31 08:43:59.028309 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:43:59.028266    4720 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"88f66af69123ea58261f9eb3722badf1188522a4fd4e0e031f2e4a95b19573b9"} err="failed to get container status \"88f66af69123ea58261f9eb3722badf1188522a4fd4e0e031f2e4a95b19573b9\": rpc error: code = NotFound desc = could not find container \"88f66af69123ea58261f9eb3722badf1188522a4fd4e0e031f2e4a95b19573b9\": container with ID starting with 88f66af69123ea58261f9eb3722badf1188522a4fd4e0e031f2e4a95b19573b9 not found: ID does not exist"
Oct 31 08:43:59.175056 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:43:59.175027    4720 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-prefill-577b5bd485-5gk9r"]
Oct 31 08:43:59.179027 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:43:59.179007    4720 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-prefill-577b5bd485-5gk9r"]
Oct 31 08:44:00.025075 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:44:00.025041    4720 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f9cc8749-3865-468d-b250-02cac0c7d050" path="/var/lib/kubelet/pods/f9cc8749-3865-468d-b250-02cac0c7d050/volumes"
Oct 31 08:44:00.160681 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:44:00.160648    4720 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-router-scheduler-69f7c679f8-nnp58" podUID="38a428b4-4abb-4176-b421-1719e7993740" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 08:44:04.875535 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:44:04.875498    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 08:44:06.457873 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:44:06.457762    4720 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-4jszp_b1db0e3b-e85d-4769-8012-cf4b6f740d9d/console-operator/2.log"
Oct 31 08:44:06.465472 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:44:06.465443    4720 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-4jszp_b1db0e3b-e85d-4769-8012-cf4b6f740d9d/console-operator/2.log"
Oct 31 08:44:10.160996 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:44:10.160954    4720 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-router-scheduler-69f7c679f8-nnp58" podUID="38a428b4-4abb-4176-b421-1719e7993740" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 08:44:10.161510 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:44:10.161061    4720 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-router-scheduler-69f7c679f8-nnp58"
Oct 31 08:44:13.461407 ip-10-0-139-83 kubenswrapper[4720]: E1031 08:44:13.461358    4720 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8m768" containerName="extractor"
Oct 31 08:44:17.284543 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:44:17.284504    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 08:44:17.450488 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:44:17.450452    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 08:44:17.982541 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:44:17.982508    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 08:44:18.133599 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:44:18.133564    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 08:44:20.161655 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:44:20.161615    4720 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-router-scheduler-69f7c679f8-nnp58" podUID="38a428b4-4abb-4176-b421-1719e7993740" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 08:44:21.938400 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:44:21.938366    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 08:44:22.215943 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:44:22.215868    4720 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-zqwkd"]
Oct 31 08:44:22.219197 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:44:22.219168    4720 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-zqwkd"]
Oct 31 08:44:22.975649 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:44:22.975607    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 08:44:23.137679 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:44:23.137645    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 08:44:23.294051 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:44:23.293952    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 08:44:24.025236 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:44:24.025202    4720 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6d2844ac-8483-400b-8044-73a9f1d6461f" path="/var/lib/kubelet/pods/6d2844ac-8483-400b-8044-73a9f1d6461f/volumes"
Oct 31 08:44:25.824300 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:44:25.824257    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 08:44:25.970931 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:44:25.970890    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 08:44:28.092825 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:44:28.092796    4720 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-router-scheduler-69f7c679f8-nnp58"
Oct 31 08:44:28.135709 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:44:28.135641    4720 generic.go:356] "Generic (PLEG): container finished" podID="38a428b4-4abb-4176-b421-1719e7993740" containerID="0757ea2e97dc1fb105d91b740a6fc0d4d2ea8d4528db101bf64e5e6512b8fa55" exitCode=0
Oct 31 08:44:28.135709 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:44:28.135686    4720 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-router-scheduler-69f7c679f8-nnp58"
Oct 31 08:44:28.135867 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:44:28.135754    4720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-router-scheduler-69f7c679f8-nnp58" event={"ID":"38a428b4-4abb-4176-b421-1719e7993740","Type":"ContainerDied","Data":"0757ea2e97dc1fb105d91b740a6fc0d4d2ea8d4528db101bf64e5e6512b8fa55"}
Oct 31 08:44:28.135867 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:44:28.135819    4720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-router-scheduler-69f7c679f8-nnp58" event={"ID":"38a428b4-4abb-4176-b421-1719e7993740","Type":"ContainerDied","Data":"2719fbf7c4d6a83aa220fa1ab5c0fd4fba1289551857a354085492b9b776748f"}
Oct 31 08:44:28.135867 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:44:28.135852    4720 scope.go:122] "RemoveContainer" containerID="0757ea2e97dc1fb105d91b740a6fc0d4d2ea8d4528db101bf64e5e6512b8fa55"
Oct 31 08:44:28.143997 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:44:28.143979    4720 scope.go:122] "RemoveContainer" containerID="0757ea2e97dc1fb105d91b740a6fc0d4d2ea8d4528db101bf64e5e6512b8fa55"
Oct 31 08:44:28.144291 ip-10-0-139-83 kubenswrapper[4720]: E1031 08:44:28.144272    4720 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0757ea2e97dc1fb105d91b740a6fc0d4d2ea8d4528db101bf64e5e6512b8fa55\": container with ID starting with 0757ea2e97dc1fb105d91b740a6fc0d4d2ea8d4528db101bf64e5e6512b8fa55 not found: ID does not exist" containerID="0757ea2e97dc1fb105d91b740a6fc0d4d2ea8d4528db101bf64e5e6512b8fa55"
Oct 31 08:44:28.144382 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:44:28.144300    4720 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0757ea2e97dc1fb105d91b740a6fc0d4d2ea8d4528db101bf64e5e6512b8fa55"} err="failed to get container status \"0757ea2e97dc1fb105d91b740a6fc0d4d2ea8d4528db101bf64e5e6512b8fa55\": rpc error: code = NotFound desc = could not find container \"0757ea2e97dc1fb105d91b740a6fc0d4d2ea8d4528db101bf64e5e6512b8fa55\": container with ID starting with 0757ea2e97dc1fb105d91b740a6fc0d4d2ea8d4528db101bf64e5e6512b8fa55 not found: ID does not exist"
Oct 31 08:44:28.168725 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:44:28.168695    4720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/38a428b4-4abb-4176-b421-1719e7993740-kube-api-access-sszrd\" (UniqueName: \"kubernetes.io/projected/38a428b4-4abb-4176-b421-1719e7993740-kube-api-access-sszrd\") pod \"38a428b4-4abb-4176-b421-1719e7993740\" (UID: \"38a428b4-4abb-4176-b421-1719e7993740\") "
Oct 31 08:44:28.168866 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:44:28.168735    4720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/38a428b4-4abb-4176-b421-1719e7993740-tls-certs\" (UniqueName: \"kubernetes.io/secret/38a428b4-4abb-4176-b421-1719e7993740-tls-certs\") pod \"38a428b4-4abb-4176-b421-1719e7993740\" (UID: \"38a428b4-4abb-4176-b421-1719e7993740\") "
Oct 31 08:44:28.170974 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:44:28.170944    4720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/38a428b4-4abb-4176-b421-1719e7993740-kube-api-access-sszrd" pod "38a428b4-4abb-4176-b421-1719e7993740" (UID: "38a428b4-4abb-4176-b421-1719e7993740"). InnerVolumeSpecName "kube-api-access-sszrd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 08:44:28.171068 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:44:28.171012    4720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/38a428b4-4abb-4176-b421-1719e7993740-tls-certs" pod "38a428b4-4abb-4176-b421-1719e7993740" (UID: "38a428b4-4abb-4176-b421-1719e7993740"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 08:44:28.269432 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:44:28.269394    4720 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sszrd\" (UniqueName: \"kubernetes.io/projected/38a428b4-4abb-4176-b421-1719e7993740-kube-api-access-sszrd\") on node \"ip-10-0-139-83.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 08:44:28.269432 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:44:28.269424    4720 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/38a428b4-4abb-4176-b421-1719e7993740-tls-certs\") on node \"ip-10-0-139-83.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 08:44:28.553782 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:44:28.553742    4720 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-router-scheduler-69f7c679f8-nnp58"]
Oct 31 08:44:28.558166 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:44:28.558136    4720 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-router-scheduler-69f7c679f8-nnp58"]
Oct 31 08:44:28.608618 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:44:28.608587    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 08:44:28.756683 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:44:28.756601    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 08:44:30.025707 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:44:30.025673    4720 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="38a428b4-4abb-4176-b421-1719e7993740" path="/var/lib/kubelet/pods/38a428b4-4abb-4176-b421-1719e7993740/volumes"
Oct 31 08:44:32.401150 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:44:32.401105    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 08:44:32.565113 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:44:32.565077    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 08:44:38.606839 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:44:38.606791    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 08:44:38.994541 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:44:38.994463    4720 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5f8f955bf6-bxb8q"]
Oct 31 08:44:38.994806 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:44:38.994792    4720 cpu_manager.go:414] "containerMap: removing container" podUID="f9cc8749-3865-468d-b250-02cac0c7d050" containerName="storage-initializer"
Oct 31 08:44:38.994854 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:44:38.994810    4720 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f9cc8749-3865-468d-b250-02cac0c7d050" containerName="storage-initializer"
Oct 31 08:44:38.994854 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:44:38.994830    4720 cpu_manager.go:414] "containerMap: removing container" podUID="f9cc8749-3865-468d-b250-02cac0c7d050" containerName="main"
Oct 31 08:44:38.994854 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:44:38.994835    4720 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f9cc8749-3865-468d-b250-02cac0c7d050" containerName="main"
Oct 31 08:44:38.994854 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:44:38.994847    4720 cpu_manager.go:414] "containerMap: removing container" podUID="38a428b4-4abb-4176-b421-1719e7993740" containerName="main"
Oct 31 08:44:38.994854 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:44:38.994853    4720 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="38a428b4-4abb-4176-b421-1719e7993740" containerName="main"
Oct 31 08:44:38.995004 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:44:38.994920    4720 memory_manager.go:365] "RemoveStaleState removing state" podUID="f9cc8749-3865-468d-b250-02cac0c7d050" containerName="main"
Oct 31 08:44:38.995004 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:44:38.994928    4720 memory_manager.go:365] "RemoveStaleState removing state" podUID="38a428b4-4abb-4176-b421-1719e7993740" containerName="main"
Oct 31 08:44:38.999570 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:44:38.999546    4720 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5f8f955bf6-bxb8q"
Oct 31 08:44:39.051382 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:44:39.051353    4720 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5f8f955bf6-bxb8q"]
Oct 31 08:44:39.052764 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:44:39.052742    4720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/311625d7-ceca-4585-b128-bd8124002a8b-home\") pod \"prestop-hook-test-kserve-5f8f955bf6-bxb8q\" (UID: \"311625d7-ceca-4585-b128-bd8124002a8b\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5f8f955bf6-bxb8q"
Oct 31 08:44:39.052890 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:44:39.052779    4720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/311625d7-ceca-4585-b128-bd8124002a8b-tls-certs\") pod \"prestop-hook-test-kserve-5f8f955bf6-bxb8q\" (UID: \"311625d7-ceca-4585-b128-bd8124002a8b\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5f8f955bf6-bxb8q"
Oct 31 08:44:39.052890 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:44:39.052809    4720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/311625d7-ceca-4585-b128-bd8124002a8b-cabundle-cert\") pod \"prestop-hook-test-kserve-5f8f955bf6-bxb8q\" (UID: \"311625d7-ceca-4585-b128-bd8124002a8b\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5f8f955bf6-bxb8q"
Oct 31 08:44:39.052994 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:44:39.052927    4720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4cqm8\" (UniqueName: \"kubernetes.io/projected/311625d7-ceca-4585-b128-bd8124002a8b-kube-api-access-4cqm8\") pod \"prestop-hook-test-kserve-5f8f955bf6-bxb8q\" (UID: \"311625d7-ceca-4585-b128-bd8124002a8b\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5f8f955bf6-bxb8q"
Oct 31 08:44:39.052994 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:44:39.052980    4720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/311625d7-ceca-4585-b128-bd8124002a8b-dshm\") pod \"prestop-hook-test-kserve-5f8f955bf6-bxb8q\" (UID: \"311625d7-ceca-4585-b128-bd8124002a8b\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5f8f955bf6-bxb8q"
Oct 31 08:44:39.053104 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:44:39.053058    4720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/311625d7-ceca-4585-b128-bd8124002a8b-model-cache\") pod \"prestop-hook-test-kserve-5f8f955bf6-bxb8q\" (UID: \"311625d7-ceca-4585-b128-bd8124002a8b\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5f8f955bf6-bxb8q"
Oct 31 08:44:39.053157 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:44:39.053099    4720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/311625d7-ceca-4585-b128-bd8124002a8b-tmp-dir\") pod \"prestop-hook-test-kserve-5f8f955bf6-bxb8q\" (UID: \"311625d7-ceca-4585-b128-bd8124002a8b\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5f8f955bf6-bxb8q"
Oct 31 08:44:39.053157 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:44:39.053129    4720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/311625d7-ceca-4585-b128-bd8124002a8b-kserve-provision-location\") pod \"prestop-hook-test-kserve-5f8f955bf6-bxb8q\" (UID: \"311625d7-ceca-4585-b128-bd8124002a8b\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5f8f955bf6-bxb8q"
Oct 31 08:44:39.080507 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:44:39.080483    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 08:44:39.097349 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:44:39.097296    4720 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"default-dockercfg-9dmmz\""
Oct 31 08:44:39.097349 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:44:39.097316    4720 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"seaweedfs-s3-creds\""
Oct 31 08:44:39.097349 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:44:39.097304    4720 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"kube-root-ca.crt\""
Oct 31 08:44:39.097617 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:44:39.097473    4720 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"openshift-service-ca.crt\""
Oct 31 08:44:39.097617 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:44:39.097497    4720 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"odh-kserve-custom-ca-bundle\""
Oct 31 08:44:39.097617 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:44:39.097507    4720 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"prestop-hook-test-kserve-self-signed-certs\""
Oct 31 08:44:39.154137 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:44:39.154109    4720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4cqm8\" (UniqueName: \"kubernetes.io/projected/311625d7-ceca-4585-b128-bd8124002a8b-kube-api-access-4cqm8\") pod \"prestop-hook-test-kserve-5f8f955bf6-bxb8q\" (UID: \"311625d7-ceca-4585-b128-bd8124002a8b\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5f8f955bf6-bxb8q"
Oct 31 08:44:39.154258 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:44:39.154163    4720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/311625d7-ceca-4585-b128-bd8124002a8b-dshm\") pod \"prestop-hook-test-kserve-5f8f955bf6-bxb8q\" (UID: \"311625d7-ceca-4585-b128-bd8124002a8b\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5f8f955bf6-bxb8q"
Oct 31 08:44:39.154258 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:44:39.154194    4720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/311625d7-ceca-4585-b128-bd8124002a8b-model-cache\") pod \"prestop-hook-test-kserve-5f8f955bf6-bxb8q\" (UID: \"311625d7-ceca-4585-b128-bd8124002a8b\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5f8f955bf6-bxb8q"
Oct 31 08:44:39.154258 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:44:39.154221    4720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/311625d7-ceca-4585-b128-bd8124002a8b-tmp-dir\") pod \"prestop-hook-test-kserve-5f8f955bf6-bxb8q\" (UID: \"311625d7-ceca-4585-b128-bd8124002a8b\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5f8f955bf6-bxb8q"
Oct 31 08:44:39.154258 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:44:39.154246    4720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/311625d7-ceca-4585-b128-bd8124002a8b-kserve-provision-location\") pod \"prestop-hook-test-kserve-5f8f955bf6-bxb8q\" (UID: \"311625d7-ceca-4585-b128-bd8124002a8b\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5f8f955bf6-bxb8q"
Oct 31 08:44:39.154511 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:44:39.154284    4720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/311625d7-ceca-4585-b128-bd8124002a8b-home\") pod \"prestop-hook-test-kserve-5f8f955bf6-bxb8q\" (UID: \"311625d7-ceca-4585-b128-bd8124002a8b\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5f8f955bf6-bxb8q"
Oct 31 08:44:39.154511 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:44:39.154310    4720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/311625d7-ceca-4585-b128-bd8124002a8b-tls-certs\") pod \"prestop-hook-test-kserve-5f8f955bf6-bxb8q\" (UID: \"311625d7-ceca-4585-b128-bd8124002a8b\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5f8f955bf6-bxb8q"
Oct 31 08:44:39.154511 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:44:39.154367    4720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/311625d7-ceca-4585-b128-bd8124002a8b-cabundle-cert\") pod \"prestop-hook-test-kserve-5f8f955bf6-bxb8q\" (UID: \"311625d7-ceca-4585-b128-bd8124002a8b\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5f8f955bf6-bxb8q"
Oct 31 08:44:39.154795 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:44:39.154668    4720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/311625d7-ceca-4585-b128-bd8124002a8b-model-cache\") pod \"prestop-hook-test-kserve-5f8f955bf6-bxb8q\" (UID: \"311625d7-ceca-4585-b128-bd8124002a8b\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5f8f955bf6-bxb8q"
Oct 31 08:44:39.154795 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:44:39.154737    4720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/311625d7-ceca-4585-b128-bd8124002a8b-tmp-dir\") pod \"prestop-hook-test-kserve-5f8f955bf6-bxb8q\" (UID: \"311625d7-ceca-4585-b128-bd8124002a8b\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5f8f955bf6-bxb8q"
Oct 31 08:44:39.154990 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:44:39.154827    4720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/311625d7-ceca-4585-b128-bd8124002a8b-home\") pod \"prestop-hook-test-kserve-5f8f955bf6-bxb8q\" (UID: \"311625d7-ceca-4585-b128-bd8124002a8b\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5f8f955bf6-bxb8q"
Oct 31 08:44:39.154990 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:44:39.154847    4720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/311625d7-ceca-4585-b128-bd8124002a8b-kserve-provision-location\") pod \"prestop-hook-test-kserve-5f8f955bf6-bxb8q\" (UID: \"311625d7-ceca-4585-b128-bd8124002a8b\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5f8f955bf6-bxb8q"
Oct 31 08:44:39.155312 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:44:39.155290    4720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/311625d7-ceca-4585-b128-bd8124002a8b-cabundle-cert\") pod \"prestop-hook-test-kserve-5f8f955bf6-bxb8q\" (UID: \"311625d7-ceca-4585-b128-bd8124002a8b\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5f8f955bf6-bxb8q"
Oct 31 08:44:39.157275 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:44:39.157258    4720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/311625d7-ceca-4585-b128-bd8124002a8b-dshm\") pod \"prestop-hook-test-kserve-5f8f955bf6-bxb8q\" (UID: \"311625d7-ceca-4585-b128-bd8124002a8b\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5f8f955bf6-bxb8q"
Oct 31 08:44:39.157690 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:44:39.157669    4720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/311625d7-ceca-4585-b128-bd8124002a8b-tls-certs\") pod \"prestop-hook-test-kserve-5f8f955bf6-bxb8q\" (UID: \"311625d7-ceca-4585-b128-bd8124002a8b\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5f8f955bf6-bxb8q"
Oct 31 08:44:39.210754 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:44:39.210734    4720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4cqm8\" (UniqueName: \"kubernetes.io/projected/311625d7-ceca-4585-b128-bd8124002a8b-kube-api-access-4cqm8\") pod \"prestop-hook-test-kserve-5f8f955bf6-bxb8q\" (UID: \"311625d7-ceca-4585-b128-bd8124002a8b\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5f8f955bf6-bxb8q"
Oct 31 08:44:39.315601 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:44:39.315509    4720 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5f8f955bf6-bxb8q"
Oct 31 08:44:39.470297 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:44:39.470257    4720 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-89ccd5fbb-qpmc8"]
Oct 31 08:44:39.473860 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:44:39.473841    4720 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-89ccd5fbb-qpmc8"
Oct 31 08:44:39.528207 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:44:39.528172    4720 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"prestop-hook-test-epp-sa-dockercfg-jthvq\""
Oct 31 08:44:39.531149 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:44:39.531123    4720 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-89ccd5fbb-qpmc8"]
Oct 31 08:44:39.556501 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:44:39.556465    4720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f7422773-b478-4959-9e99-d1bbc866f927-tls-certs\") pod \"prestop-hook-test-kserve-router-scheduler-89ccd5fbb-qpmc8\" (UID: \"f7422773-b478-4959-9e99-d1bbc866f927\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-89ccd5fbb-qpmc8"
Oct 31 08:44:39.556613 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:44:39.556516    4720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f2gbw\" (UniqueName: \"kubernetes.io/projected/f7422773-b478-4959-9e99-d1bbc866f927-kube-api-access-f2gbw\") pod \"prestop-hook-test-kserve-router-scheduler-89ccd5fbb-qpmc8\" (UID: \"f7422773-b478-4959-9e99-d1bbc866f927\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-89ccd5fbb-qpmc8"
Oct 31 08:44:39.585936 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:44:39.585916    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 08:44:39.657601 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:44:39.657575    4720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f7422773-b478-4959-9e99-d1bbc866f927-tls-certs\") pod \"prestop-hook-test-kserve-router-scheduler-89ccd5fbb-qpmc8\" (UID: \"f7422773-b478-4959-9e99-d1bbc866f927\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-89ccd5fbb-qpmc8"
Oct 31 08:44:39.657914 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:44:39.657618    4720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f2gbw\" (UniqueName: \"kubernetes.io/projected/f7422773-b478-4959-9e99-d1bbc866f927-kube-api-access-f2gbw\") pod \"prestop-hook-test-kserve-router-scheduler-89ccd5fbb-qpmc8\" (UID: \"f7422773-b478-4959-9e99-d1bbc866f927\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-89ccd5fbb-qpmc8"
Oct 31 08:44:39.660195 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:44:39.660172    4720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f7422773-b478-4959-9e99-d1bbc866f927-tls-certs\") pod \"prestop-hook-test-kserve-router-scheduler-89ccd5fbb-qpmc8\" (UID: \"f7422773-b478-4959-9e99-d1bbc866f927\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-89ccd5fbb-qpmc8"
Oct 31 08:44:39.714535 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:44:39.714509    4720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f2gbw\" (UniqueName: \"kubernetes.io/projected/f7422773-b478-4959-9e99-d1bbc866f927-kube-api-access-f2gbw\") pod \"prestop-hook-test-kserve-router-scheduler-89ccd5fbb-qpmc8\" (UID: \"f7422773-b478-4959-9e99-d1bbc866f927\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-89ccd5fbb-qpmc8"
Oct 31 08:44:39.758951 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:44:39.758922    4720 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5f8f955bf6-bxb8q"]
Oct 31 08:44:39.761099 ip-10-0-139-83 kubenswrapper[4720]: W1031 08:44:39.761074    4720 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod311625d7_ceca_4585_b128_bd8124002a8b.slice/crio-f985706102500ce0b5e120ef3791bdb0656617d746f2e82f15bf2a4454859d82 WatchSource:0}: Error finding container f985706102500ce0b5e120ef3791bdb0656617d746f2e82f15bf2a4454859d82: Status 404 returned error can't find the container with id f985706102500ce0b5e120ef3791bdb0656617d746f2e82f15bf2a4454859d82
Oct 31 08:44:39.787344 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:44:39.787307    4720 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-89ccd5fbb-qpmc8"
Oct 31 08:44:40.160930 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:44:40.160902    4720 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-89ccd5fbb-qpmc8"]
Oct 31 08:44:40.163874 ip-10-0-139-83 kubenswrapper[4720]: W1031 08:44:40.163846    4720 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf7422773_b478_4959_9e99_d1bbc866f927.slice/crio-43dadc8cf43ebf0df4c03e5b2c2c3b710b9438c6714fb4edbb2ac0455e4260cc WatchSource:0}: Error finding container 43dadc8cf43ebf0df4c03e5b2c2c3b710b9438c6714fb4edbb2ac0455e4260cc: Status 404 returned error can't find the container with id 43dadc8cf43ebf0df4c03e5b2c2c3b710b9438c6714fb4edbb2ac0455e4260cc
Oct 31 08:44:40.195622 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:44:40.195596    4720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5f8f955bf6-bxb8q" event={"ID":"311625d7-ceca-4585-b128-bd8124002a8b","Type":"ContainerStarted","Data":"cfc54015d563270b8531fa76f760ed948f485e06bea2131f3dca57abd0ee2c44"}
Oct 31 08:44:40.195744 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:44:40.195653    4720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5f8f955bf6-bxb8q" event={"ID":"311625d7-ceca-4585-b128-bd8124002a8b","Type":"ContainerStarted","Data":"f985706102500ce0b5e120ef3791bdb0656617d746f2e82f15bf2a4454859d82"}
Oct 31 08:44:40.196993 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:44:40.196967    4720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-89ccd5fbb-qpmc8" event={"ID":"f7422773-b478-4959-9e99-d1bbc866f927","Type":"ContainerStarted","Data":"43dadc8cf43ebf0df4c03e5b2c2c3b710b9438c6714fb4edbb2ac0455e4260cc"}
Oct 31 08:44:40.437923 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:44:40.437844    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 08:44:41.203406 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:44:41.203358    4720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-89ccd5fbb-qpmc8" event={"ID":"f7422773-b478-4959-9e99-d1bbc866f927","Type":"ContainerStarted","Data":"950dcec70f56ec5b473e84eff247c90d6f122ea1696b92ae261797168c467921"}
Oct 31 08:44:41.203838 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:44:41.203699    4720 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-89ccd5fbb-qpmc8"
Oct 31 08:44:41.317982 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:44:41.317916    4720 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-89ccd5fbb-qpmc8" podStartSLOduration=2.317902388 podStartE2EDuration="2.317902388s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 08:44:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 08:44:41.316852689 +0000 UTC m=+4535.447920772" watchObservedRunningTime="2026-10-31 08:44:41.317902388 +0000 UTC m=+4535.448970459"
Oct 31 08:44:42.209172 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:44:42.209135    4720 generic.go:356] "Generic (PLEG): container finished" podID="311625d7-ceca-4585-b128-bd8124002a8b" containerID="cfc54015d563270b8531fa76f760ed948f485e06bea2131f3dca57abd0ee2c44" exitCode=0
Oct 31 08:44:42.209641 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:44:42.209207    4720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5f8f955bf6-bxb8q" event={"ID":"311625d7-ceca-4585-b128-bd8124002a8b","Type":"ContainerDied","Data":"cfc54015d563270b8531fa76f760ed948f485e06bea2131f3dca57abd0ee2c44"}
Oct 31 08:44:43.215516 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:44:43.215480    4720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5f8f955bf6-bxb8q" event={"ID":"311625d7-ceca-4585-b128-bd8124002a8b","Type":"ContainerStarted","Data":"ffbda9df978943b1603b01e87238f446f9e82945e9d19b6f1f71ac6e7f570f7e"}
Oct 31 08:44:43.315853 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:44:43.315820    4720 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5f8f955bf6-bxb8q"
Oct 31 08:44:43.434245 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:44:43.434184    4720 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5f8f955bf6-bxb8q" podStartSLOduration=3.206979756 podStartE2EDuration="5.43417064s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.227190884s" isStatefulPod=false podCreationTimestamp="2026-10-31 08:44:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 08:44:43.432781543 +0000 UTC m=+4537.563849615" watchObservedRunningTime="2026-10-31 08:44:43.43417064 +0000 UTC m=+4537.565238702"
Oct 31 08:44:43.461178 ip-10-0-139-83 kubenswrapper[4720]: E1031 08:44:43.461142    4720 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8m768" containerName="extractor"
Oct 31 08:44:44.656320 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:44:44.656286    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 08:44:44.823146 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:44:44.823105    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 08:44:45.417568 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:44:45.417488    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 08:44:49.316001 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:44:49.315956    4720 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5f8f955bf6-bxb8q"
Oct 31 08:44:49.317673 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:44:49.317644    4720 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5f8f955bf6-bxb8q" podUID="311625d7-ceca-4585-b128-bd8124002a8b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/health\": dial tcp 10.132.0.82:8000: connect: connection refused"
Oct 31 08:44:59.316543 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:44:59.316494    4720 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5f8f955bf6-bxb8q" podUID="311625d7-ceca-4585-b128-bd8124002a8b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/health\": dial tcp 10.132.0.82:8000: connect: connection refused"
Oct 31 08:45:05.275868 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:45:05.275827    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 08:45:06.753301 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:45:06.753265    4720 scope.go:122] "RemoveContainer" containerID="6dd718e6cdb43e1aa7b646bb37635e4783cbdd0d115a05e124aa5ab9ef49ae72"
Oct 31 08:45:09.316702 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:45:09.316658    4720 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5f8f955bf6-bxb8q" podUID="311625d7-ceca-4585-b128-bd8124002a8b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/health\": dial tcp 10.132.0.82:8000: connect: connection refused"
Oct 31 08:45:12.211255 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:45:12.211222    4720 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-89ccd5fbb-qpmc8"
Oct 31 08:45:13.460811 ip-10-0-139-83 kubenswrapper[4720]: E1031 08:45:13.460770    4720 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8m768" containerName="extractor"
Oct 31 08:45:19.316830 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:45:19.316785    4720 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5f8f955bf6-bxb8q" podUID="311625d7-ceca-4585-b128-bd8124002a8b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/health\": dial tcp 10.132.0.82:8000: connect: connection refused"
Oct 31 08:45:26.281048 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:45:26.281005    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 08:45:29.316378 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:45:29.316295    4720 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5f8f955bf6-bxb8q" podUID="311625d7-ceca-4585-b128-bd8124002a8b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/health\": dial tcp 10.132.0.82:8000: connect: connection refused"
Oct 31 08:45:39.316655 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:45:39.316611    4720 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5f8f955bf6-bxb8q" podUID="311625d7-ceca-4585-b128-bd8124002a8b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/health\": dial tcp 10.132.0.82:8000: connect: connection refused"
Oct 31 08:45:39.550574 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:45:39.550540    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 08:45:39.725703 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:45:39.725667    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 08:45:40.254041 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:45:40.253997    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 08:45:40.425506 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:45:40.425473    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 08:45:42.316588 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:45:42.316543    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 08:45:43.460913 ip-10-0-139-83 kubenswrapper[4720]: E1031 08:45:43.460872    4720 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8m768" containerName="extractor"
Oct 31 08:45:44.220683 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:45:44.220642    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 08:45:44.588410 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:45:44.588383    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 08:45:45.322760 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:45:45.322725    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 08:45:49.315969 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:45:49.315927    4720 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5f8f955bf6-bxb8q" podUID="311625d7-ceca-4585-b128-bd8124002a8b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/health\": dial tcp 10.132.0.82:8000: connect: connection refused"
Oct 31 08:45:59.316952 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:45:59.316901    4720 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5f8f955bf6-bxb8q" podUID="311625d7-ceca-4585-b128-bd8124002a8b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/health\": dial tcp 10.132.0.82:8000: connect: connection refused"
Oct 31 08:46:05.555680 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:46:05.555640    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 08:46:05.715763 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:46:05.715703    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 08:46:06.828493 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:46:06.828451    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 08:46:09.316062 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:46:09.316017    4720 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5f8f955bf6-bxb8q" podUID="311625d7-ceca-4585-b128-bd8124002a8b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/health\": dial tcp 10.132.0.82:8000: connect: connection refused"
Oct 31 08:46:12.312090 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:46:12.312056    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 08:46:13.461486 ip-10-0-139-83 kubenswrapper[4720]: E1031 08:46:13.461447    4720 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8m768" containerName="extractor"
Oct 31 08:46:19.316708 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:46:19.316621    4720 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5f8f955bf6-bxb8q" podUID="311625d7-ceca-4585-b128-bd8124002a8b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/health\": dial tcp 10.132.0.82:8000: connect: connection refused"
Oct 31 08:46:22.314855 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:46:22.314818    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 08:46:26.472992 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:46:26.472936    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 08:46:26.650121 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:46:26.650084    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 08:46:29.316312 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:46:29.316272    4720 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5f8f955bf6-bxb8q" podUID="311625d7-ceca-4585-b128-bd8124002a8b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/health\": dial tcp 10.132.0.82:8000: connect: connection refused"
Oct 31 08:46:39.325624 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:46:39.325580    4720 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5f8f955bf6-bxb8q"
Oct 31 08:46:39.333148 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:46:39.333126    4720 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5f8f955bf6-bxb8q"
Oct 31 08:46:40.063470 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:46:40.063440    4720 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5f8f955bf6-bxb8q"]
Oct 31 08:46:41.143669 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:46:41.143633    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 08:46:43.460942 ip-10-0-139-83 kubenswrapper[4720]: E1031 08:46:43.460906    4720 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8m768" containerName="extractor"
Oct 31 08:46:55.793362 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:46:55.793285    4720 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5f8f955bf6-bxb8q" podUID="311625d7-ceca-4585-b128-bd8124002a8b" containerName="main" containerID="cri-o://ffbda9df978943b1603b01e87238f446f9e82945e9d19b6f1f71ac6e7f570f7e" gracePeriod=45
Oct 31 08:46:56.326956 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:46:56.326915    4720 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5f8f955bf6-bxb8q" podUID="311625d7-ceca-4585-b128-bd8124002a8b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/health\": dial tcp 10.132.0.82:8000: connect: connection refused"
Oct 31 08:46:56.449373 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:46:56.449348    4720 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5f8f955bf6-bxb8q"
Oct 31 08:46:56.499808 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:46:56.499747    4720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/311625d7-ceca-4585-b128-bd8124002a8b-dshm\" (UniqueName: \"kubernetes.io/empty-dir/311625d7-ceca-4585-b128-bd8124002a8b-dshm\") pod \"311625d7-ceca-4585-b128-bd8124002a8b\" (UID: \"311625d7-ceca-4585-b128-bd8124002a8b\") "
Oct 31 08:46:56.499808 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:46:56.499787    4720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/311625d7-ceca-4585-b128-bd8124002a8b-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/311625d7-ceca-4585-b128-bd8124002a8b-tmp-dir\") pod \"311625d7-ceca-4585-b128-bd8124002a8b\" (UID: \"311625d7-ceca-4585-b128-bd8124002a8b\") "
Oct 31 08:46:56.499981 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:46:56.499830    4720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/311625d7-ceca-4585-b128-bd8124002a8b-kube-api-access-4cqm8\" (UniqueName: \"kubernetes.io/projected/311625d7-ceca-4585-b128-bd8124002a8b-kube-api-access-4cqm8\") pod \"311625d7-ceca-4585-b128-bd8124002a8b\" (UID: \"311625d7-ceca-4585-b128-bd8124002a8b\") "
Oct 31 08:46:56.499981 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:46:56.499875    4720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/311625d7-ceca-4585-b128-bd8124002a8b-home\" (UniqueName: \"kubernetes.io/empty-dir/311625d7-ceca-4585-b128-bd8124002a8b-home\") pod \"311625d7-ceca-4585-b128-bd8124002a8b\" (UID: \"311625d7-ceca-4585-b128-bd8124002a8b\") "
Oct 31 08:46:56.499981 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:46:56.499909    4720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/311625d7-ceca-4585-b128-bd8124002a8b-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/311625d7-ceca-4585-b128-bd8124002a8b-kserve-provision-location\") pod \"311625d7-ceca-4585-b128-bd8124002a8b\" (UID: \"311625d7-ceca-4585-b128-bd8124002a8b\") "
Oct 31 08:46:56.499981 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:46:56.499942    4720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/311625d7-ceca-4585-b128-bd8124002a8b-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/311625d7-ceca-4585-b128-bd8124002a8b-model-cache\") pod \"311625d7-ceca-4585-b128-bd8124002a8b\" (UID: \"311625d7-ceca-4585-b128-bd8124002a8b\") "
Oct 31 08:46:56.499981 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:46:56.499976    4720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/311625d7-ceca-4585-b128-bd8124002a8b-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/311625d7-ceca-4585-b128-bd8124002a8b-cabundle-cert\") pod \"311625d7-ceca-4585-b128-bd8124002a8b\" (UID: \"311625d7-ceca-4585-b128-bd8124002a8b\") "
Oct 31 08:46:56.500236 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:46:56.500042    4720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/311625d7-ceca-4585-b128-bd8124002a8b-tls-certs\" (UniqueName: \"kubernetes.io/secret/311625d7-ceca-4585-b128-bd8124002a8b-tls-certs\") pod \"311625d7-ceca-4585-b128-bd8124002a8b\" (UID: \"311625d7-ceca-4585-b128-bd8124002a8b\") "
Oct 31 08:46:56.500429 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:46:56.500398    4720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/311625d7-ceca-4585-b128-bd8124002a8b-model-cache" pod "311625d7-ceca-4585-b128-bd8124002a8b" (UID: "311625d7-ceca-4585-b128-bd8124002a8b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 08:46:56.500653 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:46:56.500576    4720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/311625d7-ceca-4585-b128-bd8124002a8b-home" pod "311625d7-ceca-4585-b128-bd8124002a8b" (UID: "311625d7-ceca-4585-b128-bd8124002a8b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 08:46:56.500970 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:46:56.500885    4720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/311625d7-ceca-4585-b128-bd8124002a8b-cabundle-cert" pod "311625d7-ceca-4585-b128-bd8124002a8b" (UID: "311625d7-ceca-4585-b128-bd8124002a8b"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 08:46:56.502268 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:46:56.502243    4720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/311625d7-ceca-4585-b128-bd8124002a8b-dshm" pod "311625d7-ceca-4585-b128-bd8124002a8b" (UID: "311625d7-ceca-4585-b128-bd8124002a8b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 08:46:56.502379 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:46:56.502357    4720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/311625d7-ceca-4585-b128-bd8124002a8b-kube-api-access-4cqm8" pod "311625d7-ceca-4585-b128-bd8124002a8b" (UID: "311625d7-ceca-4585-b128-bd8124002a8b"). InnerVolumeSpecName "kube-api-access-4cqm8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 08:46:56.502541 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:46:56.502521    4720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/311625d7-ceca-4585-b128-bd8124002a8b-tls-certs" pod "311625d7-ceca-4585-b128-bd8124002a8b" (UID: "311625d7-ceca-4585-b128-bd8124002a8b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 08:46:56.519095 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:46:56.519042    4720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/311625d7-ceca-4585-b128-bd8124002a8b-tmp-dir" pod "311625d7-ceca-4585-b128-bd8124002a8b" (UID: "311625d7-ceca-4585-b128-bd8124002a8b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 08:46:56.529855 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:46:56.529829    4720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/311625d7-ceca-4585-b128-bd8124002a8b-kserve-provision-location" pod "311625d7-ceca-4585-b128-bd8124002a8b" (UID: "311625d7-ceca-4585-b128-bd8124002a8b"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 08:46:56.601346 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:46:56.601308    4720 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4cqm8\" (UniqueName: \"kubernetes.io/projected/311625d7-ceca-4585-b128-bd8124002a8b-kube-api-access-4cqm8\") on node \"ip-10-0-139-83.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 08:46:56.601346 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:46:56.601344    4720 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/311625d7-ceca-4585-b128-bd8124002a8b-home\") on node \"ip-10-0-139-83.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 08:46:56.601474 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:46:56.601354    4720 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/311625d7-ceca-4585-b128-bd8124002a8b-kserve-provision-location\") on node \"ip-10-0-139-83.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 08:46:56.601474 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:46:56.601363    4720 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/311625d7-ceca-4585-b128-bd8124002a8b-model-cache\") on node \"ip-10-0-139-83.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 08:46:56.601474 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:46:56.601372    4720 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/311625d7-ceca-4585-b128-bd8124002a8b-cabundle-cert\") on node \"ip-10-0-139-83.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 08:46:56.601474 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:46:56.601380    4720 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/311625d7-ceca-4585-b128-bd8124002a8b-tls-certs\") on node \"ip-10-0-139-83.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 08:46:56.601474 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:46:56.601389    4720 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/311625d7-ceca-4585-b128-bd8124002a8b-dshm\") on node \"ip-10-0-139-83.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 08:46:56.601474 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:46:56.601396    4720 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/311625d7-ceca-4585-b128-bd8124002a8b-tmp-dir\") on node \"ip-10-0-139-83.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 08:46:56.848277 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:46:56.848250    4720 generic.go:356] "Generic (PLEG): container finished" podID="311625d7-ceca-4585-b128-bd8124002a8b" containerID="ffbda9df978943b1603b01e87238f446f9e82945e9d19b6f1f71ac6e7f570f7e" exitCode=0
Oct 31 08:46:56.848693 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:46:56.848301    4720 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5f8f955bf6-bxb8q"
Oct 31 08:46:56.848693 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:46:56.848340    4720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5f8f955bf6-bxb8q" event={"ID":"311625d7-ceca-4585-b128-bd8124002a8b","Type":"ContainerDied","Data":"ffbda9df978943b1603b01e87238f446f9e82945e9d19b6f1f71ac6e7f570f7e"}
Oct 31 08:46:56.848693 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:46:56.848397    4720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5f8f955bf6-bxb8q" event={"ID":"311625d7-ceca-4585-b128-bd8124002a8b","Type":"ContainerDied","Data":"f985706102500ce0b5e120ef3791bdb0656617d746f2e82f15bf2a4454859d82"}
Oct 31 08:46:56.848693 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:46:56.848420    4720 scope.go:122] "RemoveContainer" containerID="ffbda9df978943b1603b01e87238f446f9e82945e9d19b6f1f71ac6e7f570f7e"
Oct 31 08:46:56.857366 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:46:56.857346    4720 scope.go:122] "RemoveContainer" containerID="cfc54015d563270b8531fa76f760ed948f485e06bea2131f3dca57abd0ee2c44"
Oct 31 08:46:56.864898 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:46:56.864878    4720 scope.go:122] "RemoveContainer" containerID="ffbda9df978943b1603b01e87238f446f9e82945e9d19b6f1f71ac6e7f570f7e"
Oct 31 08:46:56.865164 ip-10-0-139-83 kubenswrapper[4720]: E1031 08:46:56.865144    4720 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ffbda9df978943b1603b01e87238f446f9e82945e9d19b6f1f71ac6e7f570f7e\": container with ID starting with ffbda9df978943b1603b01e87238f446f9e82945e9d19b6f1f71ac6e7f570f7e not found: ID does not exist" containerID="ffbda9df978943b1603b01e87238f446f9e82945e9d19b6f1f71ac6e7f570f7e"
Oct 31 08:46:56.865212 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:46:56.865172    4720 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ffbda9df978943b1603b01e87238f446f9e82945e9d19b6f1f71ac6e7f570f7e"} err="failed to get container status \"ffbda9df978943b1603b01e87238f446f9e82945e9d19b6f1f71ac6e7f570f7e\": rpc error: code = NotFound desc = could not find container \"ffbda9df978943b1603b01e87238f446f9e82945e9d19b6f1f71ac6e7f570f7e\": container with ID starting with ffbda9df978943b1603b01e87238f446f9e82945e9d19b6f1f71ac6e7f570f7e not found: ID does not exist"
Oct 31 08:46:56.865212 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:46:56.865187    4720 scope.go:122] "RemoveContainer" containerID="cfc54015d563270b8531fa76f760ed948f485e06bea2131f3dca57abd0ee2c44"
Oct 31 08:46:56.865418 ip-10-0-139-83 kubenswrapper[4720]: E1031 08:46:56.865398    4720 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cfc54015d563270b8531fa76f760ed948f485e06bea2131f3dca57abd0ee2c44\": container with ID starting with cfc54015d563270b8531fa76f760ed948f485e06bea2131f3dca57abd0ee2c44 not found: ID does not exist" containerID="cfc54015d563270b8531fa76f760ed948f485e06bea2131f3dca57abd0ee2c44"
Oct 31 08:46:56.865465 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:46:56.865424    4720 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cfc54015d563270b8531fa76f760ed948f485e06bea2131f3dca57abd0ee2c44"} err="failed to get container status \"cfc54015d563270b8531fa76f760ed948f485e06bea2131f3dca57abd0ee2c44\": rpc error: code = NotFound desc = could not find container \"cfc54015d563270b8531fa76f760ed948f485e06bea2131f3dca57abd0ee2c44\": container with ID starting with cfc54015d563270b8531fa76f760ed948f485e06bea2131f3dca57abd0ee2c44 not found: ID does not exist"
Oct 31 08:46:57.012073 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:46:57.012042    4720 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5f8f955bf6-bxb8q"]
Oct 31 08:46:57.015880 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:46:57.015858    4720 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5f8f955bf6-bxb8q"]
Oct 31 08:46:57.487444 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:46:57.487416    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 08:46:57.645147 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:46:57.645122    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 08:46:58.025174 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:46:58.025142    4720 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="311625d7-ceca-4585-b128-bd8124002a8b" path="/var/lib/kubelet/pods/311625d7-ceca-4585-b128-bd8124002a8b/volumes"
Oct 31 08:47:00.530643 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:47:00.530604    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 08:47:00.968458 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:47:00.968420    4720 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-89ccd5fbb-qpmc8"]
Oct 31 08:47:01.089670 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:47:01.089627    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 08:47:01.245519 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:47:01.245439    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 08:47:02.211593 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:47:02.211560    4720 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-89ccd5fbb-qpmc8" podUID="f7422773-b478-4959-9e99-d1bbc866f927" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 08:47:05.640653 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:47:05.640620    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 08:47:05.901635 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:47:05.901557    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 08:47:12.211435 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:47:12.211402    4720 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-89ccd5fbb-qpmc8" podUID="f7422773-b478-4959-9e99-d1bbc866f927" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 08:47:13.460625 ip-10-0-139-83 kubenswrapper[4720]: E1031 08:47:13.460589    4720 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8m768" containerName="extractor"
Oct 31 08:47:15.988685 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:47:15.988628    4720 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-89ccd5fbb-qpmc8" podUID="f7422773-b478-4959-9e99-d1bbc866f927" containerName="main" containerID="cri-o://950dcec70f56ec5b473e84eff247c90d6f122ea1696b92ae261797168c467921" gracePeriod=45
Oct 31 08:47:22.211257 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:47:22.211219    4720 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-89ccd5fbb-qpmc8" podUID="f7422773-b478-4959-9e99-d1bbc866f927" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 08:47:22.211678 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:47:22.211361    4720 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-89ccd5fbb-qpmc8"
Oct 31 08:47:26.836316 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:47:26.836275    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 08:47:27.750848 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:47:27.750817    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 08:47:27.912550 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:47:27.912522    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 08:47:28.321136 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:47:28.321104    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 08:47:32.211159 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:47:32.211124    4720 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-89ccd5fbb-qpmc8" podUID="f7422773-b478-4959-9e99-d1bbc866f927" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 08:47:33.404603 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:47:33.404571    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 08:47:41.511692 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:47:41.511656    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 08:47:42.210971 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:47:42.210939    4720 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-89ccd5fbb-qpmc8" podUID="f7422773-b478-4959-9e99-d1bbc866f927" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 08:47:43.460455 ip-10-0-139-83 kubenswrapper[4720]: E1031 08:47:43.460418    4720 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8m768" containerName="extractor"
Oct 31 08:47:44.418617 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:47:44.418587    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 08:47:46.238235 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:47:46.238205    4720 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-89ccd5fbb-qpmc8"
Oct 31 08:47:46.260230 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:47:46.260184    4720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f7422773-b478-4959-9e99-d1bbc866f927-tls-certs\" (UniqueName: \"kubernetes.io/secret/f7422773-b478-4959-9e99-d1bbc866f927-tls-certs\") pod \"f7422773-b478-4959-9e99-d1bbc866f927\" (UID: \"f7422773-b478-4959-9e99-d1bbc866f927\") "
Oct 31 08:47:46.260341 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:47:46.260244    4720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f7422773-b478-4959-9e99-d1bbc866f927-kube-api-access-f2gbw\" (UniqueName: \"kubernetes.io/projected/f7422773-b478-4959-9e99-d1bbc866f927-kube-api-access-f2gbw\") pod \"f7422773-b478-4959-9e99-d1bbc866f927\" (UID: \"f7422773-b478-4959-9e99-d1bbc866f927\") "
Oct 31 08:47:46.262305 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:47:46.262278    4720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f7422773-b478-4959-9e99-d1bbc866f927-kube-api-access-f2gbw" pod "f7422773-b478-4959-9e99-d1bbc866f927" (UID: "f7422773-b478-4959-9e99-d1bbc866f927"). InnerVolumeSpecName "kube-api-access-f2gbw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 08:47:46.262423 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:47:46.262308    4720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f7422773-b478-4959-9e99-d1bbc866f927-tls-certs" pod "f7422773-b478-4959-9e99-d1bbc866f927" (UID: "f7422773-b478-4959-9e99-d1bbc866f927"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 08:47:46.360736 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:47:46.360707    4720 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f7422773-b478-4959-9e99-d1bbc866f927-tls-certs\") on node \"ip-10-0-139-83.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 08:47:46.360843 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:47:46.360739    4720 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f2gbw\" (UniqueName: \"kubernetes.io/projected/f7422773-b478-4959-9e99-d1bbc866f927-kube-api-access-f2gbw\") on node \"ip-10-0-139-83.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 08:47:47.061128 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:47:47.061090    4720 generic.go:356] "Generic (PLEG): container finished" podID="f7422773-b478-4959-9e99-d1bbc866f927" containerID="950dcec70f56ec5b473e84eff247c90d6f122ea1696b92ae261797168c467921" exitCode=0
Oct 31 08:47:47.061349 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:47:47.061140    4720 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-89ccd5fbb-qpmc8"
Oct 31 08:47:47.061349 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:47:47.061156    4720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-89ccd5fbb-qpmc8" event={"ID":"f7422773-b478-4959-9e99-d1bbc866f927","Type":"ContainerDied","Data":"950dcec70f56ec5b473e84eff247c90d6f122ea1696b92ae261797168c467921"}
Oct 31 08:47:47.061349 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:47:47.061200    4720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-89ccd5fbb-qpmc8" event={"ID":"f7422773-b478-4959-9e99-d1bbc866f927","Type":"ContainerDied","Data":"43dadc8cf43ebf0df4c03e5b2c2c3b710b9438c6714fb4edbb2ac0455e4260cc"}
Oct 31 08:47:47.061349 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:47:47.061235    4720 scope.go:122] "RemoveContainer" containerID="950dcec70f56ec5b473e84eff247c90d6f122ea1696b92ae261797168c467921"
Oct 31 08:47:47.070424 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:47:47.070405    4720 scope.go:122] "RemoveContainer" containerID="950dcec70f56ec5b473e84eff247c90d6f122ea1696b92ae261797168c467921"
Oct 31 08:47:47.070694 ip-10-0-139-83 kubenswrapper[4720]: E1031 08:47:47.070673    4720 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"950dcec70f56ec5b473e84eff247c90d6f122ea1696b92ae261797168c467921\": container with ID starting with 950dcec70f56ec5b473e84eff247c90d6f122ea1696b92ae261797168c467921 not found: ID does not exist" containerID="950dcec70f56ec5b473e84eff247c90d6f122ea1696b92ae261797168c467921"
Oct 31 08:47:47.070747 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:47:47.070703    4720 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"950dcec70f56ec5b473e84eff247c90d6f122ea1696b92ae261797168c467921"} err="failed to get container status \"950dcec70f56ec5b473e84eff247c90d6f122ea1696b92ae261797168c467921\": rpc error: code = NotFound desc = could not find container \"950dcec70f56ec5b473e84eff247c90d6f122ea1696b92ae261797168c467921\": container with ID starting with 950dcec70f56ec5b473e84eff247c90d6f122ea1696b92ae261797168c467921 not found: ID does not exist"
Oct 31 08:47:47.224886 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:47:47.224862    4720 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-89ccd5fbb-qpmc8"]
Oct 31 08:47:47.274111 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:47:47.274083    4720 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-89ccd5fbb-qpmc8"]
Oct 31 08:47:47.581907 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:47:47.581877    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 08:47:47.751695 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:47:47.751667    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 08:47:48.024948 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:47:48.024878    4720 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f7422773-b478-4959-9e99-d1bbc866f927" path="/var/lib/kubelet/pods/f7422773-b478-4959-9e99-d1bbc866f927/volumes"
Oct 31 08:48:04.348918 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:48:04.348880    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 08:48:04.575261 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:48:04.575227    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 08:48:06.243111 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:48:06.243074    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 08:48:09.895026 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:48:09.894984    4720 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-5d77dc844-jnx68"]
Oct 31 08:48:09.895527 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:48:09.895479    4720 cpu_manager.go:414] "containerMap: removing container" podUID="311625d7-ceca-4585-b128-bd8124002a8b" containerName="storage-initializer"
Oct 31 08:48:09.895527 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:48:09.895498    4720 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="311625d7-ceca-4585-b128-bd8124002a8b" containerName="storage-initializer"
Oct 31 08:48:09.895527 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:48:09.895516    4720 cpu_manager.go:414] "containerMap: removing container" podUID="311625d7-ceca-4585-b128-bd8124002a8b" containerName="main"
Oct 31 08:48:09.895527 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:48:09.895522    4720 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="311625d7-ceca-4585-b128-bd8124002a8b" containerName="main"
Oct 31 08:48:09.895746 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:48:09.895535    4720 cpu_manager.go:414] "containerMap: removing container" podUID="f7422773-b478-4959-9e99-d1bbc866f927" containerName="main"
Oct 31 08:48:09.895746 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:48:09.895541    4720 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f7422773-b478-4959-9e99-d1bbc866f927" containerName="main"
Oct 31 08:48:09.895746 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:48:09.895623    4720 memory_manager.go:365] "RemoveStaleState removing state" podUID="311625d7-ceca-4585-b128-bd8124002a8b" containerName="main"
Oct 31 08:48:09.895746 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:48:09.895633    4720 memory_manager.go:365] "RemoveStaleState removing state" podUID="f7422773-b478-4959-9e99-d1bbc866f927" containerName="main"
Oct 31 08:48:09.900087 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:48:09.900061    4720 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-5d77dc844-jnx68"
Oct 31 08:48:09.915576 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:48:09.915551    4720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d1ccb554-bc34-46a9-89b9-a8a30cd57680-dshm\") pod \"rolling-upgrade-test-kserve-5d77dc844-jnx68\" (UID: \"d1ccb554-bc34-46a9-89b9-a8a30cd57680\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-5d77dc844-jnx68"
Oct 31 08:48:09.915679 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:48:09.915578    4720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d1ccb554-bc34-46a9-89b9-a8a30cd57680-tls-certs\") pod \"rolling-upgrade-test-kserve-5d77dc844-jnx68\" (UID: \"d1ccb554-bc34-46a9-89b9-a8a30cd57680\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-5d77dc844-jnx68"
Oct 31 08:48:09.915679 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:48:09.915606    4720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d1ccb554-bc34-46a9-89b9-a8a30cd57680-tmp-dir\") pod \"rolling-upgrade-test-kserve-5d77dc844-jnx68\" (UID: \"d1ccb554-bc34-46a9-89b9-a8a30cd57680\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-5d77dc844-jnx68"
Oct 31 08:48:09.915679 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:48:09.915620    4720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d1ccb554-bc34-46a9-89b9-a8a30cd57680-home\") pod \"rolling-upgrade-test-kserve-5d77dc844-jnx68\" (UID: \"d1ccb554-bc34-46a9-89b9-a8a30cd57680\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-5d77dc844-jnx68"
Oct 31 08:48:09.915844 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:48:09.915713    4720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x57dr\" (UniqueName: \"kubernetes.io/projected/d1ccb554-bc34-46a9-89b9-a8a30cd57680-kube-api-access-x57dr\") pod \"rolling-upgrade-test-kserve-5d77dc844-jnx68\" (UID: \"d1ccb554-bc34-46a9-89b9-a8a30cd57680\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-5d77dc844-jnx68"
Oct 31 08:48:09.915844 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:48:09.915822    4720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d1ccb554-bc34-46a9-89b9-a8a30cd57680-model-cache\") pod \"rolling-upgrade-test-kserve-5d77dc844-jnx68\" (UID: \"d1ccb554-bc34-46a9-89b9-a8a30cd57680\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-5d77dc844-jnx68"
Oct 31 08:48:10.003720 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:48:10.003689    4720 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-rolling-upgrade-coordination-a9229638\"/\"openshift-service-ca.crt\""
Oct 31 08:48:10.003938 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:48:10.003694    4720 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-rolling-upgrade-coordination-a9229638\"/\"kube-root-ca.crt\""
Oct 31 08:48:10.003938 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:48:10.003736    4720 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-rolling-upgrade-coordination-a9229638\"/\"default-dockercfg-xwjcc\""
Oct 31 08:48:10.004115 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:48:10.003782    4720 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-rolling-upgrade-coordination-a9229638\"/\"rolling-upgrade-test-kserve-self-signed-certs\""
Oct 31 08:48:10.006025 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:48:10.006001    4720 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-5d77dc844-jnx68"]
Oct 31 08:48:10.012695 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:48:10.012674    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 08:48:10.016748 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:48:10.016724    4720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d1ccb554-bc34-46a9-89b9-a8a30cd57680-dshm\") pod \"rolling-upgrade-test-kserve-5d77dc844-jnx68\" (UID: \"d1ccb554-bc34-46a9-89b9-a8a30cd57680\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-5d77dc844-jnx68"
Oct 31 08:48:10.016838 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:48:10.016750    4720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d1ccb554-bc34-46a9-89b9-a8a30cd57680-tls-certs\") pod \"rolling-upgrade-test-kserve-5d77dc844-jnx68\" (UID: \"d1ccb554-bc34-46a9-89b9-a8a30cd57680\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-5d77dc844-jnx68"
Oct 31 08:48:10.016838 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:48:10.016779    4720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d1ccb554-bc34-46a9-89b9-a8a30cd57680-tmp-dir\") pod \"rolling-upgrade-test-kserve-5d77dc844-jnx68\" (UID: \"d1ccb554-bc34-46a9-89b9-a8a30cd57680\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-5d77dc844-jnx68"
Oct 31 08:48:10.016912 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:48:10.016886    4720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d1ccb554-bc34-46a9-89b9-a8a30cd57680-home\") pod \"rolling-upgrade-test-kserve-5d77dc844-jnx68\" (UID: \"d1ccb554-bc34-46a9-89b9-a8a30cd57680\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-5d77dc844-jnx68"
Oct 31 08:48:10.016950 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:48:10.016918    4720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x57dr\" (UniqueName: \"kubernetes.io/projected/d1ccb554-bc34-46a9-89b9-a8a30cd57680-kube-api-access-x57dr\") pod \"rolling-upgrade-test-kserve-5d77dc844-jnx68\" (UID: \"d1ccb554-bc34-46a9-89b9-a8a30cd57680\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-5d77dc844-jnx68"
Oct 31 08:48:10.016992 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:48:10.016954    4720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d1ccb554-bc34-46a9-89b9-a8a30cd57680-model-cache\") pod \"rolling-upgrade-test-kserve-5d77dc844-jnx68\" (UID: \"d1ccb554-bc34-46a9-89b9-a8a30cd57680\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-5d77dc844-jnx68"
Oct 31 08:48:10.017166 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:48:10.017131    4720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d1ccb554-bc34-46a9-89b9-a8a30cd57680-tmp-dir\") pod \"rolling-upgrade-test-kserve-5d77dc844-jnx68\" (UID: \"d1ccb554-bc34-46a9-89b9-a8a30cd57680\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-5d77dc844-jnx68"
Oct 31 08:48:10.017236 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:48:10.017217    4720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d1ccb554-bc34-46a9-89b9-a8a30cd57680-home\") pod \"rolling-upgrade-test-kserve-5d77dc844-jnx68\" (UID: \"d1ccb554-bc34-46a9-89b9-a8a30cd57680\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-5d77dc844-jnx68"
Oct 31 08:48:10.017392 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:48:10.017319    4720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d1ccb554-bc34-46a9-89b9-a8a30cd57680-model-cache\") pod \"rolling-upgrade-test-kserve-5d77dc844-jnx68\" (UID: \"d1ccb554-bc34-46a9-89b9-a8a30cd57680\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-5d77dc844-jnx68"
Oct 31 08:48:10.019284 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:48:10.019259    4720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d1ccb554-bc34-46a9-89b9-a8a30cd57680-dshm\") pod \"rolling-upgrade-test-kserve-5d77dc844-jnx68\" (UID: \"d1ccb554-bc34-46a9-89b9-a8a30cd57680\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-5d77dc844-jnx68"
Oct 31 08:48:10.019420 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:48:10.019400    4720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d1ccb554-bc34-46a9-89b9-a8a30cd57680-tls-certs\") pod \"rolling-upgrade-test-kserve-5d77dc844-jnx68\" (UID: \"d1ccb554-bc34-46a9-89b9-a8a30cd57680\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-5d77dc844-jnx68"
Oct 31 08:48:10.072864 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:48:10.072844    4720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x57dr\" (UniqueName: \"kubernetes.io/projected/d1ccb554-bc34-46a9-89b9-a8a30cd57680-kube-api-access-x57dr\") pod \"rolling-upgrade-test-kserve-5d77dc844-jnx68\" (UID: \"d1ccb554-bc34-46a9-89b9-a8a30cd57680\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-5d77dc844-jnx68"
Oct 31 08:48:10.212264 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:48:10.212205    4720 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-5d77dc844-jnx68"
Oct 31 08:48:10.515739 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:48:10.515669    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 08:48:10.583421 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:48:10.583399    4720 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-5d77dc844-jnx68"]
Oct 31 08:48:10.585459 ip-10-0-139-83 kubenswrapper[4720]: W1031 08:48:10.585431    4720 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd1ccb554_bc34_46a9_89b9_a8a30cd57680.slice/crio-b012eca0ab8e38f3524dcb832c656b3dcd8568674a2c66d7dec03174510e49e7 WatchSource:0}: Error finding container b012eca0ab8e38f3524dcb832c656b3dcd8568674a2c66d7dec03174510e49e7: Status 404 returned error can't find the container with id b012eca0ab8e38f3524dcb832c656b3dcd8568674a2c66d7dec03174510e49e7
Oct 31 08:48:10.678322 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:48:10.678291    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 08:48:11.161297 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:48:11.161236    4720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-5d77dc844-jnx68" event={"ID":"d1ccb554-bc34-46a9-89b9-a8a30cd57680","Type":"ContainerStarted","Data":"a8126dbb3ff7a0a2d89594024b11aee8d1240b312dc7e0434bf3e860cce9cad6"}
Oct 31 08:48:11.161709 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:48:11.161316    4720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-5d77dc844-jnx68" event={"ID":"d1ccb554-bc34-46a9-89b9-a8a30cd57680","Type":"ContainerStarted","Data":"b012eca0ab8e38f3524dcb832c656b3dcd8568674a2c66d7dec03174510e49e7"}
Oct 31 08:48:11.212638 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:48:11.212610    4720 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-5d77dc844-jnx68"
Oct 31 08:48:11.281159 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:48:11.281112    4720 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-5d77dc844-jnx68" podStartSLOduration=2.281101271 podStartE2EDuration="2.281101271s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 08:48:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 08:48:11.280544518 +0000 UTC m=+4745.411612578" watchObservedRunningTime="2026-10-31 08:48:11.281101271 +0000 UTC m=+4745.412169341"
Oct 31 08:48:13.460374 ip-10-0-139-83 kubenswrapper[4720]: E1031 08:48:13.460316    4720 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8m768" containerName="extractor"
Oct 31 08:48:16.309605 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:48:16.309567    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 08:48:16.474293 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:48:16.474263    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 08:48:18.863117 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:48:18.863086    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 08:48:20.213253 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:48:20.213218    4720 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-5d77dc844-jnx68"
Oct 31 08:48:20.225477 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:48:20.225455    4720 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-5d77dc844-jnx68"
Oct 31 08:48:21.211775 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:48:21.211745    4720 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-5d77dc844-jnx68"
Oct 31 08:48:27.169339 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:48:27.169271    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 08:48:27.779513 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:48:27.779480    4720 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-56f46b8cdd-rzrgb"]
Oct 31 08:48:27.782814 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:48:27.782799    4720 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-default-56f46b8cdd-rzrgb"
Oct 31 08:48:27.848622 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:48:27.848595    4720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d225dc99-d91e-441d-861d-1d50de64b2d8-credential-socket\") pod \"openshift-ai-inference-openshift-default-56f46b8cdd-rzrgb\" (UID: \"d225dc99-d91e-441d-861d-1d50de64b2d8\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-56f46b8cdd-rzrgb"
Oct 31 08:48:27.848742 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:48:27.848624    4720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d225dc99-d91e-441d-861d-1d50de64b2d8-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-default-56f46b8cdd-rzrgb\" (UID: \"d225dc99-d91e-441d-861d-1d50de64b2d8\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-56f46b8cdd-rzrgb"
Oct 31 08:48:27.848742 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:48:27.848644    4720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d225dc99-d91e-441d-861d-1d50de64b2d8-workload-certs\") pod \"openshift-ai-inference-openshift-default-56f46b8cdd-rzrgb\" (UID: \"d225dc99-d91e-441d-861d-1d50de64b2d8\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-56f46b8cdd-rzrgb"
Oct 31 08:48:27.848742 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:48:27.848668    4720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v9f8r\" (UniqueName: \"kubernetes.io/projected/d225dc99-d91e-441d-861d-1d50de64b2d8-kube-api-access-v9f8r\") pod \"openshift-ai-inference-openshift-default-56f46b8cdd-rzrgb\" (UID: \"d225dc99-d91e-441d-861d-1d50de64b2d8\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-56f46b8cdd-rzrgb"
Oct 31 08:48:27.848889 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:48:27.848751    4720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d225dc99-d91e-441d-861d-1d50de64b2d8-istio-podinfo\") pod \"openshift-ai-inference-openshift-default-56f46b8cdd-rzrgb\" (UID: \"d225dc99-d91e-441d-861d-1d50de64b2d8\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-56f46b8cdd-rzrgb"
Oct 31 08:48:27.848889 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:48:27.848785    4720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d225dc99-d91e-441d-861d-1d50de64b2d8-istio-envoy\") pod \"openshift-ai-inference-openshift-default-56f46b8cdd-rzrgb\" (UID: \"d225dc99-d91e-441d-861d-1d50de64b2d8\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-56f46b8cdd-rzrgb"
Oct 31 08:48:27.848889 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:48:27.848815    4720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d225dc99-d91e-441d-861d-1d50de64b2d8-workload-socket\") pod \"openshift-ai-inference-openshift-default-56f46b8cdd-rzrgb\" (UID: \"d225dc99-d91e-441d-861d-1d50de64b2d8\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-56f46b8cdd-rzrgb"
Oct 31 08:48:27.848889 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:48:27.848868    4720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d225dc99-d91e-441d-861d-1d50de64b2d8-istio-data\") pod \"openshift-ai-inference-openshift-default-56f46b8cdd-rzrgb\" (UID: \"d225dc99-d91e-441d-861d-1d50de64b2d8\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-56f46b8cdd-rzrgb"
Oct 31 08:48:27.849084 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:48:27.848981    4720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d225dc99-d91e-441d-861d-1d50de64b2d8-istio-token\") pod \"openshift-ai-inference-openshift-default-56f46b8cdd-rzrgb\" (UID: \"d225dc99-d91e-441d-861d-1d50de64b2d8\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-56f46b8cdd-rzrgb"
Oct 31 08:48:27.931307 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:48:27.931281    4720 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Oct 31 08:48:27.945720 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:48:27.945698    4720 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"openshift-ai-inference-openshift-default-dockercfg-z29g8\""
Oct 31 08:48:27.950029 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:48:27.950004    4720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d225dc99-d91e-441d-861d-1d50de64b2d8-istio-podinfo\") pod \"openshift-ai-inference-openshift-default-56f46b8cdd-rzrgb\" (UID: \"d225dc99-d91e-441d-861d-1d50de64b2d8\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-56f46b8cdd-rzrgb"
Oct 31 08:48:27.950088 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:48:27.950058    4720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d225dc99-d91e-441d-861d-1d50de64b2d8-istio-envoy\") pod \"openshift-ai-inference-openshift-default-56f46b8cdd-rzrgb\" (UID: \"d225dc99-d91e-441d-861d-1d50de64b2d8\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-56f46b8cdd-rzrgb"
Oct 31 08:48:27.950133 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:48:27.950105    4720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d225dc99-d91e-441d-861d-1d50de64b2d8-workload-socket\") pod \"openshift-ai-inference-openshift-default-56f46b8cdd-rzrgb\" (UID: \"d225dc99-d91e-441d-861d-1d50de64b2d8\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-56f46b8cdd-rzrgb"
Oct 31 08:48:27.950245 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:48:27.950223    4720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d225dc99-d91e-441d-861d-1d50de64b2d8-istio-data\") pod \"openshift-ai-inference-openshift-default-56f46b8cdd-rzrgb\" (UID: \"d225dc99-d91e-441d-861d-1d50de64b2d8\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-56f46b8cdd-rzrgb"
Oct 31 08:48:27.950317 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:48:27.950293    4720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d225dc99-d91e-441d-861d-1d50de64b2d8-istio-token\") pod \"openshift-ai-inference-openshift-default-56f46b8cdd-rzrgb\" (UID: \"d225dc99-d91e-441d-861d-1d50de64b2d8\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-56f46b8cdd-rzrgb"
Oct 31 08:48:27.950411 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:48:27.950357    4720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d225dc99-d91e-441d-861d-1d50de64b2d8-credential-socket\") pod \"openshift-ai-inference-openshift-default-56f46b8cdd-rzrgb\" (UID: \"d225dc99-d91e-441d-861d-1d50de64b2d8\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-56f46b8cdd-rzrgb"
Oct 31 08:48:27.950411 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:48:27.950398    4720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d225dc99-d91e-441d-861d-1d50de64b2d8-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-default-56f46b8cdd-rzrgb\" (UID: \"d225dc99-d91e-441d-861d-1d50de64b2d8\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-56f46b8cdd-rzrgb"
Oct 31 08:48:27.950516 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:48:27.950433    4720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d225dc99-d91e-441d-861d-1d50de64b2d8-workload-certs\") pod \"openshift-ai-inference-openshift-default-56f46b8cdd-rzrgb\" (UID: \"d225dc99-d91e-441d-861d-1d50de64b2d8\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-56f46b8cdd-rzrgb"
Oct 31 08:48:27.950516 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:48:27.950487    4720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v9f8r\" (UniqueName: \"kubernetes.io/projected/d225dc99-d91e-441d-861d-1d50de64b2d8-kube-api-access-v9f8r\") pod \"openshift-ai-inference-openshift-default-56f46b8cdd-rzrgb\" (UID: \"d225dc99-d91e-441d-861d-1d50de64b2d8\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-56f46b8cdd-rzrgb"
Oct 31 08:48:27.950621 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:48:27.950524    4720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d225dc99-d91e-441d-861d-1d50de64b2d8-workload-socket\") pod \"openshift-ai-inference-openshift-default-56f46b8cdd-rzrgb\" (UID: \"d225dc99-d91e-441d-861d-1d50de64b2d8\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-56f46b8cdd-rzrgb"
Oct 31 08:48:27.950676 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:48:27.950650    4720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d225dc99-d91e-441d-861d-1d50de64b2d8-istio-data\") pod \"openshift-ai-inference-openshift-default-56f46b8cdd-rzrgb\" (UID: \"d225dc99-d91e-441d-861d-1d50de64b2d8\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-56f46b8cdd-rzrgb"
Oct 31 08:48:27.950729 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:48:27.950699    4720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d225dc99-d91e-441d-861d-1d50de64b2d8-credential-socket\") pod \"openshift-ai-inference-openshift-default-56f46b8cdd-rzrgb\" (UID: \"d225dc99-d91e-441d-861d-1d50de64b2d8\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-56f46b8cdd-rzrgb"
Oct 31 08:48:27.950874 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:48:27.950848    4720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d225dc99-d91e-441d-861d-1d50de64b2d8-workload-certs\") pod \"openshift-ai-inference-openshift-default-56f46b8cdd-rzrgb\" (UID: \"d225dc99-d91e-441d-861d-1d50de64b2d8\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-56f46b8cdd-rzrgb"
Oct 31 08:48:27.951468 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:48:27.951447    4720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d225dc99-d91e-441d-861d-1d50de64b2d8-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-default-56f46b8cdd-rzrgb\" (UID: \"d225dc99-d91e-441d-861d-1d50de64b2d8\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-56f46b8cdd-rzrgb"
Oct 31 08:48:27.952568 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:48:27.952542    4720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d225dc99-d91e-441d-861d-1d50de64b2d8-istio-envoy\") pod \"openshift-ai-inference-openshift-default-56f46b8cdd-rzrgb\" (UID: \"d225dc99-d91e-441d-861d-1d50de64b2d8\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-56f46b8cdd-rzrgb"
Oct 31 08:48:27.952821 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:48:27.952799    4720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d225dc99-d91e-441d-861d-1d50de64b2d8-istio-podinfo\") pod \"openshift-ai-inference-openshift-default-56f46b8cdd-rzrgb\" (UID: \"d225dc99-d91e-441d-861d-1d50de64b2d8\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-56f46b8cdd-rzrgb"
Oct 31 08:48:27.984200 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:48:27.984176    4720 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-56f46b8cdd-rzrgb"]
Oct 31 08:48:28.007649 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:48:28.007627    4720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d225dc99-d91e-441d-861d-1d50de64b2d8-istio-token\") pod \"openshift-ai-inference-openshift-default-56f46b8cdd-rzrgb\" (UID: \"d225dc99-d91e-441d-861d-1d50de64b2d8\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-56f46b8cdd-rzrgb"
Oct 31 08:48:28.007882 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:48:28.007860    4720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v9f8r\" (UniqueName: \"kubernetes.io/projected/d225dc99-d91e-441d-861d-1d50de64b2d8-kube-api-access-v9f8r\") pod \"openshift-ai-inference-openshift-default-56f46b8cdd-rzrgb\" (UID: \"d225dc99-d91e-441d-861d-1d50de64b2d8\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-56f46b8cdd-rzrgb"
Oct 31 08:48:28.094853 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:48:28.094830    4720 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-default-56f46b8cdd-rzrgb"
Oct 31 08:48:28.521910 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:48:28.521887    4720 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-56f46b8cdd-rzrgb"]
Oct 31 08:48:28.523941 ip-10-0-139-83 kubenswrapper[4720]: W1031 08:48:28.523911    4720 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd225dc99_d91e_441d_861d_1d50de64b2d8.slice/crio-658268c712a335e32d5412d4cb91ad3cfc97f65f67d144fbb1f61d1815762ab9 WatchSource:0}: Error finding container 658268c712a335e32d5412d4cb91ad3cfc97f65f67d144fbb1f61d1815762ab9: Status 404 returned error can't find the container with id 658268c712a335e32d5412d4cb91ad3cfc97f65f67d144fbb1f61d1815762ab9
Oct 31 08:48:28.526039 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:48:28.526010    4720 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 31 08:48:28.526101 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:48:28.526076    4720 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 31 08:48:28.526146 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:48:28.526104    4720 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 31 08:48:29.238184 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:48:29.238150    4720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-56f46b8cdd-rzrgb" event={"ID":"d225dc99-d91e-441d-861d-1d50de64b2d8","Type":"ContainerStarted","Data":"9d09efb3c9c2833d2c8118ea6bcb02437f20251a7b6650b115cc23d0612dac4a"}
Oct 31 08:48:29.238426 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:48:29.238202    4720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-56f46b8cdd-rzrgb" event={"ID":"d225dc99-d91e-441d-861d-1d50de64b2d8","Type":"ContainerStarted","Data":"658268c712a335e32d5412d4cb91ad3cfc97f65f67d144fbb1f61d1815762ab9"}
Oct 31 08:48:30.096083 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:48:30.096052    4720 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/openshift-ai-inference-openshift-default-56f46b8cdd-rzrgb"
Oct 31 08:48:30.097721 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:48:30.097688    4720 patch_prober.go:29] interesting pod/openshift-ai-inference-openshift-default-56f46b8cdd-rzrgb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" start-of-body=
Oct 31 08:48:30.097855 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:48:30.097736    4720 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-56f46b8cdd-rzrgb" podUID="d225dc99-d91e-441d-861d-1d50de64b2d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused"
Oct 31 08:48:31.096176 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:48:31.096140    4720 patch_prober.go:29] interesting pod/openshift-ai-inference-openshift-default-56f46b8cdd-rzrgb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" start-of-body=
Oct 31 08:48:31.096592 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:48:31.096194    4720 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-56f46b8cdd-rzrgb" podUID="d225dc99-d91e-441d-861d-1d50de64b2d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused"
Oct 31 08:48:32.096286 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:48:32.096241    4720 patch_prober.go:29] interesting pod/openshift-ai-inference-openshift-default-56f46b8cdd-rzrgb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" start-of-body=
Oct 31 08:48:32.096729 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:48:32.096305    4720 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-56f46b8cdd-rzrgb" podUID="d225dc99-d91e-441d-861d-1d50de64b2d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused"
Oct 31 08:48:33.095663 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:48:33.095625    4720 patch_prober.go:29] interesting pod/openshift-ai-inference-openshift-default-56f46b8cdd-rzrgb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" start-of-body=
Oct 31 08:48:33.095841 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:48:33.095682    4720 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-56f46b8cdd-rzrgb" podUID="d225dc99-d91e-441d-861d-1d50de64b2d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused"
Oct 31 08:48:34.095835 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:48:34.095801    4720 patch_prober.go:29] interesting pod/openshift-ai-inference-openshift-default-56f46b8cdd-rzrgb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" start-of-body=
Oct 31 08:48:34.096219 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:48:34.095854    4720 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-56f46b8cdd-rzrgb" podUID="d225dc99-d91e-441d-861d-1d50de64b2d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused"
Oct 31 08:48:35.100108 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:48:35.100076    4720 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/openshift-ai-inference-openshift-default-56f46b8cdd-rzrgb"
Oct 31 08:48:35.100555 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:48:35.100341    4720 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/openshift-ai-inference-openshift-default-56f46b8cdd-rzrgb"
Oct 31 08:48:35.101025 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:48:35.101010    4720 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/openshift-ai-inference-openshift-default-56f46b8cdd-rzrgb"
Oct 31 08:48:35.268725 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:48:35.268668    4720 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/openshift-ai-inference-openshift-default-56f46b8cdd-rzrgb" podStartSLOduration=8.268651802 podStartE2EDuration="8.268651802s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 08:48:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 08:48:29.45613786 +0000 UTC m=+4763.587205930" watchObservedRunningTime="2026-10-31 08:48:35.268651802 +0000 UTC m=+4769.399719888"
Oct 31 08:48:41.816257 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:48:41.816227    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 08:48:42.884571 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:48:42.884534    4720 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-56f46b8cdd-v774p"]
Oct 31 08:48:42.887987 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:48:42.887970    4720 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-default-56f46b8cdd-v774p"
Oct 31 08:48:42.986926 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:48:42.986901    4720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/285720d1-747c-438e-9e87-2057a77e5eb3-credential-socket\") pod \"openshift-ai-inference-openshift-default-56f46b8cdd-v774p\" (UID: \"285720d1-747c-438e-9e87-2057a77e5eb3\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-56f46b8cdd-v774p"
Oct 31 08:48:42.987085 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:48:42.986932    4720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/285720d1-747c-438e-9e87-2057a77e5eb3-workload-socket\") pod \"openshift-ai-inference-openshift-default-56f46b8cdd-v774p\" (UID: \"285720d1-747c-438e-9e87-2057a77e5eb3\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-56f46b8cdd-v774p"
Oct 31 08:48:42.987085 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:48:42.986957    4720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/285720d1-747c-438e-9e87-2057a77e5eb3-istio-envoy\") pod \"openshift-ai-inference-openshift-default-56f46b8cdd-v774p\" (UID: \"285720d1-747c-438e-9e87-2057a77e5eb3\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-56f46b8cdd-v774p"
Oct 31 08:48:42.987085 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:48:42.987040    4720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k9cgq\" (UniqueName: \"kubernetes.io/projected/285720d1-747c-438e-9e87-2057a77e5eb3-kube-api-access-k9cgq\") pod \"openshift-ai-inference-openshift-default-56f46b8cdd-v774p\" (UID: \"285720d1-747c-438e-9e87-2057a77e5eb3\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-56f46b8cdd-v774p"
Oct 31 08:48:42.987085 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:48:42.987076    4720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/285720d1-747c-438e-9e87-2057a77e5eb3-istio-podinfo\") pod \"openshift-ai-inference-openshift-default-56f46b8cdd-v774p\" (UID: \"285720d1-747c-438e-9e87-2057a77e5eb3\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-56f46b8cdd-v774p"
Oct 31 08:48:42.987317 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:48:42.987165    4720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/285720d1-747c-438e-9e87-2057a77e5eb3-istio-data\") pod \"openshift-ai-inference-openshift-default-56f46b8cdd-v774p\" (UID: \"285720d1-747c-438e-9e87-2057a77e5eb3\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-56f46b8cdd-v774p"
Oct 31 08:48:42.987317 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:48:42.987184    4720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/285720d1-747c-438e-9e87-2057a77e5eb3-workload-certs\") pod \"openshift-ai-inference-openshift-default-56f46b8cdd-v774p\" (UID: \"285720d1-747c-438e-9e87-2057a77e5eb3\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-56f46b8cdd-v774p"
Oct 31 08:48:42.987317 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:48:42.987198    4720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/285720d1-747c-438e-9e87-2057a77e5eb3-istio-token\") pod \"openshift-ai-inference-openshift-default-56f46b8cdd-v774p\" (UID: \"285720d1-747c-438e-9e87-2057a77e5eb3\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-56f46b8cdd-v774p"
Oct 31 08:48:42.987317 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:48:42.987216    4720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/285720d1-747c-438e-9e87-2057a77e5eb3-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-default-56f46b8cdd-v774p\" (UID: \"285720d1-747c-438e-9e87-2057a77e5eb3\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-56f46b8cdd-v774p"
Oct 31 08:48:42.992730 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:48:42.992707    4720 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-56f46b8cdd-v774p"]
Oct 31 08:48:43.087525 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:48:43.087496    4720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/285720d1-747c-438e-9e87-2057a77e5eb3-istio-data\") pod \"openshift-ai-inference-openshift-default-56f46b8cdd-v774p\" (UID: \"285720d1-747c-438e-9e87-2057a77e5eb3\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-56f46b8cdd-v774p"
Oct 31 08:48:43.087656 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:48:43.087527    4720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/285720d1-747c-438e-9e87-2057a77e5eb3-workload-certs\") pod \"openshift-ai-inference-openshift-default-56f46b8cdd-v774p\" (UID: \"285720d1-747c-438e-9e87-2057a77e5eb3\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-56f46b8cdd-v774p"
Oct 31 08:48:43.087656 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:48:43.087547    4720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/285720d1-747c-438e-9e87-2057a77e5eb3-istio-token\") pod \"openshift-ai-inference-openshift-default-56f46b8cdd-v774p\" (UID: \"285720d1-747c-438e-9e87-2057a77e5eb3\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-56f46b8cdd-v774p"
Oct 31 08:48:43.087656 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:48:43.087579    4720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/285720d1-747c-438e-9e87-2057a77e5eb3-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-default-56f46b8cdd-v774p\" (UID: \"285720d1-747c-438e-9e87-2057a77e5eb3\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-56f46b8cdd-v774p"
Oct 31 08:48:43.087656 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:48:43.087624    4720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/285720d1-747c-438e-9e87-2057a77e5eb3-credential-socket\") pod \"openshift-ai-inference-openshift-default-56f46b8cdd-v774p\" (UID: \"285720d1-747c-438e-9e87-2057a77e5eb3\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-56f46b8cdd-v774p"
Oct 31 08:48:43.087656 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:48:43.087651    4720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/285720d1-747c-438e-9e87-2057a77e5eb3-workload-socket\") pod \"openshift-ai-inference-openshift-default-56f46b8cdd-v774p\" (UID: \"285720d1-747c-438e-9e87-2057a77e5eb3\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-56f46b8cdd-v774p"
Oct 31 08:48:43.087887 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:48:43.087695    4720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/285720d1-747c-438e-9e87-2057a77e5eb3-istio-envoy\") pod \"openshift-ai-inference-openshift-default-56f46b8cdd-v774p\" (UID: \"285720d1-747c-438e-9e87-2057a77e5eb3\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-56f46b8cdd-v774p"
Oct 31 08:48:43.087887 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:48:43.087744    4720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k9cgq\" (UniqueName: \"kubernetes.io/projected/285720d1-747c-438e-9e87-2057a77e5eb3-kube-api-access-k9cgq\") pod \"openshift-ai-inference-openshift-default-56f46b8cdd-v774p\" (UID: \"285720d1-747c-438e-9e87-2057a77e5eb3\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-56f46b8cdd-v774p"
Oct 31 08:48:43.087887 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:48:43.087774    4720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/285720d1-747c-438e-9e87-2057a77e5eb3-istio-podinfo\") pod \"openshift-ai-inference-openshift-default-56f46b8cdd-v774p\" (UID: \"285720d1-747c-438e-9e87-2057a77e5eb3\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-56f46b8cdd-v774p"
Oct 31 08:48:43.088045 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:48:43.087990    4720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/285720d1-747c-438e-9e87-2057a77e5eb3-istio-data\") pod \"openshift-ai-inference-openshift-default-56f46b8cdd-v774p\" (UID: \"285720d1-747c-438e-9e87-2057a77e5eb3\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-56f46b8cdd-v774p"
Oct 31 08:48:43.088045 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:48:43.088031    4720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/285720d1-747c-438e-9e87-2057a77e5eb3-workload-certs\") pod \"openshift-ai-inference-openshift-default-56f46b8cdd-v774p\" (UID: \"285720d1-747c-438e-9e87-2057a77e5eb3\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-56f46b8cdd-v774p"
Oct 31 08:48:43.088153 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:48:43.088108    4720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/285720d1-747c-438e-9e87-2057a77e5eb3-workload-socket\") pod \"openshift-ai-inference-openshift-default-56f46b8cdd-v774p\" (UID: \"285720d1-747c-438e-9e87-2057a77e5eb3\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-56f46b8cdd-v774p"
Oct 31 08:48:43.088208 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:48:43.088151    4720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/285720d1-747c-438e-9e87-2057a77e5eb3-credential-socket\") pod \"openshift-ai-inference-openshift-default-56f46b8cdd-v774p\" (UID: \"285720d1-747c-438e-9e87-2057a77e5eb3\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-56f46b8cdd-v774p"
Oct 31 08:48:43.088706 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:48:43.088684    4720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/285720d1-747c-438e-9e87-2057a77e5eb3-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-default-56f46b8cdd-v774p\" (UID: \"285720d1-747c-438e-9e87-2057a77e5eb3\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-56f46b8cdd-v774p"
Oct 31 08:48:43.092051 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:48:43.090420    4720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/285720d1-747c-438e-9e87-2057a77e5eb3-istio-envoy\") pod \"openshift-ai-inference-openshift-default-56f46b8cdd-v774p\" (UID: \"285720d1-747c-438e-9e87-2057a77e5eb3\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-56f46b8cdd-v774p"
Oct 31 08:48:43.092051 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:48:43.090731    4720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/285720d1-747c-438e-9e87-2057a77e5eb3-istio-podinfo\") pod \"openshift-ai-inference-openshift-default-56f46b8cdd-v774p\" (UID: \"285720d1-747c-438e-9e87-2057a77e5eb3\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-56f46b8cdd-v774p"
Oct 31 08:48:43.146473 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:48:43.146407    4720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/285720d1-747c-438e-9e87-2057a77e5eb3-istio-token\") pod \"openshift-ai-inference-openshift-default-56f46b8cdd-v774p\" (UID: \"285720d1-747c-438e-9e87-2057a77e5eb3\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-56f46b8cdd-v774p"
Oct 31 08:48:43.146589 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:48:43.146495    4720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k9cgq\" (UniqueName: \"kubernetes.io/projected/285720d1-747c-438e-9e87-2057a77e5eb3-kube-api-access-k9cgq\") pod \"openshift-ai-inference-openshift-default-56f46b8cdd-v774p\" (UID: \"285720d1-747c-438e-9e87-2057a77e5eb3\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-56f46b8cdd-v774p"
Oct 31 08:48:43.200028 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:48:43.199999    4720 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-default-56f46b8cdd-v774p"
Oct 31 08:48:43.460740 ip-10-0-139-83 kubenswrapper[4720]: E1031 08:48:43.460655    4720 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8m768" containerName="extractor"
Oct 31 08:48:43.625192 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:48:43.625168    4720 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-56f46b8cdd-v774p"]
Oct 31 08:48:43.627139 ip-10-0-139-83 kubenswrapper[4720]: W1031 08:48:43.627112    4720 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod285720d1_747c_438e_9e87_2057a77e5eb3.slice/crio-4343922ee39344d59e73935f3405dedcb04ae60da52dd66e85d44b79b5341870 WatchSource:0}: Error finding container 4343922ee39344d59e73935f3405dedcb04ae60da52dd66e85d44b79b5341870: Status 404 returned error can't find the container with id 4343922ee39344d59e73935f3405dedcb04ae60da52dd66e85d44b79b5341870
Oct 31 08:48:43.629090 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:48:43.629045    4720 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 31 08:48:43.629187 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:48:43.629132    4720 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 31 08:48:43.629258 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:48:43.629188    4720 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 31 08:48:44.307825 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:48:44.307784    4720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-56f46b8cdd-v774p" event={"ID":"285720d1-747c-438e-9e87-2057a77e5eb3","Type":"ContainerStarted","Data":"bc0f5c6ed8db23093039268f3fb0d4e87181a6aaff92b09f0c45845f050f560f"}
Oct 31 08:48:44.308239 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:48:44.307839    4720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-56f46b8cdd-v774p" event={"ID":"285720d1-747c-438e-9e87-2057a77e5eb3","Type":"ContainerStarted","Data":"4343922ee39344d59e73935f3405dedcb04ae60da52dd66e85d44b79b5341870"}
Oct 31 08:48:44.496900 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:48:44.496848    4720 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/openshift-ai-inference-openshift-default-56f46b8cdd-v774p" podStartSLOduration=2.496834229 podStartE2EDuration="2.496834229s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 08:48:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 08:48:44.49385246 +0000 UTC m=+4778.624920542" watchObservedRunningTime="2026-10-31 08:48:44.496834229 +0000 UTC m=+4778.627902303"
Oct 31 08:48:45.200319 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:48:45.200286    4720 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/openshift-ai-inference-openshift-default-56f46b8cdd-v774p"
Oct 31 08:48:45.201730 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:48:45.201700    4720 patch_prober.go:29] interesting pod/openshift-ai-inference-openshift-default-56f46b8cdd-v774p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" start-of-body=
Oct 31 08:48:45.201879 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:48:45.201748    4720 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-56f46b8cdd-v774p" podUID="285720d1-747c-438e-9e87-2057a77e5eb3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused"
Oct 31 08:48:46.201063 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:48:46.201022    4720 patch_prober.go:29] interesting pod/openshift-ai-inference-openshift-default-56f46b8cdd-v774p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" start-of-body=
Oct 31 08:48:46.201581 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:48:46.201126    4720 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-56f46b8cdd-v774p" podUID="285720d1-747c-438e-9e87-2057a77e5eb3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused"
Oct 31 08:48:47.200899 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:48:47.200865    4720 patch_prober.go:29] interesting pod/openshift-ai-inference-openshift-default-56f46b8cdd-v774p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" start-of-body=
Oct 31 08:48:47.201088 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:48:47.200915    4720 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-56f46b8cdd-v774p" podUID="285720d1-747c-438e-9e87-2057a77e5eb3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused"
Oct 31 08:48:48.200353 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:48:48.200290    4720 patch_prober.go:29] interesting pod/openshift-ai-inference-openshift-default-56f46b8cdd-v774p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" start-of-body=
Oct 31 08:48:48.200797 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:48:48.200363    4720 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-56f46b8cdd-v774p" podUID="285720d1-747c-438e-9e87-2057a77e5eb3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused"
Oct 31 08:48:49.201184 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:48:49.201137    4720 patch_prober.go:29] interesting pod/openshift-ai-inference-openshift-default-56f46b8cdd-v774p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" start-of-body=
Oct 31 08:48:49.201618 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:48:49.201205    4720 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-56f46b8cdd-v774p" podUID="285720d1-747c-438e-9e87-2057a77e5eb3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused"
Oct 31 08:48:50.204171 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:48:50.204137    4720 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/openshift-ai-inference-openshift-default-56f46b8cdd-v774p"
Oct 31 08:48:50.204583 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:48:50.204544    4720 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/openshift-ai-inference-openshift-default-56f46b8cdd-v774p"
Oct 31 08:48:50.205307 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:48:50.205269    4720 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/openshift-ai-inference-openshift-default-56f46b8cdd-v774p"
Oct 31 08:48:55.904863 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:48:55.904827    4720 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-5d77dc844-jnx68"]
Oct 31 08:48:55.914945 ip-10-0-139-83 kubenswrapper[4720]: E1031 08:48:55.914902    4720 log.go:32] "ExecSync cmd from runtime service failed" err="rpc error: code = Unknown desc = command error: , stdout: , stderr: , exit code -1" containerID="a8126dbb3ff7a0a2d89594024b11aee8d1240b312dc7e0434bf3e860cce9cad6" cmd=["/bin/sleep","15"]
Oct 31 08:48:55.915114 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:48:55.914967    4720 handlers.go:85] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-5d77dc844-jnx68" message="" err="rpc error: code = Unknown desc = command error: , stdout: , stderr: , exit code -1"
Oct 31 08:48:55.915114 ip-10-0-139-83 kubenswrapper[4720]: E1031 08:48:55.914984    4720 kuberuntime_container.go:780] "PreStop hook failed" err="rpc error: code = Unknown desc = command error: , stdout: , stderr: , exit code -1" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-5d77dc844-jnx68" podUID="d1ccb554-bc34-46a9-89b9-a8a30cd57680" containerName="main" containerID="cri-o://a8126dbb3ff7a0a2d89594024b11aee8d1240b312dc7e0434bf3e860cce9cad6"
Oct 31 08:48:55.915114 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:48:55.915041    4720 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-5d77dc844-jnx68" podUID="d1ccb554-bc34-46a9-89b9-a8a30cd57680" containerName="main" containerID="cri-o://a8126dbb3ff7a0a2d89594024b11aee8d1240b312dc7e0434bf3e860cce9cad6" gracePeriod=60
Oct 31 08:48:55.927036 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:48:55.927007    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 08:48:56.089113 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:48:56.089081    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 08:48:56.201671 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:48:56.201598    4720 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-5d77dc844-jnx68" podUID="d1ccb554-bc34-46a9-89b9-a8a30cd57680" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/health\": dial tcp 10.132.0.84:8000: connect: connection refused"
Oct 31 08:48:56.358703 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:48:56.358672    4720 generic.go:356] "Generic (PLEG): container finished" podID="d1ccb554-bc34-46a9-89b9-a8a30cd57680" containerID="a8126dbb3ff7a0a2d89594024b11aee8d1240b312dc7e0434bf3e860cce9cad6" exitCode=0
Oct 31 08:48:56.358849 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:48:56.358748    4720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-5d77dc844-jnx68" event={"ID":"d1ccb554-bc34-46a9-89b9-a8a30cd57680","Type":"ContainerDied","Data":"a8126dbb3ff7a0a2d89594024b11aee8d1240b312dc7e0434bf3e860cce9cad6"}
Oct 31 08:48:56.657268 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:48:56.657237    4720 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-5d77dc844-jnx68"
Oct 31 08:48:56.794362 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:48:56.794286    4720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d1ccb554-bc34-46a9-89b9-a8a30cd57680-tls-certs\" (UniqueName: \"kubernetes.io/secret/d1ccb554-bc34-46a9-89b9-a8a30cd57680-tls-certs\") pod \"d1ccb554-bc34-46a9-89b9-a8a30cd57680\" (UID: \"d1ccb554-bc34-46a9-89b9-a8a30cd57680\") "
Oct 31 08:48:56.794496 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:48:56.794371    4720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d1ccb554-bc34-46a9-89b9-a8a30cd57680-home\" (UniqueName: \"kubernetes.io/empty-dir/d1ccb554-bc34-46a9-89b9-a8a30cd57680-home\") pod \"d1ccb554-bc34-46a9-89b9-a8a30cd57680\" (UID: \"d1ccb554-bc34-46a9-89b9-a8a30cd57680\") "
Oct 31 08:48:56.794496 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:48:56.794410    4720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d1ccb554-bc34-46a9-89b9-a8a30cd57680-dshm\" (UniqueName: \"kubernetes.io/empty-dir/d1ccb554-bc34-46a9-89b9-a8a30cd57680-dshm\") pod \"d1ccb554-bc34-46a9-89b9-a8a30cd57680\" (UID: \"d1ccb554-bc34-46a9-89b9-a8a30cd57680\") "
Oct 31 08:48:56.794496 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:48:56.794440    4720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d1ccb554-bc34-46a9-89b9-a8a30cd57680-kube-api-access-x57dr\" (UniqueName: \"kubernetes.io/projected/d1ccb554-bc34-46a9-89b9-a8a30cd57680-kube-api-access-x57dr\") pod \"d1ccb554-bc34-46a9-89b9-a8a30cd57680\" (UID: \"d1ccb554-bc34-46a9-89b9-a8a30cd57680\") "
Oct 31 08:48:56.794496 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:48:56.794481    4720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d1ccb554-bc34-46a9-89b9-a8a30cd57680-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d1ccb554-bc34-46a9-89b9-a8a30cd57680-tmp-dir\") pod \"d1ccb554-bc34-46a9-89b9-a8a30cd57680\" (UID: \"d1ccb554-bc34-46a9-89b9-a8a30cd57680\") "
Oct 31 08:48:56.794634 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:48:56.794501    4720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d1ccb554-bc34-46a9-89b9-a8a30cd57680-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d1ccb554-bc34-46a9-89b9-a8a30cd57680-model-cache\") pod \"d1ccb554-bc34-46a9-89b9-a8a30cd57680\" (UID: \"d1ccb554-bc34-46a9-89b9-a8a30cd57680\") "
Oct 31 08:48:56.794679 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:48:56.794620    4720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d1ccb554-bc34-46a9-89b9-a8a30cd57680-home" pod "d1ccb554-bc34-46a9-89b9-a8a30cd57680" (UID: "d1ccb554-bc34-46a9-89b9-a8a30cd57680"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 08:48:56.794733 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:48:56.794720    4720 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d1ccb554-bc34-46a9-89b9-a8a30cd57680-home\") on node \"ip-10-0-139-83.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 08:48:56.794872 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:48:56.794821    4720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d1ccb554-bc34-46a9-89b9-a8a30cd57680-model-cache" pod "d1ccb554-bc34-46a9-89b9-a8a30cd57680" (UID: "d1ccb554-bc34-46a9-89b9-a8a30cd57680"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 08:48:56.794979 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:48:56.794901    4720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d1ccb554-bc34-46a9-89b9-a8a30cd57680-tmp-dir" pod "d1ccb554-bc34-46a9-89b9-a8a30cd57680" (UID: "d1ccb554-bc34-46a9-89b9-a8a30cd57680"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 08:48:56.796708 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:48:56.796685    4720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d1ccb554-bc34-46a9-89b9-a8a30cd57680-tls-certs" pod "d1ccb554-bc34-46a9-89b9-a8a30cd57680" (UID: "d1ccb554-bc34-46a9-89b9-a8a30cd57680"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 08:48:56.796799 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:48:56.796715    4720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d1ccb554-bc34-46a9-89b9-a8a30cd57680-kube-api-access-x57dr" pod "d1ccb554-bc34-46a9-89b9-a8a30cd57680" (UID: "d1ccb554-bc34-46a9-89b9-a8a30cd57680"). InnerVolumeSpecName "kube-api-access-x57dr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 08:48:56.796993 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:48:56.796973    4720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d1ccb554-bc34-46a9-89b9-a8a30cd57680-dshm" pod "d1ccb554-bc34-46a9-89b9-a8a30cd57680" (UID: "d1ccb554-bc34-46a9-89b9-a8a30cd57680"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 08:48:56.895908 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:48:56.895886    4720 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d1ccb554-bc34-46a9-89b9-a8a30cd57680-dshm\") on node \"ip-10-0-139-83.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 08:48:56.895908 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:48:56.895907    4720 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x57dr\" (UniqueName: \"kubernetes.io/projected/d1ccb554-bc34-46a9-89b9-a8a30cd57680-kube-api-access-x57dr\") on node \"ip-10-0-139-83.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 08:48:56.896034 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:48:56.895918    4720 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d1ccb554-bc34-46a9-89b9-a8a30cd57680-tmp-dir\") on node \"ip-10-0-139-83.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 08:48:56.896034 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:48:56.895927    4720 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d1ccb554-bc34-46a9-89b9-a8a30cd57680-model-cache\") on node \"ip-10-0-139-83.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 08:48:56.896034 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:48:56.895935    4720 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d1ccb554-bc34-46a9-89b9-a8a30cd57680-tls-certs\") on node \"ip-10-0-139-83.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 08:48:57.363881 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:48:57.363854    4720 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-5d77dc844-jnx68"
Oct 31 08:48:57.363881 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:48:57.363869    4720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-5d77dc844-jnx68" event={"ID":"d1ccb554-bc34-46a9-89b9-a8a30cd57680","Type":"ContainerDied","Data":"b012eca0ab8e38f3524dcb832c656b3dcd8568674a2c66d7dec03174510e49e7"}
Oct 31 08:48:57.364322 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:48:57.363921    4720 scope.go:122] "RemoveContainer" containerID="a8126dbb3ff7a0a2d89594024b11aee8d1240b312dc7e0434bf3e860cce9cad6"
Oct 31 08:48:57.525738 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:48:57.525708    4720 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-5d77dc844-jnx68"]
Oct 31 08:48:57.574611 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:48:57.574586    4720 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-5d77dc844-jnx68"]
Oct 31 08:48:58.025375 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:48:58.025350    4720 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d1ccb554-bc34-46a9-89b9-a8a30cd57680" path="/var/lib/kubelet/pods/d1ccb554-bc34-46a9-89b9-a8a30cd57680/volumes"
Oct 31 08:49:06.483888 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:49:06.483788    4720 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-4jszp_b1db0e3b-e85d-4769-8012-cf4b6f740d9d/console-operator/2.log"
Oct 31 08:49:06.494766 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:49:06.494739    4720 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-4jszp_b1db0e3b-e85d-4769-8012-cf4b6f740d9d/console-operator/2.log"
Oct 31 08:49:06.587086 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:49:06.587058    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 08:49:13.460529 ip-10-0-139-83 kubenswrapper[4720]: E1031 08:49:13.460484    4720 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8m768" containerName="extractor"
Oct 31 08:49:18.458626 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:49:18.458551    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 08:49:18.603735 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:49:18.603701    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 08:49:24.396153 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:49:24.396120    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 08:49:27.509768 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:49:27.509736    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 08:49:42.077285 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:49:42.077233    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 08:49:42.278554 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:49:42.278513    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 08:49:43.461106 ip-10-0-139-83 kubenswrapper[4720]: E1031 08:49:43.461065    4720 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8m768" containerName="extractor"
Oct 31 08:49:47.204410 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:49:47.204374    4720 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-r6ms4/must-gather-j5269"]
Oct 31 08:49:47.204788 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:49:47.204692    4720 cpu_manager.go:414] "containerMap: removing container" podUID="d1ccb554-bc34-46a9-89b9-a8a30cd57680" containerName="main"
Oct 31 08:49:47.204788 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:49:47.204702    4720 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d1ccb554-bc34-46a9-89b9-a8a30cd57680" containerName="main"
Oct 31 08:49:47.204788 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:49:47.204768    4720 memory_manager.go:365] "RemoveStaleState removing state" podUID="d1ccb554-bc34-46a9-89b9-a8a30cd57680" containerName="main"
Oct 31 08:49:47.207949 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:49:47.207931    4720 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-r6ms4/must-gather-j5269"
Oct 31 08:49:47.232294 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:49:47.232269    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 08:49:47.259361 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:49:47.259317    4720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/e5bec65f-573e-443c-ad34-a4da0d4a7987-must-gather-output\") pod \"must-gather-j5269\" (UID: \"e5bec65f-573e-443c-ad34-a4da0d4a7987\") " pod="openshift-must-gather-r6ms4/must-gather-j5269"
Oct 31 08:49:47.259467 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:49:47.259392    4720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vmblp\" (UniqueName: \"kubernetes.io/projected/e5bec65f-573e-443c-ad34-a4da0d4a7987-kube-api-access-vmblp\") pod \"must-gather-j5269\" (UID: \"e5bec65f-573e-443c-ad34-a4da0d4a7987\") " pod="openshift-must-gather-r6ms4/must-gather-j5269"
Oct 31 08:49:47.264242 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:49:47.264217    4720 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-r6ms4/must-gather-j5269"]
Oct 31 08:49:47.310879 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:49:47.310852    4720 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-r6ms4\"/\"kube-root-ca.crt\""
Oct 31 08:49:47.310972 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:49:47.310923    4720 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-r6ms4\"/\"openshift-service-ca.crt\""
Oct 31 08:49:47.310972 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:49:47.310950    4720 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-r6ms4\"/\"default-dockercfg-6n2q2\""
Oct 31 08:49:47.360722 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:49:47.360694    4720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/e5bec65f-573e-443c-ad34-a4da0d4a7987-must-gather-output\") pod \"must-gather-j5269\" (UID: \"e5bec65f-573e-443c-ad34-a4da0d4a7987\") " pod="openshift-must-gather-r6ms4/must-gather-j5269"
Oct 31 08:49:47.360819 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:49:47.360755    4720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vmblp\" (UniqueName: \"kubernetes.io/projected/e5bec65f-573e-443c-ad34-a4da0d4a7987-kube-api-access-vmblp\") pod \"must-gather-j5269\" (UID: \"e5bec65f-573e-443c-ad34-a4da0d4a7987\") " pod="openshift-must-gather-r6ms4/must-gather-j5269"
Oct 31 08:49:47.361000 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:49:47.360984    4720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/e5bec65f-573e-443c-ad34-a4da0d4a7987-must-gather-output\") pod \"must-gather-j5269\" (UID: \"e5bec65f-573e-443c-ad34-a4da0d4a7987\") " pod="openshift-must-gather-r6ms4/must-gather-j5269"
Oct 31 08:49:47.410840 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:49:47.410818    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 08:49:47.418023 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:49:47.417994    4720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vmblp\" (UniqueName: \"kubernetes.io/projected/e5bec65f-573e-443c-ad34-a4da0d4a7987-kube-api-access-vmblp\") pod \"must-gather-j5269\" (UID: \"e5bec65f-573e-443c-ad34-a4da0d4a7987\") " pod="openshift-must-gather-r6ms4/must-gather-j5269"
Oct 31 08:49:47.521589 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:49:47.521539    4720 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-r6ms4/must-gather-j5269"
Oct 31 08:49:47.892135 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:49:47.892104    4720 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-r6ms4/must-gather-j5269"]
Oct 31 08:49:47.894896 ip-10-0-139-83 kubenswrapper[4720]: W1031 08:49:47.894857    4720 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode5bec65f_573e_443c_ad34_a4da0d4a7987.slice/crio-81022860bfd0ffb1f3db4731cc55c4581e0f19b72929c12d2d49053dcc06030c WatchSource:0}: Error finding container 81022860bfd0ffb1f3db4731cc55c4581e0f19b72929c12d2d49053dcc06030c: Status 404 returned error can't find the container with id 81022860bfd0ffb1f3db4731cc55c4581e0f19b72929c12d2d49053dcc06030c
Oct 31 08:49:47.896845 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:49:47.896829    4720 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 31 08:49:48.552035 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:49:48.552005    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 08:49:48.586724 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:49:48.586694    4720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-r6ms4/must-gather-j5269" event={"ID":"e5bec65f-573e-443c-ad34-a4da0d4a7987","Type":"ContainerStarted","Data":"81022860bfd0ffb1f3db4731cc55c4581e0f19b72929c12d2d49053dcc06030c"}
Oct 31 08:49:54.013168 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:49:54.013132    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 08:49:54.186045 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:49:54.186005    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 08:49:55.622848 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:49:55.622817    4720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-r6ms4/must-gather-j5269" event={"ID":"e5bec65f-573e-443c-ad34-a4da0d4a7987","Type":"ContainerStarted","Data":"a476ce7bc1e1474dcddfaffe8b0366b153c7f00db8ca9e78ac5e64626a22fe8f"}
Oct 31 08:49:55.623280 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:49:55.622857    4720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-r6ms4/must-gather-j5269" event={"ID":"e5bec65f-573e-443c-ad34-a4da0d4a7987","Type":"ContainerStarted","Data":"8a4f3817cca69997993c3f6c566cc485569f49ca1321004136bda3ab57e9bf2c"}
Oct 31 08:49:55.736995 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:49:55.736944    4720 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-r6ms4/must-gather-j5269" podStartSLOduration=2.105141331 podStartE2EDuration="8.736931615s" totalImagesPullingTime="6.631790284s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 08:49:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 08:49:55.735222078 +0000 UTC m=+4849.866290149" watchObservedRunningTime="2026-10-31 08:49:55.736931615 +0000 UTC m=+4849.867999684"
Oct 31 08:49:59.297351 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:49:59.297271    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 08:50:06.899362 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:50:06.899311    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 08:50:13.462815 ip-10-0-139-83 kubenswrapper[4720]: E1031 08:50:13.462775    4720 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8m768" containerName="extractor"
Oct 31 08:50:27.943297 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:50:27.943265    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 08:50:42.483863 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:50:42.483830    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 08:50:43.460506 ip-10-0-139-83 kubenswrapper[4720]: E1031 08:50:43.460466    4720 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8m768" containerName="extractor"
Oct 31 08:50:59.715829 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:50:59.715755    4720 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-56f46b8cdd-rzrgb_d225dc99-d91e-441d-861d-1d50de64b2d8/istio-proxy/0.log"
Oct 31 08:51:00.035512 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:51:00.035448    4720 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-56f46b8cdd-v774p_285720d1-747c-438e-9e87-2057a77e5eb3/istio-proxy/0.log"
Oct 31 08:51:00.266141 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:51:00.266110    4720 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-6c88c45d57-c2bgn_b44ed0e0-a72e-4746-a7de-ed8610da845d/router/0.log"
Oct 31 08:51:04.555197 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:51:04.555166    4720 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-56f46b8cdd-rzrgb_d225dc99-d91e-441d-861d-1d50de64b2d8/istio-proxy/0.log"
Oct 31 08:51:04.774026 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:51:04.773995    4720 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-56f46b8cdd-v774p_285720d1-747c-438e-9e87-2057a77e5eb3/istio-proxy/0.log"
Oct 31 08:51:04.987154 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:51:04.987128    4720 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-6c88c45d57-c2bgn_b44ed0e0-a72e-4746-a7de-ed8610da845d/router/0.log"
Oct 31 08:51:07.210646 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:51:07.210611    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 08:51:08.541975 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:51:08.541949    4720 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-68df894cb4-xjstz_33530b7d-e4c2-490b-b7ea-7a7915cbe769/manager/0.log"
Oct 31 08:51:08.752214 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:51:08.752182    4720 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-wmfgm_601ccf10-fbcd-41c1-ae75-6d7c01bc1b8a/kuadrant-console-plugin/0.log"
Oct 31 08:51:09.018346 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:51:09.018303    4720 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-658c459c99-nnrhp_c4d53665-7863-48a0-a0a2-7e911345bd4f/manager/0.log"
Oct 31 08:51:11.980077 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:51:11.980039    4720 generic.go:356] "Generic (PLEG): container finished" podID="e5bec65f-573e-443c-ad34-a4da0d4a7987" containerID="8a4f3817cca69997993c3f6c566cc485569f49ca1321004136bda3ab57e9bf2c" exitCode=0
Oct 31 08:51:11.980506 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:51:11.980110    4720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-r6ms4/must-gather-j5269" event={"ID":"e5bec65f-573e-443c-ad34-a4da0d4a7987","Type":"ContainerDied","Data":"8a4f3817cca69997993c3f6c566cc485569f49ca1321004136bda3ab57e9bf2c"}
Oct 31 08:51:11.980506 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:51:11.980413    4720 scope.go:122] "RemoveContainer" containerID="8a4f3817cca69997993c3f6c566cc485569f49ca1321004136bda3ab57e9bf2c"
Oct 31 08:51:12.561750 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:51:12.561721    4720 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-r6ms4_must-gather-j5269_e5bec65f-573e-443c-ad34-a4da0d4a7987/gather/0.log"
Oct 31 08:51:13.460176 ip-10-0-139-83 kubenswrapper[4720]: E1031 08:51:13.460136    4720 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8m768" containerName="extractor"
Oct 31 08:51:14.194786 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:51:14.194762    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 08:51:14.469257 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:51:14.469156    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 08:51:14.673798 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:51:14.673761    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 08:51:19.123655 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:51:19.123627    4720 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-must-gather-r6ms4/must-gather-j5269"]
Oct 31 08:51:19.124041 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:51:19.123794    4720 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-must-gather-r6ms4/must-gather-j5269" podUID="e5bec65f-573e-443c-ad34-a4da0d4a7987" containerName="copy" containerID="cri-o://a476ce7bc1e1474dcddfaffe8b0366b153c7f00db8ca9e78ac5e64626a22fe8f" gracePeriod=2
Oct 31 08:51:19.172927 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:51:19.172902    4720 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-must-gather-r6ms4/must-gather-j5269"]
Oct 31 08:51:19.174446 ip-10-0-139-83 kubenswrapper[4720]: E1031 08:51:19.174397    4720 status_manager.go:1045] "Failed to get status for pod" err="pods \"must-gather-j5269\" is forbidden: User \"system:node:ip-10-0-139-83.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-r6ms4\": no relationship found between node 'ip-10-0-139-83.us-west-1.compute.internal' and this object" podUID="e5bec65f-573e-443c-ad34-a4da0d4a7987" pod="openshift-must-gather-r6ms4/must-gather-j5269"
Oct 31 08:51:19.408509 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:51:19.408486    4720 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-r6ms4_must-gather-j5269_e5bec65f-573e-443c-ad34-a4da0d4a7987/copy/0.log"
Oct 31 08:51:19.411033 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:51:19.411015    4720 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-r6ms4/must-gather-j5269"
Oct 31 08:51:19.461555 ip-10-0-139-83 kubenswrapper[4720]: E1031 08:51:19.461523    4720 status_manager.go:1045] "Failed to get status for pod" err="pods \"must-gather-j5269\" is forbidden: User \"system:node:ip-10-0-139-83.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-r6ms4\": no relationship found between node 'ip-10-0-139-83.us-west-1.compute.internal' and this object" podUID="e5bec65f-573e-443c-ad34-a4da0d4a7987" pod="openshift-must-gather-r6ms4/must-gather-j5269"
Oct 31 08:51:19.527263 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:51:19.527236    4720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e5bec65f-573e-443c-ad34-a4da0d4a7987-must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/e5bec65f-573e-443c-ad34-a4da0d4a7987-must-gather-output\") pod \"e5bec65f-573e-443c-ad34-a4da0d4a7987\" (UID: \"e5bec65f-573e-443c-ad34-a4da0d4a7987\") "
Oct 31 08:51:19.527375 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:51:19.527270    4720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e5bec65f-573e-443c-ad34-a4da0d4a7987-kube-api-access-vmblp\" (UniqueName: \"kubernetes.io/projected/e5bec65f-573e-443c-ad34-a4da0d4a7987-kube-api-access-vmblp\") pod \"e5bec65f-573e-443c-ad34-a4da0d4a7987\" (UID: \"e5bec65f-573e-443c-ad34-a4da0d4a7987\") "
Oct 31 08:51:19.529585 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:51:19.529561    4720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e5bec65f-573e-443c-ad34-a4da0d4a7987-kube-api-access-vmblp" pod "e5bec65f-573e-443c-ad34-a4da0d4a7987" (UID: "e5bec65f-573e-443c-ad34-a4da0d4a7987"). InnerVolumeSpecName "kube-api-access-vmblp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 08:51:19.533897 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:51:19.533874    4720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5bec65f-573e-443c-ad34-a4da0d4a7987-must-gather-output" pod "e5bec65f-573e-443c-ad34-a4da0d4a7987" (UID: "e5bec65f-573e-443c-ad34-a4da0d4a7987"). InnerVolumeSpecName "must-gather-output". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 08:51:19.628697 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:51:19.628674    4720 reconciler_common.go:299] "Volume detached for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/e5bec65f-573e-443c-ad34-a4da0d4a7987-must-gather-output\") on node \"ip-10-0-139-83.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 08:51:19.628697 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:51:19.628696    4720 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vmblp\" (UniqueName: \"kubernetes.io/projected/e5bec65f-573e-443c-ad34-a4da0d4a7987-kube-api-access-vmblp\") on node \"ip-10-0-139-83.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 08:51:19.843469 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:51:19.843440    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 08:51:20.004115 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:51:20.004085    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 08:51:20.014966 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:51:20.014945    4720 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-r6ms4_must-gather-j5269_e5bec65f-573e-443c-ad34-a4da0d4a7987/copy/0.log"
Oct 31 08:51:20.015297 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:51:20.015277    4720 generic.go:356] "Generic (PLEG): container finished" podID="e5bec65f-573e-443c-ad34-a4da0d4a7987" containerID="a476ce7bc1e1474dcddfaffe8b0366b153c7f00db8ca9e78ac5e64626a22fe8f" exitCode=143
Oct 31 08:51:20.015377 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:51:20.015322    4720 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-r6ms4/must-gather-j5269"
Oct 31 08:51:20.015377 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:51:20.015355    4720 scope.go:122] "RemoveContainer" containerID="a476ce7bc1e1474dcddfaffe8b0366b153c7f00db8ca9e78ac5e64626a22fe8f"
Oct 31 08:51:20.023750 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:51:20.023695    4720 scope.go:122] "RemoveContainer" containerID="8a4f3817cca69997993c3f6c566cc485569f49ca1321004136bda3ab57e9bf2c"
Oct 31 08:51:20.025015 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:51:20.024994    4720 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e5bec65f-573e-443c-ad34-a4da0d4a7987" path="/var/lib/kubelet/pods/e5bec65f-573e-443c-ad34-a4da0d4a7987/volumes"
Oct 31 08:51:20.035969 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:51:20.035954    4720 scope.go:122] "RemoveContainer" containerID="a476ce7bc1e1474dcddfaffe8b0366b153c7f00db8ca9e78ac5e64626a22fe8f"
Oct 31 08:51:20.036239 ip-10-0-139-83 kubenswrapper[4720]: E1031 08:51:20.036223    4720 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a476ce7bc1e1474dcddfaffe8b0366b153c7f00db8ca9e78ac5e64626a22fe8f\": container with ID starting with a476ce7bc1e1474dcddfaffe8b0366b153c7f00db8ca9e78ac5e64626a22fe8f not found: ID does not exist" containerID="a476ce7bc1e1474dcddfaffe8b0366b153c7f00db8ca9e78ac5e64626a22fe8f"
Oct 31 08:51:20.036277 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:51:20.036247    4720 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a476ce7bc1e1474dcddfaffe8b0366b153c7f00db8ca9e78ac5e64626a22fe8f"} err="failed to get container status \"a476ce7bc1e1474dcddfaffe8b0366b153c7f00db8ca9e78ac5e64626a22fe8f\": rpc error: code = NotFound desc = could not find container \"a476ce7bc1e1474dcddfaffe8b0366b153c7f00db8ca9e78ac5e64626a22fe8f\": container with ID starting with a476ce7bc1e1474dcddfaffe8b0366b153c7f00db8ca9e78ac5e64626a22fe8f not found: ID does not exist"
Oct 31 08:51:20.036277 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:51:20.036262    4720 scope.go:122] "RemoveContainer" containerID="8a4f3817cca69997993c3f6c566cc485569f49ca1321004136bda3ab57e9bf2c"
Oct 31 08:51:20.036514 ip-10-0-139-83 kubenswrapper[4720]: E1031 08:51:20.036495    4720 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8a4f3817cca69997993c3f6c566cc485569f49ca1321004136bda3ab57e9bf2c\": container with ID starting with 8a4f3817cca69997993c3f6c566cc485569f49ca1321004136bda3ab57e9bf2c not found: ID does not exist" containerID="8a4f3817cca69997993c3f6c566cc485569f49ca1321004136bda3ab57e9bf2c"
Oct 31 08:51:20.036561 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:51:20.036520    4720 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8a4f3817cca69997993c3f6c566cc485569f49ca1321004136bda3ab57e9bf2c"} err="failed to get container status \"8a4f3817cca69997993c3f6c566cc485569f49ca1321004136bda3ab57e9bf2c\": rpc error: code = NotFound desc = could not find container \"8a4f3817cca69997993c3f6c566cc485569f49ca1321004136bda3ab57e9bf2c\": container with ID starting with 8a4f3817cca69997993c3f6c566cc485569f49ca1321004136bda3ab57e9bf2c not found: ID does not exist"
Oct 31 08:51:20.066095 ip-10-0-139-83 kubenswrapper[4720]: E1031 08:51:20.066067    4720 status_manager.go:1045] "Failed to get status for pod" err="pods \"must-gather-j5269\" is forbidden: User \"system:node:ip-10-0-139-83.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-r6ms4\": no relationship found between node 'ip-10-0-139-83.us-west-1.compute.internal' and this object" podUID="e5bec65f-573e-443c-ad34-a4da0d4a7987" pod="openshift-must-gather-r6ms4/must-gather-j5269"
Oct 31 08:51:23.885218 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:51:23.885190    4720 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-kht4l_77256436-39ff-4c2a-8390-4e9fa7d68486/global-pull-secret-syncer/0.log"
Oct 31 08:51:24.821466 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:51:24.821429    4720 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-f9gtg_631cff9e-a1f1-4dd2-bdc8-ef36519c7c1b/connection-checker/0.log"
Oct 31 08:51:25.237076 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:51:25.237050    4720 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-hxmpf_323c8334-9a56-4c4e-8a6e-ae80995810d0/connection-checker/0.log"
Oct 31 08:51:25.655599 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:51:25.655573    4720 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-xcrsn_91363236-08db-4f54-ae0a-e68604fcecc6/connection-checker/0.log"
Oct 31 08:51:26.122883 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:51:26.122852    4720 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log"
Oct 31 08:51:27.800930 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:51:27.800902    4720 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-139-83.us-west-1.compute.internal_2183ffc19200ec816330ede9883f3bf6/haproxy/0.log"
Oct 31 08:51:28.180795 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:51:28.180768    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 08:51:34.427849 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:51:34.427822    4720 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-68df894cb4-xjstz_33530b7d-e4c2-490b-b7ea-7a7915cbe769/manager/0.log"
Oct 31 08:51:34.842738 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:51:34.842717    4720 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-wmfgm_601ccf10-fbcd-41c1-ae75-6d7c01bc1b8a/kuadrant-console-plugin/0.log"
Oct 31 08:51:35.314978 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:51:35.314906    4720 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-658c459c99-nnrhp_c4d53665-7863-48a0-a0a2-7e911345bd4f/manager/0.log"
Oct 31 08:51:36.740750 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:51:36.740715    4720 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-9rj77/perf-node-gather-daemonset-vnj2d"]
Oct 31 08:51:36.741124 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:51:36.741032    4720 cpu_manager.go:414] "containerMap: removing container" podUID="e5bec65f-573e-443c-ad34-a4da0d4a7987" containerName="copy"
Oct 31 08:51:36.741124 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:51:36.741043    4720 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e5bec65f-573e-443c-ad34-a4da0d4a7987" containerName="copy"
Oct 31 08:51:36.741124 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:51:36.741066    4720 cpu_manager.go:414] "containerMap: removing container" podUID="e5bec65f-573e-443c-ad34-a4da0d4a7987" containerName="gather"
Oct 31 08:51:36.741124 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:51:36.741072    4720 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e5bec65f-573e-443c-ad34-a4da0d4a7987" containerName="gather"
Oct 31 08:51:36.741253 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:51:36.741132    4720 memory_manager.go:365] "RemoveStaleState removing state" podUID="e5bec65f-573e-443c-ad34-a4da0d4a7987" containerName="gather"
Oct 31 08:51:36.741253 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:51:36.741142    4720 memory_manager.go:365] "RemoveStaleState removing state" podUID="e5bec65f-573e-443c-ad34-a4da0d4a7987" containerName="copy"
Oct 31 08:51:36.746462 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:51:36.746443    4720 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-9rj77/perf-node-gather-daemonset-vnj2d"
Oct 31 08:51:36.797804 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:51:36.797780    4720 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-9rj77\"/\"default-dockercfg-wfbtx\""
Oct 31 08:51:36.844098 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:51:36.844070    4720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/bb51d7aa-ea85-40d5-8706-a928a6794597-lib-modules\") pod \"perf-node-gather-daemonset-vnj2d\" (UID: \"bb51d7aa-ea85-40d5-8706-a928a6794597\") " pod="openshift-must-gather-9rj77/perf-node-gather-daemonset-vnj2d"
Oct 31 08:51:36.844218 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:51:36.844100    4720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lchkw\" (UniqueName: \"kubernetes.io/projected/bb51d7aa-ea85-40d5-8706-a928a6794597-kube-api-access-lchkw\") pod \"perf-node-gather-daemonset-vnj2d\" (UID: \"bb51d7aa-ea85-40d5-8706-a928a6794597\") " pod="openshift-must-gather-9rj77/perf-node-gather-daemonset-vnj2d"
Oct 31 08:51:36.844218 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:51:36.844129    4720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/bb51d7aa-ea85-40d5-8706-a928a6794597-podres\") pod \"perf-node-gather-daemonset-vnj2d\" (UID: \"bb51d7aa-ea85-40d5-8706-a928a6794597\") " pod="openshift-must-gather-9rj77/perf-node-gather-daemonset-vnj2d"
Oct 31 08:51:36.844345 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:51:36.844239    4720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/bb51d7aa-ea85-40d5-8706-a928a6794597-sys\") pod \"perf-node-gather-daemonset-vnj2d\" (UID: \"bb51d7aa-ea85-40d5-8706-a928a6794597\") " pod="openshift-must-gather-9rj77/perf-node-gather-daemonset-vnj2d"
Oct 31 08:51:36.844345 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:51:36.844266    4720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/bb51d7aa-ea85-40d5-8706-a928a6794597-proc\") pod \"perf-node-gather-daemonset-vnj2d\" (UID: \"bb51d7aa-ea85-40d5-8706-a928a6794597\") " pod="openshift-must-gather-9rj77/perf-node-gather-daemonset-vnj2d"
Oct 31 08:51:36.849695 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:51:36.849672    4720 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-9rj77/perf-node-gather-daemonset-vnj2d"]
Oct 31 08:51:36.885982 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:51:36.885954    4720 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-9rj77\"/\"kube-root-ca.crt\""
Oct 31 08:51:36.886071 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:51:36.886014    4720 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-9rj77\"/\"openshift-service-ca.crt\""
Oct 31 08:51:36.944711 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:51:36.944692    4720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/bb51d7aa-ea85-40d5-8706-a928a6794597-proc\") pod \"perf-node-gather-daemonset-vnj2d\" (UID: \"bb51d7aa-ea85-40d5-8706-a928a6794597\") " pod="openshift-must-gather-9rj77/perf-node-gather-daemonset-vnj2d"
Oct 31 08:51:36.944802 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:51:36.944738    4720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/bb51d7aa-ea85-40d5-8706-a928a6794597-lib-modules\") pod \"perf-node-gather-daemonset-vnj2d\" (UID: \"bb51d7aa-ea85-40d5-8706-a928a6794597\") " pod="openshift-must-gather-9rj77/perf-node-gather-daemonset-vnj2d"
Oct 31 08:51:36.944802 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:51:36.944759    4720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lchkw\" (UniqueName: \"kubernetes.io/projected/bb51d7aa-ea85-40d5-8706-a928a6794597-kube-api-access-lchkw\") pod \"perf-node-gather-daemonset-vnj2d\" (UID: \"bb51d7aa-ea85-40d5-8706-a928a6794597\") " pod="openshift-must-gather-9rj77/perf-node-gather-daemonset-vnj2d"
Oct 31 08:51:36.944802 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:51:36.944794    4720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/bb51d7aa-ea85-40d5-8706-a928a6794597-podres\") pod \"perf-node-gather-daemonset-vnj2d\" (UID: \"bb51d7aa-ea85-40d5-8706-a928a6794597\") " pod="openshift-must-gather-9rj77/perf-node-gather-daemonset-vnj2d"
Oct 31 08:51:36.944937 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:51:36.944812    4720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/bb51d7aa-ea85-40d5-8706-a928a6794597-proc\") pod \"perf-node-gather-daemonset-vnj2d\" (UID: \"bb51d7aa-ea85-40d5-8706-a928a6794597\") " pod="openshift-must-gather-9rj77/perf-node-gather-daemonset-vnj2d"
Oct 31 08:51:36.944937 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:51:36.944824    4720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/bb51d7aa-ea85-40d5-8706-a928a6794597-sys\") pod \"perf-node-gather-daemonset-vnj2d\" (UID: \"bb51d7aa-ea85-40d5-8706-a928a6794597\") " pod="openshift-must-gather-9rj77/perf-node-gather-daemonset-vnj2d"
Oct 31 08:51:36.944937 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:51:36.944870    4720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/bb51d7aa-ea85-40d5-8706-a928a6794597-sys\") pod \"perf-node-gather-daemonset-vnj2d\" (UID: \"bb51d7aa-ea85-40d5-8706-a928a6794597\") " pod="openshift-must-gather-9rj77/perf-node-gather-daemonset-vnj2d"
Oct 31 08:51:36.944937 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:51:36.944892    4720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/bb51d7aa-ea85-40d5-8706-a928a6794597-lib-modules\") pod \"perf-node-gather-daemonset-vnj2d\" (UID: \"bb51d7aa-ea85-40d5-8706-a928a6794597\") " pod="openshift-must-gather-9rj77/perf-node-gather-daemonset-vnj2d"
Oct 31 08:51:36.944937 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:51:36.944902    4720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/bb51d7aa-ea85-40d5-8706-a928a6794597-podres\") pod \"perf-node-gather-daemonset-vnj2d\" (UID: \"bb51d7aa-ea85-40d5-8706-a928a6794597\") " pod="openshift-must-gather-9rj77/perf-node-gather-daemonset-vnj2d"
Oct 31 08:51:37.005703 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:51:37.002656    4720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lchkw\" (UniqueName: \"kubernetes.io/projected/bb51d7aa-ea85-40d5-8706-a928a6794597-kube-api-access-lchkw\") pod \"perf-node-gather-daemonset-vnj2d\" (UID: \"bb51d7aa-ea85-40d5-8706-a928a6794597\") " pod="openshift-must-gather-9rj77/perf-node-gather-daemonset-vnj2d"
Oct 31 08:51:37.058569 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:51:37.058544    4720 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-9rj77/perf-node-gather-daemonset-vnj2d"
Oct 31 08:51:37.431052 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:51:37.431028    4720 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-9rj77/perf-node-gather-daemonset-vnj2d"]
Oct 31 08:51:37.432750 ip-10-0-139-83 kubenswrapper[4720]: W1031 08:51:37.432724    4720 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podbb51d7aa_ea85_40d5_8706_a928a6794597.slice/crio-804164e760883303fae0c0ec815ebf3078bf7022d8653c6370addb243dde2243 WatchSource:0}: Error finding container 804164e760883303fae0c0ec815ebf3078bf7022d8653c6370addb243dde2243: Status 404 returned error can't find the container with id 804164e760883303fae0c0ec815ebf3078bf7022d8653c6370addb243dde2243
Oct 31 08:51:38.094845 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:51:38.094814    4720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-9rj77/perf-node-gather-daemonset-vnj2d" event={"ID":"bb51d7aa-ea85-40d5-8706-a928a6794597","Type":"ContainerStarted","Data":"3fc9089019cf09398c42a869ab14a1b9cc2b14ba6d77d0b0ac724b1c1f19786a"}
Oct 31 08:51:38.095221 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:51:38.094866    4720 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-9rj77/perf-node-gather-daemonset-vnj2d"
Oct 31 08:51:38.095221 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:51:38.094889    4720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-9rj77/perf-node-gather-daemonset-vnj2d" event={"ID":"bb51d7aa-ea85-40d5-8706-a928a6794597","Type":"ContainerStarted","Data":"804164e760883303fae0c0ec815ebf3078bf7022d8653c6370addb243dde2243"}
Oct 31 08:51:38.207842 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:51:38.207798    4720 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-9rj77/perf-node-gather-daemonset-vnj2d" podStartSLOduration=2.207785144 podStartE2EDuration="2.207785144s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 08:51:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 08:51:38.206423106 +0000 UTC m=+4952.337491177" watchObservedRunningTime="2026-10-31 08:51:38.207785144 +0000 UTC m=+4952.338853233"
Oct 31 08:51:41.520156 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:51:41.520122    4720 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-884576c57-6k8mn_789f11d8-d951-4144-a589-a2bdbf6844e3/cluster-monitoring-operator/0.log"
Oct 31 08:51:42.815925 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:51:42.815891    4720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4k2m_2e6fa788-ffb3-4a3b-8f6c-e52c2f9655a5/konnectivity-agent/0.log" limit=1024
Oct 31 08:51:43.460119 ip-10-0-139-83 kubenswrapper[4720]: E1031 08:51:43.460080    4720 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8m768" containerName="extractor"
Oct 31 08:51:44.003721 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:51:44.003682    4720 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-p56z2_bc1939c5-394d-4174-a903-7c23b89e113b/node-exporter/0.log"
Oct 31 08:51:44.110036 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:51:44.110008    4720 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-9rj77/perf-node-gather-daemonset-vnj2d"
Oct 31 08:51:44.419305 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:51:44.419282    4720 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-p56z2_bc1939c5-394d-4174-a903-7c23b89e113b/kube-rbac-proxy/0.log"
Oct 31 08:51:44.832709 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:51:44.832686    4720 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-p56z2_bc1939c5-394d-4174-a903-7c23b89e113b/init-textfile/0.log"
Oct 31 08:51:59.281205 ip-10-0-139-83 kubenswrapper[4720]: I1031 08:51:59.281177    4720 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-86c478bb88-j94dz_314c29aa-db6b-4f7f-86b4-c8952ce09c84/networking-console-plugin/0.log"